RSSMiner
- Rssminer - a simple, intelligent RSS reader
- Live version http://rssminer.net
- Write in Clojure & Javascript & Java, by shen feng
Features
- Build for readability
- Realtime, instant fulltext search
- Realtime, personalized recommendation based on reading history
- Google Chrome plugin to add subscription
- Clean and compact code
- Super fast (In order to be fast, I write the
Web server,Database Connection Pool,Chinese segmentation,Template System,Some JS libsfrom scratch. I save every bit to make it faster. The server can handle thousands of request per seconds(Including fulltext search) with very low latency.
Technologies
Server-side
- Java. Clojure.
- Apache Lucene, use it to do fulltext, realtime, instant search.
- Ring, compojure, easier http.
- http-kit, super fast event driven HTTP server and HTTP client. Especially written for Rssminer.
- dbcp, Simple database connection pool. Especially written for Rssminer
- mmseg, A java implementation of MMSEG. Especially written for Rssminer
- MySQL, data store
- Redis, Message Queue; Per user per feed score Store. Proxy cache.
- Mustache.clj, Mustache for Clojure
Client-side
- jQuery, nicer API
- Underscore, be functional
- sass, fix css
- mustache.js, Mustache for JS
How to run it
Install Instructions
-
Install Dependencies leiningen, JDK7(JDK6 works), Redis, MySQL, sass, rake
-
clone this repo, install dependency
git clone git://github.com/shenfeng/rssminer.git && cd rssminer && lein deps- Initialize database, create user, import schema
cd rssminer && ./scripts/admin init-db- Run it
# make sure /var/rssminer/index dir exits and is writable. refer command line args for more info
rake run:dev # run server in dev profile, view it: http://127.0.0.1:9090Run unit test
rake testrake mysql_dev # replace my.cnf will a dev one, run mysql in /tmp, run it after understand it.It will create/drop a temp MySQL database for each test. If MySQL's db
path is in tmpfs, It will much faster(12s vs 40+s).
Command line args
rake prepare javac && ./scripts/run --help # rake -T list commandsUsage:
Switches Default Desc
-------- ------- ----
-p, --port 9090 Port to listen
--worker 2 Http worker thread count
--fetcher-concurrency 10
--fetch-size 20 Bulk fetch size
--profile :dev dev or prod
--redis-host 127.0.0.1 Redis host
--redis-port 6379 Redis port
--static-server //192.168.1.200 static server
--db-url jdbc:mysql://localhost/rssminer MySQL Database url
--db-user feng MySQL user name
--db-pass MySQL password
--bind-ip 0.0.0.0 Which ip to bind
--events-threshold 20 How many user feed events buffered before recompute again
--index-path /var/rssminer/index Path to store lucene index
--no-fetcher, --fetcher false Start rss fetcher
--no-proxy, --proxy false Enable Socks proxy
--no-help, --help false Print this help
Copyright
Copyright (c) Feng Shen[email protected]. All rights reserved.