kafka · spark streaming example
This is a simple dashboard example on Kafka and Spark Streaming
Prerequisites
Java 1.8 or newer version required because lambda expression used for few cases
- Java >= 1.8 (Oracle JDK has been tested)
- Maven >= 3
- Apache Spark >= 2.0.2
- Kafka >= 0.10.1.0
Installation
First of all, clone the git repository,
$ git clone [email protected]:trK54Ylmz/kafka-spark-streaming-example.gitafter you need to use Maven for creating uber jar files,
$ mvn clean package -DskipTestsuntil that moment we had created jar files and now we'll install Kafka and MySQL,
$ wget http://www-us.apache.org/dist/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
$ # or wget http://www-eu.apache.org/dist/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
$ tar -xf kafka_2.11-0.10.1.0.tgz
$ cd kafka_2.11-0.10.1.0
$ nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties > /tmp/kafka-zookeeper.out 2>&1 &
$ nohup ./bin/kafka-server-start.sh ./config/server.properties > /tmp/kafka-server.out 2>&1 &Kafka is ready we can continue to install MySQL,
$ sudo apt-get install mysql-server # for Ubuntu, Debian
$ sudo yum install mysql-server && sudo systemctl start mysqld # for CentOS, RHEL
$ brew install mysql && mysql.server restart # for macOSand finally create MySQL database and table,
CREATE DATABASE IF NOT EXISTS dashboard_test;
USE dashboard_test;
CREATE TABLE IF NOT EXISTS events (
market VARCHAR(24) NOT NULL DEFAULT '',
rate FLOAT DEFAULT NULL,
dt DATETIME NOT NULL,
PRIMARY KEY (market, dt)
)
ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;Usage
1 - Start the Spark streaming service and it'll process events from Kafka topic to MySQL,
$ cd kafka-spark-streaming-example
$ java -Dconfig=./config/common.conf -jar streaming/target/spark-streaming-0.1.jar2 - Start the Kafka producer and it'll write events to Kafka topic,
$ java -Dconfig=./config/common.conf -jar producer/target/kafka-producer-0.1.jar3 - Start the web server so you can see the dashboard
$ java -Dconfig=./config/common.conf -jar web/target/web-0.1.jar4 - If everything look fine, please enter the dashboard address,
open http://localhost:8080 # default value : 8080