tspannhw/stocks-nifi-kafka tspannhw/stocks-nifi-kafka

  • Stars
    star
    14
  • Rank 1,438,076 (Top 29 %)
  • Language
  • License
    Apache License 2.0
  • Created about 6 years ago
  • Updated about 6 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
tspannhw/stocks-nifi-kafka
github

tspannhw

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Stocks -> NiFi -> Kafka -> Profit

More Repositories

1

EverythingApacheNiFi

EverythingApacheNiFi
66
star
2

OpenSourceComputerVision

Open Source Computer Vision with TensorFlow, MiniFi, Apache NiFi, OpenCV, Apache Tika and Python For processing images from IoT devices like Raspberry Pis, NVidia Jetson TX1, NanoPi Duos and more which are equipped with attached cameras or external USB webcams, we use Python to interface via OpenCV and PiCamera. From there we run image processing at the edge on these IoT device using OpenCV and TensorFlow to determine attributes and image analytics. A pache MiniFi coordinates running these Python scripts and decides when and what to send from that analysis and the image to a remote Apache NiFi server for additional processing. At the Apache NiFi cluster in the cluster it routes the images to one processing path and the JSON encoded metadata to another flow. The JSON data (with it's schema referenced from a central Schema Registry) is routed and routed using Record Processing and SQL, this data in enriched and augment before conversion to AVRO to be send via Apache Kafka to SAM. Streaming Analytics Manager then does deeper processing on this stream and others including weather and twitter to determine what should be done on this data. References https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html https://github.com/tspannhw/rpi-noir-screen https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html https://community.hortonworks.com/articles/107379/minifi-for-image-capture-and-ingestion-from-raspbe.html https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html
Python
43
star
3

nifi-tensorflow-processor

Example Tensorflow Processor using Java API for Apache NiFi 1.2 - 1.9.1+
Java
39
star
4

nifi-extracttext-processor

Apache NiFi Custom Processor Extracting Text From Files with Apache Tika
Java
32
star
5

phoenix

Apache Phoenix / Hbase Spring Boot Microservices
Java
23
star
6

nifi-nlp-processor

Apache NiFi NLP Processor
Java
18
star
7

hive

Apache Hive / Hadoop Spring Boot Microservice
Java
17
star
8

dws2017sydney

Code, Examples, Templates and Scripts for DataWorksSummit 2017 Sydney Talk
Python
16
star
9

SpeakerProfile

My speaker profile for events and conferences based on codepo8/presenter-terms
11
star
10

nifi-corenlp-processor

Apache NiFi Custom Processor for working with Stanford CoreNLP for Sentiment Analysis in Java 8
Java
11
star
11

nifi-convertjsontoddl-processor

Apache NiFi 1.5/1.6/1.9.2+ Processor to produce DDL
Java
10
star
12

ApacheDeepLearning201

Apache Deep Learning 201 with Apache MXNet
Python
10
star
13

nifi-attributecleaner-processor

Clean up attribute names for Apache NiFi to send to Apache Avro
Java
10
star
14

FLiP-Pi-DeltaLake-Thermal

Apache Pulsar -> Sink -> DeltaLake
HTML
9
star
15

ApacheDeepLearning101

ApacheDeepLearning101
Python
8
star
16

hivereader

Apache Hive - REST API
Java
8
star
17

pulsar-thermal-pinot

Apache Pulsar - Apache Pinot - Thermal Sensor Data
Python
8
star
18

ScyllaFLiPSTheStream

Scylla FLiPS The Stream With Apache Pulsar
HTML
7
star
19

livysparkjob

Apache Livy - Apache NiFi - Example Scala Spark Job
XSLT
7
star
20

ClouderaFlowManagementWorkshop

Cloudera Flow Management Workshop with Apache NiFi
Python
7
star
21

nifi-djl-processor

Apache NiFi 1.10 DJL
Java
7
star
22

nifi-dl4j-processor

Apache NiFi - DeepLearning4J Example Processor for VGG16 based https://github.com/SkymindIO/Demos-for-sales-marketting
Java
7
star
23

rpi-mxnet-movidius-minifi

USB WebCam - Raspberry Pi - Apache MXNet - Movidius - Apache NiFi - MiniFi
Python
7
star
24

nifi-langdetect-processor

Apache NiFi + Apache Tika + OptimaizeLangDetector
Java
7
star
25

flume-to-nifi

Examples for Apache Flume to Apache NiFi
7
star
26

MmFLaNK

Mm FLaNK Stack (MXNet, MiNiFi, Flink, NiFi, Kafka, Kudu) for AI-IoT
Java
6
star
27

airquality

Spring Boot, JDK 11, Apache Pulsar 2.9.1, REST, AirNow
Java
6
star
28

FLiPStackWeekly

FLiP Stack Weekly covering Apache NiFi, Apache Flink, Apache Kafka, Apache Spark, Apache Iceberg, Apache Ozone, Apache Pulsar, and more...
6
star
29

rpi-picamera-mqtt-nifi

Ingesting Camera Images and Video from Raspberry Pi + Pi Camera Through MQTT and FTP
Python
6
star
30

FLiPC-FastStreamingIntoClickhouseWithApachePulsar

Fast Streaming into Clickhouse with Apache Pulsar - Meetup 2022
6
star
31

minifi-jetson-nano

MiNiFi Agent Configuration and Scripts for NVidia Jetson Nano device
Python
6
star
32

ApacheConAtHome2020

ApacheCon @Home Sept-Oct 2020 Materials
HTML
6
star
33

SmartWeather

FLaNK: Smart Weather
Java
5
star
34

nifi-cdsw-edge

Edge to AI: IoT
Python
5
star
35

EnterpriseNIFI

Enterprise NIFI Talks and Code
5
star
36

nifi-gluoncv-yolo3

Using Apache MXNet Gluon 0.3 for YOLO3 Pretrained model in the zoo!
Python
5
star
37

rpi-sensehat-minifi-python

Raspberry Pi Python Code for use with Apache NiFi & Apache MiniFi Java Agent
Python
5
star
38

meetup-sensors

sensors, machine learning, nifi, minifi, python, enviroplus, iot, streaming, flank, kafka, time series, cml
Shell
5
star
39

nifi-processors

All my processors (NARs) in one place
5
star
40

ApacheNiFiTemplates

Templates for NiFi
5
star
41

nifi-deepspeech

Using Mozilla TensorFlow implementation of Baidu's Deep Speech
Python
5
star
42

TensorflowOnYARN

Examples of Python TensorFlow apps running on HDP 3 / YARN 3.1
Python
5
star
43

FLiPN-NFT

Reading OpenSea NFT with Apache NiFi, Apache Pulsar and Friends. Meetup. StreamNative
HTML
5
star
44

stateless-examples

Apache NiFi 1.10 Stateless Examples
Shell
5
star
45

StreamingAnalyticsUsingFlinkSQL

FLiP: StreamNative: Cloud-Native: Streaming Analytics Using Apache Flink SQL on Apache Pulsar
Java
5
star
46

pulsar-transit-function

Apache Pulsar - Transit Enrichment and Routing Function
Java
4
star
47

FLiP-EdgeAI

Edge AI with Jetson Nano + Sensors
Python
4
star
48

rpi-sensehat-mqtt-nifi

Using SenseHat for Sensor Ingest into Raspberry Pi 3 B+ via Python to Apache NiFi
Python
4
star
49

SmartStocks

Smart Stocks
4
star
50

onionomega-mqtt-micropython

Onion Omega 2+ with OLED and Micropython
Python
4
star
51

nifi-djlsentimentanalysis-processor

nifi-djlsentimentanalysis-processor
Java
4
star
52

FLiP-Pi-BreakoutGarden

Python
4
star
53

minifi-breakoutgarden

Using MiniFi with the Pimoroni Breakout Garden Hat and Sensors
Python
4
star
54

spring-pulsar-airquality

Java
4
star
55

nifi-energy-monitoring

Updated real-time energy monitoring with Python and Apache NiFi
Python
4
star
56

iot-scripts

Scripts for IoT (mostly Python and Bash)
Python
4
star
57

linkextractorprocessor

Apache NiFi Custom Processor For Link Extracting
Java
4
star
58

FLiPN-Demos

Some FLiPN Demos
HTML
4
star
59

cdp-datahub-azure-nifikafka

Cloudera CDP Datahub on Azure for Apache NiFi and Apache Kafka Demo
TSQL
4
star
60

IoTFusion2018Talk

My Talk at IoT Fusion 2018 Philadelphia, PA
Python
4
star
61

yolo3-keras-tensorflow

A tweak to https://github.com/qqwweee/keras-yolo3 to store images and json attributes
Python
4
star
62

nifi-djlqa-processor

nifi-djlqa-processor
Java
4
star
63

IngestingDroneData

Ingesting Drone Data into Big Data Platforms
Python
4
star
64

spy

Inspired by TURN and Robin David's awesome python library.
Python
3
star
65

FLiP-PY-FakeDataPulsar

Generate Fake Data
Python
3
star
66

rpizw-nifi-mqtt-gps

GPS Sensor Reading from Raspberry Pi Zero Wireless Sending via MQTT to Apache NiFi 1.x
Python
3
star
67

FLiP-IoT

FLiP - IoT Examples with Go, Java, MiNiFi, Flink, Pulsar, StreamNative, JSON
Python
3
star
68

CloudDemo2021

HTML
3
star
69

nifi-osquery

osquery nifi integration
Shell
3
star
70

EverythingMiNiFi

Apache NiFi - MiNiFi Agents. Java - rolling into NiFi. MiNiFi C++ Agent.
3
star
71

DWS-DeepLearning-CrashCourse

TensorFlow Inception Apache Zeppelin Notebook Python
Jupyter Notebook
3
star
72

brains-kafka

Scala
3
star
73

pulsar-airquality-timeplus

Real-Time Analytics on AirQuality Data with timeplus
3
star
74

spring-pulsar-reactiveair

Reactive Spring Example for Apache Pulsar
Java
3
star
75

airline-sentiment-streaming

Streaming with Airline Sentiment. Utilizing Cloudera Machine Learning, Apache NiFi, Apache Hue, Apache Impala, Apache Kudu
Jupyter Notebook
3
star
76

hbasecoprocessor

Example HBase Coprocessor
Java
3
star
77

FLiP-Transit

Apache Flink - Apache Pulsar - Apache NiFi -> Real-Time Transit Data
3
star
78

AllThingsNiFi

Everything Apache NiFi, Cloudera DataFlow
3
star
79

nifi-cdsw

Example Apache NiFi to CDSW
Python
3
star
80

kafkaadmin-processor

Apache Kafka Apache NiFi
Java
3
star
81

HBase2

Apache HBase 2.0 with Apache NiFi 1.10, Apache Kafka 2.2, Apache Flink 1.9
3
star
82

nifi-cdsw-gluoncv

Using Apache MXNet GluonCV YOLO with Apache NiFi and Cloudera Data Science Workbench
Python
3
star
83

nifi-mxnetinference-processor

Apache NiFi Processor For Apache MXNet Inference
Java
3
star
84

FLiP-Pi-Thermal

Python
3
star
85

FLiPN-AirQuality-REST

Apache NiFi, Apache Pulsar. Air Quality Readings
HTML
3
star
86

gtfs

GTFS / ProtoBuf Data
Java
3
star
87

PulsarOnRaspberryPi

Pulsar on Pi
CMake
3
star
88

BackupRegistry

Backup the NiFi Registry
3
star
89

FLiP-Stream2Clickhouse

Stonks. Stocks. Apache Pulsar - Stream to Altinity Cloud - Clickhouse
3
star
90

phillycrime-springboot-phoenix

Apache Phoenix - Apache HBase - SpringBoot - Philly Crime Data
Java
3
star
91

mxnet-for-iot

Apache MXNet for IoT
Python
3
star
92

operations-dashboard

Apache NiFi operations-dashboard
Java
3
star
93

nifi-puttwitter-processor

Send out tweets with Twitter4J
Java
3
star
94

EverythingApacheKafka

Apache Kafka, Kafka Connect, Kafka Streams, FLaNK, Friend, Tools, Demos
3
star
95

pulsar-airquality-function

pulsar-airquality-function
Java
3
star
96

pulsar-pychat-function

Python
3
star
97

nifi-ocr

OCR with Tesseract, Deep Learning, Python, Apache NiFi, ...
Python
3
star
98

nifi-nanopi-duo

NanoPi Duo + Apache MinIFi + MiniFi + Python + USB Camera + DL/ML/IOT
Python
3
star
99

obd2-nifi

OBDII Interface with Apache NiFi, MiniFi, Raspberry Pi, Comma.AI, Panda, SenseHat, GPS, Weather, Humidity, Temperature, Altitude
3
star
100

nvidiajetsontx1-mxnet

MXNet to MiniFi Example
Python
3
star