tspannhw/EverythingApacheKafka tspannhw/EverythingApacheKafka

  • Stars
    star
    3
  • Rank 3,963,521 (Top 79 %)
  • Language
  • License
    Apache License 2.0
  • Created over 4 years ago
  • Updated over 3 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
tspannhw/EverythingApacheKafka
github

tspannhw

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Apache Kafka, Kafka Connect, Kafka Streams, FLaNK, Friend, Tools, Demos

More Repositories

1

EverythingApacheNiFi

EverythingApacheNiFi
66
star
2

OpenSourceComputerVision

Open Source Computer Vision with TensorFlow, MiniFi, Apache NiFi, OpenCV, Apache Tika and Python For processing images from IoT devices like Raspberry Pis, NVidia Jetson TX1, NanoPi Duos and more which are equipped with attached cameras or external USB webcams, we use Python to interface via OpenCV and PiCamera. From there we run image processing at the edge on these IoT device using OpenCV and TensorFlow to determine attributes and image analytics. A pache MiniFi coordinates running these Python scripts and decides when and what to send from that analysis and the image to a remote Apache NiFi server for additional processing. At the Apache NiFi cluster in the cluster it routes the images to one processing path and the JSON encoded metadata to another flow. The JSON data (with it's schema referenced from a central Schema Registry) is routed and routed using Record Processing and SQL, this data in enriched and augment before conversion to AVRO to be send via Apache Kafka to SAM. Streaming Analytics Manager then does deeper processing on this stream and others including weather and twitter to determine what should be done on this data. References https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html https://github.com/tspannhw/rpi-noir-screen https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html https://community.hortonworks.com/articles/107379/minifi-for-image-capture-and-ingestion-from-raspbe.html https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html
Python
43
star
3

nifi-tensorflow-processor

Example Tensorflow Processor using Java API for Apache NiFi 1.2 - 1.9.1+
Java
39
star
4

nifi-extracttext-processor

Apache NiFi Custom Processor Extracting Text From Files with Apache Tika
Java
32
star
5

phoenix

Apache Phoenix / Hbase Spring Boot Microservices
Java
23
star
6

nifi-nlp-processor

Apache NiFi NLP Processor
Java
18
star
7

hive

Apache Hive / Hadoop Spring Boot Microservice
Java
17
star
8

dws2017sydney

Code, Examples, Templates and Scripts for DataWorksSummit 2017 Sydney Talk
Python
16
star
9

stocks-nifi-kafka

Stocks -> NiFi -> Kafka -> Profit
14
star
10

SpeakerProfile

My speaker profile for events and conferences based on codepo8/presenter-terms
11
star
11

nifi-corenlp-processor

Apache NiFi Custom Processor for working with Stanford CoreNLP for Sentiment Analysis in Java 8
Java
11
star
12

nifi-convertjsontoddl-processor

Apache NiFi 1.5/1.6/1.9.2+ Processor to produce DDL
Java
10
star
13

ApacheDeepLearning201

Apache Deep Learning 201 with Apache MXNet
Python
10
star
14

nifi-attributecleaner-processor

Clean up attribute names for Apache NiFi to send to Apache Avro
Java
10
star
15

FLiP-Pi-DeltaLake-Thermal

Apache Pulsar -> Sink -> DeltaLake
HTML
9
star
16

ApacheDeepLearning101

ApacheDeepLearning101
Python
8
star
17

hivereader

Apache Hive - REST API
Java
8
star
18

pulsar-thermal-pinot

Apache Pulsar - Apache Pinot - Thermal Sensor Data
Python
8
star
19

ScyllaFLiPSTheStream

Scylla FLiPS The Stream With Apache Pulsar
HTML
7
star
20

livysparkjob

Apache Livy - Apache NiFi - Example Scala Spark Job
XSLT
7
star
21

ClouderaFlowManagementWorkshop

Cloudera Flow Management Workshop with Apache NiFi
Python
7
star
22

nifi-djl-processor

Apache NiFi 1.10 DJL
Java
7
star
23

nifi-dl4j-processor

Apache NiFi - DeepLearning4J Example Processor for VGG16 based https://github.com/SkymindIO/Demos-for-sales-marketting
Java
7
star
24

rpi-mxnet-movidius-minifi

USB WebCam - Raspberry Pi - Apache MXNet - Movidius - Apache NiFi - MiniFi
Python
7
star
25

nifi-langdetect-processor

Apache NiFi + Apache Tika + OptimaizeLangDetector
Java
7
star
26

flume-to-nifi

Examples for Apache Flume to Apache NiFi
7
star
27

MmFLaNK

Mm FLaNK Stack (MXNet, MiNiFi, Flink, NiFi, Kafka, Kudu) for AI-IoT
Java
6
star
28

airquality

Spring Boot, JDK 11, Apache Pulsar 2.9.1, REST, AirNow
Java
6
star
29

FLiPStackWeekly

FLiP Stack Weekly covering Apache NiFi, Apache Flink, Apache Kafka, Apache Spark, Apache Iceberg, Apache Ozone, Apache Pulsar, and more...
6
star
30

rpi-picamera-mqtt-nifi

Ingesting Camera Images and Video from Raspberry Pi + Pi Camera Through MQTT and FTP
Python
6
star
31

FLiPC-FastStreamingIntoClickhouseWithApachePulsar

Fast Streaming into Clickhouse with Apache Pulsar - Meetup 2022
6
star
32

minifi-jetson-nano

MiNiFi Agent Configuration and Scripts for NVidia Jetson Nano device
Python
6
star
33

ApacheConAtHome2020

ApacheCon @Home Sept-Oct 2020 Materials
HTML
6
star
34

SmartWeather

FLaNK: Smart Weather
Java
5
star
35

nifi-cdsw-edge

Edge to AI: IoT
Python
5
star
36

EnterpriseNIFI

Enterprise NIFI Talks and Code
5
star
37

nifi-gluoncv-yolo3

Using Apache MXNet Gluon 0.3 for YOLO3 Pretrained model in the zoo!
Python
5
star
38

rpi-sensehat-minifi-python

Raspberry Pi Python Code for use with Apache NiFi & Apache MiniFi Java Agent
Python
5
star
39

meetup-sensors

sensors, machine learning, nifi, minifi, python, enviroplus, iot, streaming, flank, kafka, time series, cml
Shell
5
star
40

nifi-processors

All my processors (NARs) in one place
5
star
41

ApacheNiFiTemplates

Templates for NiFi
5
star
42

nifi-deepspeech

Using Mozilla TensorFlow implementation of Baidu's Deep Speech
Python
5
star
43

TensorflowOnYARN

Examples of Python TensorFlow apps running on HDP 3 / YARN 3.1
Python
5
star
44

FLiPN-NFT

Reading OpenSea NFT with Apache NiFi, Apache Pulsar and Friends. Meetup. StreamNative
HTML
5
star
45

stateless-examples

Apache NiFi 1.10 Stateless Examples
Shell
5
star
46

StreamingAnalyticsUsingFlinkSQL

FLiP: StreamNative: Cloud-Native: Streaming Analytics Using Apache Flink SQL on Apache Pulsar
Java
5
star
47

pulsar-transit-function

Apache Pulsar - Transit Enrichment and Routing Function
Java
4
star
48

FLiP-EdgeAI

Edge AI with Jetson Nano + Sensors
Python
4
star
49

rpi-sensehat-mqtt-nifi

Using SenseHat for Sensor Ingest into Raspberry Pi 3 B+ via Python to Apache NiFi
Python
4
star
50

SmartStocks

Smart Stocks
4
star
51

onionomega-mqtt-micropython

Onion Omega 2+ with OLED and Micropython
Python
4
star
52

nifi-djlsentimentanalysis-processor

nifi-djlsentimentanalysis-processor
Java
4
star
53

FLiP-Pi-BreakoutGarden

Python
4
star
54

minifi-breakoutgarden

Using MiniFi with the Pimoroni Breakout Garden Hat and Sensors
Python
4
star
55

spring-pulsar-airquality

Java
4
star
56

nifi-energy-monitoring

Updated real-time energy monitoring with Python and Apache NiFi
Python
4
star
57

iot-scripts

Scripts for IoT (mostly Python and Bash)
Python
4
star
58

linkextractorprocessor

Apache NiFi Custom Processor For Link Extracting
Java
4
star
59

FLiPN-Demos

Some FLiPN Demos
HTML
4
star
60

cdp-datahub-azure-nifikafka

Cloudera CDP Datahub on Azure for Apache NiFi and Apache Kafka Demo
TSQL
4
star
61

IoTFusion2018Talk

My Talk at IoT Fusion 2018 Philadelphia, PA
Python
4
star
62

yolo3-keras-tensorflow

A tweak to https://github.com/qqwweee/keras-yolo3 to store images and json attributes
Python
4
star
63

nifi-djlqa-processor

nifi-djlqa-processor
Java
4
star
64

IngestingDroneData

Ingesting Drone Data into Big Data Platforms
Python
4
star
65

spy

Inspired by TURN and Robin David's awesome python library.
Python
3
star
66

FLiP-PY-FakeDataPulsar

Generate Fake Data
Python
3
star
67

rpizw-nifi-mqtt-gps

GPS Sensor Reading from Raspberry Pi Zero Wireless Sending via MQTT to Apache NiFi 1.x
Python
3
star
68

FLiP-IoT

FLiP - IoT Examples with Go, Java, MiNiFi, Flink, Pulsar, StreamNative, JSON
Python
3
star
69

CloudDemo2021

HTML
3
star
70

nifi-osquery

osquery nifi integration
Shell
3
star
71

EverythingMiNiFi

Apache NiFi - MiNiFi Agents. Java - rolling into NiFi. MiNiFi C++ Agent.
3
star
72

DWS-DeepLearning-CrashCourse

TensorFlow Inception Apache Zeppelin Notebook Python
Jupyter Notebook
3
star
73

brains-kafka

Scala
3
star
74

pulsar-airquality-timeplus

Real-Time Analytics on AirQuality Data with timeplus
3
star
75

spring-pulsar-reactiveair

Reactive Spring Example for Apache Pulsar
Java
3
star
76

airline-sentiment-streaming

Streaming with Airline Sentiment. Utilizing Cloudera Machine Learning, Apache NiFi, Apache Hue, Apache Impala, Apache Kudu
Jupyter Notebook
3
star
77

hbasecoprocessor

Example HBase Coprocessor
Java
3
star
78

FLiP-Transit

Apache Flink - Apache Pulsar - Apache NiFi -> Real-Time Transit Data
3
star
79

AllThingsNiFi

Everything Apache NiFi, Cloudera DataFlow
3
star
80

nifi-cdsw

Example Apache NiFi to CDSW
Python
3
star
81

kafkaadmin-processor

Apache Kafka Apache NiFi
Java
3
star
82

HBase2

Apache HBase 2.0 with Apache NiFi 1.10, Apache Kafka 2.2, Apache Flink 1.9
3
star
83

nifi-cdsw-gluoncv

Using Apache MXNet GluonCV YOLO with Apache NiFi and Cloudera Data Science Workbench
Python
3
star
84

nifi-mxnetinference-processor

Apache NiFi Processor For Apache MXNet Inference
Java
3
star
85

FLiP-Pi-Thermal

Python
3
star
86

FLiPN-AirQuality-REST

Apache NiFi, Apache Pulsar. Air Quality Readings
HTML
3
star
87

gtfs

GTFS / ProtoBuf Data
Java
3
star
88

PulsarOnRaspberryPi

Pulsar on Pi
CMake
3
star
89

BackupRegistry

Backup the NiFi Registry
3
star
90

FLiP-Stream2Clickhouse

Stonks. Stocks. Apache Pulsar - Stream to Altinity Cloud - Clickhouse
3
star
91

phillycrime-springboot-phoenix

Apache Phoenix - Apache HBase - SpringBoot - Philly Crime Data
Java
3
star
92

mxnet-for-iot

Apache MXNet for IoT
Python
3
star
93

operations-dashboard

Apache NiFi operations-dashboard
Java
3
star
94

nifi-puttwitter-processor

Send out tweets with Twitter4J
Java
3
star
95

pulsar-airquality-function

pulsar-airquality-function
Java
3
star
96

pulsar-pychat-function

Python
3
star
97

nifi-ocr

OCR with Tesseract, Deep Learning, Python, Apache NiFi, ...
Python
3
star
98

nifi-nanopi-duo

NanoPi Duo + Apache MinIFi + MiniFi + Python + USB Camera + DL/ML/IOT
Python
3
star
99

obd2-nifi

OBDII Interface with Apache NiFi, MiniFi, Raspberry Pi, Comma.AI, Panda, SenseHat, GPS, Weather, Humidity, Temperature, Altitude
3
star
100

nvidiajetsontx1-mxnet

MXNet to MiniFi Example
Python
3
star