confluentinc/kafka-connect-hdfs

Stars
465
Rank 94,287 (Top 2 %)
Language
Java
License
Other
Created over 9 years ago
Updated about 1 year ago

confluentinc/kafka-connect-hdfs

confluentinc

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Kafka Connect HDFS connector

Kafka Connect HDFS Connector

kafka-connect-hdfs is a Kafka Connector for copying data between Kafka and Hadoop HDFS.

Documentation for this connector can be found here.

Development

To build a development version you'll need a recent version of Kafka as well as a set of upstream Confluent projects, which you'll have to build from their appropriate snapshot branch. See the FAQ for guidance on this process.

You can build kafka-connect-hdfs with Maven using the standard lifecycle phases.

FAQ

Refer frequently asked questions on Kafka Connect HDFS here - https://github.com/confluentinc/kafka-connect-hdfs/wiki/FAQ

Contribute

Source Code: https://github.com/confluentinc/kafka-connect-hdfs
Issue Tracker: https://github.com/confluentinc/kafka-connect-hdfs/issues
Learn how to work with the connector's source code by reading our Development and Contribution guidelines.

License

This project is licensed under the Confluent Community License.

ksql

The database purpose-built for stream processing applications.

confluent-kafka-go

Confluent's Apache Kafka Golang client

confluent-kafka-python

Confluent's Kafka Python Client

confluent-kafka-dotnet

Confluent's Apache Kafka .NET client

kafka-streams-examples

Demo applications and code examples for Apache Kafka's Streams API.

kafka-rest

Confluent REST Proxy for Kafka

schema-registry

Confluent Schema Registry for Kafka

examples

Apache Kafka and Confluent Platform examples and demos

bottledwater-pg

Change data capture from PostgreSQL into Kafka

demo-scene

👾Scripts and samples to support Confluent Demos and Talks. ⚠️Might be rough around the edges ;-) 👉For automated tutorials and QA'd code, see https://github.com/confluentinc/examples/

cp-docker-images

[DEPRECATED] Docker images for Confluent Platform.

kafka-connect-jdbc

Kafka Connect connector for JDBC-compatible databases

cp-all-in-one

docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Kafka Confluent Platform

cp-helm-charts

The Confluent Platform Helm charts enable you to deploy Confluent Platform services on Kubernetes for development, test, and proof of concept environments.

kafka-connect-elasticsearch

Kafka Connect Elasticsearch connector

parallel-consumer

Parallel Apache Kafka client wrapper with per message ACK, client side queueing, a simpler consumer/producer API with key concurrency and extendable non-blocking IO processing.

kafka-tutorials

Tutorials and Recipes for Apache Kafka

kafka-images

Confluent Docker images for Apache Kafka

ducktape

System integration and performance tests

librdkafka

The Apache Kafka C/C++ library

kafka-rest-node

Node.js client for the Kafka REST proxy

confluent-platform-security-tools

Security tools for the Confluent Platform.

kafka-connect-datagen

Connector that generates data for demos

docker-images

DEPRECATED - Dockerfiles for Confluent Stream Data Platform

rest-utils

Utilities and a small framework for building REST services with Jersey, Jackson, and Jetty.

cli

CLI for Confluent Cloud and Confluent Platform

openmessaging-benchmark

camus

Mirror of Linkedin's Camus

common

Common utilities library containing metrics, config and utils

kafka-workshop

confluent-kafka-javascript

Confluent's Apache Kafka JavaScript client

training-developer-src

Source Code accompanying the Confluent Kafka for Developers course

ccloud-tools

Running Tools from Confluent Platform along with your Confluent Cloud™ Cluster

bincover

Easily measure code coverage of Golang binaries

libserdes

Avro Serialization/Deserialization C/C++ library with Confluent schema-registry support

kafka-connect-blog

Demo for Kafka Connect with JDBC and HDFS Connectors

confluent-cli

Confluent Platform CLI

ksqldb-graphql

Node.js GraphQL integration for ksqlDB

confluent-sigma

terraform-provider-confluentcloud

Confluent Cloud Terraform Provider is deprecated in favor of Confluent Terraform Provider

jmx-monitoring-stacks

📊 Monitoring examples for Confluent Cloud and Confluent Platform

confluent-kubernetes-examples

Example scenario workflows for Confluent for Kubernetes

qcon-microservices

Example online orders app composed of event-driven microservices. Built for QCon workshop.

securing-kafka-blog

Secure Kafka cluster (in a VM) for development and testing

cp-demo

Confluent Platform Demo including Apache Kafka, ksqlDB, Control Center, Schema Registry, Security, Schema Linking, and Cluster Linking

training-administration-src

Contains docker-compose file needed for Apache Kafka Administration by Confluent training

mox

A hybrid mock and proxy server - easily programmable and runs on express

terraform-state-s3

Terraform module to create the S3/DynamoDB backend to store the Terraform state+lock

common-docker

Confluent Commons with support for building and testing Docker images.

ksql-recipes-try-it-at-home

Files needed to try out KSQL Recipes for yourself

training-ksql-and-streams-src

Sample solutions for the exercises of the course KSQL & Kafka Streams

schema-registry-images

Docker Images for Schema Registry

terraform-provider-confluent

Terraform Provider for Confluent

confluent-docker-utils

Common Python utils for testing Confluent's Docker images

flink-cookbook

cp-ansible

Ansible playbooks for the Confluent Platform

ksql-images

KSQL platform docker images

coding-in-motion

Source code for the "Coding in Motion" series.

proto-go-setter

online-inferencing-blog-application

Source code and application accompanying the online inferencing blog

stream-me-up-scotty

A wide range of Digital Assets from Confluent's Solution Engineering team for Confluent Cloud

training-fundamentals-src

Source code accompanying the course "Apache Kafka Technical Essentials"

infoq-kafka-ksql

Code samples to go with InfoQ article

kafka-rest-images

Docker Images for Kafka REST

kafka-mqtt-images

Confluent Docker images for Kafka MQTT

learn-kafka-courses

Learn the basics of Apache Kafka® from leaders in the Kafka community with these video courses covering the Kafka ecosystem and hands-on exercises.

commercial-workshops

Confluent Commercial SE Team's Demo and Workshop Repository

training-cao-src

Source code accompanying the course "Monitoring, Troubleshooting and Tuning"

ccloud-connectivity

Setup and testing connectivity to Confluent Cloud

event-streaming-patterns

A collection of Event Streaming Patterns, including problem statements, solutions, and implementation examples.

vscode

Confluent for Visual Studio Code

ksqldb-recipes

ksql-workshop

demo-stream-designer

Current 2022 Confluent Keynote Demo covering Stream Designer, Stream Catalog, and Stream Sharing.

control-center-images

Docker images for enterprise control center images

kafka-connect-http-demo

A demo target for running the Confluent HTTP sink connector

castle

Castle is a test harness for Apache Kafka, Trogdor, and related projects.

demo-change-data-capture

This demo shows how to capture data changes from relational databases (Oracle and PostgreSQL) and stream them to Confluent Cloud, use ksqlDB for real-time stream processing, send enriched data to cloud data warehouses (Snowflake and Amazon Redshift).

kafkacat-images

Docker Images for Kafkacat

confluent-kafka-go-dev

[EXPERIMENTAL] Development / WIP / exploratory / test fork of confluent-kafka-go

confluent-hybrid-cloud-workshop

Confluent Hybrid Cloud Workshop

learn-practical-event-modeling

ksql-elasticsearch-demo

strata-tutorials

Content for Spring 2016 Strata tutorials

demo-database-modernization

This demo shows how to stream data to cloud databases with Confluent. It includes fully-managed connectors (Oracle CDC, RabbitMQ, MongoDB Atlas), ksqlDB/Flink SQL as stream processing engine.

flink-table-api-java-examples

Java Examples for running Apache Flink® Table API on Confluent Cloud

confluent-oauth-extensions

kafka-replicator-images

Docker images for Kafka Connect

etl

Code for ETL data pipelines

operator-earlyaccess

Confluent Operator Early Access docs

schema-registry-workshop

learn-building-flink-applications-in-java-exercises

demo-application-modernization

Application modernization example including Confluent Cloud, ksqlDB, Postgres, and Elasticsearch.

csid-secrets-providers

Enables use of external third-party systems for storing/retrieving key/value pairs with Confluent clusters.

support-metrics-common

Common utilities for metrics collection of proactive support

flink-table-api-python-examples

Python Examples for running Apache Flink® Table API on Confluent Cloud

confluent-kafka-go-example

Example application using the confluent-kafka-go client

learn-kafka-kraft

KRaft mode playground

ccloud-sdk-go-v2

SDK for interacting with Confluent Cloud

streaming-ops

Simulated production environment running Kubernetes targeting Apache Kafka and Confluent components on Confluent Cloud. Managed by declarative infrastructure and GitOps.