• Stars
    star
    305
  • Rank 136,879 (Top 3 %)
  • Language
  • Created over 7 years ago
  • Updated almost 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Curated by Lenses.io

Setup

The first thing to do is to run the landoop/fast-data-dev Docker image. For example:

docker run --rm -it -p 2181:2181 -p 3030:3030 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 -p 9581:9581 -p 9582:9582 -p 9583:9583 -p 9584:9584 -e ADV_HOST=127.0.0.1 landoop/fast-data-dev:latest

Note: Please follow the instructions on fast-data-dev README to customize the container.

You can execute a bash shell at the running container as follow:

docker run --rm -it --net=host landoop/fast-data-dev bash

Note: Kafka utilities are now available.

Topics

You can create a new Kafka topic named my-topic as follows:

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic my-topic

You can verify that the my-topic topic was successfully created by listing all available topics:

kafka-topics --list --zookeeper localhost:2181

You can add more partitions as follows:

kafka-topics --zookeeper localhost:2181 --alter --topic my-topic --partitions 16

You can delete a topic named my-topic as follows:

kafka-topics --zookeeper localhost:2181 --delete --topic my-topic

You can find more details about a topic named cc_payments as follows:

kafka-topics --describe --zookeeper localhost:2181 --topic cc_payments

You can see the under-replicated partitions for all topics as follows:

kafka-topics --zookeeper localhost:2181/kafka-cluster --describe --under-replicated-partitions

Producers

You can produce messages from standard input as follows:

kafka-console-producer --broker-list localhost:9092 --topic my-topic

You can produce new messages from an existing file named messages.txt as follows:

kafka-console-producer --broker-list localhost:9092 --topic test < messages.txt

You can produce Avro messages as follows:

kafka-avro-console-producer --broker-list localhost:9092 --topic my.Topic --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}' --property schema.registry.url=http://localhost:8081

You can enter a few new values from the console as follows:

{"f1": "value1"}

Consumers

Consume messages

You can begin a consumer from the beginning of the log as follows:

kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic --from-beginning

You can consume a single message as follows:

kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic  --max-messages 1

You can consume a single message from __consumer_offsets as follows:

  • kafka version 0.9.x.x ~ 0.10.x.x*
kafka-console-consumer --bootstrap-server localhost:9092 --topic __consumer_offsets --formatter 'kafka.coordinator.GroupMetadataManager$OffsetsMessageFormatter' --max-messages 1
  • kafka version 0.11.x.x+ *
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic __consumer_offsets --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --max-messages 1

You can consume and specify a consumer group as follows:

kafka-console-consumer --topic my-topic --new-consumer --bootstrap-server localhost:9092 --consumer-property group.id=my-group

Consume Avro messages

You can consume 10 Avro messages from a topic named position-reports as follows:

kafka-avro-console-consumer --topic position-reports --new-consumer --bootstrap-server localhost:9092 --from-beginning --property schema.registry.url=localhost:8081 --max-messages 10

You can consume all existing Avro messages from a topic named position-reports as follows:

kafka-avro-console-consumer --topic position-reports --new-consumer --bootstrap-server localhost:9092 --from-beginning --property schema.registry.url=localhost:8081

Consumers admin operations

You can list all groups as follows:

kafka-consumer-groups --new-consumer --list --bootstrap-server localhost:9092

You can describe a Group named testgroup as follows:

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group testgroup

Config

You can set the retention for a topic as follows:

kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name my-topic --add-config retention.ms=3600000

You can print all configuration overrides for a topic named my-topic as follows:

kafka-configs --zookeeper localhost:2181 --describe --entity-type topics --entity-name my-topic

You can delete a configuration override for retention.ms for a topic named my-topic as follows:

kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name my-topic --delete-config retention.ms 

Performance

Although Kafka is pretty fast by design, it is good to be able to test its performance. You can check the Produce performance of Kafka as follows:

kafka-producer-perf-test --topic position-reports --throughput 10000 --record-size 300 --num-records 20000 --producer-props bootstrap.servers="localhost:9092"

ACLs

You can add a new consumer ACL to an existing topic as follows:

kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --consumer --topic topicA --group groupA

You can add a new producer ACL to an existing topic as follows:

kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic topicA

You can list the ACLs of a topic named topicA as follows:

kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --list --topic topicA

Zookeeper

You can enter the zookeeper shell as follows:

zookeeper-shell localhost:2182 ls 

More Repositories

1

fast-data-dev

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors
Shell
2,019
star
2

stream-reactor

A collection of open source Apache 2.0 Kafka Connector maintained by Lenses.io.
Scala
1,005
star
3

kafka-topics-ui

Web Tool for Kafka Topics |
JavaScript
877
star
4

kafka-connect-ui

Web tool for Kafka Connect |
JavaScript
504
star
5

schema-registry-ui

Web tool for Avro Schema Registry |
JavaScript
421
star
6

lenses-docker

❤for real-time DataOps - where the application and data fabric blends - Lenses
Shell
154
star
7

kafka-connect-tools

Kafka Connect Tooling
Scala
118
star
8

schema-registry

A CLI and Go client for Kafka Schema Registry. This repository is deprecated and no longer maintained.
Go
117
star
9

kafka-connect-query-language

SQL for Kafka Connectors
Java
96
star
10

coyote

Environment, operations and runtime-meta testing tool.
Go
90
star
11

cypress-websocket-testing

Test WebSocket connections with Cypress
TypeScript
83
star
12

tableprinter

Fast and easy-to-use table printer written in Go. Parse and print anywhere go values such as structs, slices, maps or any compatible value type as a table with ease.
Go
77
star
13

kafka-helm-charts

In-deprecation. For Lenses please check lensesio/lenses-helm-charts. Soon Stream Reactor will also get its own Helm repository.
Smarty
70
star
14

secret-provider

Open Source Secret Provider plugin for the Kafka Connect framework
Scala
45
star
15

lenses-go

Lenses.io CLI (command-line interface)
Go
35
star
16

kafka-autocomplete

Bash completion for Kafka command line utilities.
Shell
34
star
17

kafka-connect-kcql-smt

Kafka-Connect SMT (Single Message Transformations) with SQL syntax (Using Apache Calcite for the SQL parsing)
Scala
32
star
18

kafka-connectors-tests

Test suite for Kafka Connect connectors based on Landoop's Coyote and docker.
Dockerfile
32
star
19

fast-avro-write

Writing an Avro file is not as fast as you might want it. This is a library to write considerably faster to an avro file.
Scala
32
star
20

avro-sql

Use SQL to transform your avro schema/records
Scala
28
star
21

kafka-testing

Repository for advanced unit-testing with embedded kafka services
Scala
25
star
22

fast-data-connect-cluster

Create Kafka-Connect clusters with docker . You put the Kafka, we put the Connect.
Shell
25
star
23

json-sql

Scala
22
star
24

kafka-connect-common

Common components used across the datamountaineer kafka connect connectors
Scala
21
star
25

lenses-jdbc

Lenses.io JDBC driver for Apache Kafka
Kotlin
20
star
26

stream-reactor-dockers

Docker images for Stream Reactor
Shell
19
star
27

lenses-python

Python library for Lenses.io / Kafka platform engineers
Python
18
star
28

lenses-helm-charts

Helm Charts for Lenses.io
HTML
18
star
29

hadoop-bash-completion

Bash completion for hadoop. Includes commands, switches and directory/filename completion.
Shell
17
star
30

lenses-topology-example

An example of streaming microservices with Apache Kafka and Data Flow Topology integration with Lenses Ⓡ DataOps Platform. You can see it in action here: https://www.youtube.com/watch?v=V15mrcA1Wxg
Java
15
star
31

datagen

A small project to allow publishing data to Apache Kafka, Apache Pulsar or any other target system
Scala
14
star
32

landoop-avro4s-ui

User interface for avro4s
HTML
14
star
33

redux-lenses-streaming-example

JavaScript
11
star
34

kafka-testkit

TestKit - Embedded Kafka, Zookeeper, Schema Registry
Java
10
star
35

docker-images

Public docker images we use for our infrastructure.
Go
10
star
36

ftp-kafka-converters

Blog on Kafka Connect [ FTP -> KAFKA ]
Scala
9
star
37

kafka-connect-transformers

Scala
8
star
38

ais-avro-mqtt-connect-tutorial

A small tutorial on Avro, MQTT and Kafka using docker and go.
Go
7
star
39

lenses-jdbc-spark

Apache Spark with Kafka via JDBC !!!
Java
6
star
40

kafka-avro-generator

Scala
6
star
41

lenses-ldap-plugin-template

Java
6
star
42

kafka-celonis-ems-connector

Contains the documentation for the Kafka Execution Management System connector.
5
star
43

lenses-serde

Java
5
star
44

bite

The Cobra's bite
Go
5
star
45

lenses-alerts-plugin

Defines interface for pluggable lenses alert services integration, along with some officially supported implementations.
Scala
4
star
46

avro4s-ui

Avro <=> Scala through a web ui => http://landoop.com/labs
Scala
4
star
47

ansible

Sample roles from Landoop DevOps team.
Shell
4
star
48

lenses-cloud-templates

Available templates to deploy Lenses in different cloud providers
Python
3
star
49

avro-kcql

Scala
3
star
50

connect-mock

This is a dummy application that mocks the Kafka Connect API.
Kotlin
3
star
51

kafka-connect-smt

A Lenses.io Kafka Connect Single Message Transform(SMT) collection
Java
3
star
52

kafka-ui-lib

JavaScript
2
star
53

lenses-udf-example

Lenses.io - User Defined Functions for Lenses SQL
Java
2
star
54

lenses-serde-protobuf-example

Java
2
star
55

json-kcql

Provides support for json projections using the sql like syntax of KCQL
Scala
1
star
56

sql-core

Scala
1
star
57

lenses-security-http-sample

A sample project using the HTTP request to handle the authentication
Java
1
star