lensesio/kafka-connect-tools lensesio/kafka-connect-tools

  • This repository has been archived on 12/May/2021
  • Stars
    star
    118
  • Rank 299,923 (Top 6 %)
  • Language
    Scala
  • License
    Apache License 2.0
  • Created over 8 years ago
  • Updated over 3 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
lensesio/kafka-connect-tools
github

lensesio

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Kafka Connect Tooling

Connect tools is Maven

<dependency>
	<groupId>com.datamountaineer</groupId>
	<artifactId>kafka-connect-cli</artifactId>
	<version>1.0.7</version>
</dependency>

Requirements

  • Java 1.8
  • Gradle 5

Kafka Connect CLI

This is a tiny command line interface (CLI) around the Kafka Connect REST Interface to manage connectors. It is used in a git like fashion where the first program argument indicates the command: it can be one of [ps|get|rm|create|run|status|status|plugins|describe|validate|restart|pause|resume].

The CLI is meant to behave as a good unix citizen: input from stdin; output to stdout; out of band info to stderr and non-zero exit status on error. Commands dealing with configuration expect or produce data in .properties style: key=value lines and comments start with a #.

connect-cli 1.0.8
Usage: connect-cli [ps|get|rm|create|run|diff|status|plugins|describe|validate|restart|pause|resume] [options] [<connector-name>]

  --help
        prints this usage text
  -e <value> | --endpoint <value>
        Kafka Connect REST URL, default is http://localhost:8083/
  -f <value> | --format <value>
        Format of the config, default is PROPERTIES. Valid options are 'properties' and 'json'.

  Command: ps
  list active connectors names.
  
  Command: get
  get the configuration of the specified connector.
  
  Command: rm
  remove the specified connector.
  
  Command: create
  create the specified connector with the config from stdin; the connector cannot already exist.
  
  Command: run
  create or update the specified connector with the config from stdin.
  
  Command: diff
  diff the specified connector with the config from stdin.
  
  Command: status
  get connector and it's task(s) state(s).
  
  Command: plugins
  list the available connector class plugins on the classpath.
  
  Command: describe
  list the configurations for a connector class plugin on the classpath.
  
  Command: pause
  pause the specified connector.
  
  Command: restart
  restart the specified connector.
  
  Command: resume
  resume the specified connector.
  
  Command: validate
  validate the connector config from stdin against a connector class plugin on the classpath.
  
  Command: task_ps
  list the tasks belonging to a connector.
  
  Command: task_status
  get the status of a connector task.
  
  Command: task_restart
  restart the specified connector task.

You can override the default endpoint by setting an environment variable KAFKA_CONNECT_REST i.e.

export KAFKA_CONNECT_REST="http://myserver:myport"

To Build

gradle buildCli

Usage

Clone this repository, do a gradle buildCli and run the jar in a way you prefer, for example with the provided cli shell script. The CLI can be used as follows.

Get Active Connectors

Command: ps

Example:

$ bin/connect-cli ps
twitter-source

Get Connector Configuration

Command: get

Example:

$ bin/connect-cli get twitter-source
#Connector `twitter-source`:
name=twitter-source
tasks.max=1

(snip)

track.terms=test
#task ids: 0

Delete a Connector

Command: rm

Example:

$ bin/connect-cli rm twitter-source

Create a New Connector

The connector cannot already exist.

Command: create

Example:

$ bin/connect-cli create twitter-source <twitter.properties
#Connector `twitter-source`:
name=twitter-source
tasks.max=1

(snip)

track.terms=test
#task ids: 0

Create or Update a Connector

Either starts a new connector if it did not exist, or update an existing connector.

Command: run

Example:

$ bin/connect-cli run twitter-source <twitter.properties
#Connector `twitter-source`:
name=twitter-source
tasks.max=1

(snip)

track.terms=test
#task ids: 0

Diff a Connector

Diffs a connector config with the provided config.

Command: diff

Example:

$ bin/connect-cli run twitter-source <twitter.properties
#Connector `twitter-source`:
name=twitter-source
tasks.max=1

(snip)

track.terms=test
#task ids: 0

Query Connector Status

Shows a connector's status and the state of its tasks.

Command: status

Example:

bin/connect-cli status my-toy-connector
connectorState: RUNNING
numberOfTasks: 3
tasks:
  - taskId: 0
    taskState: RUNNING
  - taskId: 1
    taskState: FAILED
    trace: java.lang.Exception: broken on purpose
    at java.lang.Thread.run(Thread.java:745)
  - taskId: 2
    taskState: FAILED
    trace: java.lang.Exception: broken on purpose
    at java.lang.Thread.run(Thread.java:745)

Check which Plugins are on the Classpath and available in the Connect Cluster

Shows which Connector classes are available on the classpath.

Command: plugins

Example:

    bin/connect-cli plugins
    Class name: com.datamountaineeer.streamreactor.connect.blockchain.source.BlockchainSourceConnector
    Class name: com.datamountaineer.streamreactor.connect.elastic.ElasticSinkConnector
    Class name: com.datamountaineer.streamreactor.connect.druid.DruidSinkConnector
    Class name: io.confluent.connect.hdfs.HdfsSinkConnector
    Class name: io.confluent.connect.jdbc.JdbcSourceConnector
    Class name: com.datamountaineer.streamreactor.connect.hbase.HbaseSinkConnector
    Class name: org.apache.kafka.connect.file.FileStreamSourceConnector
    Class name: com.datamountaineer.streamreactor.connect.hazelcast.sink.HazelCastSinkConnector
    Class name: com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector
    Class name: com.datamountaineer.streamreactor.connect.rethink.source.ReThinkSourceConnector
    Class name: com.datamountaineer.streamreactor.connect.jms.sink.JMSSinkConnector
    Class name: com.datamountaineer.streamreactor.connect.influx.InfluxSinkConnector
    Class name: com.datamountaineer.streamreactor.connect.redis.sink.RedisSinkConnector
    Class name: com.datamountaineer.streamreactor.connect.bloomberg.BloombergSourceConnector
    Class name: com.datamountaineer.streamreactor.connect.yahoo.source.YahooSourceConnector
    Class name: com.datamountaineer.streamreactor.connect.kudu.sink.KuduSinkConnector
    Class name: org.apache.kafka.connect.file.FileStreamSinkConnector
    Class name: com.datamountaineer.streamreactor.connect.cassandra.source.CassandraSourceConnector
    Class name: com.datamountaineer.streamreactor.connect.voltdb.VoltSinkConnector
    Class name: com.datamountaineer.streamreactor.connect.mongodb.sink.MongoSinkConnector
    Class name: com.datamountaineer.streamreactor.connect.rethink.sink.ReThinkSinkConnector
    Class name: io.confluent.connect.hdfs.tools.SchemaSourceConnector

Describe the configuration of a Connector

Describes the configuration parameters for a Connector.

Command: describe

Example:

bin/connect-cli describe ReThinkSinkConnector
{
  "name": "com.datamountaineer.streamreactor.connect.rethink.sink.ReThinkSinkConnector",
  "error_count": 3,
  "groups": ["Common", "Connection"],
  "configs": [{
    "definition": {
      "name": "connector.class",
      "display_name": "Connector class",
      "importance": "HIGH",
      "order": 2,
      "default_value": "",
      "dependents": [],
      "type": "STRING",
      "required": true,
      "group": "Common"
    },
    "value": {
      "name": "connector.class",
      "recommended_values": [],
      "errors": ["Missing required configuration \"connector.class\" which has no default value."],
      "visible": true
    }
  }, {
...........

Validate a Connectors properties file against the required Configurations

Given a properties file for an instance of a Connector validate it against the Connector configuration.

Command: validate

Example:

  bin/connect-cli validate ReThinkSinkConnector < ../conf/quickstarts/rethink-sink.properties
  ..............
     "definition": {
            "name": "connect.rethink.sink.port",
            "display_name": "connect.rethink.sink.port",
            "importance": "MEDIUM",
            "order": 3,
            "default_value": "28015",
            "dependents": [],
            "type": "INT",
            "required": false,
            "group": "Connection"
          },
          "value": {
            "name": "connect.rethink.sink.port",
            "visible": true,
            "errors": [],
            "recommended_values": [],
            "value": "28015"
          }
        }]
      }
      Validation failed.
      Missing required configuration "connect.rethink.sink.export.route.query" which has no default value.]:

Pause a Connector

Command: pause

Example:

bin/connect-cli pause cassandra-sink
Waiting for pause
connectorState:  RUNNING
workerId: 10.0.0.9:8083
numberOfTasks: 1
tasks:
 - taskId: 0
   taskState: RUNNING
   workerId: 10.0.0.9:8083

Resume a Connector

Command: resume

Example:

bin/connect-cli resume cassandra-sink
Waiting for resume
connectorState:  RUNNING
workerId: 10.0.0.9:8083
numberOfTasks: 1
tasks:
 - taskId: 0
   taskState: RUNNING
   workerId: 10.0.0.9:8083

Restart a Connector

Command: restart

Example:

bin/connect-cli restart cassandra-sink
Waiting for restart
connectorState:  RUNNING
workerId: 10.0.0.9:8083
numberOfTasks: 1
tasks:
 - taskId: 0
   taskState: RUNNING
   workerId: 10.0.0.9:8083

List all Tasks of a Connector

Command: task_ps

Example:

bin/connect-cli tasks_ps cassandra-sink
- cassandra-sink task 0
  connector.class: com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector
  bootstrap.servers: kafka-broker1:6667,kafka-broker2:6667,kafka-broker3:6667
  producer.schema.registry.url:http://schema-registry:8081
  ...

Get the status of a Connector Task

Command: task_status

Example:

bin/connect-cli task_status cassandra-sink 0
taskId: 0
taskState: RUNNING
workerId: 10.0.0.9:8083

Restart a Connector Task

Command: task_restart

Example:

bin/connect-cli task_restart cassandra-sink 0

Misc

Contributions are encouraged, feedback to rollulus at xs4all dot nl.

Thanks, enjoy!

More Repositories

1

fast-data-dev

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors
Shell
2,019
star
2

stream-reactor

A collection of open source Apache 2.0 Kafka Connector maintained by Lenses.io.
Scala
1,005
star
3

kafka-topics-ui

Web Tool for Kafka Topics |
JavaScript
877
star
4

kafka-connect-ui

Web tool for Kafka Connect |
JavaScript
504
star
5

schema-registry-ui

Web tool for Avro Schema Registry |
JavaScript
421
star
6

kafka-cheat-sheet

Curated by Lenses.io
305
star
7

lenses-docker

❀for real-time DataOps - where the application and data fabric blends - Lenses
Shell
154
star
8

schema-registry

A CLI and Go client for Kafka Schema Registry. This repository is deprecated and no longer maintained.
Go
117
star
9

kafka-connect-query-language

SQL for Kafka Connectors
Java
96
star
10

coyote

Environment, operations and runtime-meta testing tool.
Go
90
star
11

cypress-websocket-testing

Test WebSocket connections with Cypress
TypeScript
83
star
12

tableprinter

Fast and easy-to-use table printer written in Go. Parse and print anywhere go values such as structs, slices, maps or any compatible value type as a table with ease.
Go
77
star
13

kafka-helm-charts

In-deprecation. For Lenses please check lensesio/lenses-helm-charts. Soon Stream Reactor will also get its own Helm repository.
Smarty
70
star
14

secret-provider

Open Source Secret Provider plugin for the Kafka Connect framework
Scala
45
star
15

lenses-go

Lenses.io CLI (command-line interface)
Go
35
star
16

kafka-autocomplete

Bash completion for Kafka command line utilities.
Shell
34
star
17

kafka-connect-kcql-smt

Kafka-Connect SMT (Single Message Transformations) with SQL syntax (Using Apache Calcite for the SQL parsing)
Scala
32
star
18

kafka-connectors-tests

Test suite for Kafka Connect connectors based on Landoop's Coyote and docker.
Dockerfile
32
star
19

fast-avro-write

Writing an Avro file is not as fast as you might want it. This is a library to write considerably faster to an avro file.
Scala
32
star
20

avro-sql

Use SQL to transform your avro schema/records
Scala
28
star
21

kafka-testing

Repository for advanced unit-testing with embedded kafka services
Scala
25
star
22

fast-data-connect-cluster

Create Kafka-Connect clusters with docker . You put the Kafka, we put the Connect.
Shell
25
star
23

json-sql

Scala
22
star
24

kafka-connect-common

Common components used across the datamountaineer kafka connect connectors
Scala
21
star
25

lenses-jdbc

Lenses.io JDBC driver for Apache Kafka
Kotlin
20
star
26

stream-reactor-dockers

Docker images for Stream Reactor
Shell
19
star
27

lenses-python

Python library for Lenses.io / Kafka platform engineers
Python
18
star
28

lenses-helm-charts

Helm Charts for Lenses.io
HTML
18
star
29

hadoop-bash-completion

Bash completion for hadoop. Includes commands, switches and directory/filename completion.
Shell
17
star
30

lenses-topology-example

An example of streaming microservices with Apache Kafka and Data Flow Topology integration with Lenses Ⓡ DataOps Platform. You can see it in action here: https://www.youtube.com/watch?v=V15mrcA1Wxg
Java
15
star
31

datagen

A small project to allow publishing data to Apache Kafka, Apache Pulsar or any other target system
Scala
14
star
32

landoop-avro4s-ui

User interface for avro4s
HTML
14
star
33

redux-lenses-streaming-example

JavaScript
11
star
34

kafka-testkit

TestKit - Embedded Kafka, Zookeeper, Schema Registry
Java
10
star
35

docker-images

Public docker images we use for our infrastructure.
Go
10
star
36

ftp-kafka-converters

Blog on Kafka Connect [ FTP -> KAFKA ]
Scala
9
star
37

kafka-connect-transformers

Scala
8
star
38

ais-avro-mqtt-connect-tutorial

A small tutorial on Avro, MQTT and Kafka using docker and go.
Go
7
star
39

lenses-jdbc-spark

Apache Spark with Kafka via JDBC !!!
Java
6
star
40

kafka-avro-generator

Scala
6
star
41

lenses-ldap-plugin-template

Java
6
star
42

kafka-celonis-ems-connector

Contains the documentation for the Kafka Execution Management System connector.
5
star
43

lenses-serde

Java
5
star
44

bite

The Cobra's bite
Go
5
star
45

lenses-alerts-plugin

Defines interface for pluggable lenses alert services integration, along with some officially supported implementations.
Scala
4
star
46

avro4s-ui

Avro <=> Scala through a web ui => http://landoop.com/labs
Scala
4
star
47

ansible

Sample roles from Landoop DevOps team.
Shell
4
star
48

lenses-cloud-templates

Available templates to deploy Lenses in different cloud providers
Python
3
star
49

avro-kcql

Scala
3
star
50

connect-mock

This is a dummy application that mocks the Kafka Connect API.
Kotlin
3
star
51

kafka-connect-smt

A Lenses.io Kafka Connect Single Message Transform(SMT) collection
Java
3
star
52

kafka-ui-lib

JavaScript
2
star
53

lenses-udf-example

Lenses.io - User Defined Functions for Lenses SQL
Java
2
star
54

lenses-serde-protobuf-example

Java
2
star
55

json-kcql

Provides support for json projections using the sql like syntax of KCQL
Scala
1
star
56

sql-core

Scala
1
star
57

lenses-security-http-sample

A sample project using the HTTP request to handle the authentication
Java
1
star