• Stars
    star
    877
  • Rank 52,062 (Top 2 %)
  • Language
    JavaScript
  • Created over 8 years ago
  • Updated almost 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Web Tool for Kafka Topics |

kafka-topics

GitHub release docker Join the chat at https://gitter.im/Landoop/support

Browse Kafka topics and understand what's happening on your cluster. Find topics / view topic metadata / browse topic data (kafka messages) / view topic configuration / download data. This is a web tool for the confluentinc/kafka-rest proxy.

Live Demo

kafka-topics-ui.demo.lenses.io

Running it

    docker pull landoop/kafka-topics-ui
    docker run --rm -it -p 8000:8000 \
               -e "KAFKA_REST_PROXY_URL=https://kafka-rest-proxy-host:port" \
               -e "PROXY=true" \
               landoop/kafka-topics-ui

Config: If you don't use our docker image, keep in mind that Kafka-REST-Proxy CORS support can be a bit buggy, so if you have trouble setting it up, you may need to provide CORS headers through a proxy (i.e. nginx).

Note: The schema-registry is optional and topics are attempted to be read using Avro, then fall back to JSON, and finally fall back to Binary.

Build from source

    git clone https://github.com/lensesio/kafka-topics-ui.git
    cd kafka-topics-ui
    npm install -g bower
    npm install -g http-server
    npm install
    bower install
    http-server -p 8080 .

Web UI will be available at http://localhost:8080

Nginx config

If you use nginx to serve this ui, let angular manage routing with

    location / {
      add_header 'Access-Control-Allow-Origin' "$http_origin" always;
      add_header 'Access-Control-Allow-Credentials' 'true' always;
      add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
      add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Mx-ReqToken,X-Requested-With' always;

      proxy_pass https://kafka-rest-server-url:8082;
      proxy_redirect off;

      proxy_set_header  X-Real-IP  $remote_addr;
      proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header  Host $http_host;
    }

Setup Kafka Rest clusters

Use multiple Kafka Rest clusters in env.js :

var clusters = [
    {
      NAME: "prod",
      KAFKA_REST: "https://kafka-rest-ip:8082",
      MAX_BYTES: "50000",
      RECORD_POLL_TIMEOUT: "5000",
      DEBUG_LOGS_ENABLED: true,
      LAZY_LOAD_TOPIC_META: false
    },
    {
      NAME: "dev",
      KAFKA_REST: "localhost",
      MAX_BYTES: "50000",
      COLOR: "#141414", // Optional
      RECORD_POLL_TIMEOUT: "5000",
      DEBUG_LOGS_ENABLED: true,
      LAZY_LOAD_TOPIC_META: false
    }
  ];

Alternatively, you can pass multiple proxies within KAFKA_REST_PROXY_URL env. variable by separating them with a comma.

KAFKA_REST_PROXY_URL="http://kafka-rest-ip:8082,http://other-kafka-rest-ip:8082"

Config

  • Use MAX_BYTES to set the default maximum amount of bytes to fetch from each topic.
  • Use RECORD_POLL_TIMEOUT to set the timeout in ms.
  • Use COLOR to set different header colors for each set up cluster.
  • Set DEBUG_LOGS_ENABLED to true to enable the debug logs.
  • Set LAZY_LOAD_TOPIC_META to true to lazy load topic meta information.

CP Version support

Latest release is for CP 3.2.0 and above.

For versions older than CP 3.2.0 you will need kafka topics ui version 0.8.3. You can also build it from source by running:

    git clone https://github.com/lensesio/kafka-topics-ui.git
    cd kafka-topics-ui
    git checkout tags/v0.8.3 -b v0.8.3
    npm install -g bower
    npm install
    http-server .

Important: For Kafka REST Proxy 3.2.x you should set consumer.request.timeout.ms=30000. Without this option, Kafka REST Proxy will fail to return messages for large topics. Although the default value is 1000, a bug in the Kafka REST code prevents you from manually setting (depending on some other consumer options) a value lower than 30000.

Changelog

Here

Common Issues

If having "CONNECTIVITY ERROR" problems make sure the file kafka-rest.properties has CORS enabled. To enable CORS add the following configuration to that file, and restart the backend Kafka-Rest

access.control.allow.methods=GET,POST,PUT,DELETE,OPTIONS
access.control.allow.origin=*

If using a recent version of the Kafka-Topics-UI and Kafka-REST, make sure that you have properly configured Kafka-REST with the new consumer API. That requires setting up in the configuration of Kafka REST

bootstrap.servers=PLAINTEXT://ip-address-of-kafka-broker:9092

Make sure you restart Kafka REST after changing it's configuration files

License

The project is licensed under the BSL license.

Relevant Projects

  • schema-registry-ui, View, create, evolve and manage your Avro Schemas for multiple Kafka clusters
  • kafka-connect-ui, Set up and manage connectors for multiple connect clusters
  • fast-data-dev, Docker for Kafka developers (schema-registry,kafka-rest,zoo,brokers,landoop)
  • Landoop-On-Cloudera, Install and manage your kafka streaming-platform on your Cloudera CDH cluster

More Repositories

1

fast-data-dev

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors
Shell
2,019
star
2

stream-reactor

A collection of open source Apache 2.0 Kafka Connector maintained by Lenses.io.
Scala
1,005
star
3

kafka-connect-ui

Web tool for Kafka Connect |
JavaScript
504
star
4

schema-registry-ui

Web tool for Avro Schema Registry |
JavaScript
421
star
5

kafka-cheat-sheet

Curated by Lenses.io
305
star
6

lenses-docker

❤for real-time DataOps - where the application and data fabric blends - Lenses
Shell
154
star
7

kafka-connect-tools

Kafka Connect Tooling
Scala
118
star
8

schema-registry

A CLI and Go client for Kafka Schema Registry. This repository is deprecated and no longer maintained.
Go
117
star
9

kafka-connect-query-language

SQL for Kafka Connectors
Java
96
star
10

coyote

Environment, operations and runtime-meta testing tool.
Go
90
star
11

cypress-websocket-testing

Test WebSocket connections with Cypress
TypeScript
83
star
12

tableprinter

Fast and easy-to-use table printer written in Go. Parse and print anywhere go values such as structs, slices, maps or any compatible value type as a table with ease.
Go
77
star
13

kafka-helm-charts

In-deprecation. For Lenses please check lensesio/lenses-helm-charts. Soon Stream Reactor will also get its own Helm repository.
Smarty
70
star
14

secret-provider

Open Source Secret Provider plugin for the Kafka Connect framework
Scala
45
star
15

lenses-go

Lenses.io CLI (command-line interface)
Go
35
star
16

kafka-autocomplete

Bash completion for Kafka command line utilities.
Shell
34
star
17

kafka-connect-kcql-smt

Kafka-Connect SMT (Single Message Transformations) with SQL syntax (Using Apache Calcite for the SQL parsing)
Scala
32
star
18

kafka-connectors-tests

Test suite for Kafka Connect connectors based on Landoop's Coyote and docker.
Dockerfile
32
star
19

fast-avro-write

Writing an Avro file is not as fast as you might want it. This is a library to write considerably faster to an avro file.
Scala
32
star
20

avro-sql

Use SQL to transform your avro schema/records
Scala
28
star
21

kafka-testing

Repository for advanced unit-testing with embedded kafka services
Scala
25
star
22

fast-data-connect-cluster

Create Kafka-Connect clusters with docker . You put the Kafka, we put the Connect.
Shell
25
star
23

json-sql

Scala
22
star
24

kafka-connect-common

Common components used across the datamountaineer kafka connect connectors
Scala
21
star
25

lenses-jdbc

Lenses.io JDBC driver for Apache Kafka
Kotlin
20
star
26

stream-reactor-dockers

Docker images for Stream Reactor
Shell
19
star
27

lenses-python

Python library for Lenses.io / Kafka platform engineers
Python
18
star
28

lenses-helm-charts

Helm Charts for Lenses.io
HTML
18
star
29

hadoop-bash-completion

Bash completion for hadoop. Includes commands, switches and directory/filename completion.
Shell
17
star
30

lenses-topology-example

An example of streaming microservices with Apache Kafka and Data Flow Topology integration with Lenses Ⓡ DataOps Platform. You can see it in action here: https://www.youtube.com/watch?v=V15mrcA1Wxg
Java
15
star
31

datagen

A small project to allow publishing data to Apache Kafka, Apache Pulsar or any other target system
Scala
14
star
32

landoop-avro4s-ui

User interface for avro4s
HTML
14
star
33

redux-lenses-streaming-example

JavaScript
11
star
34

kafka-testkit

TestKit - Embedded Kafka, Zookeeper, Schema Registry
Java
10
star
35

docker-images

Public docker images we use for our infrastructure.
Go
10
star
36

ftp-kafka-converters

Blog on Kafka Connect [ FTP -> KAFKA ]
Scala
9
star
37

kafka-connect-transformers

Scala
8
star
38

ais-avro-mqtt-connect-tutorial

A small tutorial on Avro, MQTT and Kafka using docker and go.
Go
7
star
39

lenses-jdbc-spark

Apache Spark with Kafka via JDBC !!!
Java
6
star
40

kafka-avro-generator

Scala
6
star
41

lenses-ldap-plugin-template

Java
6
star
42

kafka-celonis-ems-connector

Contains the documentation for the Kafka Execution Management System connector.
5
star
43

lenses-serde

Java
5
star
44

bite

The Cobra's bite
Go
5
star
45

lenses-alerts-plugin

Defines interface for pluggable lenses alert services integration, along with some officially supported implementations.
Scala
4
star
46

avro4s-ui

Avro <=> Scala through a web ui => http://landoop.com/labs
Scala
4
star
47

ansible

Sample roles from Landoop DevOps team.
Shell
4
star
48

lenses-cloud-templates

Available templates to deploy Lenses in different cloud providers
Python
3
star
49

avro-kcql

Scala
3
star
50

connect-mock

This is a dummy application that mocks the Kafka Connect API.
Kotlin
3
star
51

kafka-connect-smt

A Lenses.io Kafka Connect Single Message Transform(SMT) collection
Java
3
star
52

kafka-ui-lib

JavaScript
2
star
53

lenses-udf-example

Lenses.io - User Defined Functions for Lenses SQL
Java
2
star
54

lenses-serde-protobuf-example

Java
2
star
55

json-kcql

Provides support for json projections using the sql like syntax of KCQL
Scala
1
star
56

sql-core

Scala
1
star
57

lenses-security-http-sample

A sample project using the HTTP request to handle the authentication
Java
1
star