1. Launch test services - docker-compose up setup

2. Compile: cargo build

3. Run kafka-delta-ingest against the web_requests example topic and table (customize arguments as desired):

Notes:

• The AWS_* environment variables are for S3 and are required by the delta-rs library.

  • Above, AWS_ENDPOINT_URL points to localstack.

• The Kafka broker is assumed to be at localhost:9092, use -k to override.

• To clean data from previous local runs, execute ./bin/clean-example-data.sh. You’ll need to do this if you destroy your Kafka container between runs since your delta log directory will be out of sync with Kafka offsets.

In case you have Kafka topics secured by SSL client certificates, you can specify these secrets as environment variables.

For the cert chain include the PEM content as an environment variable named KAFKA_DELTA_INGEST_CERT. For the cert private key include the PEM content as an environment variable named KAFKA_DELTA_INGEST_KEY.

These will be set as the ssl.certificate.pem and ssl.key.pem Kafka settings respectively.

Make sure to provide the additional option:

when invoking the cli command as well.

A tarball containing 100K line-delimited JSON messages is included in tests/json/web_requests-100K.json.tar.gz. Running ./bin/extract-example-json.sh will unpack this to the expected location.

Pretty-printed example from the file

{
  "meta": {
    "producer": {
      "timestamp": "2021-03-24T15:06:17.321710+00:00"
    }
  },
  "method": "DELETE",
  "session_id": "7c28bcf9-be26-4d0b-931a-3374ab4bb458",
  "status": 204,
  "url": "http://www.youku.com",
  "uuid": "831c6afa-375c-4988-b248-096f9ed101f8"
}

Inspect example output

1. Create an Azure Event Hubs Namespace and within it, an Event Hub (which corresponds to a Kafka topic).

2. Set these environment variables, they are required by the delta-rs library:

   • AZURE_STORAGE_ACCOUNT_NAME (just the storage account name, not the FQDN)

   • AZURE_STORAGE_ACCOUNT_KEY (just the key, not the connection string)

3. Create the _delta_log directory in the web_requests directory in Azure Storage and upload the first Delta transaction containing the schema to this directory.

4. In the docker command below, replace the following placeholders with your values:

   • AZURE_STORAGE_ACCOUNT_NAME (just the storage account name, not the FQDN)

   • AZURE_STORAGE_ACCOUNT_KEY (just the key, not the connection string)

   • EVENTHUBS_NAMESPACE_NAME (just the namespace name, not the FQDN)

   • EVENTHUBS_KEY_NAME

   • EVENTHUBS_KEY

5. Build the docker image

Notes:

• If this takes a long time, make sure that docker has enough memory

  1. Execute this docker command to run kafka-delta-ingest

Notes:

• In the docker command:

  • The sasl.username is the literal string $ConnectionString and not a placeholder.

  • The following --Kafka arguments are taken from here:

    • socket.keepalive.enable=true

    • metadata.max.age.ms=180000

    • heartbeat.interval.ms=3000

    • session.timeout.ms=30000

On Windows, Service Bus Explorer can be used to send data to Event Hubs.

The following payload should be sent for the web_requests Delta table:

kcat can be run on Windows via docker using this command, which will print the last message (-o -1).

Make sure to first replace the following placeholders:

• EVENTHUBS_NAMESPACE_NAME (just the namespace name, not the FQDN)

• EVENTHUBS_KEY_NAME

• EVENTHUBS_KEY

Notes:

• The following configuration settings in the command above are taken from here: -X socket.keepalive.enable=true -X metadata.max.age.ms=180000 -X heartbeat.interval.ms=3000 -X session.timeout.ms=30000