Flink docker

Apache Flink docker images to:

• Setup a standalone Apache Flink cluster running one Flink Master and multiple Flink workers
• Build Flink applications in Scala, Java or Python to run on a Flink cluster

Currently supported versions:

• Flink 1.14.5 for Hadoop 3.2 and Scala 2.12
• Flink 1.14.2 for Hadoop 3.2 and Scala 2.12
• Flink 1.14.0 for Hadoop 3.2 and Scala 2.12
• Flink 1.13.0 for Hadoop 3.2 and Scala 2.12
• Flink 1.12.2 for Hadoop 3.2 and Scala 2.12
• Flink 1.7.2 for Hadoop 2.8 and Scala 2.11
• Flink 1.7.2 for Hadoop 2.7 and Scala 2.11
• Flink 1.7.1 for Hadoop 2.8 and Scala 2.11
• Flink 1.7.1 for Hadoop 2.7 and Scala 2.11
• Flink 1.7.0 for Hadoop 2.8 and Scala 2.11
• Flink 1.7.0 for Hadoop 2.7 and Scala 2.11
• Flink 1.6.3 for Hadoop 2.8 and Scala 2.11
• Flink 1.6.3 for Hadoop 2.7 and Scala 2.11
• Flink 1.6.2 for Hadoop 2.8 and Scala 2.11
• Flink 1.6.2 for Hadoop 2.7 and Scala 2.11
• Flink 1.6.1 for Hadoop 2.8 and Scala 2.11
• Flink 1.6.1 for Hadoop 2.7 and Scala 2.11
• Flink 1.6.0 for Hadoop 2.8 and Scala 2.11
• Flink 1.6.0 for Hadoop 2.7 and Scala 2.11
• Flink 1.5.5 for Hadoop 2.8 and Scala 2.11
• Flink 1.5.5 for Hadoop 2.7 and Scala 2.11
• Flink 1.5.4 for Hadoop 2.8 and Scala 2.11
• Flink 1.5.4 for Hadoop 2.7 and Scala 2.11
• Flink 1.5.3 for Hadoop 2.8 and Scala 2.11
• Flink 1.5.3 for Hadoop 2.7 and Scala 2.11
• Flink 1.5.2 for Hadoop 2.8 and Scala 2.11
• Flink 1.5.2 for Hadoop 2.7 and Scala 2.11
• Flink 1.5.1 for Hadoop 2.8 and Scala 2.11
• Flink 1.5.1 for Hadoop 2.7 and Scala 2.11
• Flink 1.5.0 for Hadoop 2.8 and Scala 2.11
• Flink 1.5.0 for Hadoop 2.7 and Scala 2.11
• Flink 1.4.2 for Hadoop 2.8 and Scala 2.11
• Flink 1.4.2 for Hadoop 2.7 and Scala 2.11
• Flink 1.4.1 for Hadoop 2.8 and Scala 2.11
• Flink 1.4.1 for Hadoop 2.7 and Scala 2.11
• Flink 1.4.0 for Hadoop 2.8 and Scala 2.11
• Flink 1.4.0 for Hadoop 2.7 and Scala 2.11
• Flink 1.3.2 for Hadoop 2.7 and Scala 2.11
• Flink 1.3.1 for Hadoop 2.7 and Scala 2.11
• Flink 1.3.0 for Hadoop 2.7 and Scala 2.11
• Flink 1.2.1 for Hadoop 2.7 and Scala 2.11
• Flink 1.2.0 for Hadoop 2.7 and Scala 2.11
• Flink 1.1.4 for Hadoop 2.7 and Scala 2.11
• Flink 1.1.3 for Hadoop 2.7 and Scala 2.11
• Flink 0.10.2 for Hadoop 2.7 and Scala 2.11
• Flink 0.10.1 for Hadoop 2.7 and Scala 2.11

Using Docker Compose

Add the following services to your docker-compose.yml to integrate a Flink master and Flink worker in your BDE pipeline:

flink-master:
   image: bde2020/flink-master:1.14.5-hadoop3.2
   hostname: flink-master
   container_name: flink-master
   environment:
      - INIT_DAEMON_STEP=setup_flink

   ports:
     - "8080:8080"
     - "8081:8081"

 flink-worker:
   image: bde2020/flink-worker:1.14.5-hadoop3.2
   hostname: flink-worker
   container_name: flink-worker
   environment:
     - FLINK_MASTER_PORT_6123_TCP_ADDR=flink-master
#    - FLINK_NUM_TASK_SLOTS=2
   depends_on:
      - "flink-master"

Running Docker containers without the init daemon

docker network create flink-net

Flink Master

To start a Flink master:

docker run --name flink-master --net flink-net -d bde2020/flink-master:1.14.5-hadoop3.2

Flink Worker

To start a Flink worker:

docker run --name flink-worker --net flink-net -e FLINK_MASTER_PORT_6123_TCP_ADDR=flink-master -d bde2020/flink-worker:1.14.5-hadoop3.2

Launch a Flink application

Building and running your Flink application on top of the Flink cluster is as simple as extending a template Docker image. Check the template's README for further documentation.

• Maven template
• Sbt template