Fluentd Daemonset for Kubernetes
Supported tags and respective Dockerfile links
See also dockerhub tags page: https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset/tags
Debian
Current stable
As the current limitation about the number of automated builds on hub.docker.com, there are some restrictions to ship daemonset images:
stackdriver,papertrail,syslogimages (x86_64/arm64) won't be published anymorelogentries,loggly,logzio,s3arm64 images won't be published anymore (x86_64 only supported)
If you want to use above non published images, build it by yourself. Dockerfile itself is still maintained in this repository.
Multi-Arch images
Azureblobdocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-azureblob-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-azureblob-1
Elasticsearch8docker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-elasticsearch8-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch8-1
Elasticsearch7docker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-elasticsearch7-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch7-1docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
Opensearchdocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-opensearch-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-opensearch-1
Cloudwatchdocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-cloudwatch-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-cloudwatch-1
Forwarddocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-forward-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-forward-1
Gcsdocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-gcs-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-gcs-1
Graylogdocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-graylog-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-graylog-1
Kafkadocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-kafka-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-kafka-1
Kafka2docker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-kafka2-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-kafka2-1
Kinesisdocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-kinesis-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-kinesis-1
x86_64 images
AzureblobDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-azureblob-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-azureblob-amd64-1
Elasticsearch8Dockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-elasticsearch8-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch8-amd64-1
Elasticsearch7Dockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-elasticsearch7-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch7-amd64-1docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch-amd64
Elasticsearch6DockerfileOpensearchDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-opensearch-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-opensearch-amd64-1
LogglyDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-loggly-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-loggly-amd64-1
LogentriesDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-logentries-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-logentries-amd64-1
CloudwatchDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-cloudwatch-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-cloudwatch-amd64-1
StackdriverDockerfileS3Dockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-s3-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-s3-amd64-1
SyslogDockerfileForwardDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-forward-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-forward-amd64-1
GcsDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-gcs-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-gcs-amd64-1
GraylogDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-graylog-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-graylog-amd64-1
PapertrailDockerfileLogzioDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-logzio-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-logzio-amd64-1
KafkaDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-kafka-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-kafka-amd64-1
Kafka2Dockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-kafka2-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-kafka2-amd64-1
KinesisDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-kinesis-amd64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-kinesis-amd64-1
arm64 images
AzureblobDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-azureblob-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-azureblob-arm64-1
Elasticsearch8Dockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-elasticsearch8-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch8-arm64-1
Elasticsearch7Dockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-elasticsearch7-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch7-arm64-1docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch-arm64
Elasticsearch6DockerfileOpensearchDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-opensearch-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-opensearch-arm64-1
LogglyDockerfileLogentriesDockerfileCloudwatchDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-cloudwatch-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-cloudwatch-arm64-1
StackdriverDockerfileS3DockerfileSyslogDockerfileForwardDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-forward-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-forward-arm64-1
GcsDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-gcs-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-gcs-arm64-1
GraylogDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-graylog-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-graylog-arm64-1
PapertrailDockerfileLogzioDockerfileKafkaDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-kafka-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-kafka-arm64-1
Kafka2Dockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-kafka2-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-kafka2-arm64-1
KinesisDockerfiledocker pull fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-kinesis-arm64-1.0docker pull fluent/fluentd-kubernetes-daemonset:v1.16-debian-kinesis-arm64-1
You can also use v1-debian-PLUGIN tag to refer latest v1 image, e.g. v1-debian-elasticsearch. On production, strict tag is better to avoid unexpected update.
See dockerhub's tags page for older tags.
Old stable
v0.12 development has been ended. These images are never updated.
v0.12-debian-elasticsearcharchived-image/v0.12/debian-elasticsearch/Dockerfilev0.12-debian-logglyarchived-image/v0.12/debian-loggly/Dockerfilev0.12-debian-logentriesarchived-image/v0.12/debian-logentries/Dockerfilev0.12-debian-cloudwatcharchived-image/v0.12/debian-cloudwatch/Dockerfilev0.12-debian-stackdriverarchived-image/v0.12/debian-stackdriver/Dockerfilev0.12-debian-s3archived-image/v0.12/debian-s3/Dockerfilev0.12-debian-gcsarchived-image/v0.12/debian-gcs/Dockerfilev0.12-debian-papertrailarchived-image/v0.12/debian-papertrail/Dockerfilev0.12-debian-syslogarchived-image/v0.12/debian-syslog/Dockerfilev0.12-debian-graylogarchived-image/v0.12/debian-graylog/Dockerfilev0.12-debian-logzioarchived-image/v0.12/debian-logzio/Dockerfilev0.12-debian-kafkaarchived-image/v0.12/debian-kafka/Dockerfilev0.12-debian-splunkhecarchived-image/v0.12/debian-splunkhec/Dockerfilev0.12-debian-kinesisarchived-image/v0.12/debian-kinesis/Dockerfile
Alpine Linux (This is deprecated. Use Debian images instead)
v0.12-alpine-elasticsearcharchived-image/v0.12/alpine-elasticsearch/Dockerfilev0.12-alpine-logglyarchived-image/v0.12/alpine-loggly/Dockerfilev0.12-alpine-logentriesarchived-image/v0.12/alpine-logentries/Dockerfilev0.12-alpine-cloudwatcharchived-image/v0.12/alpine-cloudwatch/Dockerfilev0.12-alpine-stackdriverarchived-image/v0.12/alpine-stackdriver/Dockerfilev0.12-alpine-s3archived-image/v0.12/alpine-s3/Dockerfilev0.12-alpine-gcsarchived-image/v0.12/alpine-gcs/Dockerfilev0.12-alpine-papertrailarchived-image/v0.12/alpine-papertrail/Dockerfilev0.12-alpine-syslogarchived-image/v0.12/alpine-syslog/Dockerfilev0.12-alpine-graylogarchived-image/v0.12/alpine-graylog/Dockerfilev0.12-alpine-logzioarchived-image/v0.12/alpine-logzio/Dockerfilev0.12-alpine-kafkaarchived-image/v0.12/alpine-kafka/Dockerfilev0.12-alpine-kinesisarchived-image/v0.12/alpine-kinesis/Dockerfilev0.12-alpine-splunkhecarchived-image/v0.12/alpine-splunkhec/Dockerfile
What is Fluentd?
Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
Image versions
Fluentd versioning is as follows:
| Series | Description |
|---|---|
| v1.x | current stable |
| v0.12 | Old stable, no longer updated |
Settings
Default image version
Default YAML uses latest v1 images like fluent/fluentd-kubernetes-daemonset:v1-debian-kafka. If you want to avoid unexpected image update, specify exact version for image like fluent/fluentd-kubernetes-daemonset:v1.8.0-debian-kafka-1.0.
Run as root
This is for v0.12 images.
In Kubernetes and default setting, fluentd needs root permission to read logs in /var/log and write pos_file to /var/log.
To avoid permission error, you need to set FLUENT_UID environment variable to 0 in your Kubernetes configuration.
Use your configuration
These images have default configuration and support some environment variables for parameters but it sometimes doesn't fit your case. If you want to use your configuration, use ConfigMap feature.
Each image has following configurations:
- fluent.conf: Destination setting, Elaticsearch, kafka and etc.
- kubernetes.conf: k8s specific setting.
tailinput for log files andkubernetes_metadatafilter - tail_container_parse.conf: parser setting for
/var/log/containers/*.log. See also "Use CRI parser for containerd/cri-o" logs section - prometheus.conf: prometheus plugin for fluentd monitoring
- systemd.conf: systemd plugin for collecting systemd-journal log. See also "Disable systemd input" section.
Overwrite conf file via ConfigMap. See also several examples:
Use CRI parser for containerd/cri-o logs
This feature is available since v1.12.0-xxx-1.1.
By default, these images use json parser for /var/log/containers/ files because docker generates json formatted logs.
On the other hand, containerd/cri-o use different log format. To parse such logs, you need to use cri parser instead.
You can use cri parser by overwriting tail_container_parse.conf via ConfigMap.
# configuration example
<parse>
@type cri
</parse>
See also CRI parser README
Use FLUENT_CONTAINER_TAIL_PATH to change container logs folder
You can update the default path for the container logs i.e /var/log/container/*.log and also one can add multiple path as defined in this fluentd document https://docs.fluentd.org/input/tail#path
Use FLUENT_CONTAINER_TAIL_EXCLUDE_PATH to exclude specific container logs
Since v1.9.3 or later images.
You can exclude container logs from /var/log/containers/ with FLUENT_CONTAINER_TAIL_EXCLUDE_PATH.
If you have a trouble with specific log, use this envvar, e.g. ["/var/log/containers/logname-*"].
exclude_pathparameter document: https://docs.fluentd.org/input/tail#exclude_path- Fluentd log issue with backslash: fluent/fluentd#2545
Disable systemd input
If you don't setup systemd in the container, fluentd shows following messages by default configuration.
[warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s
You can suppress these messages by setting disable to FLUENTD_SYSTEMD_CONF environment variable in your kubernetes configuration.
Disable prometheus input plugins
By default, latest images launch prometheus plugins to monitor fluentd.
You can disable prometheus input plugin by setting disable to FLUENTD_PROMETHEUS_CONF environment variable in your kubernetes configuration.
Disable sed execution on elasticsearch image
This is for older images. Latest elasticsearch images don't use sed.
By historical reason, elasaticsearch image executes sed command during startup phase when FLUENT_ELASTICSEARCH_USER or FLUENT_ELASTICSEARCH_PASSWORD is specified. This sometimes causes a problem with read only mount.
To avoid this problem, set "true" to FLUENT_ELASTICSEARCH_SED_DISABLE environment variable in your kubernetes configuration.
Running on OpenShift
This daemonset setting mounts /var/log as service account fluentd so you need to run containers as privileged container.
Here is command example:
oc project kube-system
oc create -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml
oc adm policy add-scc-to-user privileged -z fluentd
oc patch ds fluentd -p "spec:
template:
spec:
containers:
- name: fluentd
securityContext:
privileged: true"
oc delete pod -l k8s-app=fluentd-logging
This is from nekop's japanese article.
Multiple fluentd
When you want to have multiple fluentd. For example push to multiple destination like: elsticsearch + S3.
You need to use FLUENT_POS_EXTRA_DIR add additional directory for pos file.
Otherwise they share same pos file. You may found some log only on one destination.
Note
kafka image doesn't support zookeeper parameters
zookeeper gem doesn't work on Debian 10, so kafka image doesn't include zookeeper gem.
Windows k8s daemonset not supported in this repository
Maintainers don't have k8s experience on Windows. Some users create k8s daemonset on Windows:
Please check them out.
kafka image suggestion
Using debian-kafka2/debian-kafka2-arm64 images are better than using debian-kafka/debian-kafka-arm64 images.
Because debian-kafka2/debian-kafka2-arm64 images use out_kafka2 plugin but debian-kafka/debian-kafka-arm64 images use deprecated out_kafka_buffered plugin.
Maintainers
Some images are contributed by users. If you have a problem/question for following images, ask it to contributors.
- azureblob : @elsesiy
- papertrail : @alexouzounis
- kafka : @erhudy
- graylog : @rtnpro
- gcs : @andor-pierdelacabeza
- Amazon Kinesis : @shiftky
- logz.io : @SaMnCo / @jamielennox
- splunkhec: @FutureSharks
Currently, we don't accept new destination request without contribution. See #293
References
Kubernetes Logging with Fluentd
Issues
We can't notice comments in the DockerHub so don't use them for reporting issues or asking question.
If you have any problems with or questions about this image, please contact us through a GitHub issue.
Pull Request
Update templates files instead of docker-image files.
docker-image files are automatically generated from templates.
Note: This file is generated from templates/README.md.erb