开源软件名称（OpenSource Name）：

开源软件地址(OpenSource Url)：

开源编程语言(OpenSource Language)：

开源软件介绍(OpenSource Introduction)：

kube-fluentd-operator (KFO)

Overview

Kubernetes Fluentd Operator (KFO) is a Fluentd config manager with batteries included, config validation, no needs to restart, with sensible defaults and best practices built-in. Use Kubernetes labels to filter/route logs per namespace!

kube-fluentd-operator configures Fluentd in a Kubernetes environment. It compiles a Fluentd configuration from configmaps (one per namespace) - similar to how an Ingress controller would compile nginx configuration from several Ingress resources. This way only one instance of Fluentd can handle all log shipping for an entire cluster and the cluster admin does NOT need to coordinate with namespace admins.

Cluster administrators set up Fluentd only once and namespace owners can configure log routing as they wish. KFO will re-configure Fluentd accordingly and make sure logs originating from a namespace will not be accessible by other tenants/namespaces.

KFO also extends the Fluentd configuration language making it possible to refer to pods based on their labels and the container name pattern. This enables for very fined-grained targeting of log streams for the purpose of pre-processing before shipping. Writing a custom processor, adding a new Fluentd plugin, or writing a custom Fluentd plugin allow KFO to be extendable for any use case and any external logging ingestion system.

Finally, it is possible to ingest logs from a file on the container filesystem. While this is not recommended, there are still legacy or misconfigured apps that insist on logging to the local filesystem.

Try it out

The easiest way to get started is using the Helm chart. Official images are not published yet, so you need to pass the image.repository and image.tag manually:

git clone [email protected]:vmware/kube-fluentd-operator.git
helm install kfo ./kube-fluentd-operator/charts/log-router \
  --set rbac.create=true \
  --set image.tag=v1.16.6 \
  --set image.repository=vmware/kube-fluentd-operator

Alternatively, deploy the Helm chart from a Github release:

CHART_URL='https://github.com/vmware/kube-fluentd-operator/releases/download/v1.16.6/log-router-0.4.0.tgz'

helm install kfo ${CHART_URL} \
  --set rbac.create=true \
  --set image.tag=v1.16.6 \
  --set image.repository=vmware/kube-fluentd-operator

Then create a namespace demo and a configmap describing where all logs from demo should go to. The configmap must contain an entry called "fluent.conf". Finally, point the kube-fluentd-operator to this configmap using annotations.

kubectl create ns demo

cat > fluent.conf << EOF
<match **>
  @type null
</match>
EOF

# Create the configmap with a single entry "fluent.conf"
kubectl create configmap fluentd-config --namespace demo --from-file=fluent.conf=fluent.conf


# The following step is optional: the fluentd-config is the default configmap name.
# kubectl annotate namespace demo logging.csp.vmware.com/fluentd-configmap=fluentd-config

In a minute, this configuration would be translated to something like this:

<match demo.**>
  @type null
</match>

Even though the tag ** was used in the <match> directive, the kube-fluentd-operator correctly expands this to demo.**. Indeed, if another tag which does not start with demo. was used, it would have failed validation. Namespace admins can safely assume that they has a dedicated Fluentd for themselves.

All configuration errors are stored in the annotation logging.csp.vmware.com/fluentd-status. Try replacing ** with an invalid tag like 'hello-world'. After a minute, verify that the error message looks like this:

# extract just the value of logging.csp.vmware.com/fluentd-status
kubectl get ns demo -o jsonpath='{.metadata.annotations.logging\.csp\.vmware\.com/fluentd-status}'
bad tag for <match>: hello-world. Tag must start with **, $thisns or demo

When the configuration is made valid again the fluentd-status is set to "".

To see kube-fluentd-operator in action you need a cloud log collector like logz.io, loggly, papertrail or ELK accessible from the K8S cluster. A simple loggly configuration looks like this (replace TOKEN with your customer token):

<match **>
   @type loggly
   loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/fluentd
</match>

Build

Get the code using go get or git clone this repo:

go get -u github.com/vmware/kube-fluentd-operator/config-reloader
cd $GOPATH/src/github.com/vmware/kube-fluentd-operator

# build a base-image
cd base-image && make build-image

# build helm chart
cd charts/log-router && make helm-package

# build the daemon
cd config-reloader
make install
make build-image

# run with mock data (loaded from the examples/ folder)
make run-once-fs

# run with mock data in a loop (may need to ctrl+z to exit)
make run-loop-fs

# inspect what is generated from the above command
ls -l tmp/

Project structure

• charts/log-router: Builds the Helm chart
• base-image: Builds a Fluentd 1.2.x image with a curated list of plugins
• config-reloader: Builds the daemon that generates fluentd configuration files

Config-reloader

This is where interesting work happens. The dependency graph shows the high-level package interaction and general dataflow.

• config: handles startup configuration, reading and validation
• datasource: fetches Pods, Namespaces, ConfigMaps from Kubernetes
• fluentd: parses Fluentd config files into an object graph
• processors: walks this object graph doing validations and modifications. All features are implemented as chained Processor subtypes
• generator: serializes the processed object graph to the filesystem for Fluentd to read
• controller: orchestrates the high-level datasource -> processor -> generator pipeline.

How does it work

It works be rewriting the user-provided configuration. This is possible because kube-fluentd-operator knows about the kubernetes cluster, the current namespace and also has some sensible defaults built in. To get a quick idea what happens behind the scenes consider this configuration deployed in a namespace called monitoring:

<filter $labels(server=apache)>
  @type parser
  <parse>
    @type apache2
  </parse>
</filter>

<filter $labels(app=django)>
  @type detect_exceptions
  language python
</filter>

<match **>
  @type es
</match>

It gets processed into the following configuration which is then fed to Fluentd:

<filter kube.monitoring.*.*>
  @type record_transformer
  enable_ruby true

  <record>
    kubernetes_pod_label_values ${record["kubernetes"]["labels"]["app"]&.gsub(/[.-]/, '_') || '_'}.${record["kubernetes"]["labels"]["server"]&.gsub(/[.-]/, '_') || '_'}
  </record>
</filter>

<match kube.monitoring.*.*>
  @type rewrite_tag_filter

  <rule>
    key kubernetes_pod_label_values
    pattern ^(.+)$
    tag ${tag}._labels.$1
  </rule>
</match>

<filter kube.monitoring.*.*.**>
  @type record_transformer
  remove_keys kubernetes_pod_label_values
</filter>

<filter kube.monitoring.*.*._labels.*.apache _proc.kube.monitoring.*.*._labels.*.apache>
  @type parser
  <parse>
    @type apache2
  </parse>
</filter>

<match kube.monitoring.*.*._labels.django.*>
  @type rewrite_tag_filter

  <rule>
    invert true
    key _dummy
    pattern /ZZ/
    tag 3bfd045d94ce15036a8e3ff77fcb470e0e02ebee._proc.${tag}
  </rule>
</match>

<match 3bfd045d94ce15036a8e3ff77fcb470e0e02ebee._proc.kube.monitoring.*.*._labels.django.*>
  @type detect_exceptions
  remove_tag_prefix 3bfd045d94ce15036a8e3ff77fcb470e0e02ebee
  stream container_info
</match>

<match kube.monitoring.*.*._labels.*.* _proc.kube.monitoring.*.*._labels.*.*>
  @type es
</match>

Configuration

Basic usage

To give the illusion that every namespace runs a dedicated Fluentd the user-provided configuration is post-processed. In general, expressions starting with $ are macros that are expanded. These two directives are equivalent: <match **>, <match $thisns>. Almost always, using the ** is the preferred way to match logs: this way you can reuse the same configuration for multiple namespaces.

The admin namespace

Kube-fluentd-operator defines one namespace to be the admin namespace. By default this is set to kube-system. The admin namespace is treated differently. Its configuration is not processed further as it is assumed only the cluster admin can manipulate resources in this namespace. If you don't plan to use any of the advanced features described bellow, you can just route all logs from all namespaces using this snippet in the admin namespace:

<match **>
 @type ...
 # destination configuration omitted
</match>

** in this context is not processed and it means literally everything.

Fluentd assumes it is running in a distro with systemd and generates logs with these Fluentd tags:

• systemd.{unit}: the journal of a systemd unit, for example systemd.docker.service
• docker: all docker logs, not containers. If systemd is used, the docker logs are in systemd.docker.service
• k8s.{component}: logs from a K8S component, for example k8s.kube-apiserver
• kube.{namespace}.{pod_name}.{container_name}: a log originating from (namespace, pod, container)

As the admin namespace is processed first, a match-all directive would consume all logs and any other namespace configuration will become irrelevant (unless <copy> is used). A recommended configuration for the admin namespace is this one (assuming it is set to kube-system) - it captures all but the user namespaces' logs:

<match systemd.** kube.kube-system.** k8s.** docker>
  # all k8s-internal and OS-level logs

  # destination config omitted...
</match>

Note the <match systemd.** syntax. A single * would not work as the tag is the full name - including the unit type, for example systemd.nginx.service

Using the $labels macro

A very useful feature is the <filter> and the $labels macro to define parsing at the namespace level. For example, the config-reloader container uses the logfmt format. This makes it easy to use structured logging and ingest json data into a remote log ingestion service.

<filter $labels(app=log-router, _container=reloader)>
  @type parser
  reserve_data true
  <parse>
    @type logfmt
  </parse>
</filter>

<match **>
  @type loggly
  # destination config omitted
</match>

The above config will pipe all logs from the pods labelled with app=log-router through a logfmt parser before sending them to loggly. Again, this configuration is valid in any namespace. If the namespace doesn't contain any log-router components then the <filter> directive is never activated. The _container is sort of a "meta" label and it allows for targeting the log stream of a specific container in a multi-container pod.

If you use Kubernetes recommended labels for the pods and deployments, then KFO will rewrite . characters into _.

For example, let's assume the following labels exist in the fluentd-config in the testing namespace:

This label $labels(_container=nginx-ingress-controller) will filter by container name pattern. The label will convert to this for example: kube.testing.*.nginx-ingress-controller._labels.*.*.

This label $labels(app.kubernetes.io/name=nginx-ingress, _container=nginx-ingress-controller) converts to this kube.testing.*.nginx-ingress-controller._labels.*.nginx_ingress.

This label $labels(app.kubernetes.io/name=nginx-ingress) converts to this $labels(kube.testing*.*._labels.*.nginx_ingress).

This fluentd configmap in the testing namespace:

<filter **>
  @type concat
  timeout_label @DISTILLERY_TYPES
  key message
  stream_identity_key cont_id
  multiline_start_regexp /^(\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}|\[\w+\]\s|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b|=\w+ REPORT====|\d{2}\:\d{2}\:\d{2}\.\d{3})/
  flush_interval 10
</filter>

<match **>
  @type relabel
  @label @DISTILLERY_TYPES
</match>

<label @DISTILLERY_TYPES>
  <filter $labels(app_kubernetes_io/name=kafka)>
    @type parser
    key_name log
    format json
    reserve_data true
    suppress_parse_error_log true
  </filter>

  <filter $labels(app.kubernetes.io/name=nginx-ingress, _container=controller)>
    @type parser
    key_name log

    <parse>
      @type json
      reserve_data true
      time_format %FT%T%:z
      emit_invalid_record_to_error false
    </parse>
  </filter>

  <match $labels(tag=noisy)>
    @type null
  </match>
</label>

will be rewritten inside of KFO pods as this:

<filter kube.testing.**>
  @type concat
  flush_interval 10
  key message
  multiline_start_regexp /^(\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}|\[\w+\]\s|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b|=\w+ REPORT====|\d{2}\:\d{2}\:\d{2}\.\d{3})/
  stream_identity_key cont_id
  timeout_label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba
</filter>

<match kube.testing.**>
  @label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba
  @type relabel
</match>

<match kube.testing.**>
  @label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba
  @type null
</match>

<label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba>
  <filter kube.testing.*.*._labels.*.kafka.*>
    @type parser
    format json
    key_name log
    reserve_data true
    suppress_parse_error_log true
  </filter>
  <filter kube.testing.*.controller._labels.nginx_ingress.*.*>
    @type parser
    key_name log

    <parse>
      @type json
      emit_invalid_record_to_error false
      reserve_data true
      time_format %FT%T%:z
    </parse>
  </filter>
  <match kube.testing.*.*._labels.*.*.noisy>
    @type null
  </match>
</label>

All plugins that change the fluentd tag are disabled for security reasons. Otherwise a rogue configuration may divert other namespace's logs to itself by prepending its name to the tag.

Ingest logs from a file in the container

The only allowed <source> directive is of type mounted-file. It is used to ingest a log file from a container on an emptyDir-mounted volume:

<source>
  @type mounted-file
  path /var/log/welcome.log
  labels app=grafana, _container=test-container
  <parse>
    @type none
  </parse>
</source>

The labels parameter is similar to the $labels macro and helps the daemon locate all pods that might log to the given file path. The <parse> directive is optional and if omitted the default @type none will be used. If you know the format of the log file you can explicitly specify it, for example @type apache2 or @type json.

The above configuration would translate at runtime to something similar to this:

<source>
  @type tail
  path /var/lib/kubelet/pods/723dd34a-4ac0-11e8-8a81-0a930dd884b0/volumes/kubernetes.io~empty-dir/logs/welcome.log
  pos_file /var/log/kfotail-7020a0b821b0d230d89283ba47d9088d9b58f97d.pos
  read_from_head true
  tag kube.kfo-test.welcome-logger.test-container

  <parse>
    @type none
  </parse>
</source>

Dealing with multi-line exception stacktraces (since v1.3.0)

Most log streams are line-oriented. However, stacktraces always span multiple lines. kube-fluentd-operator integrates stacktrace processing using the fluent-plugin-detect-exceptions. If a Java-based pod produces stacktraces in the logs, then the stacktraces can be collapsed in a single log event like this:

<filter $labels(app=jpetstore)>
  @type detect_exceptions
  # you can skip language in which case all possible languages will be tried: go, java, python, ruby, etc...
  language java
</filter>

# The rest of the configuration stays the same even though quite a lot of tag rewriting takes place

<match **>
 @type es
</match>

Notice how filter is used instead of match as described in fluent-plugin-detect-exceptions. Internally, this filter is translated into several match directives so that the end user doesn't need to bother with rewriting the Fluentd tag.

Also, users don't need to bother with setting the correct stream parameter. kube-fluentd-operator generates one internally based on the container id and the stream.

Reusing output plugin definitions (since v1.6.0)

Sometimes you only have a few valid options for log sinks: a dedicated S3 bucket, the ELK stack you manage, etc. The only flexibility you're after is letting namespace owners filter and parse their logs. In such cases you can abstract over an output plugin configuration - basically reducing it to a simple name which can be referenced from any namespace. For example, let's assume you have an S3 bucket for a "test" environment and you use loggly for a "staging" environment. The first thing you do is define these two output in the admin namespace:

admin-ns.conf:
<match systemd.** docker kube.kube-system.** k8s.**>
  @type loggly
  loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/fluentd
</match>

<plugin test>
  @type s3
  aws_key_id  YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET_KEY
  s3_bucket   YOUR_S3_BUCKET_NAME
  s3_region   AWS_REGION
</plugin>

<plugin staging>
  @type loggly
  loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/fluentd
</plugin>

In the above example for the admin configuration, the match directive is first defined to direct where to send logs for the systemd, docker, kube-system, and kubernetes control plane components. Below the match directive we have defined the plugin directives which define the log sinks that can be reused by namespace configurations.

A namespace can refer to the staging and test plugins oblivious to the fact where exactly the logs end up:

acme-test.conf
<match **>
  @type test
</match>


acme-staging.conf
<match **>
  @type staging
</match>

kube-fluentd-operator will insert the content of the plugin directive in the match directive. From then on, regular validation and postprocessing takes place.

Retagging based on log contents (since v1.12.0)

Sometimes you might need to split a single log stream to perform different processing based on the contents of one of the fields. To achieve this you can use the retag plugin that allows to specify a set of rules that match regular expressions against the specified fields. If one of the rules matches, the log is re-emitted with a new namespace-unique tag based on the specified tag.

Logs that are emitted by this plugin can be consequently filtered and processed by using the $tag macro when specifiying the tag:

<match $labels(app=apache)>
  @type retag
  <rule>
    key message
    pattern /^(ERROR) .*$/
    tag notifications.$1 # refer to a capturing group using $number
  </rule>
  <rule>
    key message
    pattern /^(FATAL) .*$/
    tag notifications.$1
  </rule>
  <rule>
    key message
    pattern /^(ERROR)|(FATAL) .*$/
    tag notifications.other
    invert true # rewrite tag when unmatch pattern
  </rule>
</match>

<filter $tag(notifications.ERROR)>
  # perform some extra processing
</filter>

<filter $tag(notifications.FATAL)>
  # perform different processing
</filter>

<match $tag(notifications.**)>
  # send to common output plugin
</match>

kube-fluentd-operator ensures that tags specified using the $tag macro never conflict with tags from other namespaces, even if the tag itself is equivalent.

Sharing logs between namespaces

By default, you can consume logs only from your namespaces. Often it is useful for multiple namespaces (tenants) to get access to the logs streams of a shared resource (pod, namespace). kube-fluentd-operator makes it possible using two constructs: the source namespace expresses its intent to share logs with a destination namespace and the destination namespace expresses its desire to consume logs from a source. As a result logs are streamed only when both sides agree.

A source namespace can share with another namespace using the @type share macro:

producer namespace configuration:

<match $labels(msg=nginx-ingress)>
  @type copy
  <store>
    @type share
    # share all logs matching the labels with the namespace "consumer"
    with_namespace consumer
  </store>
</match>

consumer namespace configuration:

# use $from(producer) to get all shared logs from a namespace called "producer"
<label @$from(producer)>
  <match **>
    # process all shared logs here as usual
  </match>
</match>

The consuming namespace can use the usual syntax inside the <label @$from...> directive. The fluentd tag is being rewritten as if the logs originated from the same namespace.

The producing namespace need to wrap @type share within a <store> directive. This is done on purpose as it is very easy to just redirect the logs to the destination namespace and lose them. The @type copy clones the whole stream.

Log metadata

Often you run mulitple Kubernetes clusters but you need to aggregate all logs to a single destination. To distinguish between different sources, kube-fluentd-operator can attach arbitrary metadata to every log event. The metadata is nested under a key chosen with --meta-key. Using the helm chart, metadata can be enabled like this:

helm install ... \
  --set meta.key=metadata \
  --set meta.values.region=us-east-1 \
  --set meta.values.env=staging \
  --set meta.values.cluster=legacy

Every log event, be it from a pod, mounted-file or a systemd unit, will now carry this metadata:

{
  "metadata": {
    "region": "us-east-1",
    "env": "staging",
    "cluster": "legacy",
  }
}

All logs originating from a file look exactly as all other Kubernetes logs. However, their stream field is not set to stdout but to the path to the source file:

{
    "message": "Some message from the welcome-logger pod",
    "stream": "/var/log/welcome.log",
    "kubernetes": {
        "container_name": "test-container",
        "host": "ip-11-11-11-11.us-east-2.compute.internal",
        "namespace_name": "kfo-test",
        "pod_id": "723dd34a-4ac0-11e8-8a81-0a930dd884b0",
        "pod_name": "welcome-logger",
        "labels": {
            "msg": "welcome",
            "test-case": "b"
        },
        "namespace_labels": {}
    },
    "metadata": {
        "region": "us-east-2",
        "cluster": "legacy",
        "env": "staging"
    }
}

Custom resource definition(CRD) support (since v1.13.0)

Custom resources are introduced from v1.13.0 release onwards. It allows to have a dedicated resource for fluentd configurations, which enables to manage them in a more consistent way and move away from the generic ConfigMaps. It is possible to create configs for a new application simply by attaching a FluentdConfig resource to the application manifests, rather than using a more generic ConfigMap with specific names and/or labels.

apiVersion: logs.vdp.vmware.com/v1beta1
kind: FluentdConfig
metadata:
  name: fd-config
spec:
  fluentconf: |
    <match kube.ns.**>
      @type relabel
      @label @NOTIFICATIONS
    </match>

    <label @NOTIFICATIONS>
     <match **>
       @type null
     </match>
    </label>

The "crd" has been introduced as a new datasource, configurable through the helm chart values, to allow users that are currently set up with ConfigMaps and do not want to perform the switchover to FluentdConfigs, to be able to keep on using them. The config-reloader has been equipped with the capability of installing the CRD at startup if requested, so no manual actions to enable it on the cluster are needed. The existing configurations though ConfigMaps can be migrated to CRDs through the following migration flow

• A new user, who is installing kube-fluentd-operator for the first time, should set the datasource: crd option in the chart. This enables the crd support
• A user who is already using kube-fluentd-operator with either datasource: default or datasource: multimap will have update to the new chart and set the 'crdMigrationMode' property to 'true'. This enables the config-reloader to launch with the crd datasource and the legacy datasource (either default or multimap depending on what was configured in the datasource property). The user can slowly migrate one by one all configmap resources to the corresponding fluentdconfig resources. When the migration is complete, the Helm release can be upgraded by changing the 'crdMigrationMode' property to 'false' and switching the datasource property to 'crd'. This will effectively disable the legacy datasource and set the config-reloader to only watch fluentdconfig resources.

Tracking Fluentd version

This projects tries to keep up with major releases for Fluentd docker image.

Plugins in latest release (1.16.6)

kube-fluentd-operator aims to be easy to use and flexible. It also favors sending logs to multiple destinations using <copy> and as such comes with many plugins pre-installed:

• fluentd (1.14.4)
• fluent-config-regexp-type (1.0.0)
• fluent-mixin-config-placeholders (0.4.0)
• fluent-plugin-amqp (0.14.0)
• fluent-plugin-azure-loganalytics (0.7.0)
• fluent-plugin-cloudwatch-logs (0.14.2)
• fluent-plugin-concat (2.5.0)
• fluent-plugin-datadog (0.14.0)
• fluent-plugin-detect-exceptions (0.0.14) - forked to allow fluentd v1 plugin api
• fluent-plugin-elasticsearch (5.1.0)
• fluent-plugin-gelf-hs (1.0.8)
• fluent-plugin-google-cloud (0.13.0) - forked to allow fluentd v1.14.x
• fluent-plugin-grafana-loki (1.2.16)
• fluent-plugin-grok-parser (2.6.2)
• fluent-plugin-json-in-json-2 (1.0.2)
• fluent-plugin-kafka (0.17.2)
• fluent-plugin-kinesis (3.4.1)
• fluent-plugin-kubernetes (0.3.1)