Skip to content

Audit Log Forwarding to Rsyslog

This example demonstrates how to forward Kubernetes API audit logs from an OpenShift cluster to an external rsyslog server using the ClusterLogForwarder resource. The filter is configured to capture only KubeVirt-related API activity (VirtualMachine, VirtualMachineInstance, VirtualMachineInstanceMigration), keeping the log volume focused and manageable.

Tested with:

Component Version
OpenShift v4.22.5
OpenShift Logging v6.6.0

Documentation: Chapter 2. Configuring log forwarding

Start rsyslog to file logging

First, deploy a simple rsyslog instance inside the cluster that receives forwarded logs over TCP and writes them to persistent storage.

Create new project

1
oc new-project rsyslog

Deploy rsyslog

1
curl -L -O https://examples.openshift.pub/cluster-configuration/logging/forwarding-demo/deploy-rsyslog.yaml

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rsyslog-config
data:
  rsyslog.conf: |
    global(workDirectory="/var/lib/rsyslog")

    module(load="imtcp")
    input(type="imtcp" port="24224")

    template(name="RemoteLogFile" type="string"
      string="/log/%HOSTNAME%/%PROGRAMNAME%.log")

    *.* action(type="omfile" file="/log/all.log")
    *.* action(type="omfile" dynaFile="RemoteLogFile")
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: rsyslog-log
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rsyslog
spec:
  selector:
    matchLabels:
      app: rsyslog
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: rsyslog
    spec:
      containers:
        - name: rsyslog
          image: registry.redhat.io/rhel9/rsyslog:9.8-1784815382
          ports:
            - containerPort: 24224
          volumeMounts:
            - mountPath: /log
              name: rsyslog-log
            - mountPath: /etc/rsyslog.conf
              name: rsyslog-config
              subPath: rsyslog.conf
        - name: jq
          image: registry.access.redhat.com/hi/jq:latest-builder
          command:
            - "/bin/sh"
            - "-c"
            - |
              tail --quiet -f /log/*/kubeAPI.log |  cut -d']'   -f2- |   jq -r '
                def pad(n): tostring | (n - length) as $p | (" " * ([0, $p] | max)) + .;
                def to_epoch:
                  split(".") |
                  (.[0] + "Z" | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) as $secs |
                  ("0." + (.[1] | rtrimstr("Z")) | tonumber) as $frac |
                  $secs + $frac;
                (.requestReceivedTimestamp | to_epoch) as $req |
                (.stageTimestamp | to_epoch) as $stage |
                ((($stage - $req) * 1000000 | round) / 1000) as $dur_ms |
                (.requestReceivedTimestamp | split("T")[1] | split(".")[0]) as $time |
                (.verb | ascii_upcase | pad(6)) as $verb |
                ("\($dur_ms)ms" | pad(12)) as $dur |
                "\($time) [\($verb)][\($dur)] [\(.responseStatus.code)] \(.requestURI) [\(.user.username)]"
                '
              sleep infinity
          volumeMounts:
            - mountPath: /log
              name: rsyslog-log
        - name: cdt
          image: registry.access.redhat.com/hi/go:latest-builder
          command:
            - "/bin/sh"
            - "-c"
            - |
              export HOME=/go/
              git clone https://github.com/openshift/cluster-debug-tools.git
              cd cluster-debug-tools/
              make
              ./kubectl-dev_tool -h
              sleep infinity

          volumeMounts:
            - mountPath: /log
              name: rsyslog-log
      volumes:
        - name: rsyslog-log
          persistentVolumeClaim:
            claimName: rsyslog-log
        - name: rsyslog-config
          configMap:
            name: rsyslog-config
---
apiVersion: v1
kind: Service
metadata:
  name: rsyslog
spec:
  selector:
    app: rsyslog
  ports:
    - protocol: TCP
      port: 24224
      targetPort: 24224

1
oc apply -f https://examples.openshift.pub/cluster-configuration/logging/forwarding-demo/deploy-rsyslog.yaml

The deployment includes three containers:

  • rsyslog — listens on TCP port 24224 and writes all received messages to /log/all.log as well as per-host/program files under /log/.
  • jq — tails the KubeVirt audit log and pretty-prints each entry as a single line with verb, duration, status code, request URI, and user.
  • cdt — builds cluster-debug-tools for additional troubleshooting.

Test rsyslog

Login into the pod:

1
oc rsh deployment/rsyslog

Send a test message and verify:

1
2
logger --tcp --port 24224 --server localhost "test message from rsyslog"
cat /log/all.log

Configure Cluster Logging

Install the OpenShift Logging Operator from OperatorHub before proceeding. The operator manages the ClusterLogForwarder custom resource used in the next step.

Create ServiceAccount and RBAC

Audit log collection requires a dedicated service account with the collect-audit-logs cluster role. Without this binding, the collector pods will not have permission to read the node-level audit log files:

1
2
3
oc create sa collector -n openshift-logging
oc adm policy add-cluster-role-to-user collect-audit-logs \
  -z collector -n openshift-logging

Deploy Log forwarding API

Forward audit logs to rsyslog

The ClusterLogForwarder below creates a pipeline that sends audit logs to the rsyslog service. A kubeAPIAudit filter limits the output to metadata-level events for KubeVirt resources, dropping everything else:

ClusterLogForwarder
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
oc create -f - <<EOF
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: audit-to-rsyslog
  namespace: openshift-logging
spec:
  serviceAccount:
    name: collector
  outputs:
    - name: rsyslog-server
      type: syslog
      syslog:
        url: 'tcp://rsyslog.rsyslog.svc.cluster.local:24224'
        rfc: RFC5424
  pipelines:
    - name: audit-to-rsyslog
      inputRefs:
        - audit
      filterRefs:
        - only-virt
      outputRefs:
        - rsyslog-server
  filters:
    - name: only-virt
      type: kubeAPIAudit
      kubeAPIAudit:
        omitStages:
          - "RequestReceived"
        rules:
          - level: Metadata
            resources:
            - group: "kubevirt.io"
              resources: ["virtualmachines","virtualmachineinstances","virtualmachineinstancemigrations"]
          - level: None
EOF

Create a Virtual Machine and view the audit logs

Once the forwarding pipeline is active, create a VirtualMachine through the OpenShift console or CLI. The jq sidecar in the rsyslog pod formats the incoming audit events into a compact, human-readable stream:

1
oc logs -f -n rsyslog -c jq deployment/rsyslog

1
2
09:25:35 [CREATE][    90.681ms] [201] /apis/kubevirt.io/v1/namespaces/rbohne-vms/virtualmachines [rbohne@redhat.com-admin]
09:26:30 [CREATE][    67.071ms] [201] /apis/kubevirt.io/v1/namespaces/rbohne-vms/virtualmachineinstances [system:serviceaccount:openshift-cnv:kubevirt-controller]
2026-07-30 2021-10-25 Contributors: Robert Bohne