[zhihonl]

Background

Amazon Linux 2023 (AL2023) is now the default AMI for new EKS clusters, replacing Amazon Linux 2 (AL2). AL2023 moved to a new logging system where system logs are no longer emitted as a local file. The current Fluent Bit configuration for CW Observability Add-on collect logs from log paths specific to AL2. When running on AL2023 nodes, Fluent Bit cannot find these expected log paths therefore customers lose system and container runtime logs. FluentBit can be configured to collect the same logs from AL2023 by using a different plugin called systemd.

Issue #, if available:
aws-samples/amazon-cloudwatch-container-insights#136

Description of changes:

Dmesg(Kernel) Logs

For kernel messages host.dmesg, we replaced the file-based tail plugin reading from /var/log/dmesg with the systemd plugin using _TRANSPORT=kernel filter. This ensures we continue collecting the same kernel messages but through systemd's journal instead of a log file. There is also existing issue where this is proposed as a workaround for Bottlerocket so should help there as well: aws-samples/amazon-cloudwatch-container-insights#136 (comment)

System message Logs

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

For system messages host.messages, the original configuration read from /var/log/messages which contained all info logs except mail, authpriv, and cron as defined in the rsyslog configuration. The new configuration achieves this same behavior by using the systemd plugin to collect all priority levels (0-6) which are info level and above, and then explicitly filtering out messages from mail (facility 2), cron (facility 9), and authpriv (facility 10) facilities using a grep filter.

Security Logs

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

For security logs host.secure, we replaced reading from /var/log/secure with the systemd plugin filtered on SYSLOG_FACILITY=10, which specifically captures authpriv messages. This maintains the same security log collection capability through the new logging system.

To ensure backward compatibility with existing log processing and analytics, we added a modify filter that renames systemd specific fields to match the previous log format and removes unnecessary systemd metadata fields.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[zhihonl]

This kills two birds with one stone by also helping Bottlerocket users since that's what some users are doing as a workaound: aws-samples/amazon-cloudwatch-container-insights#136 (comment)

[zhihonl]

Doing this since systemd dumps a lot of attributes and we want to retain similar log content as before. Below is an example log message without this filter

{
    "_MACHINE_ID": "[REDACTED]",
    "PRIORITY": "3",
    "_UID": "0",
    "_GID": "0",
    "_SYSTEMD_SLICE": "system.slice",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_RUNTIME_SCOPE": "system",
    "_TRANSPORT": "syslog",
    "_SELINUX_CONTEXT": "system_u:system_r:unconfined_service_t:s0",
    "_HOSTNAME": "[REDACTED]",
    "SYSLOG_FACILITY": "10",
    "SYSLOG_IDENTIFIER": "chronicled",
    "_COMM": "chronicled",
    "_EXE": "/usr/local/chronicle/sbin/chronicled",
    "_CMDLINE": "/usr/local/chronicle/sbin/chronicled",
    "_SYSTEMD_CGROUP": "/system.slice/chronicled.service",
    "_SYSTEMD_UNIT": "chronicled.service",
    "MESSAGE": "open /var/log/secure: open /var/log/secure: no such file or directory",
    "_BOOT_ID": "82a74d3dde8f4118846b734a8150bb3c",
    "SYSLOG_PID": "626090",
    "_PID": "626090",
    "_SYSTEMD_INVOCATION_ID": "9f426ae3c4994d269a6c32f19f401a9e",
    "SYSLOG_TIMESTAMP": "Jul 24 16:10:01 ",
    "SYSLOG_RAW": "<83>Jul 24 16:10:01 chronicled[626090]: open /var/log/secure: open /var/log/secure: no such file or directory\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1753373401426331",
    "az": "us-east-1f",
    "ec2_instance_id": "i-[REDACTED]"
}

[lisguo]

I would confirm by testing on an AL2 host with the previous fluent bit and then with the new changes on AL2023 that we are not changing the volume of logs.

I am a little concerned that we are making assumptions based off of AL2 specific things (like rsyslog configuration), can we test on non AL2?

[lisguo]

seems right since we are pushing logs from emergency to info...

https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/security-analytics/8-2-6/_reference_home/syslog.html

---

Have we confirmed that we are not emitting more logs than the previous tail plugin?

[zhihonl]

I created a second fluent-bit daemonset that uses new image and new config, then compare it against the original daemonset with old config. Confirmed the log load and the message they are scraping are matching for messages log group.

[lisguo]

contained all info logs except mail, authpriv, and cron as defined in the rsyslog configuration.

So we are excluding mail, authpriv, and cron logs because rsyslog had a configuration for it for AL2. Do we know this is the same for non-AL2 hosts? What if a customer was using RHEL or some other linux distro?

[zhihonl]

TLDR: Our add-on config requires rsyslog to retrieve host logs, and rsyslog comes with a default configuration. Customers with OS that have rsyslog installed won't be broken. Customers that don't have rsyslog installed will start seeing recovery after this change. Below are the EKS optimized OS I've tested

1. AL2 - Emitted host logs before, will emit with new change.
2. AL2023 - Never emitted host logs, will emit with new change
3. Ubuntu - Never emitted host logs, will emit with new change
4. Bottlerocket - Never emitted host logs, will emit with new change

Also tried RHEL9, it does not have rsyslog installed by default so it also does not emit by default but once installed it has the same rsyslog configuration as AL2.

[lisguo]

Overall looks good to me, once the fluent bit version bump gets released I can approve