Repository navigation
Replace fluentd with fluent-bit in the operator - #4910
Conversation
e121ac4 to
c81ad06
Compare
c81ad06 to
ff46afe
Compare
There was a problem hiding this comment.
Pull request overview
This PR completes the operator-side migration of the LogCollector from fluentd to fluent-bit (Calico Enterprise), including updated resource identities (namespace/workload names), rendered configuration shape, and updated policy/monitoring expectations across the rendering and controllers.
Changes:
- Replace fluentd-specific rendering/controller wiring with fluent-bit equivalents (names, ports, RBAC, ServiceMonitor behavior).
- Update expected Calico policies/selectors to reflect fluent-bit running in
calico-systemand using port2020. - Extend the LogCollector CRD/API to add
calicoFluentBitDaemonSetand treat the deprecatedfluentdDaemonSetas an alias during migration.
Reviewed changes
Copilot reviewed 47 out of 48 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/render/tiers/tiers_test.go | Updates tier rendering tests to stop expecting a separate LogCollector namespace. |
| pkg/render/testutils/expected_policies/node_local_dns_ipv6.json | Removes legacy tigera-fluentd namespace from node-local-dns selector expectations. |
| pkg/render/testutils/expected_policies/node_local_dns_ipv4.json | Removes legacy tigera-fluentd namespace from node-local-dns selector expectations. |
| pkg/render/testutils/expected_policies/node_local_dns_dual.json | Removes legacy tigera-fluentd namespace from node-local-dns selector expectations. |
| pkg/render/testutils/expected_policies/linseed.json | Updates Linseed ingress policy sources from fluentd labels/namespace to fluent-bit in calico-system. |
| pkg/render/testutils/expected_policies/linseed_ocp.json | Same as above for OpenShift policy expectations. |
| pkg/render/testutils/expected_policies/linseed_ocp_dpi_enabled.json | Same as above for OpenShift + DPI-enabled scenario. |
| pkg/render/testutils/expected_policies/linseed_dpi_enabled.json | Same as above for DPI-enabled scenario. |
| pkg/render/testutils/expected_policies/guardian.json | Updates Guardian policy expectations to allow fluent-bit instead of fluentd. |
| pkg/render/testutils/expected_policies/guardian_ocp.json | Same as above for OpenShift policy expectations. |
| pkg/render/testutils/expected_policies/fluentbit_unmanaged.json | Updates fluent-bit “allow metrics” policy expectations (name/namespace/selector/port). |
| pkg/render/testutils/expected_policies/fluentbit_unmanaged_ocp.json | Same as above for OpenShift. |
| pkg/render/testutils/expected_policies/fluentbit_managed.json | Updates managed-cluster policy expectations for fluent-bit metrics access. |
| pkg/render/testutils/expected_policies/es-gateway.json | Updates ES gateway ingress policy sources from fluentd to fluent-bit. |
| pkg/render/testutils/expected_policies/es-gateway_ocp.json | Same as above for OpenShift. |
| pkg/render/testutils/expected_policies/dns.json | Removes legacy tigera-fluentd namespace from DNS policy selector expectations. |
| pkg/render/testutils/expected_policies/dns_ocp.json | Same as above for OpenShift DNS policy expectations. |
| pkg/render/nonclusterhost/nonclusterhost.go | Adds Linseed RBAC permissions for DNS logs in non-cluster-host flow. |
| pkg/render/nonclusterhost/nonclusterhost_test.go | Updates RBAC test expectations to include dnslogs. |
| pkg/render/monitor/monitor.go | Renames fluentd monitoring constants to fluent-bit and switches ServiceMonitor to plain HTTP fluent-bit metrics endpoint. |
| pkg/render/monitor/monitor_test.go | Updates monitor rendering tests for renamed ServiceMonitor and the removal of TLS/scheme relabeling. |
| pkg/render/manager.go | Updates manager-rendered NetworkPolicy destination service name and cluster-wide namespace list (includes a problematic entry noted in comments). |
| pkg/render/manager_test.go | Updates manager tests to expect fluent-bit service names. |
| pkg/render/logstorage/linseed/linseed.go | Updates Linseed policy source entity rule reference to fluent-bit. |
| pkg/render/logstorage/esgateway/esgateway.go | Updates ES gateway policy source entity rule reference to fluent-bit. |
| pkg/render/intrusion_detection.go | Updates comments to reflect fluent-bit consuming IDS logs. |
| pkg/render/guardian.go | Updates Guardian policy source entity rule reference to fluent-bit. |
| pkg/render/fluentd.go | Removes the fluentd renderer implementation. |
| pkg/render/fluentd_test.go | Removes fluentd rendering unit tests. |
| pkg/render/common/elasticsearch/service.go | Renames isFluentd flag to isFluentBit and updates related commentary for managed-cluster Linseed routing. |
| pkg/render/applicationlayer/applicationlayer.go | Updates comment about WAF logs being consumed by fluent-bit. |
| pkg/imports/crds/operator/operator.tigera.io_logcollectors.yaml | Adds calicoFluentBitDaemonSet and updates fluentdDaemonSet docs/enums for the migration (some descriptions still say “Fluentd”, noted in comments). |
| pkg/controller/tiers/tiers_controller.go | Stops treating LogCollector as its own namespace when computing tier namespace lists. |
| pkg/controller/monitor/prometheus.go | Watches fluent-bit ServiceMonitor name instead of fluentd. |
| pkg/controller/monitor/monitor_controller.go | Switches monitored TLS secret from fluentd to fluent-bit. |
| pkg/controller/monitor/monitor_controller_test.go | Updates controller tests to reference fluent-bit ServiceMonitor and secrets. |
| pkg/controller/logstorage/secrets/secret_controller.go | Updates upstream cert collection references from fluentd TLS secret to fluent-bit TLS secret. |
| pkg/controller/logstorage/secrets/secret_controller_test.go | Updates tests to create/check fluent-bit TLS secret scenarios. |
| pkg/controller/logcollector/logcollector_controller.go | Rewires controller from fluentd to fluent-bit rendering (secrets, filters, services, namespace ownership behavior). |
| pkg/controller/logcollector/logcollector_controller_test.go | Updates controller tests for fluent-bit DaemonSet naming, config-driven outputs, and non-creation of calico-system. |
| pkg/components/enterprise.go | Renames enterprise component definitions from fluentd to fluent-bit (linux/windows). |
| hack/gen-versions/enterprise.go.tpl | Updates version generation template keys/structs for fluent-bit components. |
| config/enterprise_versions.yml | Replaces fluentd component entries with fluent-bit equivalents. |
| api/v1/zz_generated.deepcopy.go | Adds deepcopy support for CalicoFluentBitDaemonSet on LogCollectorSpec. |
| api/v1/logcollector_types.go | Adds calicoFluentBitDaemonSet field and documents fluentdDaemonSet as deprecated alias. |
| api/v1/fluentd_daemonset_types.go | Updates override container/initContainer enums to fluent-bit naming (backward-compat concerns noted in comments). |
Files not reviewed (1)
- api/v1/zz_generated.deepcopy.go: Generated file
2403d11 to
98e7752
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 49 out of 50 changed files in this pull request and generated 1 comment.
Files not reviewed (1)
- api/v1/zz_generated.deepcopy.go: Generated file
Comments suppressed due to low confidence (2)
pkg/controller/monitor/monitor_controller.go:156
- Monitor no longer scrapes fluent-bit over mTLS (serviceMonitorFluentBit is plain HTTP), so the controller shouldn't watch or require the fluent-bit TLS secret. Keeping it here creates an unnecessary dependency on the LogCollector controller and can degrade Monitor reconciliation when LogCollector isn't installed (secret not found).
for _, secret := range []string{
certificatemanagement.CASecretName,
esmetrics.ElasticsearchMetricsServerTLSSecret,
monitor.PrometheusServerTLSSecretName,
render.FluentBitTLSSecretName,
render.NodePrometheusTLSServerSecret,
kubecontrollers.KubeControllerPrometheusTLSSecret,
render.EKSLogForwarderTLSSecretName,
} {
pkg/controller/monitor/monitor_controller.go:345
- Since fluent-bit metrics scraping is now plain HTTP, Monitor's trusted bundle no longer needs the fluent-bit certificate. Including it makes reconciliation fail with NotFound if LogCollector hasn't created the secret yet (or isn't installed).
for _, certificateName := range []string{
esmetrics.ElasticsearchMetricsServerTLSSecret,
render.FluentBitTLSSecretName,
render.NodePrometheusTLSServerSecret,
render.CalicoAPIServerTLSSecretName,
kubecontrollers.KubeControllerPrometheusTLSSecret,
} {
98e7752 to
51596b3
Compare
f0561ca to
473c516
Compare
…-6164) Calico Enterprise removes the fluentd log collector in favour of Fluent Bit (calico-fluent-bit in calico-system; tigera/operator#4910). Update the next-version Calico Enterprise and Calico Cloud docs to match: - Rewrite "Filter flow logs" / "Filter DNS logs": examples move from fluentd <filter> syntax to Fluent Bit YAML filter lists, the ConfigMap is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note explains that old fluentd-syntax filters are not translated (the operator raises a tigera status warning naming the offending key). - Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020, /api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent plus a dropped-chunks alert on fluentbit_output_retries_failed_total. - BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP behind the allow-calico-fluent-bit policy); ServiceMonitor sample is fluent-bit-metrics-service-monitor.yaml. - Sweep of remaining pages: tigera-fluentd namespace -> calico-system, fluentd-node -> calico-fluent-bit (labels, secrets, packet capture retrieval via calico-node, diags output, resource override examples now use calicoFluentBitDaemonSet, architecture/overview descriptions, cc-arch-diagram ports, operator checklist). Not touched: versioned docs, releases.json (historical), and the generated installation API reference (regenerates from the operator release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
473c516 to
0c9d6bc
Compare
| log.Info("Managed kubernetes EKS found, getting necessary credentials and config") | ||
| if instance.Spec.AdditionalSources != nil { | ||
| if instance.Spec.AdditionalSources.EksCloudwatchLog != nil { | ||
| esClusterConfig, err = utils.GetElasticsearchClusterConfig(ctx, r.client) |
There was a problem hiding this comment.
Might be worth touching base with @caseydavenport on the steps taken in this PR to remove direct ES connections - just to make sure you are aligned with what he had in mind
There was a problem hiding this comment.
Will do — the ES retirement steps here follow the EV-6164 design (ES output dropped with fluentd, Linseed as sole structured-log sink). Casey, the specific surface in this PR: fluentd's ES env plumbing and the ElasticsearchEksLogForwarderUserSecret/cluster-config watches are removed from the LogCollector controller; nothing else touches ES.
0c9d6bc to
33be6fd
Compare
| @@ -129,18 +129,27 @@ func add(mgr manager.Manager, c ctrlruntime.Controller) error { | |||
| } | |||
|
|
|||
| for _, secretName := range []string{ | |||
There was a problem hiding this comment.
I know this wasn't there before, but you might want to add a daemonset watch to trigger re reconciliation if the daemonset was removed. This is missing in a lot of places currently.
There was a problem hiding this comment.
Done. Added watches for the workloads this controller renders — the calico-fluent-bit and calico-fluent-bit-windows DaemonSets and the eks-log-forwarder Deployment (all in calico-system) — so deleting or editing any of them out-of-band now enqueues a reconcile that restores it. Reconcile already loads the singleton LogCollector and ignores the request key, so a workload-triggered event drives a full re-render. Wrapped it in a small utils.AddDaemonsetWatch helper mirroring the existing AddDeploymentWatch, since this gap exists in other controllers too.
…t cert coupling Live validation surfaced that every fluent-bit Prometheus target was down with scrape timeouts: the calico-system.prometheus egress policy still allowed only the fluentd-era metrics port 9081 (which stays — it is also the elasticsearch-metrics port) and was never taught fluent-bit's 2020. Add the egress rule and update the policy fixtures. Verified end-to-end on a GCP kubeadm cluster: targets went from 0 up / 6 down to all up. Since the fluent-bit metrics endpoint is scraped over plain HTTP, the monitor controller has no use for the fluent-bit certificate: drop it from Prometheus's trusted bundle and remove the fluent-bit and EKS-log-forwarder secret watches (the latter was dead even in the fluentd era — never in the bundle), decoupling the monitor controller from the LogCollector controller entirely. Linseed's trust of the fluent-bit client certificate (logstorage secrets controller) is unchanged — that hop is real mTLS. Also use the ServiceMonitor name constant in the watch error message (review feedback) and refresh serviceMonitorFluentBit's godoc, which still explained a TLS-era rationale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…EKS RBAC
Legacy overrides (review: the container rename is a breaking change):
route the deprecated fluentdDaemonSet field through the shared
containerNameAliases mechanism in pkg/render/common/components instead
of a bespoke translation, mapping the fluentd-era container and
init-container names to their calico-fluent-bit equivalents, and add
render tests for the alias path and for calicoFluentBitDaemonSet
precedence when both fields are set.
EKS forwarder RBAC: the legacy-cleanup list deleted the cluster-scoped
eks-log-forwarder ClusterRoleBinding that the EKS render re-creates
under the same (fluentd-era) name; creates are processed before deletes,
so on EKS clusters every reconcile deleted the forwarder's Linseed RBAC.
The binding is reused in place, so drop it from the cleanup list (and
add the missed fluentd-node-windows Linseed token secret).
Additional-store parity with fluentd (verified against ee_entrypoint.sh,
syslog-config.sh, splunk-config.sh and the out-s3-*.conf templates):
- hostScope gates only cluster *flow* logs for all three stores, and
non-cluster flows always ship when a store is enabled; the previous
render dropped every cluster type but NCH flows under NonClusterOnly
and never shipped NCH flows under All.
- S3: preserve the legacy archive prefixes (audit_tsee, ...), gzip the
objects to match their .gz suffix (fluent-plugin-s3's default
store_as), and drop waf, which the entrypoint never archived. NCH
flows get their own non_cluster_flows/ prefix — out_s3's $INDEX is
per-output, so sharing flows/ would overwrite objects.
- syslog: drop syslog_severity_preset "info" — the property is an
integer (atoi("info") = 0 = Emergency) and the default 6 already is
info — and add syslog_hostname_preset ${NODENAME} as the node-name
fallback for tags whose records carry no host key, matching fluentd's
static SYSLOG_HOSTNAME.
- parse audit.* and ids.events event time (time_key) like the fluentd
sources did, so S3 day partitions and syslog timestamps use event
time. Rendered configs (default and all-stores) validated with the
real fluent-bit image via --dry-run.
Windows: pass Tenant/ExternalElastic into the Windows render so the
shared allow-calico-fluent-bit policy cannot flap on multi-tenant
management clusters with Windows nodes, and the Windows Linseed outputs
target the right namespace and tenant header.
Cleanup: remove the ES-era ElasticsearchEksLogForwarderUserSecret and
cluster-config watches (nothing reads them since the ES output was
dropped) and the unused /etc/fluent-bit/certs trusted-bundle mounts.
Add a controller test for the fluent-bit-filters warning path
(SetWarning on unparseable content, cleared once fixed, rendering
continues) and tighten the store-output render assertions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oller Shared component (review): the resources both OS renders previously duplicated — the allow-calico-fluent-bit NetworkPolicy, GKE ResourceQuota, S3/Splunk credential copies, managed-cluster tigera-linseed Service/RoleBinding, PacketCapture RBAC and the legacy cleanup — move into a FluentBitShared component rendered exactly once per reconcile, removing the duplicate-render contention class. The Windows configuration no longer carries NonClusterHost at all. Legacy cleanup: rewritten to the namespace-cascade pattern used for the guardian/apiserver/policy-recommendation migrations — delete the tigera-fluentd Namespace plus the cluster-scoped ClusterRole/Bindings and the operator-namespace copy of tigera-fluentd-prometheus-tls. Never-in-production entries (the allow-tigera-prefixed policy name and the calico-system copy of the filters ConfigMap) are dropped. DaemonSet: startup/liveness probes move to /api/v1/uptime so sustained delivery errors cannot restart pods and drop in-memory buffers (readiness keeps the error-aware /api/v1/health); terminationGracePeriodSeconds 0 → 30 (grace 0 assumed the pos file made buffers durable, which was never true); pos-migrator init container unified across OS variants. Outputs/config: S3 keys switch to directory-style <bucketPath>/<type>/%Y%m%d_$UUID.gz — a deliberate, release-noted layout change from fluentd's flat concatenation (full parity was impossible anyway: out_s3's $INDEX is not restart-safe, so the suffix had to become a UUID) — with non-cluster flows in their own directory; the :9880 http input gains buffer_max_size 100M / buffer_chunk_size 5M (fluentd's body_size_limit parity — the ~4M default silently rejected larger relayed non-cluster-host batches); splunk drops l7, which deployed fluentd never enabled; renderFluentBitConf now reads as inputs → filters → outputs with the Linseed helpers moved next to the store outputs; buffer-durability comments corrected (fluentd buffered in memory only). Controller: the rendered fluent-bit ConfigMaps are watched so tampering triggers reconciliation, and syslog endpoints are validated to tcp/udp schemes (fluent-bit's syslog mode accepts nothing else; TLS comes from the encryption field). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review ask: represent each case from fluentd's config test harness (calico-private fluentd/test/test.sh) in this repo, with the expectation as a raw file mapped to its fluentd equivalent. TestRenderedConfigGoldens pins the full rendered fluent-bit config for 17 scenarios byte-for-byte under testdata/rendered-configs/, each table entry naming its test.sh ancestor and reusing test.sh's endpoint, bucket and log-type values so a golden reads side-by-side with its fluentd .cfg counterpart. The README maps all 20 test.sh cases: the six ES-era cases are n/a (the Elasticsearch output was retired with fluentd per EV-6164), -nch cases map to hostScope NonClusterOnly, and the divergences that cannot be mirrored are documented (the Audit syslog type always enables both audit kinds, the CRD never exposed the L7/runtime/WAF syslog toggles, and the dual-scheme splunk case splits into splunk-http/splunk-https). UPDATE_RENDERED_CONFIGS=1 regenerates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove the never-in-production calico-fluent-bit-metrics PodMonitor from the monitor toDelete list (the pre-v1.25 PodMonitor was named fluentd-metrics; the fluent-bit name never shipped) and simplify the ServiceMonitor comment that duplicated the function godoc. Drop the fluent-bit and eks-log-forwarder ingress rules from the es-gateway policy: both components ship only to Linseed in every topology (managed clusters use the tigera-linseed ExternalService via Guardian, never tigera-secure-es-gateway-http), so the rules were relics of fluentd's direct-Elasticsearch path. The equivalent source rules on the Linseed policy are the live path and remain. Remove an author-focused rationale comment from the tiers controller. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add utils.AddDaemonsetWatch (mirroring AddDeploymentWatch) and wire watches for the calico-fluent-bit / calico-fluent-bit-windows DaemonSets and the eks-log-forwarder Deployment so deleting or editing them out-of-band triggers a reconcile that restores them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nits Address round-2 review feedback (pasanw) on PR tigera#4910: - Rename FluentBit -> FluentBitOSSpecific and lift OSType out of FluentBitConfiguration into a constructor argument. The controller now builds one configuration and passes it to all three instances (shared, Linux, Windows); each applies its OS-specific logic internally (the Windows render already gates NonClusterHost on the OS, and PacketCapture RBAC only ever comes from the shared component), so rendered output is unchanged. Add a test asserting the three components produce disjoint resource sets from one config. - Move the config-pipeline rendering (renderFluentBitConf, addInputs, addFilters) out of config.go into pipeline.go (renamed from outputs.go) so the inputs -> filters -> outputs assembly lives in one file; config.go keeps the ConfigMap object, input catalog and path helpers. - Relocate the non-cluster-host ingress comment above its if, and drop the reviewer-facing "like fluentd" framing on the client-cert comment. - Validate the syslog Endpoint scheme before fillDefaults so that, once the CRD enforces the tcp/udp pattern, an upgrade carrying a stored legacy scheme degrades with a clear message instead of failing the defaulting patch with an opaque apiserver error. - Clarify that the PacketCapture Role grants the packetcapture-api ServiceAccount exec into the per-node collector pods (not the fluent-bit SA), and rename golden_test.go -> rendered_config_test.go. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a kubebuilder Pattern (^(tcp|udp)://.+$) to SyslogStoreSpec.Endpoint so the apiserver rejects an unsupported scheme at write time, matching the controller-side validation. TLS is selected via the Encryption field, not the scheme. Regenerated the bundled CRD. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
86e9b09 to
af52c27
Compare
pasanw
left a comment
There was a problem hiding this comment.
Approved with some comments to consider before merging
There was a problem hiding this comment.
Regarding release notes, I'd suggest the following:
- The tigera-fluentd namespace is removed; all log-collector resources are renamed and move to calico-system (DaemonSet/container calico-fluent-bit, TLS secret calico-fluent-bit-tls). Tooling referencing the old names (e.g. kubectl logs -c fluentd) must be updated.
- The LogCollector fluentdDaemonSet override field is deprecated in favor of calicoFluentBitDaemonSet; existing overrides (including legacy container names) keep applying for one release.
- User flow/DNS log filters: the fluentd-filters ConfigMap is no longer read. Filters must be recreated under the new fluent-bit-filters name as fluent-bit YAML filter lists; unparseable content raises a TigeraStatus warning while log shipping continues.
- Log-collector metrics are now fluent-bit's native Prometheus metrics (fluentbit_* metric names, plain HTTP on port 2020 at /api/v2/metrics/prometheus, guarded by NetworkPolicy). Dashboards and alerts keyed on fluentd_* metric names or the 9081 mTLS endpoint must be updated.
- S3 archive object keys change from fluentd's flat layout (<bucketPath>/flows20260101_<n>.gz) to directory-style keys (<bucketPath>/flows/20260101_<uuid>.gz), with non-cluster-host flows archived under their own non_cluster_flows/ directory. Downstream tooling anchored to the old flat patterns needs a one-time update.
- Log buffering moves from fluentd's in-memory buffers to fluent-bit filesystem storage under /var/log/calico/calico-fluent-bit/ on each node: buffered-but-unsent chunks now survive pod restarts, and host disk usage grows accordingly (capped per output by storage.total_limit_size).
- Non-cluster hosts can now forward DNS logs in addition to flow and policy-activity logs.
- Moved "The LogCollector fluentdDaemonSet override field is deprecated..." onto its own line
- Removed the line "The direct Elasticsearch log output is removed...", these are internal changes
There was a problem hiding this comment.
Adopted your wording, with two changes:
- Dropped the "Non-cluster hosts can now forward DNS logs…" bullet — NCH DNS forwarding will be delivered and tested under its own epic, so it shouldn't ship as a release note here.
- Added a syslog packet-size bullet (see the thread on the syslog output): the 1024-byte fluentd default is gone; when
packetSizeis unset the fluent-bit syslog output uses its RFC5424 default (2048 bytes).
I'll fold the finalized block into the PR description's release-note section in the next update.
There was a problem hiding this comment.
Claude is flagging that we have lost the defaulting of syslog packet size - we used to explicitly set it to 1024. If this is true we should probably set it and reflect that in our golden tests
There was a problem hiding this comment.
It's true the operator no longer defaults it to 1024 — but that's intentional, not a regression. fluentd's 1024-byte packet size truncated our own log lines, which is exactly the failure mode we want to avoid. The operator sets syslog_maxsize only when the user provides packetSize; when it's unset, fluent-bit's out_syslog applies its own default for the RFC5424 format we render (2048 bytes), which truncates less. So restoring an explicit 1024 would reintroduce the truncation.
I've corrected the stale Default: 1024 on the CRD field to describe the real behavior, and added a release note documenting the change. No golden update is needed — the goldens never set packetSize, so they already show no syslog_maxsize line (matching "fluent-bit default applies").
…capture component The Role/RoleBinding granting the PacketCapture API pod/exec in calico-system were rendered by the log-collector component only because, in the fluentd era, the grant had to live in the fluentd-owned tigera-fluentd namespace and targeted the fluentd DaemonSet pods (the per-node pod that mounts /var/log/calico, where captures are written). With the fluent-bit migration the API execs into calico-node instead, so the grant no longer has anything to do with the log collector. Move it to the packet-capture component, which owns the PacketCapture API's ServiceAccount and renders only when the PacketCaptureAPI CR exists (the same condition that gated it before). The Role/RoleBinding keep their names (packetcapture-api-role / -binding) and namespace (calico-system) so the objects are adopted in place on upgrade. Drop the now-unused PacketCapture field, its controller fetch, the constants and rbac.go helpers from the log collector. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The field doc claimed "Default: 1024", but the operator never defaults it — when packetSize is unset, no syslog_maxsize is rendered and the fluent-bit syslog output applies its own default for the RFC5424 format the operator uses (2048 bytes). Describe the real behavior and regenerate the CRD. Fluentd's 1024-byte limit is deliberately not restored: it truncated longer log lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-6164) Calico Enterprise removes the fluentd log collector in favour of Fluent Bit (calico-fluent-bit in calico-system; tigera/operator#4910). Update the next-version Calico Enterprise and Calico Cloud docs to match: - Rewrite "Filter flow logs" / "Filter DNS logs": examples move from fluentd <filter> syntax to Fluent Bit YAML filter lists, the ConfigMap is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note explains that old fluentd-syntax filters are not translated (the operator raises a tigera status warning naming the offending key). - Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020, /api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent plus a dropped-chunks alert on fluentbit_output_retries_failed_total. - BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP behind the allow-calico-fluent-bit policy); ServiceMonitor sample is fluent-bit-metrics-service-monitor.yaml. - Sweep of remaining pages: tigera-fluentd namespace -> calico-system, fluentd-node -> calico-fluent-bit (labels, secrets, packet capture retrieval via calico-node, diags output, resource override examples now use calicoFluentBitDaemonSet, architecture/overview descriptions, cc-arch-diagram ports, operator checklist). Not touched: versioned docs, releases.json (historical), and the generated installation API reference (regenerates from the operator release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Document the fluentd -> Fluent Bit migration (tigera/operator#4910) in the next-version Calico Enterprise release notes: - New feature entry for the Fluent Bit log collector. - Deprecated: the fluentdDaemonSet override, superseded by calicoFluentBitDaemonSet. - Upgrade notes with the breaking changes: tigera-fluentd namespace/resource rename to calico-system, fluentd-filters -> fluent-bit-filters ConfigMap, fluentbit_* Prometheus metrics on port 2020, and the new S3 archive key layout, plus the non-breaking moves to filesystem buffering and the larger syslog packet-size default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-6164) Calico Enterprise removes the fluentd log collector in favour of Fluent Bit (calico-fluent-bit in calico-system; tigera/operator#4910). Update the next-version Calico Enterprise and Calico Cloud docs to match: - Rewrite "Filter flow logs" / "Filter DNS logs": examples move from fluentd <filter> syntax to Fluent Bit YAML filter lists, the ConfigMap is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note explains that old fluentd-syntax filters are not translated (the operator raises a tigera status warning naming the offending key). - Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020, /api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent plus a dropped-chunks alert on fluentbit_output_retries_failed_total. - BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP behind the allow-calico-fluent-bit policy); ServiceMonitor sample is fluent-bit-metrics-service-monitor.yaml. - Sweep of remaining pages: tigera-fluentd namespace -> calico-system, fluentd-node -> calico-fluent-bit (labels, secrets, packet capture retrieval via calico-node, diags output, resource override examples now use calicoFluentBitDaemonSet, architecture/overview descriptions, cc-arch-diagram ports, operator checklist). Not touched: versioned docs, releases.json (historical), and the generated installation API reference (regenerates from the operator release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Document the fluentd -> Fluent Bit migration (tigera/operator#4910) in the next-version Calico Enterprise release notes: - New feature entry for the Fluent Bit log collector. - Deprecated: the fluentdDaemonSet override, superseded by calicoFluentBitDaemonSet. - Upgrade notes with the breaking changes: tigera-fluentd namespace/resource rename to calico-system, fluentd-filters -> fluent-bit-filters ConfigMap, fluentbit_* Prometheus metrics on port 2020, and the new S3 archive key layout, plus the non-breaking moves to filesystem buffering and the larger syslog packet-size default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-6164) Calico Enterprise removes the fluentd log collector in favour of Fluent Bit (calico-fluent-bit in calico-system; tigera/operator#4910). Update the next-version Calico Enterprise and Calico Cloud docs to match: - Rewrite "Filter flow logs" / "Filter DNS logs": examples move from fluentd <filter> syntax to Fluent Bit YAML filter lists, the ConfigMap is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note explains that old fluentd-syntax filters are not translated (the operator raises a tigera status warning naming the offending key). - Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020, /api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent plus a dropped-chunks alert on fluentbit_output_retries_failed_total. - BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP behind the allow-calico-fluent-bit policy); ServiceMonitor sample is fluent-bit-metrics-service-monitor.yaml. - Sweep of remaining pages: tigera-fluentd namespace -> calico-system, fluentd-node -> calico-fluent-bit (labels, secrets, packet capture retrieval via calico-node, diags output, resource override examples now use calicoFluentBitDaemonSet, architecture/overview descriptions, cc-arch-diagram ports, operator checklist). Not touched: versioned docs, releases.json (historical), and the generated installation API reference (regenerates from the operator release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Document the fluentd -> Fluent Bit migration (tigera/operator#4910) in the next-version Calico Enterprise release notes: - New feature entry for the Fluent Bit log collector. - Deprecated: the fluentdDaemonSet override, superseded by calicoFluentBitDaemonSet. - Upgrade notes with the breaking changes: tigera-fluentd namespace/resource rename to calico-system, fluentd-filters -> fluent-bit-filters ConfigMap, fluentbit_* Prometheus metrics on port 2020, and the new S3 archive key layout, plus the non-breaking moves to filesystem buffering and the larger syslog packet-size default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document the fluentd -> Fluent Bit migration (tigera/operator#4910) in the next-version Calico Enterprise release notes: - New feature entry for the Fluent Bit log collector. - Deprecated: the fluentdDaemonSet override, superseded by calicoFluentBitDaemonSet. - Upgrade notes with the breaking changes: tigera-fluentd namespace/resource rename to calico-system, fluentd-filters -> fluent-bit-filters ConfigMap, fluentbit_* Prometheus metrics on port 2020, and the new S3 archive key layout, plus the non-breaking moves to filesystem buffering and the larger syslog packet-size default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-6164) Calico Enterprise removes the fluentd log collector in favour of Fluent Bit (calico-fluent-bit in calico-system; tigera/operator#4910). Update the next-version Calico Enterprise and Calico Cloud docs to match: - Rewrite "Filter flow logs" / "Filter DNS logs": examples move from fluentd <filter> syntax to Fluent Bit YAML filter lists, the ConfigMap is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note explains that old fluentd-syntax filters are not translated (the operator raises a tigera status warning naming the offending key). - Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020, /api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent plus a dropped-chunks alert on fluentbit_output_retries_failed_total. - BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP behind the allow-calico-fluent-bit policy); ServiceMonitor sample is fluent-bit-metrics-service-monitor.yaml. - Sweep of remaining pages: tigera-fluentd namespace -> calico-system, fluentd-node -> calico-fluent-bit (labels, secrets, packet capture retrieval via calico-node, diags output, resource override examples now use calicoFluentBitDaemonSet, architecture/overview descriptions, cc-arch-diagram ports, operator checklist). Not touched: versioned docs, releases.json (historical), and the generated installation API reference (regenerates from the operator release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Document the fluentd -> Fluent Bit migration (tigera/operator#4910) in the next-version Calico Enterprise release notes: - New feature entry for the Fluent Bit log collector. - Deprecated: the fluentdDaemonSet override, superseded by calicoFluentBitDaemonSet. - Upgrade notes with the breaking changes: tigera-fluentd namespace/resource rename to calico-system, fluentd-filters -> fluent-bit-filters ConfigMap, fluentbit_* Prometheus metrics on port 2020, and the new S3 archive key layout, plus the non-breaking moves to filesystem buffering and the larger syslog packet-size default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-6164) Calico Enterprise removes the fluentd log collector in favour of Fluent Bit (calico-fluent-bit in calico-system; tigera/operator#4910). Update the next-version Calico Enterprise and Calico Cloud docs to match: - Rewrite "Filter flow logs" / "Filter DNS logs": examples move from fluentd <filter> syntax to Fluent Bit YAML filter lists, the ConfigMap is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note explains that old fluentd-syntax filters are not translated (the operator raises a tigera status warning naming the offending key). - Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020, /api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent plus a dropped-chunks alert on fluentbit_output_retries_failed_total. - BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP behind the allow-calico-fluent-bit policy); ServiceMonitor sample is fluent-bit-metrics-service-monitor.yaml. - Sweep of remaining pages: tigera-fluentd namespace -> calico-system, fluentd-node -> calico-fluent-bit (labels, secrets, packet capture retrieval via calico-node, diags output, resource override examples now use calicoFluentBitDaemonSet, architecture/overview descriptions, cc-arch-diagram ports, operator checklist). Not touched: versioned docs, releases.json (historical), and the generated installation API reference (regenerates from the operator release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Description
New feature (Calico Enterprise): the LogCollector controller now deploys fluent-bit (
calico-fluent-bitincalico-system) in place of fluentd, completing the log-collector migration on the operator side.tigera-fluentd→calico-system; DaemonSet/ServiceAccountfluentd-node→calico-fluent-bit; TLS secret →calico-fluent-bit-tls; metrics port 9081 → 2020 (fluent-bit's built-in HTTP server).calico-fluent-bit-conf/-windows), subPath-mounted on Linux and directory-mounted on Windows, started with-c. The render loads the Go plugins viaplugins_file, defines parsers inline, applies the record-transform Lua filter, and inlines user-provided fluent-bit YAML filter lists. A rendered-config hash annotation rolls the pods on config-only changes./var/log/calico/calico-fluent-bit; apos-migratorinit container (Linux and Windows) seeds offsets from the legacy fluentd.posfiles and pre-creates the tailed directories. Windows tails the same log types the fluentd Windows variant shipped.:9880HTTP input with client-certificate verification; the input Service is cleaned up when the resource is removed.eks-log-forwarderruns the fluent-bit image with a renderedin_eks→linseedpipeline and health probes (no startup init container; the input plugin resolves its resume point from Linseed).:2020/api/v1/health; the ServiceMonitor scrapes plain HTTP (fluent-bit's monitoring server has no TLS) with access restricted by the component NetworkPolicy, and legacy fluentd monitors are deleted.calico-systemnamespace (deleting the LogCollector must not garbage-collect it); the deprecatedfluentdDaemonSetoverride is honored as an alias of the newcalicoFluentBitDaemonSetfield (with container-name translation); legacytigera-fluentdresources — the namespace last — are cleaned up idempotently.Testing: render, controller, and monitor unit suites updated/extended (ConfigMap-content assertions replacing the env-var assertions); the rendered configuration was validated against the real fluent-bit binary; the full migration was validated end-to-end on a test cluster — all log types flowing to Linseed/Elasticsearch, fluentd resources fully removed, tail-offset handover without re-shipping, NonClusterHost ingestion with client-certificate enforcement, and EKS/Windows render shapes verified.
Release Note
For PR author
make gen-filesmake gen-versionsFor PR reviewers
A note for code reviewers - all pull requests must have the following:
kind/bugif this is a bugfix.kind/enhancementif this is a a new feature.enterpriseif this PR applies to Calico Enterprise only.