Skip to content

[v3.23] fix(operator): drop emSign roots from the system CA bundle - #5365

Merged
rene-dekker merged 1 commit into
tigera:release-v1.42from
electricjesus:seth/emsign-distrust-v1.42
Oct 22, 2026
Merged

rene-dekker merged 1 commit into
tigera:release-v1.42from
electricjesus:seth/emsign-distrust-v1.42

Conversation

@electricjesus

Copy link
Copy Markdown
Member

Cherry-pick of https://github.com/tigera/calico-private/pull/14857.

Applied with git am -p2. The only conflict was one context line under the new function (TrustedBundleName differs on this branch); the function body matches master. The test import path is github.com/tigera/operator/... here. go test ./pkg/tls/certificatemanagement/ ./pkg/controller/certificatemanager/ passes on release-v1.42.

Description

The operator copies its own UBI root store into the tigera-ca-bundle configmap (key ca-bundle.crt). It mounts that store into every pod that needs public roots. That store still has the four eMudhra (emSign) roots. emSign ECC Root CA - G3 anchors the eMudhra intermediates whose private keys were recovered (RHSB-2026-012, https://bugzilla.mozilla.org/show_bug.cgi?id=2084101). Red Hat has not shipped a fixed ca-certificates yet.

This change drops the four emSign roots from the bundle. getSystemCertificates() now runs the file through filterDistrustedCertificates(). That function drops any PEM block whose SHA-256 fingerprint is on a short list and keeps every other byte.

Why this is enough for the pods that use the bundle:

  • The weak intermediates have two signed copies. One chains to emSign ECC Root CA - G3, the other to Sectigo's AAA Certificate Services. The UBI bundle does not hold the Sectigo root. So once G3 is gone, no path to the cracked keys is left.
  • The bundle is mounted over /etc/pki/tls/certs/ and /etc/pki/tls/cert.pem. Go and OpenSSL read those first, so the unfiltered copies elsewhere in an image are never read.
  • This matters most for fluentd. It is Ruby, so the Go 1.26.10 fix does not cover it, and its image has no root store of its own. Its Splunk HEC, syslog and S3 outputs use this bundle.

What this does not cover (follow-ups):

  • Elasticsearch trusts its JDK cacerts, which holds both roots. Kibana trusts Node's built-in roots. Neither uses this bundle.
  • On Windows, the bundle's cert.pem mount is skipped. Go uses CryptoAPI and Ruby uses RubyInstaller's own cert.pem.
  • The operator process itself still trusts its own image store. Known keys are covered by Go 1.26.10. A rebuild on the fixed UBI covers the rest.
  • Splunk HEC has no CA pin option today (SplunkFluentdSecretCertificateKey is never wired).

Also fixes the error in the same function: fmt.Errorf had no %w, so a read error came out garbled and was not wrapped.

Rollout: hash.operator.tigera.io/system changes once, because every current image has these roots. So node, typha, dex, fluentd, eks-log-forwarder, intrusion detection, webhooks-processor and guardian (Public CA) restart once on upgrade.

Breaking for a few setups: a customer endpoint whose certificate chains to an emSign root will fail TLS after this. That covers Splunk HEC, syslog over TLS and S3-compatible stores. For syslog, the syslog-ca configmap is the workaround today.

operator/ is shared with OSS, so projectcalico/calico needs the same change.

Picking to tigera/operator release-v1.42: the paths there have no operator/ prefix, so use git am -p2. Expect a one-line context conflict under the new function, because TrustedBundleName differs there.

Test plan:

  • New specs in pkg/tls/certificatemanagement: each of the four emSign roots is dropped, another root is kept, and the output parses. A bundle with no emSign root comes back byte for byte, and nil stays nil.
  • Changing one hex digit of the G3 fingerprint fails the G3 spec.
  • Ran the filter on the real operator image bundle (v1.42.6-25-g4d78391b829c): 146 roots in, 142 out, 0 emSign left.
  • go test passes for pkg/tls/certificatemanagement, pkg/controller/certificatemanager and pkg/enterprise/render/dex. gofmt and vet are clean.

Release Note

The operator now removes the four emSign (eMudhra) root certificates from the system CA bundle it mounts into node, typha, dex, fluentd, intrusion detection, webhooks-processor and guardian. Endpoints with certificates that chain to an emSign root will no longer be trusted.

AI assistance: This PR was written in part with the assistance of generative AI.

For PR author

  • Tests for change.
  • If changing pkg/apis/, run make gen-files (N/A, no pkg/apis/ changes)
  • If changing versions, run make gen-versions (N/A, no version changes)

For PR reviewers

A note for code reviewers - all pull requests must have the following:

  • Milestone set according to targeted release.
  • Appropriate labels:
    • kind/bug if this is a bugfix.
    • kind/enhancement if this is a a new feature.
    • enterprise if this PR applies to Calico Enterprise only.

The operator mounts its own UBI root store into pods that need public
roots. That store still carries the four eMudhra (emSign) roots, and
emSign ECC Root CA - G3 anchors intermediates whose private keys were
recovered (RHSB-2026-012). Filter all four out by SHA-256 fingerprint
in getSystemCertificates(), keeping every other byte of the bundle.

Also wrap the read error with %w instead of passing it to Sprintf.

(cherry picked from commit 5f70d1e8f3 in tigera/calico-private)
@rene-dekker
rene-dekker merged commit e6a1295 into tigera:release-v1.42 Oct 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants