Skip to content

EV-6919: Fix multi owner write issues - #5124

Merged
alexh-tigera merged 2 commits into
tigera:masterfrom
alexh-tigera:multi-owner-write-convergence
Jul 29, 2026
Merged

alexh-tigera merged 2 commits into
tigera:masterfrom
alexh-tigera:multi-owner-write-convergence

Conversation

@alexh-tigera

@alexh-tigera alexh-tigera commented Jul 29, 2026 •

Copy link
Copy Markdown
Member

Description

Objects shared by multiple owners (operator.tigera.io/multipleOwners label —
pull-secret copies, the tigera-operator-secrets RoleBinding, and the gateway trust-bundle
ConfigMap) were rewritten on every alternating reconcile by their owning controllers,
indefinitely. Two independent per-writer fields caused this:

  1. common.MergeOwnerReferences put the writer's own owner reference first, so each controller
    reordered the references written by the previous one. Fixed by sorting the merged references
    by UID so every writer converges on identical bytes; the dedup key also moves from the whole
    struct (which compared *bool pointers by identity) to the UID, and slices.Concat replaces
    an append that could mutate the caller's slice.
  2. The component handler stamped app.kubernetes.io/instance and /component from the writing
    CR, so shared objects flipped identity labels between owners. Fixed by skipping those two
    per-owner labels on multi-owner objects; writer-independent labels still apply, and values an
    earlier writer stamped are left in place.

Release Note

Fixed continuous rewrites of operator-managed objects shared by multiple owners (pull-secret copies, the tigera-operator-secrets RoleBinding, and gateway trust-bundle ConfigMaps): owner references are now merged in a stable order and per-owner identity labels are no longer written to shared objects, eliminating the resulting API server write and audit-log churn.

For PR author

  • Tests for change.
  • If changing pkg/apis/, run make gen-files (not applicable)
  • If changing versions, run make gen-versions (not applicable)

For PR reviewers

A note for code reviewers - all pull requests must have the following:

  • Milestone set according to targeted release.
  • Appropriate labels:
    • kind/bug if this is a bugfix.
    • kind/enhancement if this is a a new feature.
    • enterprise if this PR applies to Calico Enterprise only.

alexh-tigera and others added 2 commits July 29, 2026 10:22
The component handler stamps app.kubernetes.io/instance and /component from
the CR it is writing for. An object carrying MultipleOwnersLabel is written
by more than one controller, each passing its own CR — the waypoint
controller's Istio CR and the gatewayapi controller's per-Gateway passes
stamp different values onto the same RoleBinding, pull-secret copy, and
trust-bundle ConfigMap, so every reconcile rewrites the object with
whichever identity wrote last. Measured on a live cluster: a namespace
holding a waypoint and two of our Gateways rewrote all three shared objects
on every periodic tick, cycling instance between the two Gateway names and
the Istio CR's name.

Stop stamping the two per-owner labels on multi-owner objects. The
writer-independent labels (name, k8s-app, part-of, managed-by) are still
applied, and a value an earlier writer already stamped is left in place:
the label merge keeps labels it did not set, so existing values freeze
instead of flipping.

Owner-reference ordering still makes these writes non-convergent; that is
addressed separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6ff005a4bbf17c0949b08fccb9a62a9626fc0158)
MergeOwnerReferences put desired first, so each controller writing a shared
object (MultipleOwnersLabel) put its own owner reference first: waypoint
wrote [Istio, Gateway], gatewayapi wrote [Gateway, Istio], and two
EgressGateways in one namespace flipped between [A, B] and [B, A]. Every
alternating reconcile was a genuine content change, so the writers rewrote
the object in turn for as long as both kept reconciling. Measured on a live
cluster: a shared RoleBinding and pull-secret copy rewritten within a
second of every reconcile trigger, indefinitely.

Sort the merged references by UID so every writer converges on identical
bytes and the second write becomes a no-op. A UID identifies an owner, so
desired's version of a reference wins over current's and a re-rendered
reference with corrected fields still lands. Order carries no meaning to
Kubernetes: owner references are a set, and the controller reference is
identified by its Controller field rather than its position.

Also fixed while here:
- dedup key changed from the whole OwnerReference struct to UID. The struct
  contains *bool Controller/BlockOwnerDeletion, so the old map key compared
  pointers and kept two references for one owner if one writer set
  Controller and another did not.
- slices.Concat instead of append(desired, ...), which could mutate the
  caller's slice.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes reconcile-loop churn for operator-managed objects that are intentionally shared by multiple owners (marked with operator.tigera.io/multipleOwners). It ensures both owner reference merging and label stamping converge to stable object bytes, avoiding alternating rewrites by different controllers.

Changes:

  • Make common.MergeOwnerReferences deduplicate by owner UID and return owner references in a stable (UID-sorted) order to prevent reorder-based write loops.
  • Prevent stamping per-owner identity labels (app.kubernetes.io/instance and app.kubernetes.io/component) onto multi-owner objects so different controllers don’t flip these values back and forth.
  • Add focused unit tests covering convergence, dedup semantics, non-mutation, and the multi-owner label behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
pkg/controller/utils/component.go Passes multi-owner context into standard labeling so per-owner identity labels are skipped for shared objects.
pkg/controller/utils/component_test.go Adds a test asserting multi-owner shared objects don’t get per-owner identity labels and don’t persist the directive label.
pkg/common/components.go Updates owner-reference merging to dedupe by UID, avoid slice mutation, and sort deterministically.
pkg/common/components_test.go Adds unit tests verifying deterministic/convergent merge behavior and UID-based deduplication.

@alexh-tigera
alexh-tigera merged commit f36611f into tigera:master Jul 29, 2026
9 checks passed
@alexh-tigera
alexh-tigera deleted the multi-owner-write-convergence branch July 29, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants