Skip to content

Fix deferred HybridCache tag invalidation - #7786

Merged
svick merged 5 commits into
dotnet:mainfrom
svick:fix/hybrid-cache-pending-tag-invalidation
Sep 30, 2026
Merged

svick merged 5 commits into
dotnet:mainfrom
svick:fix/hybrid-cache-pending-tag-invalidation

Conversation

@svick

@svick svick commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

  • use the cached payload creation timestamp when checking tag invalidations that complete after payload parsing
  • group successful payload parse outputs into a structured result
  • add a two-node shared-L2 regression test that delays the tag timestamp read

Fixes #7771

svick and others added 2 commits September 25, 2026 14:44
Use the cached payload creation timestamp when validating pending tag invalidations and cover the cross-node L2 race with a regression test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Group successful payload parse outputs into a structured result to keep production and test call sites readable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

๐ŸŸก Changes recommended

The regression test does not deterministically exercise the deferred tag-check path.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Fixes deferred HybridCache tag invalidation by comparing against the cached payloadโ€™s creation timestamp.

Changes:

  • Groups parsed payload metadata into ParsedPayload.
  • Uses payload creation time for deferred tag checks.
  • Adds payload and shared-L2 regression coverage.
File Description
HybridCachePayload.cs Returns structured parse results with creation time.
DefaultHybridCache.StampedeStateT.cs Uses payload timestamp for deferred invalidation.
PayloadTests.cs Updates parser tests for structured results.
L2Tests.cs Adds a two-node invalidation regression test.

๐Ÿ’ก Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

@dotnet-comment-bot

Copy link
Copy Markdown
Collaborator

โ€ผ๏ธ Found issues โ€ผ๏ธ

Project Coverage Type Expected Actual
Microsoft.Extensions.Diagnostics.Testing Line 99 98.65 ๐Ÿ”ป
Microsoft.Extensions.Telemetry Line 93 92.64 ๐Ÿ”ป
Microsoft.Extensions.AI Line 89 88.99 ๐Ÿ”ป
Microsoft.Extensions.AI.OpenAI Line 75 68.32 ๐Ÿ”ป
Microsoft.Extensions.AI.OpenAI Branch 75 57.59 ๐Ÿ”ป
Microsoft.Extensions.DataIngestion.MarkItDown Line 75 4.46 ๐Ÿ”ป
Microsoft.Extensions.DataIngestion.MarkItDown Branch 75 0 ๐Ÿ”ป
Microsoft.Extensions.Diagnostics.ResourceMonitoring Line 99 96.03 ๐Ÿ”ป
Microsoft.Extensions.Diagnostics.ResourceMonitoring Branch 99 92.76 ๐Ÿ”ป
Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes Line 99 97.73 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Abstractions Line 75 42.11 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Abstractions Branch 75 42.86 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Dns Line 75 68.32 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery Line 75 68.57 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery Branch 75 71.43 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Yarp Line 75 73.85 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Yarp Branch 75 70 ๐Ÿ”ป
Microsoft.Extensions.VectorData.Abstractions Line 75 37.39 ๐Ÿ”ป
Microsoft.Extensions.VectorData.Abstractions Branch 75 22.73 ๐Ÿ”ป

๐ŸŽ‰ Good job! The coverage increased ๐ŸŽ‰
Update MinCodeCoverage in the project files.

Project Expected Actual
Microsoft.Extensions.Http.Diagnostics 94 95
Microsoft.Gen.BuildMetadata 97 100
Microsoft.Gen.MetadataExtractor 57 73
Microsoft.Gen.MetricsReports 67 69
Microsoft.Extensions.AI.Abstractions 82 86
Microsoft.Extensions.AI.Evaluation.NLP 0 78
Microsoft.Extensions.Caching.Hybrid 82 85
Microsoft.Extensions.DataIngestion 75 89
Microsoft.Extensions.DataIngestion.Markdig 75 90
Microsoft.Extensions.Http.Resilience 97 100

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1612268&view=codecoverage-tab

svick and others added 2 commits September 25, 2026 16:59
Gate the tag read while returning the entry synchronously so payload parsing must reach the deferred pending-tag check before the test releases it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rename the wrapped distributed cache from tail to inner in the regression test helper.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

๐ŸŸข Approval recommended

The fix correctly addresses the stale-read defect with focused regression coverage; the remaining comment typo is non-blocking.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
Resolved since last review (1)

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@svick
svick marked this pull request as ready for review September 25, 2026 15:40
@svick
svick requested a review from a team as a code owner September 25, 2026 15:40
@dotnet-comment-bot

Copy link
Copy Markdown
Collaborator

โ€ผ๏ธ Found issues โ€ผ๏ธ

Project Coverage Type Expected Actual
Microsoft.Extensions.Diagnostics.Testing Line 99 98.65 ๐Ÿ”ป
Microsoft.Extensions.Telemetry Line 93 92.64 ๐Ÿ”ป
Microsoft.Extensions.AI.OpenAI Line 75 68.32 ๐Ÿ”ป
Microsoft.Extensions.AI.OpenAI Branch 75 57.59 ๐Ÿ”ป
Microsoft.Extensions.DataIngestion.MarkItDown Line 75 4.46 ๐Ÿ”ป
Microsoft.Extensions.DataIngestion.MarkItDown Branch 75 0 ๐Ÿ”ป
Microsoft.Extensions.Diagnostics.ResourceMonitoring Line 99 96.03 ๐Ÿ”ป
Microsoft.Extensions.Diagnostics.ResourceMonitoring Branch 99 92.76 ๐Ÿ”ป
Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes Line 99 97.73 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Dns Line 75 68.32 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Abstractions Line 75 42.11 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Abstractions Branch 75 42.86 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery Line 75 67.51 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery Branch 75 71.43 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Yarp Line 75 73.85 ๐Ÿ”ป
Microsoft.Extensions.ServiceDiscovery.Yarp Branch 75 70 ๐Ÿ”ป
Microsoft.Extensions.VectorData.Abstractions Line 75 37.39 ๐Ÿ”ป
Microsoft.Extensions.VectorData.Abstractions Branch 75 22.73 ๐Ÿ”ป

๐ŸŽ‰ Good job! The coverage increased ๐ŸŽ‰
Update MinCodeCoverage in the project files.

Project Expected Actual
Microsoft.Extensions.Http.Diagnostics 94 95
Microsoft.Gen.BuildMetadata 97 100
Microsoft.Gen.MetadataExtractor 57 73
Microsoft.Gen.MetricsReports 67 69
Microsoft.Extensions.AI.Abstractions 82 86
Microsoft.Extensions.AI.Evaluation.NLP 0 78
Microsoft.Extensions.Caching.Hybrid 82 85
Microsoft.Extensions.DataIngestion 75 89
Microsoft.Extensions.DataIngestion.Markdig 75 90
Microsoft.Extensions.Http.Resilience 97 100

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1612414&view=codecoverage-tab

@rosebyte rosebyte left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you benchmarked the passing of the large struct through the out parameter, so LGTM.

@svick
svick merged commit 52cffc2 into dotnet:main Sep 30, 2026
5 checks passed
@svick
svick deleted the fix/hybrid-cache-pending-tag-invalidation branch September 30, 2026 14:31
thomhurst added a commit to thomhurst/Respire that referenced this pull request Oct 12, 2026
Microsoft.Extensions.Caching.Hybrid 10.10.0 validates a payload tag whose
invalidation marker is still loading against the reader's start time, not
the payload's creation time, and skips a wildcard marker that is still
loading (fixed upstream in dotnet/extensions#7786, not yet released). A new
context could therefore return an L2 entry that had already been removed by
tag when its own marker read completed after the payload read.

Tag propagation relies on new contexts (per-request fallbacks and new
observations) to reload authoritative tag metadata, so it hit this race
under load. Read the request tags' and the wildcard's markers first and
replay them into the context, keeping a newer remembered invalidation.

This is the cause of the intermittent "old" results in
SubscriberReconnectDropsCachedTagMetadataAcrossAMissedInvalidation.
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 31, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HybridCache: a tag removal is missed and the invalidated L2 entry served, when the tag's invalidation timestamp is still in flight

4 participants