Skip to content

Fix database dedupe performance on large channels - #751

Merged
rusq merged 1 commit into
masterfrom
750
Oct 20, 2026
Merged

rusq merged 1 commit into
masterfrom
750

Conversation

@rusq

@rusq rusq commented Oct 19, 2026

Copy link
Copy Markdown
Owner

Large-channel database deduplication can select a channel/chunk range lookup that ignores message identity, making preview and cleanup impractically slow. Add a migrated MESSAGE (CHANNEL_ID, ID, CHUNK_ID) index so candidate searches constrain the message ID while preserving NULL-safe matching, edited payload history, and both dedupe modes. Existing archives build the index automatically on first open after upgrading.

Add migration, functional, and query-plan regression coverage plus separate Preview/Deduplicate benchmarks. Document the first-open migration cost and ignore trace files.

Validation:

  • Focused database, dedupe-command, and resume race tests passed; all 98 non-scratch repository packages passed race/coverage tests and vet. Standard make targets encounter an existing compile error in ignored tmp/security/repro; the full suite was run with local listener access enabled.
  • On a copy of a 620,723-message archive, CLI cleanup completed in 38.5 seconds, removing 25,543 messages, 310 files, and 267 chunks. Independent survivor comparisons, integrity/foreign-key checks, and a second no-op run passed. The original archive remained unchanged.
  • Both message modes passed synthetic CLI checks. Two independent empty-context reviews reported no actionable findings.

Closes #750.

@rusq rusq added the ai-assisted This PR or Issue were created with the help of AI agent(-s). label Oct 19, 2026
@rusq
rusq merged commit 1cddd73 into master Oct 20, 2026
4 checks passed
@rusq
rusq deleted the 750 branch October 20, 2026 03:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted This PR or Issue were created with the help of AI agent(-s).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dedupe never finishes on a large MESSAGE table since 4.4.5 (duplicates join now uses MESSAGE_I1)

1 participant