Repository navigation
Avoid shared language_detector and locale cache in search_dates - #1371
Merged
serhii73 merged 1 commit intoSep 3, 2026
Merged
Conversation
Fixes scrapinghub#1369. search_dates uses a process-wide DateSearchWithDetection singleton. detect_language stashed a single-use FullTextLanguageDetector on self, and _ExactLanguageSearch kept a 1-slot locale cache. Concurrent calls that disagree about the language could return None or a date from the wrong locale. Use a local detector and return the locale from get_current_language so neither value is instance state. LocaleDataLoader.get_locale already caches. Thanks @serhii73 for the report.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1371 +/- ##
==========================================
- Coverage 97.18% 97.18% -0.01%
==========================================
Files 236 236
Lines 3059 3057 -2
==========================================
- Hits 2973 2971 -2
Misses 86 86 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
serhii73
approved these changes
Sep 3, 2026
Collaborator
|
Thanks! |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1369.
search_datesruns through a process-wideDateSearchWithDetectionsingleton. Two leftover store-then-load races from #1346 can make concurrent calls returnNoneor a date built from the wrong locale:detect_languagestashed aFullTextLanguageDetectoronselfand then loaded it. That detector is single-use:_best_languagenarrowsself.languagesin place. A concurrent call can observe another call's already-narrowed detector._ExactLanguageSearchkeptself.languageas a 1-slot cache on the same singleton. A store from another thread betweenget_current_languageandtranslate_searchuses the wrong locale.Neither value needs to be instance state. Build the detector as a local, and have
get_current_languagereturn the locale (LocaleDataLoader.get_localealready caches). Public API is unchanged.Thanks @serhii73 for the report and the diagnosis.