[AdrianAtZyte]

Fixes #1180

[codspeed]

Merging this PR will not alter performance

✅ 7 untouched benchmarks

---

_{Comparing AdrianAtZyte:date-order-swap-1180 (066b1a5) with master (3ec5fe3)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.23%. Comparing base (3ec5fe3) to head (066b1a5).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1431      +/-   ##
==========================================
+ Coverage   97.21%   97.23%   +0.01%     
==========================================
  Files         236      236              
  Lines        3093     3113      +20     
==========================================
+ Hits         3007     3027      +20     
  Misses         86       86              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[serhii73]

Thanks! This fixes #1180, CI is green, and it merges cleanly onto current master. But comparing it against master turned up regressions that return wrong dates without any error, several of them in search_dates on ordinary text, so I don't think it's ready to merge yet.

search_dates, with languages=["en"], settings={"DATE_ORDER": "DMY"}:

Text	master	This PR
"Dates: 12/24, 15 March 2021"	two dates, including 2021-03-15	a single date for the whole span, 2021-03-12
"Offer valid 12/24 and 5 January 2021"	2021-01-05	None
"5 January 2021, 04/31"	2021-01-05 and 04/31	only January and 2021; 2021-01-05 is lost

The first and third also happen under the vi locale's own order, with no DATE_ORDER set.

parse:

• "12 13 December" with DMY, or "12 13 décembre" with fr: 2013-12-12 on master, December 12 of the current year here, because the 13 is dropped. With STRICT_PARSING, None.
• MYD: "31 DEC 09" goes from 2031-12-09 to 2009-12-31, "13 December 2nd" becomes None, and ja "31年10月28日" becomes 2028-10-31, so the year marked with 年 is read as the day.
• Hijri under DMY: HijriCalendar("01/13/21"), this PR's own DMY test string, gives Hijri 1413-01-21 instead of 1421-01-13.

These have four causes:

• The swap is committed per token, before the rest of the string has been read.
• A month name later in the string makes parse_alpha move the swapped month into the day, over the number the swap put there.
• A later number that then fits nowhere makes the whole parse fail. Day and month are checked separately, so April 31 passes.
• The wider %y deferral in at_year_position is what changes MYD.

A simpler approach would be to keep the token loop as on master and, when _parser fails, retry once with day and month swapped in the date order, unless the month is given by name. It can only turn a None into a date.

On a 527,380-parse differential run against master, it gives 14,531 new dates, 12,546 of them identical to this PR's, and no date becomes None. The only 6 changed dates are auto-detected nonsense like "10年20月00日". Every example above comes out as on master, and the Hijri one gives the correct 1421-01-13. It also fixes YDM "13/01/2021" and DYM "01/13/2021", which are still None with this PR, and the suite passes with this PR's tests.

Sketch, tested on current master

--- a/dateparser/parser.py
+++ b/dateparser/parser.py
@@ -377,6 +377,19 @@ class _parser:
                     setattr(self, "_token_%s" % attr, token)
                     setattr(self, attr, int(token))
 
+    @classmethod
+    def _has_month_name(cls, tokens):
+        for token, type in tokens:
+            if type != 1:
+                continue
+            for directive in cls.alpha_directives["month"]:
+                try:
+                    strptime(token.strip(), directive)
+                    return True
+                except ValueError:
+                    pass
+        return False
+
     def _get_period(self):
         if self.settings.RETURN_TIME_AS_PERIOD:
             if getattr(self, "time", None):
@@ -607,9 +620,19 @@ class _parser:
 
     @classmethod
     def parse(cls, datestring, settings, tz=None, date_order=None):
-        tokens = tokenizer(datestring)
-        po = cls(tokens.tokenize(), settings, date_order=date_order)
-        dateobj = po._results()
+        tokens = list(tokenizer(datestring).tokenize())
+        date_order = date_order or settings.DATE_ORDER
+        try:
+            po = cls(tokens, settings, date_order=date_order)
+            dateobj = po._results()
+        except ValueError as error:
+            if str(error).startswith("Fields missing") or cls._has_month_name(tokens):
+                raise
+            # The numbers do not fit the date order: read them with the day
+            # and month swapped, like YMD already does for "2021-13-01" (#1180).
+            swapped = date_order.translate(str.maketrans("DM", "MD"))
+            po = cls(tokens, settings, date_order=swapped)
+            dateobj = po._results()
 
         # correction for past, future if applicable
         dateobj = po._correct_for_time_frame(dateobj, tz)

Two caveats:

• Inside _parser.parse, the retry runs before _try_parser's REQUIRE_PARTS=["year"] retry with MYD and YMD, and changes results that retry used to find. For example, parse("1/13/20", languages=["fr"], settings={"REQUIRE_PARTS": ["year"]}) is 2013-01-20 on master and 2020-01-13 with the sketch. The swapped order should go last among _try_parser's candidates instead.
• One change comes with either approach and needs a decision: DMY locales now accept dates that only fit MDY. With several languages a different locale can win, and with try_previous_locales later dates flip. DateDataParser(languages=["fr", "en"], try_previous_locales=True, settings={"USE_GIVEN_LANGUAGE_ORDER": True}) on "01/13/2021" then "02/03/2021" gives en 2021-01-13 and en 2021-02-03 on master, and fr 2021-01-13 and fr 2021-03-02 with this PR. Applying the swap only to an explicit DATE_ORDER would avoid both caveats.

[serhii73]

Thanks, this addresses everything from my review. I re-ran the same checks on 0bcba20 merged into current master:

• The suite and ruff pass.
• Every example from my review gives master's result.
• The 527,380-parse run changes 14,766 results, all of them under an explicit DATE_ORDER, and none goes from a date to None.
• REQUIRE_PARTS without DATE_ORDER is unchanged (0 of 86,640).

One thing before merging: Codecov is red because the tests never reach the month-name guard (_has_month_name, lines 385-390). "12 13 December" parses under DMY without an error, so it never gets to the retry. This test fails without the guard, where the retry returns 2021-03-12 (the merge from my review), and it covers all of the guard's lines:

    def test_order_is_not_swapped_with_a_month_name(self):
        self.given_parser(languages=["en"], settings={"DATE_ORDER": "DMY"})
        self.when_date_is_parsed("12/24, Monday 15 March 2021")
        self.then_date_obj_exactly_is(None)

The test needs languages=["en"]. With auto-detection, fr reads the Mar in "12/24, 15 Mar 2021" as mardi, so the guard does not apply and parse returns 2021-12-24 where master returns None. search_dates is not affected.

With that, LGTM.

[serhii73]

Thanks! I re-ran the same checks on 066b1a5:

• The merge commit adds nothing beyond master: its tree is identical to merging 9b4d797 into master locally.
• The suite and ruff pass, and Codecov is green.
• test_order_is_not_swapped_with_a_month_name fails without the guard, where it gets 2021-03-12, and passes with it.
• The 527,380-parse run changes exactly the same 14,766 results as 0bcba20. All of them are under an explicit DATE_ORDER, and none goes from a date to None.

Two follow-ups, neither blocking:

• Nothing tests the Fields missing check, and the suite passes without it. Without it, parse("01/2021", settings={"DATE_ORDER": "DMY", "REQUIRE_PARTS": ["month"]}) reads the 01 given as the day as the month, and returns a date in January 2021 instead of None. This test fails without the check:

      def test_order_is_not_swapped_for_missing_parts(self):
          self.given_parser(
              languages=["en"], settings={"DATE_ORDER": "DMY", "REQUIRE_PARTS": ["month"]}
          )
          self.when_date_is_parsed("01/2021")
          self.then_date_obj_exactly_is(None)

• A bug already on master, which this PR reaches in more strings. Without RELATIVE_BASE, search_dates stops applying an explicit DATE_ORDER after the first date it finds. The cause is parse_item in search.py: it re-parses with settings.replace(RELATIVE_BASE=...), and that drops _mod_settings.

  • On master, search_dates("Shipped 25/12/2020, delivered 02/01/2021", languages=["en"], settings={"DATE_ORDER": "DMY"}) reads the second date as February 1.
  • With this PR, the first date of "Shipped 12/31/2020, delivered 02/01/2021" is found too, so the result goes from January 2 alone to December 31 and February 1.
  • Passing mod_settings=parser._settings._mod_settings to that replace() fixes both. It needs its own PR.

LGTM.