[ArockiaRajamanickam]

Description

Closes #3505.

Black reindents to 4 spaces but leaves # fmt: off regions alone, so its output can end up mixing indent widths. @JelleZijlstra said as much on the issue:

So the reason this happen is that Black (correctly) reindents the code to use 4-space indents, except within the # fmt: off part, because the user asked us not to format it. So the code it tries to output then has a syntax error from the remaining 2-space indents.

I still don't think there is a way we can format these cases correctly, but we should give a better error.

This is the better error, rather than an attempt to format those cases.

The forced second pass in format_str() parses Black's own output, but the InvalidInput it raised escaped uncaught, so it surfaced against the user's file:

$ cat reindent.py
if __name__ == "__main__":
  foo = 3
  # fmt: off
  bar = 1
  baz  =  2
  # fmt: on
  qux = 4

$ python -c "import ast; ast.parse(open('reindent.py').read())"   # the file is valid

$ black reindent.py
error: cannot parse: reindent.py:5:0
      baz  =  2
    ^
ParseError: bad input

That is byte-for-byte the shape of a genuine syntax error. def f(: produces error: cannot parse: genuine.py:1:6 and the same ParseError: bad input, so there is nothing for a user to tell them apart, and it points at a line of their code that is fine.

After:

$ black reindent.py
error: cannot format reindent.py: INTERNAL ERROR: Black 0.1.dev20 on Python (CPython) 3.12.13
produced invalid code: cannot parse: 5:0
      baz  =  2
    ^
ParseError: bad input. Please report a bug on https://github.com/psf/black/issues.
This invalid output might be helpful: /tmp/blk_iwhhpmsr.log

This also restores what @ichard26 described as missing:

Usually Black would emit a helpful INTERNAL ERROR: Black produced invalid code error, but that's been broken for a while now.

The change is one try/except around the second pass. ASTSafetyError, InvalidInput, dump_to_file and _black_info() are all already imported, and the message reuses the exact wording assert_equivalent() uses for the same class of failure, so there is one phrasing for "Black produced invalid code", not two.

Scope

Only the forced second pass is wrapped. The first pass parses the user's source, so a genuine syntax error there still reports the way it always did, and test_invalid_input_error_includes_path_location passes unchanged. Exit code stays 123 in both cases.

This does not make the affected files formattable, which Jelle judged not possible. #3702 and #3705 report the same underlying situation and would get the same clearer error.

Checklist

• Add tests
• Update CHANGES.md (follow-up commit, since it wants the PR number)

test_fmt_off_reindent_reports_black_not_the_user asserts the new message, and asserts the old cannot parse: <path>: form is absent, so a regression back to blaming the user fails the test. Reverting only src/black/__init__.py and keeping the test fails it.

Full suite: 555 passed / 2 skipped on this branch against 554 passed / 2 skipped on main on the same machine, no failures either side, the difference being the added test.

I used an AI assistant while working on this. The reproduction, the comparison against a genuine syntax error, and the decision to reuse the existing INTERNAL ERROR wording rather than invent a second one are mine, and I ran every result quoted here.

[github-actions]

diff-shades results comparing this PR (c23728c) to main (8d5a2d9):

--preview style: no changes

--stable style: no changes

---

What is this? | Workflow run | diff-shades documentation

[cobaltt7]

This is the current output:

error: cannot parse: ..\a.py:5:0
      baz  =  2
    ^
ParseError: bad input

With this PR:

error: cannot format ..\a.py: INTERNAL ERROR: Black 26.5.2 on Python (CPython) 3.12.4 produced invalid code: cannot parse: 5:0
      baz  =  2
    ^
ParseError: bad input. Please report a bug on https://github.com/psf/black/issues.  This invalid output might be helpful: C:\Users\username\AppData\Local\Temp\blk_wcm8wg6q.log

This output somewhat obscures the actual error. I'd prefer something like this:

error: INTERNAL ERROR: Black 26.5.2 on Python (CPython) 3.12.4 produced invalid code:
cannot parse ..\a.py:5:0: 
      baz  =  2
    ^
ParseError: bad input.
Please report a bug on https://github.com/psf/black/issues.  This invalid output might be helpful: C:\Users\username\AppData\Local\Temp\blk_wcm8wg6q.log

To accomplish this, I believe the best way would be to prepend to exc.context (originally "cannot parse") and append to exc.details (originally ends with ParseError: bad input.) and re-raise the same InvalidInput. If you find a better way go for it!

[ArockiaRajamanickam]

Thanks, that reads much better and it is a smaller change too. Done in c23728c, using the approach you described rather than wrapping it in an ASTSafetyError: prepend to exc.context, append to exc.details, fix up exc.args so str(exc) stays consistent with them, and re-raise the same InvalidInput.

Output now, on the reproducer from #3505:

error: INTERNAL ERROR: Black 26.5.2.dev130+g8d5a2d9f4 on Python (CPython) 3.12.13 produced invalid code:
cannot parse: /tmp/real.py:7:0
        baz = 4
    ^
ParseError: bad input
Please report a bug on https://github.com/psf/black/issues.  This invalid output might be helpful: /var/folders/.../blk_cn90p5od.log

One deliberate difference from your sketch: you had cannot parse ..\a.py:5:0: with the colon after the position, but Report.failed() builds that line as f"{message.context}: {src}:{message.lineno}:{message.column}{details}", which is shared with ordinary parse errors. Reshaping it would have changed those too, so I left the renderer alone and only fed it a different context and details. Say the word if you would rather I touched the renderer.

Keeping InvalidInput also means lineno, column and the caret survive, so the location still points at the offending line of Black's own output.

Checked the three cases that matter:

input	result
#3505 reproducer, valid Python	new INTERNAL ERROR banner
def f(: , a real syntax error	unchanged, plain cannot parse
genuinely bad indentation, fails the first pass	unchanged, plain cannot parse

That last one is worth calling out, since it is the thing this could plausibly have broken: only the forced second pass is wrapped, so a user's own broken file still gets a normal parse error rather than being blamed on Black.

Also rebased onto current main, which had moved under this PR: the second pass now assigns dst_contents instead of returning it, so the old patch no longer applied. tests/test_black.py is 174 passed, reverting only src/black/__init__.py fails the new test, black --check is clean on both changed files, and flake8 and mypy are clean.

I used an AI assistant on this. The verification of the three cases above, and the decision not to touch Report.failed(), are mine.

[cobaltt7]

Thanks!