Skip to content

Removing leading blank lines can move a coding cookie onto line 1 and change how the file is decoded #5486

Closed

Description

@jiteshgulati

Describe the bug

If a file starts with blank lines and has a coding cookie on line 3 or later then python ignores that cookie and reads the file as utf-8. Black removes the leading blank lines so the cookie ends up on line 1 where python does read it. The file is still written back as utf-8 though so any non ascii text gets decoded differently the next time it runs

The AST safety check doesnt catch it because it compares the decoded text and there the cookie is just a comment. Black exits 0 and says the file was reformatted

To Reproduce

For example, take this code (saved as utf-8)

# -*- coding: latin-1 -*-
x = "é"
print(repr(x), len(x))

And run it with these arguments

$ python file.py
'é' 1
$ black file.py
reformatted file.py
$ echo $?
0
$ python file.py
'é' 2

Same thing with a vim modeline like # vim: set fileencoding=cp1252 : and with # coding: ascii the file stops importing at all (SyntaxError: encoding problem: ascii). It also happens through stdin with --stdin-filename, with --fast, with --line-ranges and through blackd

Expected behavior

Black shouldnt change what the file means. It should either keep the file decoding the same way or refuse to format it with an error like it does when the AST check fails

Environment

  • Black's version: main (a7c2368)
  • OS and Python version: Linux (WSL2) Python 3.14.4 and also checked on 3.13

Additional context

The blank lines get removed by EmptyLineTracker (no empty lines before the first line in the file) and the encoding used to write the file comes from decode_bytes which only looks at the first two lines of the original. Nothing checks that the output still decodes back to the same text

If the file is plain ascii nothing actually changes so those cases are fine. It only breaks when there is non ascii text or when the file has a utf-8 BOM

Happy to open a PR for this. I have a fix that refuses to format only when the output wouldnt decode back to the same text plus tests for it

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions