Skip to content

[air] Fix NoneType error loading TorchCheckpoint through from_uri. - #32386

Merged
xwjiang2010 merged 9 commits into
ray-project:masterfrom
xwjiang2010:fix_batch_predictor
Feb 21, 2023
Merged

[air] Fix NoneType error loading TorchCheckpoint through from_uri.#32386
xwjiang2010 merged 9 commits into
ray-project:masterfrom
xwjiang2010:fix_batch_predictor

Conversation

@xwjiang2010

@xwjiang2010 xwjiang2010 commented Feb 9, 2023

Copy link
Copy Markdown
Contributor
  • encode/decode logic of TorchCheckpoint is only relevant in the context of training
  • it's currently intercepting __getstate__ and __setstate__
  • training worker first sends TorchCheckpoint to Trainable (through ser/deser). The TorchCheckpoint that Trainable receives is already "decoded"
  • the last statement may not hold true anymore if training worker uploads TorchCheckpoint directly with to_uri, at which point we should revisit where to put the encode/decode logic.

Signed-off-by: xwjiang2010 xwjiang2010@gmail.com

Why are these changes needed?

Related issue number

Closes #32284

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

@Yard1 Yard1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this, the model should be already encoded when it was put into the URI. I think the only thing that needs to be fixed is the missing check for the case where _data_dict is None:

    def __setstate__(self, state: dict):
        if state.get("_data_dict", None):
            state = state.copy()
            state["_data_dict"] = self._decode_data_dict(state["_data_dict"])
        super().__setstate__(state)

@krfricke krfricke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good!

@xwjiang2010

Copy link
Copy Markdown
Contributor Author

I don't think we need this, the model should be already encoded when it was put into the URI. I think the only thing that needs to be fixed is the missing check for the case where _data_dict is None:

    def __setstate__(self, state: dict):
        if state.get("_data_dict", None):
            state = state.copy()
            state["_data_dict"] = self._decode_data_dict(state["_data_dict"])
        super().__setstate__(state)

isn't this for decoding ? We encode it when putting into the URI and decode it when retrieving?

@krfricke krfricke assigned Yard1 and unassigned krfricke Feb 10, 2023
Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
@xwjiang2010

Copy link
Copy Markdown
Contributor Author

waiting for https://github.com/ray-project/ray/pull/32479/files to be landed first...

@xwjiang2010 xwjiang2010 changed the title [air] Eagerly load TorchCheckpoint upon from_uri. [air] Fix NoneType error loading TorchCheckpoint through from_uri. Feb 13, 2023
Comment thread python/ray/train/torch/torch_checkpoint.py Outdated
Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
Comment thread python/ray/train/tests/test_torch_predictor.py Outdated
Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
@xwjiang2010
xwjiang2010 merged commit f374554 into ray-project:master Feb 21, 2023
@xwjiang2010
xwjiang2010 deleted the fix_batch_predictor branch February 21, 2023 17:50
edoakes pushed a commit to edoakes/ray that referenced this pull request Mar 22, 2023
…ay-project#32386)

* Eagerly load TorchCheckpoint upon from_uri.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* [no_ci] remove eager load

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* avoid shallow copy

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* Use `mock_s3_bucket_uri` fixture.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* remove conftest import

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* revert changes in tune/.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

---------

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
peytondmurray pushed a commit to peytondmurray/ray that referenced this pull request Mar 22, 2023
…ay-project#32386)

* Eagerly load TorchCheckpoint upon from_uri.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* [no_ci] remove eager load

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* avoid shallow copy

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* Use `mock_s3_bucket_uri` fixture.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* remove conftest import

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* revert changes in tune/.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

---------

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
…ay-project#32386)

* Eagerly load TorchCheckpoint upon from_uri.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* [no_ci] remove eager load

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* avoid shallow copy

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* Use `mock_s3_bucket_uri` fixture.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* remove conftest import

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* revert changes in tune/.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

---------

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
Signed-off-by: elliottower <elliot@elliottower.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Torch batch prediction fails

3 participants