Skip to content

[AIR/Docs] Large model inference examples - #32874

Merged
Yard1 merged 9 commits into
ray-project:masterfrom
Yard1:big_model_prediction_examples
Mar 2, 2023
Merged

[AIR/Docs] Large model inference examples#32874
Yard1 merged 9 commits into
ray-project:masterfrom
Yard1:big_model_prediction_examples

Conversation

@Yard1

@Yard1 Yard1 commented Feb 27, 2023

Copy link
Copy Markdown
Member

Why are these changes needed?

This PR adds two simple examples showing how to do batch inference with Ray Data using Stable Diffusion and GPT-J models.

Note that the datasets used are very small, allowing them to be ran quickly by both the user and CI. Examples with large scale datasets (>=1 TB) are left as a TODO.

DeepSpeed Inference example will be added in a separate PR.

We are using Ray Data instead of BatchPredictor due to #32923.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Comment thread doc/source/ray-air/examples/BUILD
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
@Yard1
Yard1 marked this pull request as ready for review February 28, 2023 23:48
@Yard1
Yard1 requested a review from richardliaw February 28, 2023 23:49

@ericl ericl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good at a high level. Should we add a TODO to add a larger-scale example? This one is quite small.

@Yard1

Yard1 commented Mar 1, 2023

Copy link
Copy Markdown
Member Author

@ericl I think it's fine to keep them small - after all, it is that easy. The DeepSpeed example I am working on will be bigger and more advanced, and the same goes for fine-tuning examples that are next in line.

Comment thread doc/source/ray-air/examples/gptj_batch_prediction.ipynb
Comment thread doc/source/ray-air/examples/gptj_batch_prediction.ipynb Outdated
@ericl ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Mar 1, 2023
Comment thread doc/source/ray-air/examples/gptj_batch_prediction.ipynb
Comment thread doc/source/ray-air/examples/gptj_batch_prediction.ipynb Outdated
Comment thread doc/source/ray-air/examples/gptj_batch_prediction.ipynb Outdated

@richardliaw richardliaw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all same comments apply across notebooks. nice work!

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
@Yard1
Yard1 requested review from ericl and richardliaw March 1, 2023 20:24
@Yard1 Yard1 removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Mar 1, 2023
Comment thread doc/source/ray-air/examples/gptj_batch_prediction.ipynb Outdated
Yard1 added 2 commits March 1, 2023 21:36
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
@ericl ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Mar 1, 2023
@Yard1
Yard1 merged commit ff66e0e into ray-project:master Mar 2, 2023
@Yard1
Yard1 deleted the big_model_prediction_examples branch March 2, 2023 04:53
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request Mar 21, 2023
* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update versions

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update toc

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Cleanup

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Fix

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

---------

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Signed-off-by: Jack He <jackhe2345@gmail.com>
edoakes pushed a commit to edoakes/ray that referenced this pull request Mar 22, 2023
* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update versions

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update toc

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Cleanup

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Fix

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

---------

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
peytondmurray pushed a commit to peytondmurray/ray that referenced this pull request Mar 22, 2023
* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update versions

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update toc

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Cleanup

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Fix

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

---------

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update versions

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update toc

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Cleanup

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Fix

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

---------

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Signed-off-by: elliottower <elliot@elliottower.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update versions

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* WIP

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Update toc

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Cleanup

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Apply feedback

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

* Fix

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

---------

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Signed-off-by: Jack He <jackhe2345@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

@author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants