[data] Add streaming execution documentation - #33941
Merged
Merged
Conversation
…project#32493)" (ray-project#33485)" This reverts commit 5c79954.
jianoaix
requested review from
a team,
c21,
clarkzinzow,
ericl,
jjyao,
maxpumperla and
scv119
as code owners
March 30, 2023 20:36
ericl
reviewed
Mar 30, 2023
|
|
||
| This section covers the Datasets execution model and performance considerations. | ||
|
|
||
| .. _datasets_lazy_execution: |
Contributor
There was a problem hiding this comment.
I think we should add/edit this page instead of adding a new page.
ericl
reviewed
Mar 30, 2023
ericl
reviewed
Mar 30, 2023
| The DatasetPipeline is expected to be deprecated in Ray 2.5. If your use case doesn't | ||
| need per-window shuffle, we recommend using the streaming execution of Dataset. By | ||
| setting the resource limits, you can cap the resource usage to run the operations | ||
| as the DatasetPipeline window, and achieve even better performance and stability. |
Contributor
There was a problem hiding this comment.
Resources limits are too much detail to mention here.
Contributor
Author
There was a problem hiding this comment.
Makes sense, dropped.
ericl
reviewed
Mar 30, 2023
ericl
left a comment
ericl
left a comment
Contributor
There was a problem hiding this comment.
I think you also need to edit the key concepts page to mention streaming.
ericl
added
the
@author-action-required
jianoaix
removed
the
@author-action-required
ericl
reviewed
Apr 5, 2023
|
|
||
| ctx.execution_options.preserve_order = True | ||
|
|
||
| To enable deterministic execution, set the above to True. This may decrease performance, but will ensure block ordering is preserved through execution. This flag defaults to True in Ray 2.3 but False in |
Contributor
There was a problem hiding this comment.
We can probably simplify to say it defaults to False, since streaming isn't enabled in 2.3 by default.
ericl
reviewed
Apr 5, 2023
|
|
||
| .. code-block:: | ||
|
|
||
| ctx.execution_options.actor_locality_enabled = True |
Contributor
There was a problem hiding this comment.
Can note this is enabled by default.
ericl
approved these changes
Apr 5, 2023
ericl
added
the
@author-action-required
zhe-thoughts
approved these changes
Apr 5, 2023
zhe-thoughts
left a comment
zhe-thoughts
left a comment
Contributor
There was a problem hiding this comment.
Approved for cherry picking into 2.4.0
jianoaix
removed
the
@author-action-required
Contributor
Author
|
The test failure (https://buildkite.com/ray-project/oss-ci-build-pr/builds/17601#01875443-53d7-48eb-bfcd-f7a48e01bbfd) is not relevant and actually fixed (#34115). |
jianoaix
added a commit
to jianoaix/ray
that referenced
this pull request
Apr 6, 2023
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)" This reverts commit 5c79954. * Add streaming execution documentation * fix * feedback * remove new file * fix * fix * key concept * fix * fix * fix * wording * feedback
Contributor
|
Approved for cherry picking into 2.4.0 |
elliottower
pushed a commit
to elliottower/ray
that referenced
this pull request
Apr 22, 2023
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)" This reverts commit 5c79954. * Add streaming execution documentation * fix * feedback * remove new file * fix * fix * key concept * fix * fix * fix * wording * feedback Signed-off-by: elliottower <elliot@elliottower.com>
ProjectsByJackHe
pushed a commit
to ProjectsByJackHe/ray
that referenced
this pull request
May 4, 2023
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)" This reverts commit 5c79954. * Add streaming execution documentation * fix * feedback * remove new file * fix * fix * key concept * fix * fix * fix * wording * feedback Signed-off-by: Jack He <jackhe2345@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Streaming execution will be enabled in 2.4 so we need to have documentation coverage to guide users.
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.