Skip to content

Enable streaming executor by default - #32493

Merged
ericl merged 61 commits into
ray-project:masterfrom
jianoaix:enablestreamingexec
Mar 20, 2023
Merged

Enable streaming executor by default#32493
ericl merged 61 commits into
ray-project:masterfrom
jianoaix:enablestreamingexec

Conversation

@jianoaix

@jianoaix jianoaix commented Feb 13, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

jianoaix added 30 commits December 8, 2022 23:20
@jianoaix
Fix read_tfrecords_benchmark nightly test
edc51bd 
Signed-off-by: jianoaix <iamjianxiao@gmail.com>
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
61f4d6d
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
a33a943
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
36ebe52
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
ce6763e
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
0e2c29e
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
f2b6ed0
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
bb6c5c4
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
540fe79
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
edad7d0
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
60cc079
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
a3d3980
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
001579c
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
8aeed6c
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
7a9a49b
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
ef97167
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
6f0563c
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
bcec4d6
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
ddef4e5
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
fc9a175
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
f0e90b7
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
999d1de
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
d8159e3
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
d81cd02
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
bc831bb
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
c444395
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
642da6f
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
f713f2f
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
d416a73
@jianoaix
Merge branch 'master' of https://github.com/ray-project/ray
da5acee
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
[data] Fix test failure caused by lack of ordring in default streamin…
bf076f9 
…g executor (#34120)

* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Fix test failure caused by lack of ordring in default streaming executor
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 6, 2023
@jianoaix
Add objects GC in dataset iterator (ray-project#34030)
af1da0c 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add objects GC in dataset iterator

* test it

* more tests

* fix comment

* add a little more memory as it's close to the limit and may make test flaky

* feedback
@jianoaix jianoaix mentioned this pull request Apr 6, 2023
Merged
8 tasks
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
Add objects GC in dataset iterator (#34030) (#34141)
1e92474 
* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Add objects GC in dataset iterator

* test it

* more tests

* fix comment

* add a little more memory as it's close to the limit and may make test flaky

* feedback
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 6, 2023
@jianoaix
Fix tensorarray to numpy conversion (ray-project#34115)
0838993 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 6, 2023
@jianoaix
Fix tensorarray to numpy conversion (ray-project#34115)
1887fcf 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
[data] Add streaming execution documentation (#33941)
968f6e9 
* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 6, 2023
@jianoaix
[data] Add streaming execution documentation (ray-project#33941)
4e4c4bc 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
Fix tensorarray to numpy conversion (#34115) (#34142)
f74e3c9 
* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion
jianoaix added a commit that referenced this pull request Apr 6, 2023
@jianoaix
[data] Add streaming execution documentation (#33941) (#34143)
c4371cf 
* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback
jianoaix added a commit that referenced this pull request Apr 12, 2023
@jianoaix
[data] Make sure the tf and tensor iteration work in dataset pipeline (…
66d3aaf 
…#34248)

* Revert "[Datasets] Revert "Enable streaming executor by default (#32493)" (#33485)"

This reverts commit 5c79954.

* make sure tf and tensor iteration in datapipeline work

* Fix

* fix

* fix

* fix

* feedback

* feedback

* fix
jianoaix added a commit to jianoaix/ray that referenced this pull request Apr 12, 2023
@jianoaix
[data] Make sure the tf and tensor iteration work in dataset pipeline (…
f7dc21b 
…ray-project#34248)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* make sure tf and tensor iteration in datapipeline work

* Fix

* fix

* fix

* fix

* feedback

* feedback

* fix
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
Enable streaming executor by default (ray-project#32493)
f3feb1d 
Signed-off-by: elliottower <elliot@elliottower.com>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@bveeramani @elliottower
[Datasets] Revert "Enable streaming executor by default (ray-project#…
33e33d7 
…32493)" (ray-project#33485)

The reverted commit broke some RL tests.

![image](https://user-images.githubusercontent.com/26107013/226484663-c30f3466-80cf-4e98-84dc-b6250cf60a13.png)

Signed-off-by: elliottower <elliot@elliottower.com>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
Add objects GC in dataset iterator (ray-project#34030)
d015068 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add objects GC in dataset iterator

* test it

* more tests

* fix comment

* add a little more memory as it's close to the limit and may make test flaky

* feedback

Signed-off-by: elliottower <elliot@elliottower.com>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
Fix tensorarray to numpy conversion (ray-project#34115)
982a4c0 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion

Signed-off-by: elliottower <elliot@elliottower.com>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
[data] Fix test failure caused by lack of ordring in default streamin…
c8c2471 
…g executor (ray-project#34120)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix test failure caused by lack of ordring in default streaming executor

Signed-off-by: elliottower <elliot@elliottower.com>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
[data] Add streaming execution documentation (ray-project#33941)
235b314 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback

Signed-off-by: elliottower <elliot@elliottower.com>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@jianoaix @elliottower
[data] Make sure the tf and tensor iteration work in dataset pipeline (…
46012cc 
…ray-project#34248)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* make sure tf and tensor iteration in datapipeline work

* Fix

* fix

* fix

* fix

* feedback

* feedback

* fix

Signed-off-by: elliottower <elliot@elliottower.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
Enable streaming executor by default (ray-project#32493)
f493425 
Signed-off-by: Jack He <jackhe2345@gmail.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@bveeramani @ProjectsByJackHe
[Datasets] Revert "Enable streaming executor by default (ray-project#…
2fb8af5 
…32493)" (ray-project#33485)

The reverted commit broke some RL tests.

![image](https://user-images.githubusercontent.com/26107013/226484663-c30f3466-80cf-4e98-84dc-b6250cf60a13.png)

Signed-off-by: Jack He <jackhe2345@gmail.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
Add objects GC in dataset iterator (ray-project#34030)
4beb948 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add objects GC in dataset iterator

* test it

* more tests

* fix comment

* add a little more memory as it's close to the limit and may make test flaky

* feedback

Signed-off-by: Jack He <jackhe2345@gmail.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
Fix tensorarray to numpy conversion (ray-project#34115)
bcc6353 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix tensorarray to numpy conversion

Signed-off-by: Jack He <jackhe2345@gmail.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
[data] Fix test failure caused by lack of ordring in default streamin…
e1ef7df 
…g executor (ray-project#34120)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Fix test failure caused by lack of ordring in default streaming executor

Signed-off-by: Jack He <jackhe2345@gmail.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
[data] Add streaming execution documentation (ray-project#33941)
5adfdc0 
* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* Add streaming execution documentation

* fix

* feedback

* remove new file

* fix

* fix

* key concept

* fix

* fix

* fix

* wording

* feedback

Signed-off-by: Jack He <jackhe2345@gmail.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@jianoaix @ProjectsByJackHe
[data] Make sure the tf and tensor iteration work in dataset pipeline (…
f3c3810 
…ray-project#34248)

* Revert "[Datasets] Revert "Enable streaming executor by default (ray-project#32493)" (ray-project#33485)"

This reverts commit 5c79954.

* make sure tf and tensor iteration in datapipeline work

* Fix

* fix

* fix

* fix

* feedback

* feedback

* fix

Signed-off-by: Jack He <jackhe2345@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericl ericl ericl approved these changes
@scv119 scv119 Awaiting requested review from scv119
@clarkzinzow clarkzinzow Awaiting requested review from clarkzinzow
@jjyao jjyao Awaiting requested review from jjyao
@c21 c21 Awaiting requested review from c21

Assignees

@ericl ericl

@c21 c21

Labels

tests-ok The tagger certifies test failures are unrelated and assumes personal liability.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jianoaix @ericl @c21