Skip to content

[core][release] threaded_actors_stress_test release lock before sleeping - #33221

Merged
rkooo567 merged 1 commit into
ray-project:masterfrom
rickyyx:pr-fix-threaded-actor-test
Mar 13, 2023
Merged

[core][release] threaded_actors_stress_test release lock before sleeping#33221
rkooo567 merged 1 commit into
ray-project:masterfrom
rickyyx:pr-fix-threaded-actor-test

Conversation

@rickyyx

@rickyyx rickyyx commented Mar 11, 2023

Copy link
Copy Markdown
Member

Why are these changes needed?

The other writer do_compute is being starved.

Related issue number

Closes #33219

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@rickyyx
release lock before sleeping
a0f3eff 
Signed-off-by: rickyyx <rickyx@anyscale.com>
@rickyyx

rickyyx commented Mar 11, 2023

Copy link
Copy Markdown
Member Author

TODO: run the release test.

@rickyyx

rickyyx commented Mar 11, 2023

Copy link
Copy Markdown
Member Author

https://buildkite.com/ray-project/release-tests-pr/builds/30545

rkooo567
rkooo567 approved these changes Mar 11, 2023
View reviewed changes

@rkooo567 rkooo567 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lmk the result.

@rkooo567

rkooo567 commented Mar 13, 2023

Copy link
Copy Markdown
Contributor

release test failed?

@rkooo567 rkooo567 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Mar 13, 2023
@rickyyx

rickyyx commented Mar 13, 2023

Copy link
Copy Markdown
Member Author

Timeout waiting on wheels I beleived. Reran.

@rickyyx

rickyyx commented Mar 13, 2023

Copy link
Copy Markdown
Member Author

Oh actually - there's no wheel built, using master wheel: https://buildkite.com/ray-project/release-tests-pr/builds/30676

@rickyyx

rickyyx commented Mar 13, 2023

Copy link
Copy Markdown
Member Author

The output definitely makes more sense now

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.63it/s]

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.47it/s]

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.63it/s]

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.00s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.50it/s]
Kill all actors.
Create actors.
Took 3.196321964263916 to create 24 actors
Start computation.

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:02<00:46,  2.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:02<00:00,  9.82it/s]

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.68it/s]

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.57it/s]

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s](MemoryMonitorActor pid=1833) INFO:root:Memory usage: 2.9203948974609375 / 62.77336502075195
(MemoryMonitorActor pid=1833) INFO:root:Top 10 process memory usage:
(MemoryMonitorActor pid=1833) INFO:root:PID     MEM     COMMAND
(MemoryMonitorActor pid=1833) 244       0.3GiB  /home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/gcs/gcs_server --log_dir=/tmp/ray/s
(MemoryMonitorActor pid=1833) 29743     0.18GiB ray::PiCalculator.get_pi
(MemoryMonitorActor pid=1833) 29744     0.16GiB ray::PiCalculator.get_pi
(MemoryMonitorActor pid=1833) 29745     0.15GiB ray::PiCalculator.get_pi
(MemoryMonitorActor pid=1833) 29742     0.13GiB ray::PiCalculator.get_pi
(MemoryMonitorActor pid=1833) 1743      0.11GiB python stress_tests/test_threaded_actors.py --test-runtime 1800 --kill-interval_s 30
(MemoryMonitorActor pid=1833) 609       0.11GiB /home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/agen
(MemoryMonitorActor pid=1833) 27        0.09GiB /home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/anyscale session web_terminal_server --deploy
(MemoryMonitorActor pid=1833) 350       0.07GiB /home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/dashboa
(MemoryMonitorActor pid=1833) 14        0.07GiB /home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/jupyter-lab --allow-root --ip=127.0.0.1 --no-

Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.55it/s]

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.62it/s]

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.54it/s]

  0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   0%|          | 0/24.0 [00:00<?, ?it/s]
Computing Pi:   4%|▍         | 1/24.0 [00:01<00:23,  1.01s/it]
Computing Pi:  83%|████████▎ | 20/24.0 [00:01<00:00, 19.66it/s]
....

@rickyyx rickyyx removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Mar 13, 2023
@rickyyx

rickyyx commented Mar 13, 2023 •
edited
Loading

Copy link
Copy Markdown
Member Author

It's now 1800 secs as expected:
image

@rkooo567
rkooo567 merged commit c484975 into ray-project:master Mar 13, 2023
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request Mar 21, 2023
@rickyyx @ProjectsByJackHe
release lock before sleeping (ray-project#33221)
f0ee09a 
Signed-off-by: rickyyx <rickyx@anyscale.com>
Signed-off-by: Jack He <jackhe2345@gmail.com>
edoakes pushed a commit to edoakes/ray that referenced this pull request Mar 22, 2023
@rickyyx @edoakes
release lock before sleeping (ray-project#33221)
5a50812 
Signed-off-by: rickyyx <rickyx@anyscale.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
peytondmurray pushed a commit to peytondmurray/ray that referenced this pull request Mar 22, 2023
@rickyyx @peytondmurray
release lock before sleeping (ray-project#33221)
df834fd 
Signed-off-by: rickyyx <rickyx@anyscale.com>
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@rickyyx @elliottower
release lock before sleeping (ray-project#33221)
549a68b 
Signed-off-by: rickyyx <rickyx@anyscale.com>
Signed-off-by: elliottower <elliot@elliottower.com>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@rickyyx @ProjectsByJackHe
release lock before sleeping (ray-project#33221)
4909377 
Signed-off-by: rickyyx <rickyx@anyscale.com>
Signed-off-by: Jack He <jackhe2345@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rkooo567 rkooo567 rkooo567 approved these changes

Assignees

@rkooo567 rkooo567

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[core][nightly] threaded_actors_stress_test failed

2 participants

@rickyyx @rkooo567