[Doc][Tune][Restore] Split "Tune stopping and resuming" into two user guides

[justinvyu]

Summary

This PR splits the "Tune Stopping and Resuming" guide into two user guides: (1) Tune Stopping, and (2) Tune Fault Tolerance. This PR also adds a FAQ post about the "iterative experimentation" workflow not being supported by Tuner.restore, but rather through new Tune experiments.

Why?

The scope of Tuner.restore functionality is unclear. Users think that it should be used for iterative experimentation where hyperparameters/configs are changed before resuming training. However, its usage should be limited to providing fault tolerance at the experiment level, without any changes to the experiment specification. [github, slack, slack, slack]

In previous discussions (Tuner API Design), we have decided against allowing modifications to things like trial stopping criteria upon resume. The idea is that an experiment should follow a single spec, and experiment restore is used for fault-tolerance to continue retrying the job if it’s interrupted. We have not communicated this decision through to outward-facing docs and usage examples (both in the open-source and product documentation), which is the reason for the continued user questions (see links above).

--> For reviewers <--

There are 3 new pieces of the docs to review:

• New Tune stopping guide
• New Tune fault tolerance guide
• New FAQ section

Here's what the old guide looks like: https://docs.ray.io/en/releases-2.3.0/tune/tutorials/tune-stopping.html

Related issue number

Closes #30223
Closes #31830

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gjoliver]

very nice man.
so super quick questions

[gjoliver]

is start_from_checkpoint a new config name that is introduced?

[justinvyu]

Nope, this is just some random config name for users to seed the new run with a checkpoint. This actually only works if the checkpoint is available in s3. Otherwise, if it's a local path, some nodes will not find it.

[gjoliver]

but if we run new Tune experiments, how do you seed the initial model with whatever checkpint avaialble to us?
shouldn't we add an example for that too?

[justinvyu]

See the first point. Without introducing some API to pass an initial checkpoint to Tuner, then accessing it through session.get_checkpoint, we need to just pass it through the config.