[Tune] Add Tuner.can_restore(path) utility for checking if an experiment exists at a path/uri

[justinvyu]

Why are these changes needed?

This PR adds a utility to check if a given path (either local or remote) exists and can be restored from. It includes some simple validation that this is the root of the experiment directory (can't restore from the trial level directory).

Example workflow that this should be used in:

import os
from ray.tune import Tuner
from ray.air import RunConfig

def train_fn(config):
    # Make sure to implement checkpointing so that progress gets
    # saved on restore.
    pass

name = "exp_name"
local_dir = "~/ray_results"
exp_dir = os.path.join(local_dir, name)

if Tuner.can_restore(exp_dir):
    tuner = Tuner.restore(exp_dir, resume_errored=True)
else:
    tuner = Tuner(
        "PPO",
        run_config=RunConfig(name=name, local_dir=local_dir),
    )
tuner.fit()

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[Yard1]

LGTM!

[krfricke]

LGTM

[krfricke]

Nit for readability

Suggested change

	path = tmp_path / name
	if upload_dir:
	path = Path(upload_dir) / name
	if upload_dir:
	path = Path(upload_dir) / name
	else:
	path = tmp_path / name