[core] Corrupted cluster state due to restarting actors associated with finished jobs prevents autoscaler from scaling down

[rickyyx]

What happened + What you expected to happen

We have seen issues where a cluster is idle (all job finishes), but some actors from the job stuck inthe RESTARTING states.

We believe this is likely a race condition, where the clean up logic to kill a worker never triggers when the job finishes (and thus the owner goes out of the scope).

Versions / Dependencies

2.3

Reproduction script

NA

Issue Severity

None

[rickyyx]

Current hypothesis is this code path never gets triggered (callback not called somehow for a long running cluster):

ray/src/ray/gcs/gcs_server/gcs_actor_manager.cc

Lines 739 to 760 in 174f157

	it->second.client->WaitForActorOutOfScope(
	wait_request,
	[this, owner_node_id, owner_id, actor_id](
	Status status, const rpc::WaitForActorOutOfScopeReply &reply) {
	if (!status.ok()) {
	RAY_LOG(INFO) << "Worker " << owner_id
	<< " failed, destroying actor child, job id = "
	<< actor_id.JobId();
	} else {
	RAY_LOG(INFO) << "Actor " << actor_id
	<< " is out of scope, destroying actor, job id = "
	<< actor_id.JobId();
	}
	auto node_it = owners_.find(owner_node_id);
	if (node_it != owners_.end() && node_it->second.count(owner_id)) {
	// Only destroy the actor if its owner is still alive. The actor may
	// have already been destroyed if the owner died.
	DestroyActor(
	actor_id, GenActorOutOfScopeCause(GetActor(actor_id)), /*force_kill=*/true);
	}
	});

Could be a similar issue to #31640