[BUG] cloudwatch_logs sink: Uploader thread silently terminates on unchecked Throwable #6887

Describe the bug

When an unchecked Throwable escapes CloudWatchLogsDispatcher.Uploader.upload(), the executor worker thread terminates with no Data Prepper signal. Specifically:

• Both cloudWatchLogsEventsSucceeded and cloudWatchLogsEventsFailed stay flat for the affected batch
• No structured Data Prepper log entry is emitted
• EventHandle.release() is never called, so the acknowledgement-aware source never receives an ack and stalls indefinitely on that batch
• The only visible artifact is an unstructured stack trace from java.util.concurrent.ThreadPoolExecutor.runWorker() on stderr

The thread pool replaces the worker and continues processing other batches, so the pipeline appears healthy from outside. The affected batch eventually gets re-delivered (e.g., on SQS visibility-timeout expiry) and fails the same way — repeating indefinitely.

The root cause is the missing top-level Throwable catch in Uploader.run():

   @Override
   public void run() {
       upload();   // no safety net
   }

The inner try { ... } catch (Exception e) inside upload() does not catch:

• Error subclasses: NoSuchMethodError, NoClassDefFoundError, etc. These can surface from real-world conditions like AWS SDK version mismatches on the runtime classpath, native-library issues, or memory pressure.
• A RuntimeException thrown from the post-loop cleanup code (after the inner try/catch returns) — for example from response handling, DLQ-object construction, event-handle release, or DLQ push.

To Reproduce

The minimal reproduction is a unit test that stubs CloudWatchLogsClient.putLogEvents(...) to throw an Error:

   when(cloudWatchLogsClient.putLogEvents(any(PutLogEventsRequest.class)))
           .thenThrow(new NoSuchMethodError("simulated linkage failure"));

   // dispatch a batch via the Uploader, then observe:
   //   - no Data Prepper metric was incremented
   //   - EventHandle.release(...) was never called
   //   - the Uploader thread returned without any structured log

In production, the same outcome is reached whenever any unchecked Throwable escapes Uploader.upload(). Concrete triggers seen in the wild include:

1. AWS SDK version mismatch between Data Prepper's bundled SDK and a transitive dependency, surfacing as NoSuchMethodError or NoClassDefFoundError from the first call into cloudWatchLogsClient.putLogEvents(...).

2. A RuntimeException from PutLogEventsResponse.rejectedLogEventsInfo() access during response handling — for example if a future SDK release changes response shape.

Expected behavior

When an unchecked Throwable escapes the Uploader, Data Prepper should:

• Increment a dedicated metric so dashboards/alarms can detect the failure
• Emit a structured ERROR-level log identifying the affected log group, log stream, and event count
• Release the batch's EventHandles so the source can make forward progress

Screenshots

N/A — the failure mode is the absence of metrics and logs.

Environment

• OS: Any (JVM-level error handling)
• Data Prepper: current main (verified)
• Java: 11

Additional context

• Source: data-prepper-plugins/cloudwatch-logs/.../CloudWatchLogsDispatcher.java, methods Uploader.run() and Uploader.upload()
• Severity: high — no operator-visible signal; only surfaces when downstream consumers report missing data.