Streaming Lambda response support in the OSI (Data Prepper) Lambda processor #5973

Is your feature request related to a problem? Please describe.
Currently, the Lambda processor in OpenSearch Ingestion (OSI) uses the standard AWS Lambda invoke function and waits for the full synchronous response. This approach restricts lambda functions to only respond with non-streaming payloads. As a result, any function that performs heavy computation or needs to progressively return data (due to large payload handling and lambda restricting the synchronous response to 6MB) is constrained by payload size. It would be very helpful to support streaming Lambda responses to allow larger response payloads to be returned and ingested into OpenSearch.

Describe the solution you'd like
Enable the Lambda processor in OSI to support streaming responses by using the InvokeWithResponseStream API or Lambda Function URLs, allowing incremental processing of data returned by the Lambda function. Add a configuration flag (e.g., streaming: true) to opt into this behavior.

Describe alternatives you've considered (Optional)
Considered preprocessing data outside OSI to not run into lambda response payload limitations, but that adds latency, complexity, and defeats the purpose of inline, real-time processing.

Additional context
Add any other context or screenshots about the feature request here.