Repository navigation
Support foreach (looping) over input arrays to split event #5707
Description
Is your feature request related to a problem? Please describe.
Currently, Data Prepper lacks a dedicated processor to iterate over array elements and apply transformations to each element individually. This limitation makes it challenging to process array fields efficiently, especially when dealing with nested data structures or when needing to apply consistent transformations across all elements in an array.
For example, when processing log data with arrays of strings or objects, users currently need to create complex workarounds or multiple processors to handle array transformations, which is neither efficient nor maintainable.
It would also be good to split each element into an individual event.
Describe the solution you'd like
Add a new "foreach" processor to Data Prepper that would:
- Accept an array field path as input
- Allow defining a set of sub-processors to be applied to each element
- Support options for:
- Transforming elements in-place
- Creating new events for each element (array splitting)
- Handling null or missing values
- Controlling parallel processing of elements
Describe alternatives you've considered (Optional)
Can't find any
Additional context
Activity
dlvenable commentedon May 20, 2025
@architkshk , We have a
split_eventsprocessor. But, it only works for splitting on strings. If this were to split on an array, would it work for you?dlvenable commentedon May 20, 2025
If you
split_event, then you could use the normal processor chain for the processing that you want to do on each part of the array.architkshk commentedon May 20, 2025
Yes, splitting on array should work, if it splits it into multiple events and rest of the processors acting on each one of them (just like a for loop).
besha100 commentedon May 28, 2025
@dlvenable We have multiple cases where this would be useful, so I’m giving this a thumbs up.
zgouda commentedon Feb 5, 2026
We're encountering this limitation across multiple implementations where we need to transform nested arrays with conditional filtering and object building. Currently we have to implement workarounds, but a foreach/array iteration processor would make these transformations much more maintainable and scalable. Would be great to see this implemented. @dlvenable
srikanthpadakanti commentedon Apr 5, 2026
Hello @dlvenable Can you please assign this to me. Thanks.
dlvenable commentedon Apr 6, 2026
@srikanthpadakanti , Thank you for your interest in working on this.
I think one thing we would need to consider is how iterating over an array works with the JSON pointer structure. We have had some aspects in which processors need both the ability to load from the root of an event and from a relative position. There is a good discussion #6609 that would inform any solution to a foreach processor.
srikanthpadakanti commentedon Apr 6, 2026
@dlvenable Thanks for linking #6609, I went through the discussion there. I understand the JSON pointer context issue is relevant if we build a foreach processor with sub-processors that run inside the array element context.
But if we extend split_event to support array fields, each element becomes its own independent event and the normal processor chain handles it from there. No parent/child context ambiguity since each split event is a standalone event. Would that be sufficient to close this issue, or you looking for the full foreach with inline sub-processors as well?
dlvenable commentedon Apr 7, 2026
I think this is a good way to go.