Repository navigation
[FEATURE] Add filter_entries processor for array element filtering #6610
Description
Is your feature request related to a problem? Please describe.
There is no way to remove elements from an array based on a condition. While existing processors can modify keys within array elements or drop entire events, none can filter the array itself — keeping only elements that match a condition and discarding
the rest.
For example, given an array of objects at /items, there is no way to produce a new array containing only the elements where /type == "a".
Describe the solution you'd like
A new filter_entries processor that evaluates a condition against each element of an array and keeps only those elements where the condition is true.
| Option | Required | Description |
|---|---|---|
| source | Yes | Key of the array field to filter |
| target | No | Key to write the filtered array to. Defaults to source (in-place) |
| condition | Yes | Expression evaluated per element. Elements where this is true are kept |
| filter_entries_when | No | Expression evaluated against the root event. If false, the processor is skipped entirely |
Follows the mutate-event naming convention: add_entries, delete_entries, select_entries
Example
Input:
{
"items": [
{"type": "a", "id": "1"},
{"type": "b", "id": "2"},
{"type": "a", "id": "3"}
]
}Pipeline:
- filter_entries:
source: "/items"
target: "/filtered"
condition: '/type == "a"'Expected output:
{
"filtered": [
{"type": "a", "id": "1"},
{"type": "a", "id": "3"}
]
}Describe alternatives you've considered (Optional)
-
Extending select_entries with iterate_on: select_entries filters keys (columns) within an event, while the needed operation is filtering elements (rows) within an array. These are semantically distinct — adding both behaviors to one processor would
make configuration confusing. Each mutate-event processor today does one thing: add_entries adds keys, delete_entries removes keys, select_entries keeps keys. filter_entries keeping elements is a natural extension of this pattern. -
Using delete_entries with iterate_on and delete_from_element_when: This can conditionally delete keys from array elements but cannot remove the element itself from the array. An element with all its keys deleted still remains in the array as an
empty object {}. -
Using drop_events with a condition: drop_events operates at the event level. It can drop an entire event but has no way to target individual elements within an array field inside the event.
Additional context
N/A
Activity
dlvenable commentedon Mar 6, 2026
I think this should be named
filter_listsince it operates on lists.The value
conditionshould probably be namedkeep_when.I do have some concerns about using the JSON pointer inside the list elements. We do this in the
translateprocessor and it has caused some issues both with the user experience and performance.dlvenable commentedon Mar 6, 2026
What we don't currently support is a relative JSON pointer. I think that would be the ideal way to handle accessing data inside lists. @bagmarnikhil you opened this issue #6609 about how you can't access the root when you expect to. I think the behavior you pointed out is currently incorrect (as you said) and we should be able to read either from the root or relative.
yavmanis commentedon Mar 23, 2026
I'd like to extend the scope slightly to also cover filtering arrays of primitives (strings, numbers, booleans), not just arrays of objects.
For primitive arrays, each element would be accessible via a /value key in the condition expression. For example:
Input:
Pipeline:
Expected output:
This keeps the processor general-purpose — it works for both [{"type": "a"}, {"type": "b"}] and ["foo", "bar", ""] style arrays with the same configuration pattern.
Note: I have started working on this issue and will soon raise PR for this
dlvenable commentedon Apr 20, 2026
Completed by #6659.