[BUG] rename_keys processor: json pointers with escaped syntax fail to validate

Describe the bug
The escaped syntax for json pointers define how to build json pointers for fields that include special characters.

However, the isValidKey() method in JacksonEventKey only checks the basic character set and keys defined with the escaped syntax are rejected.

To Reproduce
Steps to reproduce the behavior:

Create a pipeline with a rename_keys processor using an escaped syntax:

my-file-pipeline:
  source:
    file:
      path: run/data/events.jsonl
      record_type: event
      format: json
  sink:
    - file:
        path: "run/data/result.jsonl"
  processor:
    - rename_keys:
        entries:
          - from_key: host
            to_key: '"cs(host)"'

Run data-prepper
data-prepper cannot start with the error:

2024-10-28T16:35:39,367 [main] ERROR org.opensearch.dataprepper.core.validation.LoggingPluginErrorsHandler - 1. rp-pipeline-file.processor.rename_keys: caused by: Parameter "entries.null.to_key" for plugin "rename_keys" is invalid: key "cs(host)" must contain only alphanumeric chars with .-_@/ and must follow JsonPointer (ie. 'field/to/key')

Expected behavior
The to_key argument "cs(host)" should be accepted as it conforms to the documented syntax.

Screenshots
N/A

Environment (please complete the following information):

OS: macOs
Version 14.5

Additional context
N/A

@joelmarty

@joelmarty , Do you want to produce key names with parenthesis in them?

Perhaps we can make use of escape sequences to allow parenthesis. Right now, our validation just looks for the characters themselves. But, we do not allow them to be escaped.

We have some related work in #5111.

@dlvenable

@dlvenable yes, I am trying to produce field names compatible with w3c's extended log file format, that uses the format prefix(header) to designate headers sent in the request or the response. For instance, cs(user-agent) is the field for the user-agent header sent in the request.

@joelmarty

@joelmarty , I don't think we are working this, but if you'd like to work this, we can help with guiding you on what to change.

@dlvenable

Hello @dlvenable
If no one is working on this yet, I can work on this.

This does work in OpenSearch, so we should support this natively in Data Prepper:

PUT test_index
POST test_index/_doc
{
  "value" : "abc",
  "cs(host)" : "127.0.0.1"
}
GET test_index/_search
{
  "query": {
    "match_all": {}
  }
}

results in:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "test_index",
        "_id": "cXFOFJgBIv8y3Y3ZCjb8",
        "_score": 1,
        "_source": {
          "value": "abc",
          "cs(host)": "127.0.0.1"
        }
      }
    ]
  }
}

@Utkarsh-Aga

@Utkarsh-Aga , if you can work this, that would be great.

We need to include parenthesis here:

data-prepper/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEventKey.java

Lines 172 to 183 in 48d5135

    
           if (!(c >= 48 && c <= 57 
        
                   || c >= 65 && c <= 90 
        
                   || c >= 97 && c <= 122 
        
                   || c == '.' 
        
                   || c == '-' 
        
                   || c == '_' 
        
                   || c == '~' 
        
                   || c == '@' 
        
                   || c == '/' 
        
                   || c == '[' 
        
                   || c == ']' 
        
           )) {

and here:

data-prepper/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEventKey.java

Lines 30 to 31 in 48d5135

    
           private static final Pattern INVALID_KEY_CHARS_PATTERN = 
        
                   Pattern.compile("[^A-Za-z0-9._~@/\\[\\]-]");

We would definitely want some unit tests as well.

Okay Sure, Would work on it.
Thank you

Resolved by #6233 .

Navigation Menu

Repository navigation

[BUG] rename_keys processor: json pointers with escaped syntax fail to validate #5121

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if (!(c >= 48 && c <= 57
	\|\| c >= 65 && c <= 90
	\|\| c >= 97 && c <= 122
	\|\| c == '.'
	\|\| c == '-'
	\|\| c == '_'
	\|\| c == '~'
	\|\| c == '@'
	\|\| c == '/'
	\|\| c == '['
	\|\| c == ']'
	)) {

	private static final Pattern INVALID_KEY_CHARS_PATTERN =
	Pattern.compile("[^A-Za-z0-9._~@/\\[\\]-]");

[BUG] rename_keys processor: json pointers with escaped syntax fail to validate #5121

Description

Activity

dlvenable commented on Oct 29, 2024

dlvenable commented on Oct 29, 2024

joelmarty commented on Oct 30, 2024

dlvenable commented on Mar 11, 2025

Utkarsh-Aga commented on Jul 9, 2025

dlvenable commented on Jul 16, 2025

dlvenable commented on Jul 16, 2025

Utkarsh-Aga commented on Jul 17, 2025

dlvenable commented on Apr 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions