Repository navigation
[otel_apm_service_map] derived.environment always resolves to generic:default #6786
Description
Description
The APM environment derivation logic in Data Prepper fails to resolve deployment.environment.name and AWS platform attributes like cloud.platform, from OTel resource attributes. All spans and service map nodes get generic:default as their environment regardless of what resource attributes are set.
This affects both:
derived.environmenton raw spans (set byotel_tracesprocessor)keyAttributes.environmenton service map nodes (set byotel_apm_service_mapprocessor)
Steps to Reproduce
- Run Data Prepper with the
otlpsource,otel_tracesprocessor, andotel_apm_service_mapprocessor - Send a trace via OTLP with
deployment.environment.nameset as a resource attribute:
curl -X POST http://localhost:4318/v1/traces \
-H "Content-Type: application/json" \
-d '{
"resourceSpans": [{
"resource": {
"attributes": [
{"key": "service.name", "value": {"stringValue": "my-service"}},
{"key": "deployment.environment.name", "value": {"stringValue": "production"}}
]
},
"scopeSpans": [{
"scope": {"name": "my-tracer"},
"spans": [{
"traceId": "aabbccdd11223344aabbccdd11223344",
"spanId": "1122334455667788",
"parentSpanId": "",
"name": "GET /api/test",
"kind": 2,
"startTimeUnixNano": "1745905500000000000",
"endTimeUnixNano": "1745905500500000000",
"attributes": [
{"key": "http.method", "value": {"stringValue": "GET"}},
{"key": "http.status_code", "value": {"intValue": "200"}}
],
"status": {"code": 1}
}]
}]
}]
}'- Query the raw span index:
curl -k -u admin:admin "https://localhost:9200/otel-v1-apm-span-*/_search" \
-H "Content-Type: application/json" \
-d '{"query": {"term": {"serviceName": "my-service"}}, "size": 1,
"_source": ["attributes.derived.environment", "resource.attributes.deployment.environment.name"]}'Expected Behavior
- Raw span:
attributes.derived.environment="production" - Service map node:
keyAttributes.environment="production"
Similarly, setting cloud.platform: aws_ec2 as a resource attribute should yield ec2:default, and cloud.platform: aws_api_gateway with aws.api_gateway.stage: prod should yield api-gateway:prod.
Actual Behavior
- Raw span:
attributes.derived.environment="generic:default" - Service map node:
keyAttributes.environment="generic:default"
The resource attributes are correctly stored in the span (resource.attributes.deployment.environment.name = "production") but are never consulted during environment derivation.
Root Cause
Three bugs in the environment derivation code path:
Bug 1 — Variable name typo in OTelApmServiceMapProcessor.extractSpanAttributes()
In OTelApmServiceMapProcessor.java, the method that builds the combined attributes map for the service map processor has a typo:
final Map<String, Object> scopeAttributes = (Map<String, Object>) scope.get("attributes");
if (attributes != null) { // BUG: should be 'scopeAttributes'
combinedAttributes.putAll(scopeAttributes);
}The condition checks attributes (the span-level attributes variable, which is almost always non-null) instead of scopeAttributes (which is null when the instrumentation scope has no attributes — the common case). This causes putAll(null) → NullPointerException, which is caught by the outer try-catch and returns Collections.emptyMap(), discarding all previously collected data including the resource.
Bug 2 — Resource not included in OTelSpanDerivationUtil.deriveAttributesForSpan()
In OTelSpanDerivationUtil.java, the method that derives attributes for raw spans only passes span.getAttributes() to computeEnvironment():
final Map<String, Object> spanAttributes = span.getAttributes();
// ...
final String environment = computeEnvironment(spanAttributes);span.getAttributes() returns only span-level attributes (http.method, http.target, etc.). The resource attributes where deployment.environment.name lives are in span.getResource(), which is never consulted. computeEnvironment() receives a map with no "resource" key and falls through to "generic:default".
Bug 3 — getAwsServiceEnvironment() doesn't check resource attributes for AWS platform detection
In ServiceEnvironmentProviders.java, the getAwsServiceEnvironment() method looks for cloud.platform and cloud.resource_id only at the top level of the combined attributes map:
String cloudPlatform = getStringAttribute(spanAttributes, "cloud.platform");With the standard OTel codec (used by the otlp source), these are resource attributes nested under spanAttributes["resource"]["attributes"], not at the top level. The method already navigates into resource.attributes for the cloud.provider + faas.name Lambda check, but doesn't do so for cloud.platform, cloud.resource_id, or aws.api_gateway.stage.
Affected environment types:
| Environment | Attribute | Location | Working? |
|---|---|---|---|
| API Gateway | cloud.platform |
resource | ❌ Not found |
| EC2 | cloud.platform |
resource | ❌ Not found |
| Lambda | cloud.resource_id |
resource | ❌ Not found |
| Lambda | aws.lambda.invoked_arn |
span | ✅ Works (span-level) |
| Lambda | cloud.provider + faas.name |
resource | ✅ Works (already navigates resource) |
| Deployment env | deployment.environment.name |
resource | ❌ Blocked by Bug 1 & 2 |
Proposed Fixes
Fix 1 — OTelApmServiceMapProcessor.java
Change the variable name in the null check from attributes to scopeAttributes:
final Map<String, Object> scopeAttributes = (Map<String, Object>) scope.get("attributes");
- if (attributes != null) {
+ if (scopeAttributes != null) {
combinedAttributes.putAll(scopeAttributes);
}Fix 2 — OTelSpanDerivationUtil.java
In deriveAttributesForSpan(), build a combined attributes map that includes the resource before passing to computeEnvironment():
public static void deriveAttributesForSpan(final Span span) {
try {
final Map<String, Object> spanAttributes = span.getAttributes();
final ErrorFaultResult errorFault = computeErrorAndFault(span.getStatus(), spanAttributes);
// ... operationName logic unchanged ...
- final String environment = computeEnvironment(spanAttributes);
+ // Build combined attributes including resource for environment derivation
+ final Map<String, Object> combinedAttributes = spanAttributes != null
+ ? new HashMap<>(spanAttributes) : new HashMap<>();
+ final Map<String, Object> resource = span.getResource();
+ if (resource != null) {
+ combinedAttributes.put("resource", resource);
+ }
+ final String environment = computeEnvironment(combinedAttributes);Fix 3 — ServiceEnvironmentProviders.java
Refactor getAwsServiceEnvironment() to extract resource attributes once and check both span-level and resource attributes for each key:
+ private static String getAttributeFromSpanOrResource(
+ final Map<String, Object> spanAttributes,
+ final Map<String, Object> resourceAttributes,
+ final String key) {
+ String value = getStringAttribute(spanAttributes, key);
+ if (value == null && resourceAttributes != null) {
+ value = getStringAttribute(resourceAttributes, key);
+ }
+ return value;
+ }
+
+ private static Map<String, Object> extractResourceAttributes(
+ final Map<String, Object> spanAttributes) {
+ try {
+ return (Map<String, Object>)
+ ((Map<String, Object>) spanAttributes.get("resource")).get("attributes");
+ } catch (Exception ignored) {
+ return null;
+ }
+ }
public static String getAwsServiceEnvironment(final Map<String, Object> spanAttributes) {
try {
- String cloudPlatform = getStringAttribute(spanAttributes, "cloud.platform");
+ final Map<String, Object> resourceAttributes = extractResourceAttributes(spanAttributes);
+
+ String cloudPlatform = getAttributeFromSpanOrResource(
+ spanAttributes, resourceAttributes, "cloud.platform");
if (cloudPlatform != null && cloudPlatform.equals("aws_api_gateway")) {
- return "api-gateway:" + getStringAttribute(spanAttributes, "aws.api_gateway.stage");
+ String stage = getAttributeFromSpanOrResource(
+ spanAttributes, resourceAttributes, "aws.api_gateway.stage");
+ return "api-gateway:" + stage;
}
if (cloudPlatform != null && cloudPlatform.equals("aws_ec2")) {
return "ec2:default";
}
- String cloudResourceId = getStringAttribute(spanAttributes, "cloud.resource_id");
- String invokedArn = getStringAttribute(spanAttributes, "aws.lambda.invoked_arn");
+ String cloudResourceId = getAttributeFromSpanOrResource(
+ spanAttributes, resourceAttributes, "cloud.resource_id");
+ String invokedArn = getAttributeFromSpanOrResource(
+ spanAttributes, resourceAttributes, "aws.lambda.invoked_arn");
if (cloudResourceId != null && cloudResourceId.startsWith("arn:aws:lambda:") || invokedArn != null) {
return "lambda:default";
}
- Map<String, Object> resourceAttributes = (Map<String, Object>)
- ((Map<String, Object>) spanAttributes.get("resource")).get("attributes");
- String cloudProvider = getStringAttribute(resourceAttributes, "cloud.provider");
- String faasName = getStringAttribute(resourceAttributes, "faas.name");
+ String cloudProvider = resourceAttributes != null
+ ? getStringAttribute(resourceAttributes, "cloud.provider") : null;
+ String faasName = resourceAttributes != null
+ ? getStringAttribute(resourceAttributes, "faas.name") : null;
if (cloudProvider != null && cloudProvider.equals("aws") && faasName != null) {
return "lambda:default";
}
} catch (Exception ignored) {
}
return null;
}Affected Files
data-prepper-plugins/otel-apm-service-map-processor/.../OTelApmServiceMapProcessor.javadata-prepper-plugins/otel-proto-common/.../OTelSpanDerivationUtil.javadata-prepper-plugins/otel-proto-common/.../ServiceEnvironmentProviders.java
Activity
[-]`derived.environment` always resolves to `generic:default` — resource attributes not consulted during environment derivation[/-][+][otel_apm_service_map] `derived.environment` always resolves to `generic:default`[/+]