Support automatic plugin loading in Data Prepper core

Is your feature request related to a problem? Please describe.

Data Prepper plugins can depend on other Data Prepper plugins. They load these plugins manually using the PluginModel.

This results in duplicated logic such as the following:

data-prepper/data-prepper-plugins/aggregate-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/aggregate/AggregateProcessor.java

Lines 83 to 87 in 91b6666

    
           private AggregateAction loadAggregateAction(final PluginFactory pluginFactory) { 
        
               final PluginModel actionConfiguration = aggregateProcessorConfig.getAggregateAction(); 
        
               final PluginSetting actionPluginSetting = new PluginSetting(actionConfiguration.getPluginName(), actionConfiguration.getPluginSettings()); 
        
               return pluginFactory.loadPlugin(AggregateAction.class, actionPluginSetting); 
        
           }

More importantly, it is also not compatible with our new support for schema and documentation generation.

Describe the solution you'd like

Provide a feature in Data Prepper core for loading plugins on behalf of other plugins. Data Prepper can have a new @UsesDataPrepperPlugin annotation which states that the type is a Data Prepper plugin.

In plugin configurations, use the desired plugin interface type. For example, replace

data-prepper/data-prepper-plugins/aggregate-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/aggregate/AggregateProcessorConfig.java

Lines 31 to 34 in 91b6666

    
           @JsonPropertyDescription("The action to be performed on each group. One of the available aggregate actions must be provided, or you can create custom aggregate actions. remove_duplicates and put_all are the available actions. For more information, see Creating New Aggregate Actions.") 
        
           @JsonProperty("action") 
        
           @NotNull 
        
           private PluginModel aggregateAction;

with:

    @JsonProperty("action")
    @NotNull
    @UsesDataPrepperPlugin
    private AggregateAction aggregateAction;

When loading plugins in Data Prepper core, detect this annotation and load the plugin.

@dlvenable

@dlvenable I have a slightly different proposal which makes implementation easier without intrusive change in data-prepper-core:

We will keep the original

@JsonProperty("action")
@NotNull
@UsesDataPrepperPlugin(pluginType = AggregateAction.class)
private PluginModel aggregateAction;

Then we will register a custom deserializer and schema generator for PluginModel.class.

The schema generator will identify the PluginModel type, retrieves its pluginType in the annotation and then pull the schemas of each option (implementation on AggregateAction) model configuration.
The custom deserializer will validate PluginModel::name against the allowed pluginName in the pluginType of the annotation (allowed plugin names can be retrieved through reflection since they are implementations of the AggregateAction interface)

@chenqi0805

@chenqi0805 , The problem with that alternative is that the top-level plugin still doesn't have the interface it needs. Ultimately it needs to call getAggregateAction() and get a AggregateAction interface - not a PluginModel.

Consider the aggregate processor. It has this line:

data-prepper/data-prepper-plugins/aggregate-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/aggregate/AggregateProcessor.java

Line 75 in 2f21000

this.aggregateAction = loadAggregateAction(pluginFactory);

It really should be:

this.aggregateAction = config.getAggregateAction();

To support this, the configs need to have:

private AggregateAction aggregateAction;

public AggregateAction getAggregateAction() {
  return aggregateAction;
}

Support automatic plugin loading in Data Prepper core.

Support automatic plugin loading in Data Prepper core. (opensearch-pr…

Navigation Menu

Repository navigation

Support automatic plugin loading in Data Prepper core #4838

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	private AggregateAction loadAggregateAction(final PluginFactory pluginFactory) {
	final PluginModel actionConfiguration = aggregateProcessorConfig.getAggregateAction();
	final PluginSetting actionPluginSetting = new PluginSetting(actionConfiguration.getPluginName(), actionConfiguration.getPluginSettings());
	return pluginFactory.loadPlugin(AggregateAction.class, actionPluginSetting);
	}

	@JsonPropertyDescription("The action to be performed on each group. One of the available aggregate actions must be provided, or you can create custom aggregate actions. remove_duplicates and put_all are the available actions. For more information, see Creating New Aggregate Actions.")
	@JsonProperty("action")
	@NotNull
	private PluginModel aggregateAction;

Support automatic plugin loading in Data Prepper core #4838

Description

Activity

chenqi0805 commented on Sep 17, 2024

dlvenable commented on May 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions