Skip to content

Reduce the Data Prepper tar.gz and Docker image sizes #3356

New issue
New issue
Closed
#6520
Closed
Reduce the Data Prepper tar.gz and Docker image sizes#3356
#6520
Assignees
dlvenable
Labels
maintenanceIssues to help maintain the project, such as improving builds, testing, etc.
Milestone
v2.14

Description

@dlvenable
dlvenable
opened on Sep 18, 2023

Is your feature request related to a problem? Please describe.

It appears that the Data Prepper image has grown quite large.

ls -l release/archives/linux/build/distributions/ |  awk '{print $5,$9}'

664652152 opensearch-data-prepper-2.5.0-SNAPSHOT-linux-x64.tar.gz
857129267 opensearch-data-prepper-jdk-2.5.0-SNAPSHOT-linux-x64.tar.gz

Even without the JDK, it comes to 633 MB.

Describe the solution you'd like

Find why it is so large and see how we can reduce the size.

Tasks

Activity

  1. dlvenable
    added
    untriaged
    on Sep 18, 2023
  2. github-project-automation
    added this to Data Prepper Tracking Boardon Sep 18, 2023
  3. github-project-automation
    moved this to Unplanned in Data Prepper Tracking Boardon Sep 18, 2023
  4. dlvenable
    added
    maintenanceIssues to help maintain the project, such as improving builds, testing, etc.
    and removed
    untriaged
    on Sep 18, 2023
  5. github-actions
    added
    untriaged
    on Sep 18, 2023

  6. KarstenSchnitter commented on Sep 19, 2023

    @KarstenSchnitter
    KarstenSchnitter
    on Sep 19, 2023
    Collaborator

    I took a look at the 2.4.0 tag and the current main branch builds. I did a ./gradlew assemble and a ls -lSh release/archives/linux/build/install/opensearch-data-prepper-<version>/lib afterwards, since the lib folder contains most of the used disk space. This gives a nice list of all the dependencies sorted in descending order by size. This makes finding new a large dependencies easier. Overall 2.4.0 has 514 jars, while 2.5.0 currently has 867. The top 20 lines are:

    2.4.0

    total 516680
-rwxr-xr-x  1 user  staff    13M Sep 19 13:33 icu4j-69.1.jar
-rwxr-xr-x  1 user  staff    12M Sep 19 13:33 opensearch-1.3.8.jar
-rwxr-xr-x  1 user  staff    10M Sep 19 13:33 armeria-1.22.1.jar
-rwxr-xr-x  1 user  staff    10M Sep 19 13:33 eclipse-collections-12.0.0.M3.jar
-rwxr-xr-x  1 user  staff   6.0M Sep 19 13:33 glue-2.20.67.jar
-rwxr-xr-x  1 user  staff   5.7M Sep 19 13:34 scala-library-2.13.10.jar
-rwxr-xr-x  1 user  staff   5.6M Sep 19 13:33 zstd-jni-1.5.2-1.jar
-rwxr-xr-x  1 user  staff   5.6M Sep 19 13:33 bcprov-jdk15on-1.70.jar
-rwxr-xr-x  1 user  staff   5.3M Sep 19 13:33 hadoop-hdfs-client-3.3.5.jar
-rwxr-xr-x  1 user  staff   5.2M Sep 19 13:33 kafka_2.13-7.3.3-ccs.jar
-rwxr-xr-x  1 user  staff   4.8M Sep 19 13:33 kafka-clients-7.4.0-ccs.jar
-rwxr-xr-x  1 user  staff   4.8M Sep 19 13:33 opensearch-java-2.5.0.jar
-rwxr-xr-x  1 user  staff   4.4M Sep 19 13:33 hadoop-common-3.3.6.jar
-rwxr-xr-x  1 user  staff   3.7M Sep 19 13:33 aws-java-sdk-glue-1.12.506.jar
-rwxr-xr-x  1 user  staff   3.6M Sep 19 13:33 byte-buddy-1.12.6.jar
-rwxr-xr-x  1 user  staff   3.6M Sep 19 13:34 scala-reflect-2.13.10.jar
-rwxr-xr-x  1 user  staff   3.5M Sep 19 13:33 hadoop-yarn-api-3.3.5.jar
-rwxr-xr-x  1 user  staff   3.4M Sep 19 13:34 lucene-core-8.10.1.jar
-rwxr-xr-x  1 user  staff   3.2M Sep 19 13:34 hadoop-shaded-guava-1.1.1.jar

    2.5.0

    otal 1432008
-rwxr-xr-x  1 user  staff    17M Sep 19 13:16 ec2-2.20.67.jar
-rwxr-xr-x  1 user  staff    13M Sep 19 13:16 icu4j-69.1.jar
-rwxr-xr-x  1 user  staff    12M Sep 19 13:16 opensearch-1.3.8.jar
-rwxr-xr-x  1 user  staff    11M Sep 19 13:16 aws-crt-0.21.17.jar
-rwxr-xr-x  1 user  staff    10M Sep 19 13:16 armeria-1.22.1.jar
-rwxr-xr-x  1 user  staff    10M Sep 19 13:16 eclipse-collections-12.0.0.M3.jar
-rwxr-xr-x  1 user  staff   9.7M Sep 19 13:16 sagemaker-2.20.67.jar
-rwxr-xr-x  1 user  staff   8.9M Sep 19 13:16 elasticsearch-java-7.17.0.jar
-rwxr-xr-x  1 user  staff   8.0M Sep 19 13:16 quicksight-2.20.67.jar
-rwxr-xr-x  1 user  staff   8.0M Sep 19 13:16 bcprov-jdk18on-1.76.jar
-rwxr-xr-x  1 user  staff   6.0M Sep 19 13:16 glue-2.20.67.jar
-rwxr-xr-x  1 user  staff   5.8M Sep 19 13:16 iot-2.20.67.jar
-rwxr-xr-x  1 user  staff   5.7M Sep 19 13:16 securityhub-2.20.67.jar
-rwxr-xr-x  1 user  staff   5.7M Sep 19 13:16 scala-library-2.13.10.jar
-rwxr-xr-x  1 user  staff   5.6M Sep 19 13:16 zstd-jni-1.5.2-1.jar
-rwxr-xr-x  1 user  staff   5.2M Sep 19 13:16 kafka_2.13-7.3.3-ccs.jar
-rwxr-xr-x  1 user  staff   5.0M Sep 19 13:16 ssm-2.20.67.jar
-rwxr-xr-x  1 user  staff   4.8M Sep 19 13:16 kafka-clients-7.4.0-ccs.jar
-rwxr-xr-x  1 user  staff   4.8M Sep 19 13:16 opensearch-java-2.5.0.jar

    The largest additional jars in 2.5.0 are:

    -rwxr-xr-x  1 user  staff    17M Sep 19 13:16 ec2-2.20.67.jar
-rwxr-xr-x  1 user  staff    11M Sep 19 13:16 aws-crt-0.21.17.jar
-rwxr-xr-x  1 user  staff   9.7M Sep 19 13:16 sagemaker-2.20.67.jar
-rwxr-xr-x  1 user  staff   8.9M Sep 19 13:16 elasticsearch-java-7.17.0.jar
-rwxr-xr-x  1 user  staff   8.0M Sep 19 13:16 quicksight-2.20.67.jar
-rwxr-xr-x  1 user  staff   8.0M Sep 19 13:16 bcprov-jdk18on-1.76.jar
-rwxr-xr-x  1 user  staff   5.8M Sep 19 13:16 iot-2.20.67.jar
-rwxr-xr-x  1 user  staff   5.7M Sep 19 13:16 securityhub-2.20.67.jar
-rwxr-xr-x  1 user  staff   5.0M Sep 19 13:16 ssm-2.20.67.jar
-rwxr-xr-x  1 user  staff   4.8M Sep 19 13:16 connect-2.20.67.jar
-rwxr-xr-x  1 user  staff   4.4M Sep 19 13:16 rds-2.20.67.jar
-rwxr-xr-x  1 user  staff   4.0M Sep 19 13:16 chime-2.20.67.jar
-rwxr-xr-x  1 user  staff   3.8M Sep 19 13:16 lightsail-2.20.67.jar
-rwxr-xr-x  1 user  staff   3.7M Sep 19 13:16 cloudfront-2.20.67.jar
-rwxr-xr-x  1 user  staff   3.7M Sep 19 13:16 redshift-2.20.67.jar
-rwxr-xr-x  1 user  staff   3.6M Sep 19 13:16 pinpoint-2.20.67.jar

    You can see, that a lot of AWS libraries with version 2.20.67 are added with 2.5.0. The above sum up to 90M of uncompressed space with even more such libraries with smaller sizes. Maybe this is an starting point to reduce the size of the tarball and Docker image.

  7. KarstenSchnitter commented on Sep 19, 2023

    @KarstenSchnitter
    KarstenSchnitter
    on Sep 19, 2023
    Collaborator

    I can calculate the overall size of the 2.20.67 libraries with data-prepper % du -ch release/archives/linux/build/install/opensearch-data-prepper-2.5.0-SNAPSHOT-linux-x64/lib/*2.20.67* and get 442M in total. Still this does not account for the entire difference between 2.4.0 and 2.5.0, which was 893M.

  8. KarstenSchnitter commented on Sep 19, 2023

    @KarstenSchnitter
    KarstenSchnitter
    on Sep 19, 2023
    Collaborator

    A probable cause might be the inclusion of aws-sdk-java with the GeoIp Plugin (#2811) in 9f2150e or the update of the awssdk:bom with b7661e6. There are more aws dependencies, that might have pulled far too much libraries, e.g. awssdk:s3. I cannot judge if everything is really needed.

  9. dlvenable commented on Sep 19, 2023

    @dlvenable
    dlvenable
    on Sep 19, 2023
    MemberAuthor

    @KarstenSchnitter , Thanks for the data points here! I just took a look and GeoIp appears to only be using the s3 SDK, so I think we can reduce that to just S3. That should eliminate EC2, SageMaker, QuickSight which are in the top-list, as well as remove many other dependencies.

    Separately, we should also see if we can remove the AWS CRT library. We are using Netty elsewhere and it would probably be ideal to stick with Netty. A recent PR (#3338) consolidated the OpenSearch sink to use Netty instead of the AWS CRT. The GeoIP processor is using the S3 Transfer Manager. That may require AWS CRT.

  10. dlvenable
    added this to the v2.5 milestone on Sep 19, 2023
  11. dlvenable
    moved this from Unplanned to To do in Data Prepper Tracking Boardon Sep 19, 2023
  12. dlvenable
    self-assigned this
    on Sep 19, 2023
  13. dlvenable
    removed
    untriaged
    on Sep 19, 2023
  14. dlvenable
    modified the milestones: v2.5, v2.6 on Oct 9, 2023

  15. dlvenable commented on Oct 9, 2023

    @dlvenable
    dlvenable
    on Oct 9, 2023
    MemberAuthor

    I'm moving this to 2.6 because the large files are not included in 2.5.

  16. dlvenable
    modified the milestones: v2.6, v2.7 on Nov 21, 2023

  17. dlvenable commented on Nov 21, 2023

    @dlvenable
    dlvenable
    on Nov 21, 2023
    MemberAuthor

    Those projects are not included in 2.6, so I'm pushing this out.

  18. dlvenable
    assigned
    asifsmohammed
    and unassigned
    dlvenable
    on Jan 16, 2024
  19. dlvenable
    unassigned
    asifsmohammed
    on Jan 30, 2024
  20. dlvenable
    removed this from the v2.7 milestone on Jan 30, 2024
  21. dlvenable
    self-assigned this
    on Feb 16, 2026
  22. dlvenable
    added this to the v2.14 milestone on Feb 16, 2026
  23. dlvenable
    moved this from To do to In progress in Data Prepper Tracking Boardon Feb 17, 2026
  24. dlvenable
    closed this as completedin #6520on Feb 18, 2026
  25. github-project-automation
    moved this from In progress to Done in Data Prepper Tracking Boardon Feb 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

maintenanceIssues to help maintain the project, such as improving builds, testing, etc.

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions