AWS Glue is Amazon’s serverless data integration service, and by August 2026 its engine has moved twice in twelve months: AWS Glue 5.0 shipped on Apache Spark 3.5.4 in late 2024, and AWS Glue 5.1 followed on November 26, 2025 with Apache Spark 3.5.6, Python 3.11, and Scala 2.12.18. If you’re still running Glue 4.0 jobs or have never touched Glue at all, this tutorial gets you from zero to a working, scheduled ETL pipeline in 12 steps and about 90 minutes, using the current 5.1 runtime the whole way through.
By the end you’ll have a complete working project: an S3 data lake crawled into the Glue Data Catalog, a Spark ETL job that transforms raw CSV into partitioned Parquet, a Data Catalog you can query straight from Athena, and a schedule that runs it all automatically. We’ll also cover exactly what it costs in DPU-hours, the pitfalls that burn a surprising amount of first-week budget, and troubleshooting for the errors you’re most likely to hit along the way.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What Is AWS Glue and Why It Matters in 2026
AWS Glue is a serverless data integration service built around three pieces: the Glue Data Catalog (a shared metadata store), Glue Crawlers (automated schema discovery), and Glue Jobs (the actual ETL compute, mostly running Apache Spark). Instead of standing up and patching a Spark cluster yourself, you point Glue at your data, let it infer the schema, write a transformation script, and Glue runs it on managed infrastructure billed by the second. The Data Catalog it builds isn’t Glue-only either — Athena, Redshift Spectrum, EMR, and Lake Formation all read from the same catalog, so cataloging your data once with Glue pays off across the rest of the AWS analytics stack.
What changed with the move to Glue 5.0 and then 5.1 is meaningful for anyone still on 4.0. According to AWS’s own Glue 5.0 announcement, the 5.0 runtime jumped from Spark 3.3.0 to Spark 3.5.4, alongside Python 3.11, Scala 2.12.18, and Java 17, plus updated open table format support: Hudi 0.15.0, Iceberg 1.7.1, and Delta Lake 3.3.0. Glue 5.0 also added support for Amazon SageMaker Unified Studio and SageMaker Lakehouse. Glue 5.1 followed with a further Spark bump to 3.5.6, described in AWS’s Glue 5.1 announcement as bringing performance and security enhancements on top of the same Python and Scala versions. Separately, a June 2025 AWS update added support for Glue 5.0 Spark jobs to read and write directly against AWS Lake Formation-registered tables, provided the job’s IAM role has full table access — a meaningful change for teams running lakehouse architectures with fine-grained access control.
None of that changes how you build a Glue pipeline day to day, which is what the rest of this tutorial walks through. But if you’re deciding which Glue version to target for a new project, there’s no reason to start on anything older than 5.1. The full version matrix, including the exact Hadoop, Iceberg, Hudi, and Delta Lake versions bundled with each Glue release, is maintained in AWS’s Glue versions release notes, worth bookmarking before you pin a version in a production job.
Where Glue Fits in a Real Data Platform
In practice, teams reach for Glue for a fairly consistent set of jobs: turning a messy S3 data lake of raw CSVs or JSON logs into a clean, partitioned, queryable table; syncing data out of an operational database like Amazon RDS into a data warehouse without hand-rolling extraction code; running scheduled batch transformations that feed a BI dashboard or a machine learning feature store; and increasingly, preparing data for the lakehouse pattern with Iceberg or Delta Lake tables shared across Glue, Athena, and Redshift Spectrum. If your workload looks like any of those and doesn’t need a persistent, always-on cluster, Glue’s serverless, pay-per-DPU-hour model is usually cheaper and less operational overhead than standing up and babysitting a dedicated Spark or EMR cluster sized for peak load it hits only occasionally.
Prerequisites: Tools, Accounts, and Versions You Need
Glue is entirely serverless, so there’s nothing to install to run it — everything runs on AWS-managed infrastructure. You do need a small set of local tools to script and deploy it, plus an S3 bucket to hold the sample data.
| Tool or resource | Version / tier | Install / setup | Why you need it |
|---|---|---|---|
| AWS account | Free tier or pay-as-you-go | Sign up at aws.amazon.com | Hosts every Glue resource in this tutorial |
| AWS CLI | Latest version (v2) | curl “https://awscli.amazonaws.com/AWSCLIV2.pkg” -o AWSCLIV2.pkg | Creates databases, crawlers, and jobs from the terminal |
| Python | 3.11 (matches Glue 5.1’s runtime) | Download from python.org or use pyenv | Writing and locally linting Glue ETL scripts |
| boto3 | Latest version | pip install boto3 | Scripting Glue API calls directly from Python |
| An S3 bucket | N/A | aws s3 mb s3://your-bucket-name | Stores raw data, scripts, and job output |
| An IAM role for Glue | N/A | Attach the AWSGlueServiceRole managed policy | Lets Glue jobs and crawlers read/write your resources |
This tutorial writes Glue ETL scripts in Python, which is the most common path into Glue, but the same Spark jobs run equally well written in Scala if that’s what your team standardizes on. Everything else — the Data Catalog, crawlers, scheduling, IAM — is identical regardless of the scripting language you pick. If your workload is lightweight orchestration or scripting rather than a Spark transformation, Glue’s Python Shell job type is also worth knowing about: it runs plain Python without spinning up a Spark cluster at all, which is cheaper and faster to start for tasks that don’t need distributed processing.
Step 1: Create an S3 Bucket and Upload Sample Data
Glue needs raw data to work with. Create a bucket with a folder structure that separates raw input from Glue’s own script storage and the transformed output the ETL job will produce.
aws s3 mb s3://glue-tutorial-2026
# Folder layout Glue will read from and write to
aws s3api put-object --bucket glue-tutorial-2026 --key raw/
aws s3api put-object --bucket glue-tutorial-2026 --key scripts/
aws s3api put-object --bucket glue-tutorial-2026 --key output/
# Upload a sample CSV to the raw/ prefix
aws s3 cp orders.csv s3://glue-tutorial-2026/raw/orders.csvAny CSV works for this tutorial — a handful of columns like order_id, customer_id, order_date, and amount is enough to demonstrate a real transform in Step 6. Keep raw data and Glue scripts in separate prefixes; it makes IAM scoping and cleanup much simpler once you have more than one pipeline in the same bucket.
Step 2: Create an IAM Role for Glue
Every Glue crawler and job runs with an IAM role that determines what it can read and write. Attach the AWS-managed AWSGlueServiceRole policy as a baseline, then add a scoped inline policy granting access to your specific bucket.
aws iam create-role \
--role-name AWSGlueServiceRole-tutorial \
--assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "glue.amazonaws.com"},
"Action": "sts:AssumeRole"
}]
}'
aws iam attach-role-policy \
--role-name AWSGlueServiceRole-tutorial \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
aws iam put-role-policy \
--role-name AWSGlueServiceRole-tutorial \
--policy-name GlueTutorialS3Access \
--policy-document '{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
"Resource": [
"arn:aws:s3:::glue-tutorial-2026",
"arn:aws:s3:::glue-tutorial-2026/*"
]
}]
}'Scoping the S3 policy to just your tutorial bucket, rather than granting broad S3 access, is the difference between a Glue job that can only touch what it needs and one that could silently read or overwrite unrelated data in your account.
Step 3: Create a Glue Database in the Data Catalog
The Data Catalog organizes tables into databases, the same way a traditional relational database would, except a Glue database is just a metadata namespace — it holds no data itself, only pointers to where the data lives in S3 and what its schema looks like.
aws glue create-database \
--database-input '{
"Name": "glue_tutorial_db",
"Description": "Tutorial database for the Glue walkthrough",
"LocationUri": "s3://glue-tutorial-2026/"
}'Once this exists, both a crawler and any Glue job can reference glue_tutorial_db as the destination for whatever tables they define or discover, and downstream tools like Athena will see the same database immediately since they all read from the same catalog.
Step 4: Create and Run a Glue Crawler
A crawler inspects the data at a given path, infers its schema, and registers it as a table in the Data Catalog — no manual schema definition required. Point one at the raw/ prefix from Step 1.
aws glue create-crawler \
--name glue-tutorial-crawler \
--role arn:aws:iam::123456789012:role/AWSGlueServiceRole-tutorial \
--database-name glue_tutorial_db \
--targets '{
"S3Targets": [
{"Path": "s3://glue-tutorial-2026/raw/"}
]
}' \
--table-prefix "raw_"
# Trigger it to run once immediately
aws glue start-crawler --name glue-tutorial-crawlerA crawler run typically takes a couple of minutes for a small sample dataset. Check its status and confirm the table landed in the catalog:
aws glue get-crawler --name glue-tutorial-crawler --query 'Crawler.State'
# "READY" (once it finishes)
aws glue get-tables --database-name glue_tutorial_db --query 'TableList[].Name'
# ["raw_orders"]Substitute the IAM role ARN with the one you created in Step 2, using your own account ID. Replace the placeholder account ID shown above with your actual AWS account ID before running these commands.
Step 5: Explore the Table Schema With Glue Studio or the CLI
Before writing any transformation logic, confirm the crawler inferred the schema correctly. The Glue Studio visual editor in the AWS console shows this graphically, or you can pull it directly with the CLI.
aws glue get-table --database-name glue_tutorial_db --name raw_orders \
--query 'Table.StorageDescriptor.Columns'[
{"Name": "order_id", "Type": "string"},
{"Name": "customer_id", "Type": "string"},
{"Name": "order_date", "Type": "string"},
{"Name": "amount", "Type": "double"}
]If a column comes back typed as string when you expected a date or number, that’s usually a sign the source CSV has inconsistent formatting in that column — worth fixing at the source before it becomes a transformation-time headache in Step 6.
Step 6: Write a Glue ETL Job in Python
With a cataloged source table, write the actual transformation: read the raw orders, cast order_date to a real date type, filter out any rows with a null customer_id, and write the result back to S3 as partitioned Parquet — a far more efficient format for downstream querying than the original CSV.
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import to_date, col
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
# Read the crawled table from the Data Catalog
raw_orders = glueContext.create_dynamic_frame.from_catalog(
database="glue_tutorial_db",
table_name="raw_orders"
).toDF()
# Transform: cast order_date, drop rows with missing customer_id
cleaned = (
raw_orders
.withColumn("order_date", to_date(col("order_date"), "yyyy-MM-dd"))
.filter(col("customer_id").isNotNull())
)
# Write partitioned Parquet output back to S3
cleaned.write.mode("overwrite").partitionBy("order_date") \
.parquet("s3://glue-tutorial-2026/output/orders_cleaned/")
job.commit()Upload the script to the scripts/ prefix, then register the job pointing to it, targeting the Glue 5.1 runtime.
aws s3 cp orders_etl.py s3://glue-tutorial-2026/scripts/orders_etl.py
aws glue create-job \
--name orders-etl-job \
--role arn:aws:iam::123456789012:role/AWSGlueServiceRole-tutorial \
--command '{
"Name": "glueetl",
"ScriptLocation": "s3://glue-tutorial-2026/scripts/orders_etl.py",
"PythonVersion": "3"
}' \
--glue-version "5.1" \
--number-of-workers 2 \
--worker-type G.1X \
--default-arguments '{
"--job-language": "python",
"--enable-metrics": "true",
"--enable-continuous-cloudwatch-log": "true",
"--enable-glue-datacatalog": "true"
}'Two G.1X workers is a reasonable starting size for a small dataset. Each Data Processing Unit (DPU) provides 4 vCPUs and 16 GB of memory, so start modest and scale up only if the job’s CloudWatch metrics show it’s memory- or CPU-bound, which we’ll cover in the pricing and troubleshooting sections. Glue also offers a G.2X worker type with double the vCPUs and memory per DPU for memory-intensive jobs, and G.025X for lightweight jobs that don’t need a full DPU’s worth of compute — picking the right worker type matters as much as picking the right worker count once you move past a small sample dataset.
Step 7: Run the Job and Check the Output
Start the job and poll its status until it succeeds.
aws glue start-job-run --job-name orders-etl-job
# Returns a JobRunId, e.g. "jr_8f2e1c9a4b7d6e3f2a1b0c9d8e7f6a5b"
aws glue get-job-run --job-name orders-etl-job --run-id jr_8f2e1c9a4b7d6e3f2a1b0c9d8e7f6a5b \
--query 'JobRun.JobRunState'
# "RUNNING" then "SUCCEEDED"Once it succeeds, confirm the partitioned Parquet output landed in S3:
aws s3 ls s3://glue-tutorial-2026/output/orders_cleaned/ --recursive
# output/orders_cleaned/order_date=2026-08-15/part-00000-...snappy.parquet
# output/orders_cleaned/order_date=2026-08-16/part-00000-...snappy.parquetThe partition-per-date folder structure is exactly what makes this data cheap and fast to query later — Athena or Redshift Spectrum can skip entire partitions that don’t match a query’s date filter instead of scanning the whole dataset.
Step 8: Catalog the Transformed Output With a Second Crawler
The ETL job wrote data to S3, but nothing has cataloged it yet. Run a second crawler against the output/ prefix so the cleaned, partitioned table becomes queryable.
aws glue create-crawler \
--name glue-tutorial-output-crawler \
--role arn:aws:iam::123456789012:role/AWSGlueServiceRole-tutorial \
--database-name glue_tutorial_db \
--targets '{
"S3Targets": [
{"Path": "s3://glue-tutorial-2026/output/orders_cleaned/"}
]
}' \
--table-prefix "clean_"
aws glue start-crawler --name glue-tutorial-output-crawlerThis crawler also picks up the order_date partitions automatically and registers them as a partitioned column in the catalog, which is what lets partition pruning work when you query the table.
Step 9: Query the Result With Amazon Athena
Because Athena reads from the same Glue Data Catalog, the cleaned table is queryable with standard SQL the moment the output crawler finishes — no loading step required.
SELECT order_date, COUNT(*) AS order_count, SUM(amount) AS total_amount
FROM glue_tutorial_db.clean_orders_cleaned
WHERE order_date >= DATE '2026-08-01'
GROUP BY order_date
ORDER BY order_date;This is the payoff of the whole pipeline: raw CSV in, queryable partitioned Parquet out, with the schema and partitioning handled automatically by the two crawlers instead of hand-written DDL.
Step 10: Schedule the Pipeline to Run Automatically
A pipeline you have to trigger by hand isn’t a pipeline. Attach a schedule directly to the crawler and job so new data lands and gets processed without manual intervention.
# Schedule the raw crawler to run nightly at 2 AM UTC
aws glue update-crawler \
--name glue-tutorial-crawler \
--schedule "cron(0 2 * * ? *)"
# Create a Glue trigger to run the ETL job right after the crawler finishes
aws glue create-trigger \
--name orders-etl-after-crawl \
--type CONDITIONAL \
--predicate '{
"Conditions": [{
"CrawlerName": "glue-tutorial-crawler",
"CrawlState": "SUCCEEDED",
"LogicalOperator": "EQUALS"
}]
}' \
--actions '[{"JobName": "orders-etl-job"}]' \
--start-on-creationChaining the ETL job to fire off a successful crawler run, rather than scheduling both independently, avoids the race condition where the job runs against yesterday’s crawl because the crawler hadn’t finished yet.
Step 11: Monitor With CloudWatch Metrics and Logs
Every Glue job with --enable-continuous-cloudwatch-log and --enable-metrics set, as configured in Step 6, streams both driver and executor logs into CloudWatch and publishes job-level metrics like memory usage, CPU load, and active executors.
aws logs tail /aws-glue/jobs/output --follow
aws cloudwatch get-metric-statistics \
--namespace Glue \
--metric-name glue.driver.aggregate.numFailedTasks \
--dimensions Name=JobName,Value=orders-etl-job Name=JobRunId,Value=jr_8f2e1c9a4b7d6e3f2a1b0c9d8e7f6a5b \
--start-time 2026-08-21T00:00:00Z --end-time 2026-08-21T23:59:59Z \
--period 300 --statistics SumSet a CloudWatch alarm on job failures so a broken pipeline surfaces immediately rather than being discovered when a downstream Athena query returns stale data days later.
Step 12: Lock Down IAM and Cost Controls Before Production
Before this pipeline handles anything beyond sample data, run through a short hardening checklist:
- Narrow the Glue job’s IAM role to only the specific S3 prefixes it reads from and writes to, instead of the whole bucket.
- Set job bookmarks (
--job-bookmark-option job-bookmark-enable) so reruns only process new data instead of reprocessing everything from scratch. - Tag every Glue job, crawler, and database with a cost-allocation tag so DPU-hour spend is traceable in Cost Explorer.
- Set a maximum job timeout and a reasonable
--number-of-workersceiling so a runaway or stuck job doesn’t silently burn DPU-hours for hours. - If the job touches AWS Lake Formation-registered tables, confirm the job role has full table access, since Glue 5.0+ Spark jobs require that for read/write access to Lake Formation tables.
- Turn on AWS CloudTrail logging for the Glue API if you need an audit trail of who created or modified jobs and crawlers.
Job bookmarks in particular save real money on recurring pipelines — without them, every scheduled run reprocesses the entire dataset instead of just what’s new since the last run.
Complete Working Project: A Scheduled Data Lake Pipeline
Put together, the 12 steps above produce a complete, self-running data lake pipeline. Here’s the full shape of it:
glue-tutorial-2026/ (S3 bucket)
├── raw/ ← new CSVs land here
├── scripts/
│ └── orders_etl.py ← the Spark ETL script from Step 6
└── output/
└── orders_cleaned/
├── order_date=2026-08-15/
└── order_date=2026-08-16/
Glue Data Catalog: glue_tutorial_db
├── raw_orders ← from glue-tutorial-crawler
└── clean_orders_cleaned ← from glue-tutorial-output-crawler
Automation:
glue-tutorial-crawler (nightly, cron)
→ orders-etl-after-crawl (conditional trigger)
→ orders-etl-job (Glue 5.1, 2x G.1X)
→ glue-tutorial-output-crawler (manual or scheduled)
→ queryable in AthenaDrop a new CSV into the raw/ prefix, and the whole chain — crawl, transform, catalog, query — runs itself on the next scheduled cycle, with CloudWatch watching for failures the entire way. Extend it by swapping the CSV source for a JDBC connection to a production database, or adding a Glue Data Quality ruleset to reject rows that fail validation before they ever reach the output table.
The pattern scales well beyond a single orders table too. Add a second raw source, a second crawler, and a second transformation script, and you have a multi-table pipeline feeding the same catalog — the kind of setup that typically grows into a proper data lake as more teams start landing data in the raw/ prefix. Because the Data Catalog is shared infrastructure rather than something tied to a single job, adding sources doesn’t require redesigning anything you’ve already built; each new pipeline just adds its own tables alongside the existing ones.
AWS Glue Pricing in 2026: What It Actually Costs
Glue bills almost everything by the DPU-hour. According to the official AWS Glue pricing page, the rate is $0.44 per DPU-hour, billed per second with a one-minute minimum duration per run, and each DPU provides 4 vCPUs and 16 GB of memory. Crawlers, ETL jobs, and interactive sessions all bill at that same $0.44 per DPU-hour rate.
| Component | Rate | Free tier | Notes |
|---|---|---|---|
| ETL jobs (Spark) | $0.44 per DPU-hour | None | Billed per second, 1-minute minimum per run |
| Crawlers | $0.44 per DPU-hour | None | Same rate as ETL; runs bill for actual crawl duration |
| Interactive sessions / Glue Studio | $0.44 per DPU-hour | None | No separate Glue Studio license fee |
| Data Catalog storage | $1.00 per 100,000 objects/month beyond free tier | First 1,000,000 objects free/month | Objects = tables, partitions, and database entries |
| Data Catalog requests | $1.00 per 1,000,000 requests/month beyond free tier | First 1,000,000 requests free/month | Covers API calls like GetTable and GetPartitions |
For the tutorial project in this article, running two G.1X workers (2 DPUs) for a few minutes per job run costs a small fraction of a dollar, and the Data Catalog usage stays well inside the free 1-million-object, 1-million-request tier for anything short of a genuinely large production catalog. The real cost driver in production is DPU-hours on large, frequent Spark jobs — that’s where right-sizing worker count and worker type (Step 6) and enabling job bookmarks (Step 12) actually move the bill.
AWS Glue vs Databricks vs Apache Airflow vs Fivetran
Glue rarely competes head-to-head with these tools; more often it’s paired with one of them. Glue is positioned as serverless, AWS-native ETL with a catalog that’s shared across Athena, Redshift Spectrum, EMR, and Lake Formation, which makes it the default choice for teams already committed to AWS looking for managed Spark ETL without operating a cluster. Databricks is typically preferred for large-scale collaborative lakehouse analytics and ML workflows, especially across multiple clouds, and interoperates with Glue-managed data through shared Delta Lake or Iceberg tables rather than replacing it outright. Apache Airflow is primarily an orchestrator, not an ETL engine — many teams use Airflow to schedule and chain Glue jobs (or use AWS Step Functions for the same purpose within AWS), while Glue does the actual Spark transformation work; see our Step Functions vs Airflow comparison for how those two orchestration options stack up. Fivetran, by contrast, focuses on managed ELT connectors from SaaS apps into a warehouse, where Glue is the better fit for custom, heavier transformations rather than plug-and-play ingestion.
| Tool | Category | Where it fits with Glue |
|---|---|---|
| AWS Glue | Serverless ETL + catalog | The transformation and cataloging engine itself |
| Databricks | Lakehouse analytics + ML platform | Often reads the same Iceberg/Delta tables Glue writes |
| Apache Airflow | Workflow orchestration | Schedules and chains Glue jobs alongside other tasks |
| Fivetran | Managed SaaS-to-warehouse ELT | Handles ingestion; Glue handles custom transformation |
If your pipeline eventually needs to move data between warehouses too, it’s worth comparing the destination platforms directly — see our AWS RDS vs Azure Database vs Google Cloud SQL breakdown for the relational side, or our look at streaming ingestion in Kafka vs Kinesis if your raw/ prefix should really be a stream instead of a batch drop.
Common Pitfalls When Building on AWS Glue
These are the mistakes that eat the most time and budget on a first Glue project. None of them are exotic — they’re defaults that look harmless in a small tutorial dataset and turn into real cost or data-quality problems the moment production-scale data shows up.
- Skipping job bookmarks on recurring jobs. Without
--job-bookmark-option job-bookmark-enable, every scheduled run reprocesses the entire source dataset instead of only new data, quietly multiplying your DPU-hour bill. - Over-provisioning DPUs before checking actual usage. Jumping straight to 10 workers for a small dataset wastes money; start with 2 and scale up only after CloudWatch metrics show the job is genuinely memory- or CPU-bound.
- Granting the Glue IAM role broad S3 access. Scoping to the exact bucket and prefixes the job needs, as shown in Step 2, limits the blast radius if a script has a bug or the role is ever compromised.
- Not re-crawling after a schema change at the source. If upstream columns are added, removed, or change type, the catalog won’t reflect it until the crawler runs again, which can cause silent data quality issues downstream.
- Writing output without partitioning. Skipping
partitionByon the write step, as used in Step 6, means every downstream Athena query scans the entire dataset instead of pruning to relevant partitions — slower and more expensive at any real scale. - Assuming Glue Studio and hand-written Spark scripts are interchangeable without review. Glue Studio-generated jobs are a great starting point, but they’re worth reading through before production, since visual-editor output can include steps or defaults you didn’t intend.
Troubleshooting: 8 AWS Glue Errors and How to Fix Them
Here are the errors most likely to interrupt a Glue project, and the fastest fix for each. When a fix isn’t obvious from the error message alone, the Glue job’s CloudWatch logs from Step 11, combined with the official Glue job monitoring documentation, are the fastest path to a root cause.
- Job fails with an OutOfMemoryError or ExecutorLostFailure. The job is under-provisioned for its data volume. Increase the number of workers or move to a larger worker type (G.2X instead of G.1X), and check whether a wide, unpruned join is inflating memory use.
- Crawler finishes but the table doesn’t appear in the catalog. Confirm the crawler’s IAM role actually has read access to the S3 path — a silent permissions failure is a far more common cause than a crawler bug.
- Job reprocesses all data instead of only new records. Job bookmarks are disabled, misconfigured, or were reset. Check the
--job-bookmark-optionargument and confirm it’s set tojob-bookmark-enable, notjob-bookmark-disable. - “Access Denied” reading or writing a Lake Formation-registered table. Since Glue 5.0, Spark jobs need the job’s IAM role to have full table access under Lake Formation permissions specifically, separate from the underlying S3 IAM policy.
- Schema mismatch between the catalog and the actual S3 data. Usually means the crawler hasn’t run since the source data’s structure changed. Re-run the crawler, or manually edit the table definition if the change is deliberate and you want to skip a re-crawl.
- Job runs successfully but output is empty. Check the transformation logic for an overly aggressive filter — a null-check or date-cast that silently drops every row is the most common cause, and it won’t throw an error, it’ll just produce zero output rows.
- Crawler and ETL job run out of order. If they’re scheduled independently rather than chained, the job can run against a stale or incomplete crawl. Use a conditional trigger, as shown in Step 10, so the job only fires after the crawler succeeds.
- Job takes far longer than expected on a small dataset. Startup and cluster provisioning time adds a baseline delay to every Glue job regardless of data size. If that startup overhead is a problem for latency-sensitive pipelines, evaluate whether a Lambda-based approach or a persistent EMR cluster fits better than a Glue Spark job for that specific workload.
Advanced Tips: Iceberg Tables, Lake Formation, and Glue for Ray
Once the basic crawl-transform-catalog loop from this tutorial is solid, three more advanced capabilities are worth exploring for larger or more demanding pipelines.
SageMaker Lakehouse and Unified Studio Integration
Glue 5.0 added support for Amazon SageMaker Unified Studio and SageMaker Lakehouse, which brings Glue-cataloged tables directly into SageMaker’s notebook and ML tooling without a separate export or copy step. If your pipeline’s ultimate destination is a machine learning feature set rather than a BI dashboard, that integration is worth exploring once the base ETL pattern from this tutorial is working, since it removes an entire data-movement step between cataloging and model training.
Open table formats. Glue 5.0 and 5.1 ship with updated support for Apache Iceberg 1.7.1, Apache Hudi 0.15.0, and Delta Lake 3.3.0. Writing your output as Iceberg instead of plain partitioned Parquet, as this tutorial’s project does, adds schema evolution, time travel, and safe concurrent writes — genuinely useful once more than one job writes to the same table.
Lake Formation integration. For fine-grained, column- and row-level access control on top of the Data Catalog, Lake Formation lets you grant permissions per table, column, or even row filter rather than relying purely on IAM. Since a June 2025 AWS update, Glue 5.0+ Spark jobs can read and write Lake Formation-registered tables directly, provided the job role has full table access — worth adopting if multiple teams share the same catalog and need different visibility into the same tables.
Glue for Ray. For distributed Python workloads that don’t map cleanly onto Spark’s DataFrame model — custom feature engineering, certain ML preprocessing steps — Glue supports running that logic on Ray instead, still billed on the same DPU model as Spark jobs. It’s a narrower use case than the Spark ETL in this tutorial, but worth knowing about if a future pipeline needs distributed Python rather than distributed SQL-like transforms.
Frequently Asked Questions
Is AWS Glue free to use?
There’s no dedicated free tier for Glue compute — ETL jobs, crawlers, and interactive sessions all bill at $0.44 per DPU-hour from the first second. The Data Catalog does have a real free tier: the first 1,000,000 objects stored and 1,000,000 requests per month.
What Glue version should I use for a new project in 2026?
Glue 5.1, the current latest release as of November 2025, running Apache Spark 3.5.6, Python 3.11, and Scala 2.12.18. There’s no reason to start a new project on 4.0 or earlier.
Do I need to write Spark code to use Glue?
Not necessarily. Glue Studio’s visual editor generates Spark jobs from a drag-and-drop interface, and Glue DataBrew offers a no-code path for data preparation and cleaning. This tutorial writes the ETL script by hand for clarity and control, but either path produces a job that runs the same way underneath.
How is Glue different from Amazon EMR?
EMR gives you a managed Hadoop/Spark cluster you control the sizing, tuning, and lifecycle of. Glue is serverless — you don’t manage cluster infrastructure at all, just DPU count per job — at the cost of some of the fine-grained tuning control EMR offers.
What’s the difference between a Glue crawler and a Glue job?
A crawler discovers schema and registers tables in the Data Catalog; it doesn’t transform data. A job actually processes data — reading, transforming, and writing it — typically using Spark. Most real pipelines use both, as this tutorial’s project does.
Can Glue read from a database instead of S3?
Yes, via JDBC connections to sources like PostgreSQL, MySQL, or Amazon RDS instances, in addition to S3. The crawler and job patterns in this tutorial work the same way against a JDBC source once the connection is configured.
Why does my job keep reprocessing the same data?
Job bookmarks are almost certainly disabled or misconfigured. Set --job-bookmark-option job-bookmark-enable in the job’s default arguments, as covered in Step 12 and the troubleshooting section.
Is there a formal comparison of Glue’s market position against Databricks or Airflow?
Not from a major independent research firm in current public data. What’s clear from AWS’s own product positioning and third-party pricing guides is that Glue is typically paired with, rather than pitted against, tools like Airflow (orchestration) and Databricks (broader lakehouse analytics), each covering a different layer of the same pipeline.
How do I know if my Glue job needs more DPUs?
Check the CloudWatch metrics enabled in Step 6 and Step 11. Consistently high memory utilization or executor failures point to needing more workers or a larger worker type; consistently low utilization across a long-running job usually means you’re over-provisioned and paying for idle capacity.
Where to Go Next
You now have a complete, scheduled, monitored Glue pipeline running on the current 5.1 runtime: raw CSV crawled into the catalog, transformed into partitioned Parquet by a Spark ETL job, re-cataloged, and queryable in Athena on a nightly schedule. From here, the natural next steps are moving the output table to Iceberg for schema evolution, layering in Lake Formation for column-level access control, and right-sizing DPU usage against your actual production data volume.
Related Coverage
- How to Set Up AWS Lambda: 12 Steps, 90 Min [2026]
- How to Set Up AWS Step Functions: 12 Steps, 90 Min [2026]
- Step Functions vs Airflow: $0 vs $357/mo Floor [2026]
- Kafka vs Kinesis 2026: $0.015/Shard-Hour and 5x Latency Gap [Tested]
- Snowflake vs Databricks vs BigQuery 2026: $6.25/TB Showdown
- AWS RDS vs Azure Database vs Google Cloud SQL: 25% Price Gap [2026]
