Skip to Content
Cloud Engineering Academy
7. Data Engineering & ETL Pipelines

Lab 7.4: a Glue Spark job with job bookmarks and a data-quality quarantine

Run a Glue 5.0 job that reads only new raw files through a job bookmark, evaluates a DQDL ruleset row by row, writes passing rows to partitioned Parquet and failing rows with their broken rules to quarantine, and compare its verdict with the Python validator.

intermediateโฑ 1 h 30 min๐ŸŸก Low costDEA-C01

Assumes: AWS Glue 5.0 (Spark 3.5, Python 3) ยท Glue Data Quality (DQDL) ยท AWS CDK v2 (C#) ยท AWS CLI v2 ยท Region us-east-1 ยท labs/m07-transaction-pipeline/glue. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lab you can:

  • Read a Glue job script that uses a job bookmark (transformation_ctx, job.init, job.commit) and explain exactly what a bookmark remembers.
  • Prove with three runs that the second run does nothing and the third reads only the newly uploaded file.
  • Write and read a DQDL ruleset, and say which checks it can express and which need code.
  • Split a dataset into passing and failing rows using row-level data-quality outcomes, and write the failing rows with the names of the rules they broke.
  • Explain the difference between a bookmark (progress) and idempotency (safe repetition), and how reset-job-bookmark can duplicate data.
  • Compare the DQDL verdict on the messy file with the Python validator's, and explain the gap.

Prerequisites

  • Validation and data quality and PySpark in Glue for beginners.
  • Lab 7.3 Part A, which deploys the stack (HlM07-DataLake). You need the same tools: AWS CLI v2 with learn-sandbox, the .NET 10 SDK, Node.js and the CDK CLI.
  • The messy sample labs/m07-transaction-pipeline/data/samples/transactions_messy.csv (1,036 rows, 97 planted defects, from Lab 7.1).

Cost label

๐ŸŸก Credits. A Glue Spark job bills per DPU-hour with a 1-minute minimum. The lab job uses 2 workers of type G.1X (the minimum), so each run costs about USD 0.03 at USD 0.44 per DPU-hour (the module README's estimate, checked 2026-11-18 โš  VERIFY on the Glue pricing page). Three or four runs plus the evaluation overhead is a few cents. Add the standing cost of the stack (KMS key about USD 1 per month prorated, one secret, pennies of S3) and the lab fits inside USD 0.25 if destroyed the same day.

Free alternative: the same rules run offline. Use hl-pipeline validate (Lab 7.1) or python -m pytest glue/tests in the Spark container (Lab 7.2). You do not get bookmarks or the Glue Data Quality engine, but you do get the validation logic.

Scenario

A partner drops a transactions file every hour into incoming/. Each run of your Glue job must read only the files that have arrived since the last run, test every row against written rules, load the good rows to the curated zone as typed Parquet, and put the bad rows in quarantine with the reason, so an operator can fix and replay them. Nobody should read the same file twice and nobody should lose a row.

Architecture

The job runs under hl-m07-glue-job-role (read on raw, write on curated and quarantine, the data key, glue:PublishDataQuality) and the security configuration hl-m07-glue-security, which encrypts S3 output, CloudWatch logs and the bookmark.

The files

FileRole in the lab
glue/jobs/transactions_dq_bookmarked.pyThe job
glue/dq/transactions_raw.dqdlThe DQDL ruleset, uploaded as a CDK asset and read from S3 at run time
infra/cdk/.../DataLakeStack.csDefines the job hl-m07-raw-dq-bookmarked (Glue 5.0, G.1X, 2 workers, bookmark enabled)
glue/src/harbourline_etl/transactions.pyThe Python rules you compare against

Part A: read the ruleset

glue/dq/transactions_raw.dqdl
Rules = [
    RowCount > 0,
    IsComplete "transaction_id",
    IsComplete "account_id",
    IsComplete "idempotency_key",
    IsUnique "transaction_id",
    ColumnValues "account_id" matches "ACC-[0-9]{10}",
    ColumnValues "booking_date" matches "[0-9]{4}-[0-9]{2}-[0-9]{2}",
    ColumnValues "value_date" matches "[0-9]{4}-[0-9]{2}-[0-9]{2}",
    ColumnValues "created_at" matches "[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z",
    ColumnValues "amount" matches "-?[0-9]{1,16}(\.[0-9]{1,2})?",
    ColumnValues "currency" in ["GBP", "EUR"],
    ColumnValues "type" in ["card_payment", "transfer_in", ...],
    ColumnValues "channel" in ["mobile", "web", "branch", "card", "api", "batch"],
    ColumnValues "status" in ["booked", "pending", "reversed", "rejected"]
]

DQDL (Data Quality Definition Language) is a declarative list of rules. Every raw column arrives as a string, so the format checks are regular expressions. Three kinds of rule appear: dataset rules (RowCount > 0), completeness and uniqueness (IsComplete, IsUnique) and value rules (matches, in). What DQDL cannot express here is documented in the file's header comment: the IBAN mod-97 check, sign rules by transaction type (credits positive, debits negative), a real calendar check such as 2026-02-30, and duplicates by idempotency_key (retries with a new id). Those live in the Python and C# validators. A good pipeline uses both: DQDL for cheap, declarative, auditable checks at scale; code for rules that need logic.

Part B: deploy and prepare

Deploy the stack if it is not already up

cd labs/m07-transaction-pipeline/infra/cdk
aws sso login --profile learn-sandbox
cdk deploy HlM07-DataLake --profile learn-sandbox -c owner=<your-name>
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1 MSYS_NO_PATHCONV=1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
RAW="hl-m07-raw-${ACCOUNT}"; CUR="hl-m07-curated-${ACCOUNT}"; QUA="hl-m07-quarantine-${ACCOUNT}"
cd ../..

Stop the project pipeline from firing on your test uploads

The raw bucket sends an EventBridge event for every object under incoming/, and the rule hl-m07-raw-object-created starts the project's Step Functions workflow and its own Glue job (hl-m07-transactions-etl) for each one. This lab's job reads the same incoming/ prefix, so if you leave the rule on, each upload also starts the project job: two jobs, double the cost, and the project's curated and gold zones fill with your test data. Disable the rule for the lab and re-enable it for the project:

aws events disable-rule --name hl-m07-raw-object-created

Upload the messy file under its own folder

aws s3 cp data/samples/transactions_messy.csv "s3://${RAW}/incoming/lab74/run1/transactions_messy.csv"

Part C: three runs

Run 1: the first run reads the file

RUN=$(aws glue start-job-run --job-name hl-m07-raw-dq-bookmarked --query JobRunId --output text); echo $RUN
until s=$(aws glue get-job-run --job-name hl-m07-raw-dq-bookmarked --run-id "$RUN" --query JobRun.JobRunState --output text); \
  [ "$s" != RUNNING ] && [ "$s" != STARTING ] && [ "$s" != WAITING ]; do echo "$s"; sleep 15; done; echo "final: $s"

Expect a start time of a minute or two (Glue starts a Spark cluster) and final: SUCCEEDED. Read the job's output: Glue console โ†’ ETL jobs โ†’ hl-m07-raw-dq-bookmarked โ†’ Runs โ†’ (run) โ†’ Output logs, or:

aws logs tail /aws-glue/jobs/output --since 30m

The log group name can differ when continuous logging or a security configuration is on โš  VERIFY; the Output logs link in the run details always works.

You should see the rule outcomes table printed by rule_outcomes.show(truncate=False), then a line like passed=<n> failed=<m>. In my offline simulation of the same rules on this file, about 55 rows fail (โš  VERIFY on your run; I did not execute Glue while writing this lab). The exact number depends on how Glue Data Quality treats IsUnique on duplicates and on whether matches is a full or a partial match, which my simulation answered the same way both ways.

Look at the two outputs

aws s3 ls "s3://${CUR}/transactions_dq/" --recursive | head
aws s3 ls "s3://${QUA}/dq_rejects/" --recursive
aws s3 cp "s3://${QUA}/dq_rejects/" - --recursive 2>/dev/null | head -c 1500

Curated holds Parquet files under year=2026/month=10/day=20/ and so on. Quarantine holds JSON Lines files: every original column plus DataQualityRulesPass, DataQualityRulesFail, DataQualityEvaluationResult and a reason_codes text made from the failing rules, for example ColumnValues "currency" in ["GBP","EUR"]. The raw values are kept so an operator can fix the row and resubmit it. That also means quarantine holds raw personal data: handle it like the raw zone.

Run 2: nothing new, so nothing happens

Start the job again without uploading anything:

RUN=$(aws glue start-job-run --job-name hl-m07-raw-dq-bookmarked --query JobRunId --output text)

The output now contains No new files since the last bookmarked run. and the run succeeds in about a minute. The bookmark remembers that run1/transactions_messy.csv was already read by the source named raw_csv. Check that curated did not grow: aws s3 ls "s3://${CUR}/transactions_dq/" --recursive --summarize | tail -2 shows the same count as before.

Run 3: one new file

Upload the clean sample as a second file and run again:

aws s3 cp data/samples/transactions.csv "s3://${RAW}/incoming/lab74/run3/transactions.csv"
RUN=$(aws glue start-job-run --job-name hl-m07-raw-dq-bookmarked --query JobRunId --output text)

The output reports only the new file's rows: passed=1012 failed=0 if every row of the clean file passes. The messy file is not read again.

Reset the bookmark and see the trap

aws glue reset-job-bookmark --job-name hl-m07-raw-dq-bookmarked
RUN=$(aws glue start-job-run --job-name hl-m07-raw-dq-bookmarked --query JobRunId --output text)

After the reset the job reads both files again and appends their rows to curated and quarantine a second time. mode("append") plus a reset means duplicates. A bookmark tracks progress; it is not idempotency. A job that must be safe to repeat needs an idempotent write (overwrite the partition, MERGE into Iceberg, or the run-id and audit-record pattern the project job uses in the module project).

Part D: read the job

glue/jobs/transactions_dq_bookmarked.py (the parts that matter)
job = Job(glue_context)
job.init(ARGS["JOB_NAME"], ARGS)  # loads the bookmark state for this job

raw = glue_context.create_dynamic_frame.from_options(
    connection_type="s3",
    connection_options={"paths": [f"s3://{ARGS['raw_bucket']}/incoming/"], "recurse": True},
    format="csv",
    format_options={"withHeader": True, "separator": ","},
    transformation_ctx="raw_csv",
)

results = EvaluateDataQuality().process_rows(
    frame=raw_with_file, ruleset=ruleset,
    publishing_options={"dataQualityEvaluationContext": "hl_m07_raw_check",
                        "enableDataQualityCloudWatchMetrics": True,
                        "enableDataQualityResultsPublishing": True},
    additional_options={"performanceTuning.caching": "CACHE_NOTHING"})
rows = SelectFromCollection.apply(dfc=results, key="rowLevelOutcomes", transformation_ctx="rows").toDF()
passed = rows.filter(F.col("DataQualityEvaluationResult") == "Passed").drop(*dq_cols)
failed = rows.filter(F.col("DataQualityEvaluationResult") != "Passed")
...
job.commit()  # saves the bookmark: these files will not be read again
  1. job.init(...) โ€” loads this job's bookmark state, keyed by job name. Without it, no bookmark.
  2. create_dynamic_frame.from_options with recurse: True โ€” reads every CSV under incoming/ including sub-folders such as lab74/run1/. Bookmarks work on S3 sources read with this API and a transformation_ctx.
  3. transformation_ctx="raw_csv" โ€” the name of the bookmark entry. Glue records which objects this source has consumed (by path and last-modified time โš  VERIFY the exact tracking details). Rename it and the job forgets its history. Use a stable, unique name per source.
  4. if raw.count() == 0: job.commit(); sys.exit(0) โ€” the "no new files" branch still commits, so the empty run is recorded as a success.
  5. F.input_file_name() added as source_file โ€” lineage: every row knows which file it came from.
  6. EvaluateDataQuality().process_rows(...) โ€” evaluates the ruleset and returns a collection holding ruleOutcomes (one line per rule with pass or fail) and rowLevelOutcomes (the input rows plus the pass/fail columns). enableDataQualityResultsPublishing sends the results to the Glue console and CloudWatch metrics. The row-level outcome is available for row-level rules; dataset-level rules such as RowCount appear only in ruleOutcomes โš  VERIFY which of your rules are evaluated per row.
  7. Filter on DataQualityEvaluationResult โ€” the job splits the frame on the outcome column. Check the exact outcome values in your run before trusting a filter โš  VERIFY (Passed versus Failed versus other values).
  8. The typing block โ€” only rows that passed are converted: to_date with a strict pattern, cast("decimal(18,2)") and the partition columns. The IBAN becomes a masked column and the raw column is dropped. This lab masks but does not tokenise; the project job tokenises with a secret key.
  9. mode("append") โ€” adds new files to the existing data. This is right for one run and wrong for a re-run of the same input.
  10. job.commit() โ€” saves the bookmark. If the job fails before this line, no progress is saved and the same files are read next time, which is the safe behaviour for a crash.

Part E: compare with the Python validator

CheckDQDL rulesetPython and C# validator
Missing ids, formats of dates, account id, amountYes (IsComplete, matches)Yes
Allowed values for currency, type, channel, statusYes (in [...])Yes
Duplicate transaction_idIsUnique flags rowsYes, in-file tracker
Duplicate by idempotency_key (retry with a new id)NoYes
IBAN check digits (mod-97)NoYes
Credits positive, debits negativeNoYes
Real calendar date (2026-02-30)No (only a pattern)Yes
Mojibake and Latin-1 repairNoYes (repair and warn)
Late-arriving flagNoYes (warn)
Reason codes an operator can act onThe failing rule textStable codes such as INVALID_IBAN

That is why the Python validator rejects 67 rows of this file and the ruleset fails fewer: DQDL has no rule for IBAN checks, sign rules or idempotency-key retries. A rule set that "passes" proves only that the rules you wrote pass. The project pipeline therefore runs both: the Python rules decide row by row with reason codes, and a smaller DQDL ruleset runs on the curated rows as a second line of defence.

Security considerations

โš  Learning shortcut โ€” not for production

The job is given one broad role for all three outputs, appends to curated without a duplicate guard, masks the IBAN instead of tokenising it with a protected key, and writes the quarantined rows with raw values into a bucket that anyone with the data key and S3 access could read. A bank splits the roles, makes writes idempotent, uses keyed tokens and keeps quarantine under a break-glass policy (see Secure pipeline design).

  • The bookmark is encrypted (CSE-KMS in the security configuration) because it lists object names, which can themselves carry sensitive context such as partner ids.
  • Least privilege: the role may read raw but not delete it, and glue:PublishDataQuality is the one action that needs * as the resource.
  • Logs: the job prints counts and the rule table, not rows. Do not add df.show() of raw data "for debugging".
  • Ruleset as code: it lives in the repository, is uploaded as a versioned asset and is read from S3 at run time. A ruleset edited by hand in the console has no review trail.

Common mistakes

  • Leaving the EventBridge rule on โ†’ every test upload also runs the project job. Disable it for the lab.
  • Forgetting transformation_ctx โ†’ the job has no bookmark entry and re-reads everything silently.
  • Calling job.commit() before the writes finish โ†’ progress is saved for work that failed.
  • Resetting the bookmark and re-running with append โ†’ duplicates in curated.
  • Treating DQDL as complete validation โ†’ IBAN, sign and retry rules need code.
  • Using G.2X workers "to be safe" for a 1,000-row file โ†’ four times the DPU cost for nothing.
  • Reading incoming/ recursively when unrelated test files sit there โ†’ unrelated files get processed and bookmarked.

Troubleshooting

SymptomLikely causeFix
ModuleNotFoundError: awsgluedqThe job runs on a Glue version without the libraryThe stack sets Glue 5.0; do not change GlueVersion
AccessDeniedException from KMS on write or on the bookmarkRole lacks key permissions or the key policy changedThe stack grants the role kms:Decrypt and GenerateDataKey; re-add them if you edited the key policy
Run stays WAITINGConcurrent-run limit or worker capacityWait; check the job's maximum concurrency
Run 2 re-reads the fileBookmark disabled, transformation_ctx missing, or the job failed before commitCheck --job-bookmark-option is job-bookmark-enable and look at the failed run's error
Output logs are emptyLooking in the wrong log group; continuous logging may name it differentlyUse the Output logs link in the job run details
FileNotFoundException on the rulesetThe asset URI argument is wrong after a redeployRedeploy the stack so --ruleset_s3_uri is current
Failed count differs from 67The rule sets differ (see Part E)Expected. Compare by rule, not by total
The project workflow also startedThe EventBridge rule was still enabledaws events disable-rule --name hl-m07-raw-object-created; stop the extra Step Functions execution if needed

Cost and cleanup

# 1. Remove the lab's data (raw has versioning, so use the destroy below for a full clean).
aws s3 rm "s3://${RAW}/incoming/lab74/" --recursive
aws s3 rm "s3://${CUR}/transactions_dq/" --recursive
aws s3 rm "s3://${QUA}/dq_rejects/" --recursive

# 2. Put the project trigger back (skip only if you will destroy the stack now).
aws events enable-rule --name hl-m07-raw-object-created

# 3. Reset the bookmark so a future run starts clean.
aws glue reset-job-bookmark --job-name hl-m07-raw-dq-bookmarked

# 4. Destroy everything if you are not continuing to the project today.
cd infra/cdk && cdk destroy HlM07-DataLake --profile learn-sandbox

# 5. Check that no billing resources remain.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query "ResourceTagMappingList[?contains(ResourceARN, 'hl-m07')].ResourceARN"

If you destroy the stack, the key enters its seven-day pending-deletion window (the minimum). Glue Data Quality results kept in the service are covered by the job's costs; there is no separate always-on charge for them โš  VERIFY.

Exercises

  • ๐ŸŸข Guided: Repeat the three-run experiment and write down, for each run, the number of files read, passed and failed. Explain every number.
  • ๐ŸŸก Challenge: Add a DQDL rule that requires amount to be present, and a rule that bounds the length of reference (look up the DQDL syntax in the AWS reference). Redeploy, reset the bookmark, re-run and show which new rows are quarantined. Say why the Python rules would still catch something DQDL does not.
  • ๐Ÿ”ด Live problem: The job failed halfway through writing curated after reading three files. The on-call engineer re-runs it and finance reports duplicate transactions. Explain exactly what happened with the bookmark and the append write, how to clean up without losing data, and the design change that prevents it.

Assignment

Create docs/dq-rules.md in your portfolio repository: your DQDL ruleset, a table of what it can and cannot check compared with code-based rules, the three-run evidence (counts) and a paragraph on bookmark versus idempotency.

Solutions

Knowledge check

Interview questions

  • "How do you process only new files in Glue?" โ€” Job bookmarks with a transformation_ctx on S3 sources and job.commit(), or an explicit file manifest or event-driven design where each file triggers a run.
  • "Bookmarks or idempotency?" โ€” Bookmarks track progress; idempotent writes make repeats safe. You usually need both.
  • "Where does Glue Data Quality fit with custom validation?" โ€” Use DQDL for declarative, auditable checks and metrics, and code for rules DQDL cannot express. Quarantine bad rows with reason codes rather than failing the whole file.

Certification practice

Completion checklist

  • I disabled the project's EventBridge rule for the lab and re-enabled it after
  • I ran the job three times and explained each result
  • I looked at the passing Parquet and the quarantined JSON Lines
  • I can explain bookmark versus idempotency and what a reset does
  • I compared DQDL with the Python validator and can name three gaps
  • I cleaned up the data or destroyed the stack and ran the tag check

Key takeaways

  • A bookmark remembers progress per transformation_ctx and is saved only by job.commit().
  • Row-level data-quality outcomes let one job split good rows from bad and keep the broken rule names.
  • DQDL is declarative and cheap but cannot express every banking rule; combine it with code.
  • Bookmarks are not idempotency: reset plus append equals duplicates.
  • Quarantine holds raw data; protect it like the raw zone.

Further reading

Independent learning resource โ€” not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page