Lab 7.4: a Glue Spark job with job bookmarks and a data-quality quarantine
Run a Glue 5.0 job that reads only new raw files through a job bookmark, evaluates a DQDL ruleset row by row, writes passing rows to partitioned Parquet and failing rows with their broken rules to quarantine, and compare its verdict with the Python validator.
Assumes: AWS Glue 5.0 (Spark 3.5, Python 3) ยท Glue Data Quality (DQDL) ยท AWS CDK v2 (C#) ยท AWS CLI v2 ยท Region us-east-1 ยท labs/m07-transaction-pipeline/glue. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lab you can:
- Read a Glue job script that uses a job bookmark (
transformation_ctx,job.init,job.commit) and explain exactly what a bookmark remembers. - Prove with three runs that the second run does nothing and the third reads only the newly uploaded file.
- Write and read a DQDL ruleset, and say which checks it can express and which need code.
- Split a dataset into passing and failing rows using row-level data-quality outcomes, and write the failing rows with the names of the rules they broke.
- Explain the difference between a bookmark (progress) and idempotency (safe repetition), and how
reset-job-bookmarkcan duplicate data. - Compare the DQDL verdict on the messy file with the Python validator's, and explain the gap.
Prerequisites
- Validation and data quality and PySpark in Glue for beginners.
- Lab 7.3 Part A, which deploys the stack (
HlM07-DataLake). You need the same tools: AWS CLI v2 withlearn-sandbox, the .NET 10 SDK, Node.js and the CDK CLI. - The messy sample
labs/m07-transaction-pipeline/data/samples/transactions_messy.csv(1,036 rows, 97 planted defects, from Lab 7.1).
Cost label
๐ก Credits. A Glue Spark job bills per DPU-hour with a 1-minute minimum. The lab job uses 2 workers of type G.1X (the minimum), so each run costs about USD 0.03 at USD 0.44 per DPU-hour (the module README's estimate, checked 2026-11-18 โ VERIFY on the Glue pricing page). Three or four runs plus the evaluation overhead is a few cents. Add the standing cost of the stack (KMS key about USD 1 per month prorated, one secret, pennies of S3) and the lab fits inside USD 0.25 if destroyed the same day.
Free alternative: the same rules run offline. Use hl-pipeline validate (Lab 7.1) or python -m pytest glue/tests in the Spark container (Lab 7.2). You do not get bookmarks or the Glue Data Quality engine, but you do get the validation logic.
Scenario
A partner drops a transactions file every hour into incoming/. Each run of your Glue job must read only the files that have arrived since the last run, test every row against written rules, load the good rows to the curated zone as typed Parquet, and put the bad rows in quarantine with the reason, so an operator can fix and replay them. Nobody should read the same file twice and nobody should lose a row.
Architecture
The job runs under hl-m07-glue-job-role (read on raw, write on curated and quarantine, the data key, glue:PublishDataQuality) and the security configuration hl-m07-glue-security, which encrypts S3 output, CloudWatch logs and the bookmark.
The files
| File | Role in the lab |
|---|---|
glue/jobs/transactions_dq_bookmarked.py | The job |
glue/dq/transactions_raw.dqdl | The DQDL ruleset, uploaded as a CDK asset and read from S3 at run time |
infra/cdk/.../DataLakeStack.cs | Defines the job hl-m07-raw-dq-bookmarked (Glue 5.0, G.1X, 2 workers, bookmark enabled) |
glue/src/harbourline_etl/transactions.py | The Python rules you compare against |
Part A: read the ruleset
Rules = [
RowCount > 0,
IsComplete "transaction_id",
IsComplete "account_id",
IsComplete "idempotency_key",
IsUnique "transaction_id",
ColumnValues "account_id" matches "ACC-[0-9]{10}",
ColumnValues "booking_date" matches "[0-9]{4}-[0-9]{2}-[0-9]{2}",
ColumnValues "value_date" matches "[0-9]{4}-[0-9]{2}-[0-9]{2}",
ColumnValues "created_at" matches "[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z",
ColumnValues "amount" matches "-?[0-9]{1,16}(\.[0-9]{1,2})?",
ColumnValues "currency" in ["GBP", "EUR"],
ColumnValues "type" in ["card_payment", "transfer_in", ...],
ColumnValues "channel" in ["mobile", "web", "branch", "card", "api", "batch"],
ColumnValues "status" in ["booked", "pending", "reversed", "rejected"]
]DQDL (Data Quality Definition Language) is a declarative list of rules. Every raw column arrives as a string, so the format checks are regular expressions. Three kinds of rule appear: dataset rules (RowCount > 0), completeness and uniqueness (IsComplete, IsUnique) and value rules (matches, in). What DQDL cannot express here is documented in the file's header comment: the IBAN mod-97 check, sign rules by transaction type (credits positive, debits negative), a real calendar check such as 2026-02-30, and duplicates by idempotency_key (retries with a new id). Those live in the Python and C# validators. A good pipeline uses both: DQDL for cheap, declarative, auditable checks at scale; code for rules that need logic.
Part B: deploy and prepare
Deploy the stack if it is not already up
cd labs/m07-transaction-pipeline/infra/cdk
aws sso login --profile learn-sandbox
cdk deploy HlM07-DataLake --profile learn-sandbox -c owner=<your-name>
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1 MSYS_NO_PATHCONV=1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
RAW="hl-m07-raw-${ACCOUNT}"; CUR="hl-m07-curated-${ACCOUNT}"; QUA="hl-m07-quarantine-${ACCOUNT}"
cd ../..Stop the project pipeline from firing on your test uploads
The raw bucket sends an EventBridge event for every object under incoming/, and the rule hl-m07-raw-object-created starts the project's Step Functions workflow and its own Glue job (hl-m07-transactions-etl) for each one. This lab's job reads the same incoming/ prefix, so if you leave the rule on, each upload also starts the project job: two jobs, double the cost, and the project's curated and gold zones fill with your test data. Disable the rule for the lab and re-enable it for the project:
aws events disable-rule --name hl-m07-raw-object-createdUpload the messy file under its own folder
aws s3 cp data/samples/transactions_messy.csv "s3://${RAW}/incoming/lab74/run1/transactions_messy.csv"Part C: three runs
Run 1: the first run reads the file
RUN=$(aws glue start-job-run --job-name hl-m07-raw-dq-bookmarked --query JobRunId --output text); echo $RUN
until s=$(aws glue get-job-run --job-name hl-m07-raw-dq-bookmarked --run-id "$RUN" --query JobRun.JobRunState --output text); \
[ "$s" != RUNNING ] && [ "$s" != STARTING ] && [ "$s" != WAITING ]; do echo "$s"; sleep 15; done; echo "final: $s"Expect a start time of a minute or two (Glue starts a Spark cluster) and final: SUCCEEDED. Read the job's output: Glue console โ ETL jobs โ hl-m07-raw-dq-bookmarked โ Runs โ (run) โ Output logs, or:
aws logs tail /aws-glue/jobs/output --since 30mThe log group name can differ when continuous logging or a security configuration is on โ VERIFY; the Output logs link in the run details always works.
You should see the rule outcomes table printed by rule_outcomes.show(truncate=False), then a line like passed=<n> failed=<m>. In my offline simulation of the same rules on this file, about 55 rows fail (โ VERIFY on your run; I did not execute Glue while writing this lab). The exact number depends on how Glue Data Quality treats IsUnique on duplicates and on whether matches is a full or a partial match, which my simulation answered the same way both ways.
Look at the two outputs
aws s3 ls "s3://${CUR}/transactions_dq/" --recursive | head
aws s3 ls "s3://${QUA}/dq_rejects/" --recursive
aws s3 cp "s3://${QUA}/dq_rejects/" - --recursive 2>/dev/null | head -c 1500Curated holds Parquet files under year=2026/month=10/day=20/ and so on. Quarantine holds JSON Lines files: every original column plus DataQualityRulesPass, DataQualityRulesFail, DataQualityEvaluationResult and a reason_codes text made from the failing rules, for example ColumnValues "currency" in ["GBP","EUR"]. The raw values are kept so an operator can fix the row and resubmit it. That also means quarantine holds raw personal data: handle it like the raw zone.
Run 2: nothing new, so nothing happens
Start the job again without uploading anything:
RUN=$(aws glue start-job-run --job-name hl-m07-raw-dq-bookmarked --query JobRunId --output text)The output now contains No new files since the last bookmarked run. and the run succeeds in about a minute. The bookmark remembers that run1/transactions_messy.csv was already read by the source named raw_csv. Check that curated did not grow: aws s3 ls "s3://${CUR}/transactions_dq/" --recursive --summarize | tail -2 shows the same count as before.
Run 3: one new file
Upload the clean sample as a second file and run again:
aws s3 cp data/samples/transactions.csv "s3://${RAW}/incoming/lab74/run3/transactions.csv"
RUN=$(aws glue start-job-run --job-name hl-m07-raw-dq-bookmarked --query JobRunId --output text)The output reports only the new file's rows: passed=1012 failed=0 if every row of the clean file passes. The messy file is not read again.
Reset the bookmark and see the trap
aws glue reset-job-bookmark --job-name hl-m07-raw-dq-bookmarked
RUN=$(aws glue start-job-run --job-name hl-m07-raw-dq-bookmarked --query JobRunId --output text)After the reset the job reads both files again and appends their rows to curated and quarantine a second time. mode("append") plus a reset means duplicates. A bookmark tracks progress; it is not idempotency. A job that must be safe to repeat needs an idempotent write (overwrite the partition, MERGE into Iceberg, or the run-id and audit-record pattern the project job uses in the module project).
Part D: read the job
job = Job(glue_context)
job.init(ARGS["JOB_NAME"], ARGS) # loads the bookmark state for this job
raw = glue_context.create_dynamic_frame.from_options(
connection_type="s3",
connection_options={"paths": [f"s3://{ARGS['raw_bucket']}/incoming/"], "recurse": True},
format="csv",
format_options={"withHeader": True, "separator": ","},
transformation_ctx="raw_csv",
)
results = EvaluateDataQuality().process_rows(
frame=raw_with_file, ruleset=ruleset,
publishing_options={"dataQualityEvaluationContext": "hl_m07_raw_check",
"enableDataQualityCloudWatchMetrics": True,
"enableDataQualityResultsPublishing": True},
additional_options={"performanceTuning.caching": "CACHE_NOTHING"})
rows = SelectFromCollection.apply(dfc=results, key="rowLevelOutcomes", transformation_ctx="rows").toDF()
passed = rows.filter(F.col("DataQualityEvaluationResult") == "Passed").drop(*dq_cols)
failed = rows.filter(F.col("DataQualityEvaluationResult") != "Passed")
...
job.commit() # saves the bookmark: these files will not be read againjob.init(...)โ loads this job's bookmark state, keyed by job name. Without it, no bookmark.create_dynamic_frame.from_optionswithrecurse: Trueโ reads every CSV underincoming/including sub-folders such aslab74/run1/. Bookmarks work on S3 sources read with this API and atransformation_ctx.transformation_ctx="raw_csv"โ the name of the bookmark entry. Glue records which objects this source has consumed (by path and last-modified time โ VERIFY the exact tracking details). Rename it and the job forgets its history. Use a stable, unique name per source.if raw.count() == 0: job.commit(); sys.exit(0)โ the "no new files" branch still commits, so the empty run is recorded as a success.F.input_file_name()added assource_fileโ lineage: every row knows which file it came from.EvaluateDataQuality().process_rows(...)โ evaluates the ruleset and returns a collection holdingruleOutcomes(one line per rule with pass or fail) androwLevelOutcomes(the input rows plus the pass/fail columns).enableDataQualityResultsPublishingsends the results to the Glue console and CloudWatch metrics. The row-level outcome is available for row-level rules; dataset-level rules such asRowCountappear only inruleOutcomesโ VERIFY which of your rules are evaluated per row.- Filter on
DataQualityEvaluationResultโ the job splits the frame on the outcome column. Check the exact outcome values in your run before trusting a filter โ VERIFY (PassedversusFailedversus other values). - The typing block โ only rows that passed are converted:
to_datewith a strict pattern,cast("decimal(18,2)")and the partition columns. The IBAN becomes a masked column and the raw column is dropped. This lab masks but does not tokenise; the project job tokenises with a secret key. mode("append")โ adds new files to the existing data. This is right for one run and wrong for a re-run of the same input.job.commit()โ saves the bookmark. If the job fails before this line, no progress is saved and the same files are read next time, which is the safe behaviour for a crash.
Part E: compare with the Python validator
| Check | DQDL ruleset | Python and C# validator |
|---|---|---|
| Missing ids, formats of dates, account id, amount | Yes (IsComplete, matches) | Yes |
| Allowed values for currency, type, channel, status | Yes (in [...]) | Yes |
Duplicate transaction_id | IsUnique flags rows | Yes, in-file tracker |
Duplicate by idempotency_key (retry with a new id) | No | Yes |
| IBAN check digits (mod-97) | No | Yes |
| Credits positive, debits negative | No | Yes |
Real calendar date (2026-02-30) | No (only a pattern) | Yes |
| Mojibake and Latin-1 repair | No | Yes (repair and warn) |
| Late-arriving flag | No | Yes (warn) |
| Reason codes an operator can act on | The failing rule text | Stable codes such as INVALID_IBAN |
That is why the Python validator rejects 67 rows of this file and the ruleset fails fewer: DQDL has no rule for IBAN checks, sign rules or idempotency-key retries. A rule set that "passes" proves only that the rules you wrote pass. The project pipeline therefore runs both: the Python rules decide row by row with reason codes, and a smaller DQDL ruleset runs on the curated rows as a second line of defence.
Security considerations
โ Learning shortcut โ not for production
The job is given one broad role for all three outputs, appends to curated without a duplicate guard, masks the IBAN instead of tokenising it with a protected key, and writes the quarantined rows with raw values into a bucket that anyone with the data key and S3 access could read. A bank splits the roles, makes writes idempotent, uses keyed tokens and keeps quarantine under a break-glass policy (see Secure pipeline design).
- The bookmark is encrypted (
CSE-KMSin the security configuration) because it lists object names, which can themselves carry sensitive context such as partner ids. - Least privilege: the role may read raw but not delete it, and
glue:PublishDataQualityis the one action that needs*as the resource. - Logs: the job prints counts and the rule table, not rows. Do not add
df.show()of raw data "for debugging". - Ruleset as code: it lives in the repository, is uploaded as a versioned asset and is read from S3 at run time. A ruleset edited by hand in the console has no review trail.
Common mistakes
- Leaving the EventBridge rule on โ every test upload also runs the project job. Disable it for the lab.
- Forgetting
transformation_ctxโ the job has no bookmark entry and re-reads everything silently. - Calling
job.commit()before the writes finish โ progress is saved for work that failed. - Resetting the bookmark and re-running with
appendโ duplicates in curated. - Treating DQDL as complete validation โ IBAN, sign and retry rules need code.
- Using
G.2Xworkers "to be safe" for a 1,000-row file โ four times the DPU cost for nothing. - Reading
incoming/recursively when unrelated test files sit there โ unrelated files get processed and bookmarked.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
ModuleNotFoundError: awsgluedq | The job runs on a Glue version without the library | The stack sets Glue 5.0; do not change GlueVersion |
AccessDeniedException from KMS on write or on the bookmark | Role lacks key permissions or the key policy changed | The stack grants the role kms:Decrypt and GenerateDataKey; re-add them if you edited the key policy |
Run stays WAITING | Concurrent-run limit or worker capacity | Wait; check the job's maximum concurrency |
| Run 2 re-reads the file | Bookmark disabled, transformation_ctx missing, or the job failed before commit | Check --job-bookmark-option is job-bookmark-enable and look at the failed run's error |
| Output logs are empty | Looking in the wrong log group; continuous logging may name it differently | Use the Output logs link in the job run details |
FileNotFoundException on the ruleset | The asset URI argument is wrong after a redeploy | Redeploy the stack so --ruleset_s3_uri is current |
| Failed count differs from 67 | The rule sets differ (see Part E) | Expected. Compare by rule, not by total |
| The project workflow also started | The EventBridge rule was still enabled | aws events disable-rule --name hl-m07-raw-object-created; stop the extra Step Functions execution if needed |
Cost and cleanup
# 1. Remove the lab's data (raw has versioning, so use the destroy below for a full clean).
aws s3 rm "s3://${RAW}/incoming/lab74/" --recursive
aws s3 rm "s3://${CUR}/transactions_dq/" --recursive
aws s3 rm "s3://${QUA}/dq_rejects/" --recursive
# 2. Put the project trigger back (skip only if you will destroy the stack now).
aws events enable-rule --name hl-m07-raw-object-created
# 3. Reset the bookmark so a future run starts clean.
aws glue reset-job-bookmark --job-name hl-m07-raw-dq-bookmarked
# 4. Destroy everything if you are not continuing to the project today.
cd infra/cdk && cdk destroy HlM07-DataLake --profile learn-sandbox
# 5. Check that no billing resources remain.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query "ResourceTagMappingList[?contains(ResourceARN, 'hl-m07')].ResourceARN"If you destroy the stack, the key enters its seven-day pending-deletion window (the minimum). Glue Data Quality results kept in the service are covered by the job's costs; there is no separate always-on charge for them โ VERIFY.
Exercises
- ๐ข Guided: Repeat the three-run experiment and write down, for each run, the number of files read, passed and failed. Explain every number.
- ๐ก Challenge: Add a DQDL rule that requires
amountto be present, and a rule that bounds the length ofreference(look up the DQDL syntax in the AWS reference). Redeploy, reset the bookmark, re-run and show which new rows are quarantined. Say why the Python rules would still catch something DQDL does not. - ๐ด Live problem: The job failed halfway through writing curated after reading three files. The on-call engineer re-runs it and finance reports duplicate transactions. Explain exactly what happened with the bookmark and the
appendwrite, how to clean up without losing data, and the design change that prevents it.
Assignment
Create docs/dq-rules.md in your portfolio repository: your DQDL ruleset, a table of what it can and cannot check compared with code-based rules, the three-run evidence (counts) and a paragraph on bookmark versus idempotency.
Solutions
Run 1: one file read (1,036 rows); about 55 fail the ruleset and the rest pass (your number is the one to report). Run 2: no files, so the job logs "No new files" and writes nothing. Run 3: one new file (the clean one, 1,012 rows) read and, if every row passes, passed=1012 failed=0. The pattern is the point: reads follow the bookmark, not the folder contents.
Examples: IsComplete "amount" and ColumnLength "reference" <= 100. Rows with an empty amount are already caught by the regex rule (an empty string does not match), so the completeness rule overlaps; a long reference is a new rejection class. Reset the bookmark and re-run to apply the new rules to existing files (and remember it appends). The Python rules still catch IBAN check-digit failures, wrong signs by type, real calendar dates and retries with a new id, none of which DQDL as written expresses.
The job wrote some curated partitions with append, then failed before job.commit(), so the bookmark did not advance. The re-run read the same three files again and appended the rows again: duplicates for every file the first attempt had already written. Cleanup: identify the affected run's output (partition paths and file modified times, or the source_file column), delete the duplicate files or rewrite the affected partitions from raw, and reconcile counts against the source. Prevent it by making the write idempotent: write to a run-specific staging prefix and promote it on success, overwrite the partition for the dates in the file, MERGE into an Iceberg table, or record a run id and an audit record last, as the project job does.
Knowledge check
Answer: Which input data the job has already processed, per source (the transformation_ctx), so a later run reads only new data. It is saved by job.commit().
Answer: The bookmark did not advance, so the next run reads the same input again. Safe for reads, but writes that already happened are repeated unless they are idempotent.
Answer: IBAN mod-97 validation, sign rules by transaction type, real calendar validity, and idempotency-key retry detection are all outside what the ruleset expresses; any two are correct.
Answer: A versioned file is reviewed like code, deployed the same way everywhere and gives an audit trail of why a rule changed.
Interview questions
- "How do you process only new files in Glue?" โ Job bookmarks with a
transformation_ctxon S3 sources andjob.commit(), or an explicit file manifest or event-driven design where each file triggers a run. - "Bookmarks or idempotency?" โ Bookmarks track progress; idempotent writes make repeats safe. You usually need both.
- "Where does Glue Data Quality fit with custom validation?" โ Use DQDL for declarative, auditable checks and metrics, and code for rules DQDL cannot express. Quarantine bad rows with reason codes rather than failing the whole file.
Certification practice
A Glue job reads new CSV files hourly from an S3 prefix. After a code change the job re-reads every old file. The transformation_ctx string was renamed in the change. What is the cause?
A. The job role lost S3 access.
B. The bookmark is keyed by the transformation_ctx, so a new name has no history.
C. S3 deleted the old files.
D. Glue disabled bookmarks automatically.
Answer: B. The bookmark entry is stored under the source's context name. Renaming it starts a fresh history. A would cause failures, C would show no files, D does not happen.
A team needs failed records written to a separate S3 location with the reason, while valid records continue to the target. Which approach fits?
A. Fail the whole job on the first bad row. B. Evaluate row-level data-quality outcomes and route failing rows to a quarantine location. C. Ignore bad rows. D. Convert everything to string.
Answer: B. Row-level outcomes let the job split good from bad and keep the reasons. A stops all processing, C hides problems, D defers them.
Completion checklist
- I disabled the project's EventBridge rule for the lab and re-enabled it after
- I ran the job three times and explained each result
- I looked at the passing Parquet and the quarantined JSON Lines
- I can explain bookmark versus idempotency and what a reset does
- I compared DQDL with the Python validator and can name three gaps
- I cleaned up the data or destroyed the stack and ran the tag check
Key takeaways
- A bookmark remembers progress per
transformation_ctxand is saved only byjob.commit(). - Row-level data-quality outcomes let one job split good rows from bad and keep the broken rule names.
- DQDL is declarative and cheap but cannot express every banking rule; combine it with code.
- Bookmarks are not idempotency: reset plus append equals duplicates.
- Quarantine holds raw data; protect it like the raw zone.
Further reading
glue/jobs/transactions_dq_bookmarked.pyandglue/dq/transactions_raw.dqdlin the lab- Encrypting data written by AWS Glue
- Previous: Lab 7.3 ยท Next: Lab 7.5: Lake Formation analyst
Independent learning resource โ not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab 7.3: Athena on CSV versus Parquet, by bytes scanned
Upload the same transactions as CSV and as partitioned Parquet to S3, query both through a capped, encrypted Athena workgroup, and compare the bytes each query scans, with partition projection instead of a crawler.
Lab 7.5: an analyst who sees only GBP rows and no free-text columns
Register the curated bucket with Lake Formation, switch the transactions table from IAM-only access to Lake Formation permissions, create a data cells filter, and prove from the analyst's own session that the row and column restrictions hold.