Validation and data quality: rules, quarantine and reconciliation
Build Harbourline's data-quality gate: row-level rules with reason codes in Python and C#, a Glue Data Quality (DQDL) ruleset, a quarantine zone with a reject report, control totals that prove nothing was lost, and an audit record for every run.
Assumes: .NET 10 LTS · Python 3.11 / PySpark 3.5 · AWS Glue 5.0 Data Quality (DQDL) · xUnit v3 4.0.3 · pytest 8.4.2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Classify data-quality problems (schema, completeness, validity, uniqueness, consistency, timeliness) and decide for each whether to reject, repair or warn.
- Write row-level validation rules that emit reason codes, and keep a Python and a C# implementation in step by grading both against the same answer key.
- Write a Glue Data Quality ruleset in DQDL, evaluate it in a Glue job, and split passing and failing rows.
- Design a quarantine zone and a reject report that operations can act on.
- Prove completeness with control totals and reconciliation, and record each run in an audit trail.
Prerequisites
- Module 7 overview, File formats, PySpark in Glue
- Lab 7.1 — the messy dataset and its answer key
- Tools: Docker (for .NET and Spark tests); no AWS account needed for most of this lesson
Why this matters in banking
A partner payroll file arrives with 40 salary payments. Line 17 has an IBAN with a typo in the check digits. Line 23 says 1,433.12 with a thousands separator. Line 31 is a copy of line 30. If the pipeline loads the file as is, a salary goes to a non-existent account, an amount becomes 1 or NULL, and a person is paid twice. If the pipeline rejects the whole file, 39 people are not paid on time.
Banks need a third path: accept what is valid, quarantine what is not with a precise reason, report the rejects to someone who can fix them, and prove — with numbers — that every row and every penny is accounted for. That proof is what auditors and finance mean by reconciliation.
The idea in plain language
Data quality is "is this data fit for its purpose?". In practice it is a list of rules and what to do when a rule fails:
| Dimension | Example rule (transactions) | Typical action |
|---|---|---|
| Schema | Header has the 15 expected columns | Reject the file |
| Completeness | transaction_id, account_id, idempotency_key not empty | Reject the row |
| Validity | booking_date is a real ISO date; currency in GBP/EUR; IBAN passes mod-97 | Reject the row |
| Consistency | transfer_in amounts positive; card payments negative | Reject the row |
| Uniqueness | No repeated transaction_id or idempotency_key in a file | Reject the duplicate |
| Repairable format | UTF-8 text read as Latin-1 (Café) | Repair, keep a warning |
| Timeliness | Created more than a day after the value date | Accept, warn (LATE_ARRIVING) |
A reason code is a short, stable code (INVALID_IBAN, DUPLICATE) attached to every rejected row. Codes can be counted, alerted on, trended and translated into partner-facing messages; free-text error messages cannot.
Control totals are numbers both sides agree on: row counts and sums per currency. If the sender says "1,036 rows, £97,784.91" and you have 969 accepted plus 67 quarantined rows whose amounts add up to the same total, nothing was lost. If not, you have a break.
Analogy: a bank's cash-in-transit check. Every bag is counted at the branch and again at the centre. Bags with torn seals go to a separate cage (quarantine) with a form saying why, but they are still counted. The centre signs off only when "good bags + caged bags = bags sent".
What problem the AWS service solves
AWS Glue Data Quality evaluates rules written in DQDL (Data Quality Definition Language) against Glue Data Catalog tables or inside Glue ETL jobs. It can:
- compute a score and per-rule outcomes, and publish them to the Glue console and CloudWatch metrics;
- in ETL jobs, return row-level outcomes: each row gets the lists of rules it passed and failed, so you can route failures to quarantine;
- recommend a starter ruleset from data statistics, and detect anomalies over time with analyzers.
It is built on the open-source library Deequ (Amazon's "unit tests for data" on Spark). Its limits: DQDL rules are generic — it cannot express an IBAN mod-97 check or "duplicate by idempotency key within this file but not across files" without custom SQL, and custom logic is harder to unit test than code. That is why this module uses two lines of defence: tested code for business rules with reason codes, and DQDL as an independent check on the output.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| SSIS conditional split + error output | Glue job splitting rows by rule outcome | Row-level outcomes come from EvaluateDataQuality or your own code |
| ADF data flow assert transformation | Glue Data Quality rules in an ETL job | DQDL is a declarative rule language |
| Microsoft Purview data quality | Glue Data Quality (catalog rulesets, scores) + SageMaker Catalog / DataZone quality display ⚠ VERIFY integration naming | Rulesets attach to Glue tables |
| Great Expectations / custom C# validators | Deequ, PyDeequ, or your own code | Same concepts: checks, constraints, metrics |
SQL Server CHECK constraints | Validation in the pipeline | The lake has no constraints: files accept anything |
Architecture
Security-boundary view. The quarantine bucket holds raw values (including clear IBANs) because operators need them to fix and replay rows. It is therefore protected like the raw zone: separate bucket, KMS, versioning, access only for the pipeline role and a small operations group, short retention. Curated never contains the clear IBAN.
The rules in code
The lab's rules live in labs/m07-transaction-pipeline/glue/src/harbourline_etl/transactions.py, with an identical C# implementation in src/Harbourline.DataPipeline.Core/Validation/TransactionValidator.cs. Both are graded against transactions_messy_defects.csv, the generator's answer key.
| Reason code | Severity | Rule |
|---|---|---|
SCHEMA_COLUMN_COUNT | reject | Row does not have 15 fields |
MISSING_REQUIRED | reject | transaction_id, account_id, idempotency_key, type, channel, status present |
INVALID_ID | reject | transaction_id is a lower-case UUID; account_id is ACC- + 10 digits |
DUPLICATE | reject | transaction_id or idempotency_key seen earlier in the file |
INVALID_DATE | reject | booking_date, value_date match yyyy-MM-dd and are real dates |
INVALID_TIMESTAMP | reject | created_at is yyyy-MM-ddTHH:mm:ssZ and real |
INVALID_AMOUNT | reject | Decimal with up to 2 places that fits DECIMAL(18,2) |
INVALID_CURRENCY | reject | GBP or EUR exactly |
INVALID_CODE | reject | type, channel, status in their enums; MCC is 4 digits |
AMOUNT_SIGN | reject | transfer_in > 0; debit types < 0 |
INVALID_IBAN | reject | Counterparty IBAN passes ISO 13616 mod-97 when present |
ENCODING_INVALID | reject | Text looks double-encoded and cannot be repaired |
ENCODING_REPAIRED | warn | A Latin-1 line or mojibake was repaired |
LATE_ARRIVING | warn | created_at more than 1 day after value_date |
Notice the choices: duplicates are rejected (paying twice is worse than paying late), encoding is repaired (the data is right, only the bytes are wrong), late data is accepted (it is real, but reports for past dates must be re-run).
Step-by-step implementation
Grade the rules against the answer key (free)
Run the pytest suite and the xUnit suite. The key test asserts that every injected defect is found and that no clean row is flagged.
Read the reject report
hl-pipeline validate data/samples/transactions_messy.csv --quarantine rejects.csv prints counts by reason code and writes one line per rejected row.
Write the DQDL ruleset
glue/dq/transactions_raw.dqdl checks completeness, uniqueness, formats and allowed values on raw strings. It is the Spark-native gate used in Lab 7.4.
Evaluate in Glue and split rows
In the job, EvaluateDataQuality().process_rows(...) returns ruleOutcomes and rowLevelOutcomes; rows with DataQualityEvaluationResult = 'Passed' go to curated, others to quarantine with the failed rule names.
Reconcile and audit
Compute counts and sums per currency for accepted and quarantined rows, compare with the sender's control figures, write gold.reconciliation, and write the audit record last.
AWS Console instructions
- AWS Glue → Data Catalog → Tables → hl_m07_curated.transactions → Data quality → Create data quality rules. Choose Recommend rules to see what Glue proposes from the data, then compare with your ruleset; paste your DQDL and Save ruleset.
- Run the ruleset against the table with the lab's IAM role. Open the run to see each rule's outcome and the overall score.
- In Glue Studio, a visual job can add the Evaluate Data Quality transform; under its options choose to output original data with row-level results to route failures. The lab's script does the same in code.
- CloudWatch → Metrics → Glue Data Quality to chart pass rates over time; create an alarm when the rule pass rate drops below 100%.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# Create a ruleset attached to the curated table (a compact version of the job's curated checks).
aws glue create-data-quality-ruleset --name hl-m07-curated-transactions \
--ruleset 'Rules = [ RowCount > 0, IsComplete "transaction_id", IsUnique "transaction_id", ColumnValues "currency" in ["GBP","EUR"] ]' \
--target-table TableName=transactions,DatabaseName=hl_m07_curated \
--tags Project=cea-lab,Lab=m07-l4,Environment=learn,Owner=learner
# Run it against the catalog table.
RUN=$(aws glue start-data-quality-ruleset-evaluation-run \
--data-source '{"GlueTable":{"DatabaseName":"hl_m07_curated","TableName":"transactions"}}' \
--role arn:aws:iam::111122223333:role/hl-m07-glue-job-role \
--ruleset-names hl-m07-curated-transactions --query RunId --output text)
aws glue get-data-quality-ruleset-evaluation-run --run-id "$RUN" --query '{Status:Status,Results:ResultIds}'
aws glue get-data-quality-result --result-id <result-id> --query 'RuleResults[].{Rule:Name,Result:Result}'Infrastructure as Code
using Amazon.CDK.AWS.Glue;
_ = new CfnDataQualityRuleset(this, "CuratedRuleset", new CfnDataQualityRulesetProps
{
Name = "hl-m07-curated-transactions",
Description = "Second-line checks on curated transactions",
Ruleset = File.ReadAllText(Path.Combine(labRoot, "glue", "dq", "transactions_raw.dqdl")),
TargetTable = new CfnDataQualityRuleset.DataQualityTargetTableProperty
{
DatabaseName = "hl_m07_curated",
TableName = "transactions",
},
});Complete code
The DQDL ruleset
Rules = [
RowCount > 0,
IsComplete "transaction_id",
IsComplete "account_id",
IsComplete "idempotency_key",
IsUnique "transaction_id",
ColumnValues "account_id" matches "ACC-[0-9]{10}",
ColumnValues "booking_date" matches "[0-9]{4}-[0-9]{2}-[0-9]{2}",
ColumnValues "value_date" matches "[0-9]{4}-[0-9]{2}-[0-9]{2}",
ColumnValues "created_at" matches "[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z",
ColumnValues "amount" matches "-?[0-9]{1,16}(\.[0-9]{1,2})?",
ColumnValues "currency" in ["GBP", "EUR"],
ColumnValues "type" in ["card_payment", "transfer_in", "transfer_out", "direct_debit", "standing_order", "atm_withdrawal", "fee", "interest"],
ColumnValues "channel" in ["mobile", "web", "branch", "card", "api", "batch"],
ColumnValues "status" in ["booked", "pending", "reversed", "rejected"]
]Splitting rows by DQ outcome in a Glue job (from glue/jobs/transactions_dq_bookmarked.py)
results = EvaluateDataQuality().process_rows(
frame=raw_with_file,
ruleset=ruleset,
publishing_options={
"dataQualityEvaluationContext": "hl_m07_raw_check",
"enableDataQualityCloudWatchMetrics": True,
"enableDataQualityResultsPublishing": True,
},
additional_options={"performanceTuning.caching": "CACHE_NOTHING"},
)
rows = SelectFromCollection.apply(dfc=results, key="rowLevelOutcomes", transformation_ctx="rows").toDF()
dq_cols = ["DataQualityRulesPass", "DataQualityRulesFail", "DataQualityRulesSkip", "DataQualityEvaluationResult"]
passed = rows.filter(F.col("DataQualityEvaluationResult") == "Passed").drop(*dq_cols)
failed = rows.filter(F.col("DataQualityEvaluationResult") != "Passed")
failed.withColumn("reason_codes", F.concat_ws(",", "DataQualityRulesFail")) \
.write.mode("append").json(f"s3://{ARGS['quarantine_bucket']}/dq_rejects/")The C# test that grades the validator against the answer key
[Fact]
public void Every_injected_defect_is_found_and_nothing_else()
{
var rows = Samples.Validate("transactions_messy.csv");
// Answer key: row_number,key,defect,column,original_value,messy_value,note
var expected = File.ReadAllLines(Samples.Path("transactions_messy_defects.csv")).Skip(1)
.Select(CsvLine.Parse)
.GroupBy(f => int.Parse(f[0], CultureInfo.InvariantCulture))
.ToDictionary(g => g.Key, g => g.Select(f => f[2]).ToHashSet(StringComparer.Ordinal));
var found = rows.Where(r => r.Issues.Count > 0)
.ToDictionary(r => r.RowNumber, r => r.Issues.Select(i => i.Defect).ToHashSet(StringComparer.Ordinal));
var missed = expected.Where(e => !found.TryGetValue(e.Key, out var f) || !e.Value.IsSubsetOf(f)).Select(e => e.Key).ToList();
var unexpected = found.Keys.Except(expected.Keys).ToList();
Assert.Empty(missed); // recall: every planted defect is caught
Assert.Empty(unexpected); // precision: no clean row is flagged
}
[Fact]
public void Messy_file_control_totals_match_the_python_job()
{
var rows = Samples.Validate("transactions_messy.csv");
Assert.Equal(1036, rows.Count);
Assert.Equal(969, rows.Count(r => r.Accepted));
Assert.Equal(67, rows.Count(r => !r.Accepted));
Assert.Equal(124187.38m, rows.Where(r => r.Curated?.Currency == "GBP").Sum(r => r.Curated!.Amount));
}Reconciliation (from glue/src/harbourline_etl/reports.py)
def reconcile(rows, source_file, run_id, control=None, currencies=("EUR", "GBP")):
accepted_count = sum(1 for r in rows if r.accepted)
quarantined_count = len(rows) - accepted_count
source_count = control.record_count if control else len(rows)
counts_ok = source_count == accepted_count + quarantined_count
lines = []
for cur in currencies:
accepted_total = sum((Decimal(str(r.curated["amount"])) for r in rows
if r.accepted and r.curated and r.curated["currency"] == cur), Decimal("0.00"))
quarantined_total = sum((_raw_amount(r) for r in rows
if not r.accepted and r.raw.get("currency") == cur), Decimal("0.00"))
source_total = (control.control_sums.get(cur, Decimal("0.00")) if control else
sum((_raw_amount(r) for r in rows if r.raw.get("currency") == cur), Decimal("0.00")))
difference = source_total - accepted_total - quarantined_total
lines.append({"currency": cur, "source_row_count": source_count,
"accepted_row_count": accepted_count, "quarantined_row_count": quarantined_count,
"source_control_total": source_total, "accepted_amount_total": accepted_total,
"quarantined_amount_total": quarantined_total, "difference": difference,
"status": "MATCHED" if counts_ok and difference == 0 else "BREAK"})
return linesCode explained line by line
DQDL
Rules = [ ... ]— a ruleset is a list of rules; each rule has a type, a target column and a condition.RowCount > 0— dataset-level: an empty file is suspicious (did the partner send the header only?).IsComplete "transaction_id"— no nulls/empties in the column. Row-level outcomes mark the offending rows.IsUnique "transaction_id"— no repeated values. It cannot see duplicates byidempotency_keywith a new id — the Python rule can.ColumnValues "booking_date" matches "..."— regular expression on a string column. It cannot tell that2026-02-30is not a real date; the code rule does.ColumnValues "currency" in [...]— allowed values. Lower-casegbpfails, which is what we want.
Row split in Glue
process_rows— evaluates the ruleset and returns a collection withruleOutcomes(one row per rule) androwLevelOutcomes(every input row with outcome columns).publishing_options— results appear in the Glue console under the evaluation context name and as CloudWatch metrics for alarms.DataQualityEvaluationResult—PassedorFailedfor each row;DataQualityRulesFaillists the rules that row broke, which we turn into its reason codes.- Dataset-level rules such as
RowCountdo not mark individual rows; they appear only inruleOutcomes.
C# grading test
- Reads the answer key, groups by
row_number, and builds the set of expected defects per row. foundmaps each flagged row to the defect names derived from its reason codes (ReasonCodes.ToDefect).missedchecks recall: every planted defect is found.unexpectedchecks precision: no clean row is flagged. Both must be empty — a validator that rejects everything would pass a recall-only test.- The control-total test uses the same numbers as the Python test, so the two implementations cannot drift apart silently.
Reconciliation
counts_ok— every row read must be either accepted or quarantined.- Totals use
Decimal, neverfloat:0.1 + 0.2must be0.3in a bank. quarantined_totaluses the raw amount when it parses, so a row rejected for a bad IBAN still contributes its money to the reconciliation.- With a
control(from the sender), the source figures come from outside the pipeline — the strong check. Without one, the job reconciles against its own reading of the file, which proves only that nothing was lost inside the job. - A row with currency
gbpis rejected and belongs to no allowed currency, so a sender's GBP control total will show a break of exactly that amount — correct behaviour: someone must look at it.
Security considerations
⚠ Learning shortcut — not for production
The lab quarantine stores the full raw record as JSON so you can see what failed. In production, consider storing only the row number, reason codes and a pointer to the raw object version (which is already retained in the raw zone), so PII is not copied a second time; restrict who can read quarantine and expire it quickly once rows are resolved.
- Reject reports go to people outside the data team (operations, partners). Send reason codes and row numbers, not full records, and never e-mail clear IBANs.
- Do not log rejected values in CloudWatch. Log counts by reason code.
- DQ results are evidence. Keep the published results and the audit records for the period your audit policy requires.
- Rules are code: review them, version them, test them; a silently weakened rule is a control failure.
Banking use case
Every file that lands in Harbourline's raw zone produces: a curated partition with accepted rows; a quarantine object with rejected rows and reason codes; a reconciliation line per currency; and an audit record. At 06:00 the operations team receives the list of breaks and the count of rejects by reason and partner. Saltmarsh Payroll gets an automated reply listing line numbers with INVALID_IBAN, and corrected lines come back as a new file with a new batch id. Month-end, finance signs off only runs with status MATCHED or with a documented explanation for each break.
Common mistakes
- Silently dropping bad rows → numbers do not add up and nobody knows why. Quarantine with reason codes and reconcile.
- Rejecting a whole file for one bad row → valid payments are delayed. Reject rows; reject files only for schema problems.
- Validation rules only in DQDL → no unit tests and no business-specific checks. Keep business rules in tested code; use DQDL as a second line.
- Floating-point money in checks → off-by-a-penny breaks. Use
decimal/Decimal. - No precision check in tests → a validator that rejects everything "passes". Test both directions.
- Treating late data as bad data → real transactions lost. Accept, flag, recompute affected reports.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Recall test fails for bad_date rows | Regex accepts 2026-02-30 | Parse the date after the regex (date.fromisoformat / DateOnly.TryParseExact) |
Precision test flags clean rows with ENCODING_REPAIRED | Mojibake regex too broad | Match only Ã/Â followed by U+0080–U+00BF |
| C# and Python totals differ by pennies | One side used double/float | Use decimal and Decimal end to end |
rowLevelOutcomes missing in Glue | Using apply (rule outcomes only) instead of process_rows | Call process_rows and select rowLevelOutcomes |
DQ run fails with AccessDenied | Role lacks permissions to publish results or read the table | Add glue:PublishDataQuality and table/S3 read |
Reconciliation BREAK with difference equal to one row's amount | That row's currency code is invalid (for example gbp) | Expected; resolve via the reject report |
Cost and cleanup
Everything in this lesson except running rulesets in AWS is free: tests run in Docker. Glue Data Quality runs bill as Glue DPU time (USD 0.44 per DPU-hour in us-east-1, with a minimum per run) ⚠ VERIFY the current minimum billing for data quality runs on the Glue pricing page. A single evaluation on the lab table costs cents.
aws glue delete-data-quality-ruleset --name hl-m07-curated-transactions --profile learn-sandboxExercises
- 🟢 Guided: Run both test suites. Then break one rule on purpose (for example allow
gbp) and watch which test fails and why. - 🟡 Challenge: Add a rule
FUTURE_DATED: reject rows whosebooking_dateis more than 5 days aftercreated_at. Implement it in Python and C#, add the code to both reason-code tables, and extend both tests with a hand-made row. - 🔴 Live problem: The partner says "we sent 40 payments totalling £89,281.75" (the trailer of
HLPG_PTN0042_PAYMENTS_20261116_001_messy.csv). Your pipeline accepted 37 and quarantined 2 — and the totals do not match. Explain the possible causes and how you would demonstrate the reconciliation to the partner.
Assignment
Write a data-quality specification for the payments dataset: dimensions, rules, reason codes, severities and actions, DQDL for what DQDL can express, and the tests you would write. Use payments_messy_defects.csv as your answer key.
Solutions
Allowing gbp makes the recall test fail: the rows planted with wrong_currency are no longer flagged, so missed contains their row numbers. The control-total test also fails because more rows are accepted (the accepted GBP total no longer equals 124,187.38 — and gbp rows end up in neither currency bucket in curated). This is the value of graded tests: a "small, helpful" change to a rule shows up immediately as a change in recall and totals, and is visible in review.
Python: after parsing booking and created, if booking and created and (booking - created.date()).days > 5: row.reject("FUTURE_DATED", "booking_date", "booked more than 5 days after creation"). C#: the same with booking.Value.DayNumber - DateOnly.FromDateTime(created.Value).DayNumber > 5. Add "FUTURE_DATED": "future_dated" to both defect maps. Because the answer key has no such defect, the precision test must still pass on the sample (no clean row is future-dated); add a separate test with a synthetic row to prove the rule fires. Keeping both implementations in step is the discipline the twin-test design enforces.
Counts: 37 + 2 = 39, not 40 — one line was lost or is a duplicate the partner counted (the messy SFTP file contains a duplicated line, so the trailer no longer matches the lines either). Amounts: a line with 1,433.12 (thousands separator) cannot be parsed, so its amount is missing from both accepted and quarantined parseable totals. Demonstration: produce the reconciliation per line — line numbers accepted, line numbers rejected with reason codes and raw amounts, and the arithmetic showing source total − accepted − quarantined = the unparsed amount. Ask the partner to correct and resend those lines with a new batch id. The point is that every penny is explained, not that every penny is accepted.
Knowledge check
Answer: Warnings (repaired encoding, late arrival) mean the data is correct and usable but notable; rejecting them would lose real transactions. They are stored in dq_warnings so reports can filter or recompute.
Answer: Recall alone rewards rejecting everything; precision alone rewards accepting everything. A useful validator catches every defect and flags nothing else.
Answer: That every row and every amount the sender sent is accounted for in accepted plus quarantined data. With sender-provided figures they prove the whole chain; with self-computed figures only the job's internals.
Answer: Domain checks like IBAN mod-97, real-calendar date validity, sign rules per transaction type and duplicate detection by idempotency key. Those belong in tested code.
Interview questions
- "How do you handle bad data in a pipeline?" — Classify by dimension; reject rows with reason codes to a quarantine zone; repair only when safe and record a warning; reconcile counts and totals; alert on breaks; report rejects to the owner.
- "How do you know your validation is right?" — Graded tests against a dataset with known, planted defects; both recall and precision; the same tests across implementations.
- "What is Deequ / Glue Data Quality?" — Declarative data checks on Spark; Glue Data Quality uses DQDL, publishes scores and can mark rows for routing.
- "What is reconciliation?" — Proving with independent control figures that the output equals the input, minus documented exceptions.
Certification practice
A Glue ETL job must load valid rows into a curated table and write invalid rows, with the names of the failed rules, to a separate S3 location. The rules check completeness and allowed values. What is the most efficient approach?
A. Run a Glue crawler after loading and delete invalid rows with Athena.
B. Use the EvaluateDataQuality transform with row-level outcomes in the job, filter on the evaluation result, and write each subset to its location.
C. Load everything and ask analysts to filter invalid rows in their queries.
D. Run a separate data quality ruleset on the catalog table once a week.
Answer: B. Row-level outcomes route failures in the same job. A loads bad data first. C pushes the problem to users. D is too late and does not route rows.
A team wants an alert when the share of passing data-quality rules for a daily load falls below 100%. Which solution needs the least custom code?
A. Parse Glue job logs with a Lambda function. B. Enable publishing of data-quality results to CloudWatch metrics and create a CloudWatch alarm. C. Query the curated table in Athena every hour. D. Ask the job to e-mail a CSV of results.
Answer: B. Published metrics plus an alarm. A, C and D all require custom code and are fragile.
Rejected rows in a quarantine prefix contain full raw values, including bank account numbers. What is the best way to protect them?
A. Store quarantine in the same bucket as curated data so analysts can debug. B. Use a separate bucket encrypted with a customer managed KMS key, with access limited to the pipeline role and an operations group, and a short lifecycle expiration. C. Make the quarantine prefix public-read so partners can download their rejects. D. Convert quarantine files to CSV.
Answer: B. Separate, encrypted, least-privilege, short-lived. A exposes raw PII to analysts. C is a data breach. D changes format, not protection.
Completion checklist
- I can name the data-quality dimensions and the action for each
- I ran both test suites and understand recall and precision
- I can read and write a DQDL ruleset and know its limits
- I can explain a reconciliation line, including a break
- I deleted any rulesets I created in AWS
Key takeaways
- Validate rows, not files; reject with stable reason codes; repair only when safe and warn.
- Grade validators against a dataset with planted defects — both recall and precision — and keep twin implementations in step with shared tests.
- Use Glue Data Quality (DQDL) as an independent second line and for metrics and alarms.
- Quarantine is sensitive: protect it like raw.
- Reconcile counts and totals with independent control figures; write the audit record last.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Change data capture from SQL Server with AWS DMS
Stream inserts, updates and deletes from Harbourline's SQL Server core-banking tables into the S3 data lake with AWS DMS full load plus CDC, then merge them into a current-state table that copes with late and out-of-order changes.
Governance, lineage and cataloguing with Lake Formation
Govern Harbourline's data lake: register S3 locations with AWS Lake Formation, replace coarse IAM access with table, column, row and tag-based permissions, give a UK analyst only GBP rows without free-text columns, share safely across accounts, and record ownership and lineage.