Reporting and regulatory reporting architecture
Design a governed reporting pipeline on S3, Glue and Athena where every submitted figure can be reproduced, traced to its sources, approved by a second person and retained immutably, and verify that with a small C# manifest tool.
Assumes: .NET 10 LTS (file-based apps) · AWS SDK for .NET v4 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain why a regulatory report is a reproducible, approved, retained artefact and not just a query result.
- Design a reporting pipeline on S3, Glue, Athena and Step Functions with lineage, data-quality gates and a human approval step.
- Pin a report to exact source versions (Iceberg snapshots or object versions) so it can be rebuilt months later.
- Store the final report and its manifest in an immutable evidence location using S3 Object Lock.
- Apply the 15-section template to a reporting workload, including failure scenarios such as late data and resubmission.
- Mark every regulatory statement as needing review by the regulatory-reporting and compliance teams.
Prerequisites
- S3 data lake design and Amazon Athena
- Data quality and validation
- Lake Formation governance
- S3 protection, Object Lock and Macie
- How to read and write a banking architecture
- Tools: .NET 10 SDK or Docker for the manifest tool; AWS CLI v2 with profile
learn-sandboxonly for the optional deploy
Why this matters in banking
A regulator, an auditor or your own finance director asks: "Show me how you produced the liquidity figure you submitted in October, and prove it has not changed since." Three things go wrong in teams without a reporting architecture:
- The query changed. Someone "improved" the SQL in the console. Nobody can say which version produced the submitted numbers.
- The data changed. Late-arriving transactions were back-filled into the same partition. Re-running the query now gives a different total.
- The approval is a chat message. The person who built the report also approved it, and the evidence is a screenshot.
A reporting architecture removes all three by making the report a versioned, hashed, approved and retained artefact. This is also the part of the platform where finance, risk and compliance meet engineering, so the design document is read by people who do not read code.
Educational content, not regulatory advice
Harbourline Bank is fictional. Report names, deadlines, retention periods and sign-off rules in this lesson are illustrative. Which returns your bank must file, in what format, by when, and how long to keep them are decisions for your regulatory-reporting and compliance functions ⚠ VERIFY every regulatory detail locally. Nothing here claims that a design satisfies any reporting regime.
The idea in plain language
A regulatory report is a function with a signature: report(period, version, source snapshot, query) -> figures. If you fix all four inputs, you must always get the same figures. Everything in the architecture exists to fix those inputs and to record that you did:
- Period and version: "October 2026, submission 1" versus "October 2026, resubmission 2".
- Source snapshot: the exact state of the data, not "whatever is in the table today".
- Query: stored in Git, referenced by commit hash, never typed into a console.
- Output: hashed, approved by a different person than the preparer, then locked.
Analogy: a court exhibit. It has a reference number, a chain of custody (who handled it and when), a seal that shows tampering, and it goes in a locked store for a fixed period. A report without those properties is a rumour.
Reproducibility needs three kinds of pinning
| What to pin | Why | How on AWS |
|---|---|---|
| Data | Late data and corrections change tables | Apache Iceberg snapshot ID (Athena time travel), or S3 object version IDs for file extracts |
| Logic | SQL and transformations evolve | Git commit hash in the manifest; deployed through the pipeline, never edited in place |
| Reference data | Exchange rates, product mappings, counterparty lists change | Version the reference tables the same way; record their snapshot IDs too |
What problem the AWS services solve
| Need | Service | Honest limits |
|---|---|---|
| Governed storage | S3 + Lake Formation | Lake Formation permissions and S3 bucket policies are separate layers; test both |
| SQL over the lake without servers | Athena (engine v3) | Per-query data-scanned pricing; Iceberg time travel works only on Iceberg tables ⚠ VERIFY current feature set |
| Catalogue and schemas | Glue Data Catalog | A catalogue records structure, not business lineage |
| Data quality gates | AWS Glue Data Quality (DQDL rules) | Rules you write are only as good as your knowledge of the data |
| Orchestration with approval | Step Functions (Standard) with a task token | Execution history is kept 90 days by default, so copy what you need as evidence |
| Immutable retention | S3 Object Lock (Governance or Compliance mode) | Requires versioning; Compliance mode cannot be shortened by anyone, including root |
| Calendar triggers | EventBridge Scheduler | A schedule starts a run; it does not know whether the data is ready |
Not covered by AWS: the meaning of the report. Definitions such as "high-quality liquid asset" come from the regulator's instructions and your finance function, not from a service.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| SQL Server Reporting Services / SSIS packages | Athena + Glue jobs + Step Functions | No server to patch; a report run is a workflow execution |
| Azure Data Lake + Synapse serverless SQL | S3 + Athena | Same "query files in place" model; Athena bills per data scanned |
| Microsoft Purview (catalogue, lineage) | Glue Data Catalog, Lake Formation; lineage is mostly your manifest | Native column-level lineage is limited; build the manifest discipline yourself |
| Azure Immutable Blob Storage (WORM) | S3 Object Lock | Governance mode allows privileged override; Compliance mode does not |
| Logic Apps approval step | Step Functions .waitForTaskToken | Callback pattern: the state machine pauses until something calls SendTaskSuccess |
| Power BI for management reporting | Amazon Quick Sight | Quick Sight is fine for management dashboards; keep the regulatory submission path separate |
Architecture
The reporting pipeline
Security-boundary view
The pipeline role can write to the evidence bucket but cannot delete or overwrite. Approvers and auditors use separate roles. Putting evidence in its own account means a compromised reporting account cannot erase it.
The 15-section design
This is the design pack for Harbourline's fictional monthly liquidity return.
Sections 1 to 3: requirements
| Section | Content |
|---|---|
| 1 Business | Produce the monthly liquidity return by working day 5 with an auditable trail; support resubmission if an error is found; retire the spreadsheet process. Out of scope: the regulator's submission portal itself |
| 2 Functional | F1 Schedule a run per period. F2 Refuse to run if the source cut-off marker is missing. F3 Pin data and reference snapshots. F4 Block on failed data-quality rules. F5 Produce the report and a manifest. F6 Require approval by someone other than the preparer. F7 Publish immutably. F8 Allow a numbered resubmission that links to the original |
| 3 Non-functional | Reproducibility: re-run from manifest gives byte-identical output. Run time: under 60 minutes for the month. Availability: batch, so 99.5% of scheduled runs start on time. Retention: 7 years ⚠ VERIFY. RPO for evidence: 0 (acknowledged writes are durable). RTO: 1 business day, because the deadline is days away |
Section 5: service selection
| Choice | Why | Rejected alternative | Trade-off accepted |
|---|---|---|---|
| Athena on Iceberg | No cluster; time travel pins the snapshot | Redshift provisioned | Slower for very heavy joins; far less to operate for a monthly job |
| Step Functions Standard | Visible state, task token for approval, 1-year maximum duration | Airflow (MWAA) | Less flexible DAG tooling; no environment to patch |
| Object Lock in a separate account | Tamper resistance with separation of duties | Versioning only | Slightly harder teardown in the lab |
| Manifest JSON in the evidence bucket | Simple, portable, hash-based | A lineage product | Lineage is at run level, not column level |
Sections 6 and 7: network and security
- Network. Athena, Glue and Step Functions are regional managed services, so there is no VPC for the pipeline. Use gateway endpoint for S3 and interface endpoints where jobs run in a VPC (VPC endpoints). Block public access on every bucket.
- Identity. Separate roles:
report-pipeline(runs the workflow),report-approver(can only call the approval API),report-auditor(read evidence). The approver cannot be the principal that started the run: enforce this in the approval API, not by convention. - Keys. One KMS key per environment for evidence; key policy limits decrypt to the auditor and pipeline roles. See KMS and envelope encryption.
- Data minimisation. Reports are aggregates. Row-level extracts for supervisory queries go through a separate, logged, approved path (UK GDPR and data residency ⚠ VERIFY with your data-protection officer).
Sections 8 to 10: API, data and flow
- API. One small approval API:
POST /reports/{id}/approvalwith{ "decision": "approve|reject", "comment": "..." }. It validates that the caller differs from the preparer, then callsSendTaskSuccessorSendTaskFailure. - Database. A DynamoDB table
report_runs(keyreportId, sortversion) stores state, preparer, approver, manifest location. The lake holds data; DynamoDB holds workflow facts. - Data flow. (1) Scheduler starts run. (2) Readiness check reads a cut-off marker object. (3) Snapshot IDs are captured. (4) DQ gate runs. (5) CTAS writes staging output. (6) Manifest written. (7) Approval. (8) Copy to evidence with Object Lock retention.
Section 11: failure scenarios
| Failure | Detection | Automatic response | Customer or regulator impact | Manual step |
|---|---|---|---|---|
| Late-arriving source data | Cut-off marker missing at start | Run waits, retries on a timer, then alarms | None if resolved before deadline | Data owner confirms cut-off |
| Data-quality rule fails | Glue DQ result below threshold | Run stops before building | Report delayed | Fix source, re-run (new run ID, same version) |
| Approver absent | Task token times out after 2 days | Escalation notification | Deadline risk | Delegate approver per policy |
| Query changed since approval | Commit hash in manifest differs from deployed | Publish step refuses | None | Re-approve |
| Evidence write fails | Step Functions error, alarm | Retry with backoff | None | Investigate IAM or KMS |
| Error found after submission | Finance review | None; this is a process | Resubmission | Run version 2 linked to version 1 |
| Execution history expiry | 90 days after completion | Manifest and logs already copied | None | n/a |
Sections 12 to 15
- Scalability. Monthly volume is small. The first bottleneck is Athena concurrent query quotas ⚠ VERIFY and the 256 KiB state payload limit: pass S3 locations between states, never data.
- Cost. Athena scan cost scales with data read; partition by
booking_dateand use Parquet or Iceberg so a monthly query reads one month. Step Functions Standard bills per state transition, which is negligible at about 30 transitions a month. Evidence storage is tiny. A cost estimate must list Athena scan volume, S3 storage and KMS requests ⚠ VERIFY prices on the AWS pricing pages. - Implementation. Phase 1 evidence bucket and keys. Phase 2 one report end to end with synthetic data. Phase 3 approval API. Phase 4 parallel run against the legacy process for two cycles; reconcile differences before cut-over.
- Compliance touchpoints.
| Area | Where it touches the design | Reviewing team | Evidence |
|---|---|---|---|
| Regulatory return definitions and deadlines | Query logic, calendar | Regulatory reporting | Signed-off report specification |
| Record retention | Object Lock period | Compliance, legal | Retention schedule ⚠ VERIFY |
| Data protection | Aggregation level, extract access | Data protection officer | DPIA entry |
| Segregation of duties | Approver rule | Internal control | Role design and logs |
| Model and data governance | DQ rules, reference data owners | Data governance | Rule catalogue |
Step-by-step implementation
Create the evidence bucket with Object Lock
Object Lock must be enabled when the bucket is created in practice (see the AWS documentation for current guidance on enabling it later ⚠ VERIFY). Turn on versioning, KMS encryption and Block Public Access. Use Governance mode with a one-day default retention in this lab so you can clean up; production retention is a compliance decision.
Create an Athena workgroup with guard rails
Set an enforced result location (encrypted with your KMS key) and a per-query scan limit. This stops a runaway query from becoming a surprise bill, and keeps results out of personal buckets.
Store the query in Git and reference it by commit
The Step Functions definition receives queryCommit as an input parameter. The manifest records it. A pipeline step compares it with the deployed version.
Add the approval step
Use arn:aws:states:::sqs:sendMessage.waitForTaskToken or a Lambda that stores the token in DynamoDB. The approval API reads the token and completes the task. Set a timeout so a forgotten approval raises an alarm.
Write and verify the manifest
Compute SHA-256 hashes of inputs and output. Run the manifest tool in this lesson as a unit-level proof, then adapt it to run in the publish Lambda.
Rehearse a resubmission
Change a source figure, run version 2, and confirm version 1 evidence is untouched and version 2's manifest points back to it.
AWS Console instructions
- Console → S3 → Create bucket. Enter a name, leave Block all public access on, enable Bucket Versioning, choose SSE-KMS with your key, expand Advanced settings, enable Object Lock.
- Console → Athena → Workgroups → Create workgroup. Choose Athena SQL, set Query result configuration with the S3 path and KMS encryption, tick Override client-side settings, and set Data usage controls → Per query data usage control to a small limit.
- Console → Step Functions → State machines. Open the execution of a run to see each state's input and output.
AWS CLI
# Create the workgroup that every report query must use (results encrypted, scan cap enforced).
aws athena create-work-group --name hl-m14-reporting \
--configuration "ResultConfiguration={OutputLocation=s3://hl-m14-evidence-staging/athena/,EncryptionConfiguration={EncryptionOption=SSE_KMS,KmsKey=alias/hl-m14-reports}},EnforceWorkGroupConfiguration=true,BytesScannedCutoffPerQuery=10000000000" \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m14-l9 Key=Environment,Value=learn Key=Owner,Value=learner \
--profile learn-sandbox --region us-east-1
# Read an Iceberg table as it was at a point in time (Athena time travel).
aws athena start-query-execution --work-group hl-m14-reporting \
--query-string "SELECT sum(amount) FROM curated.transactions FOR TIMESTAMP AS OF TIMESTAMP '2026-11-03 00:00:00 UTC' WHERE booking_month = '2026-10'" \
--profile learn-sandbox --region us-east-1
# Apply a default retention (Governance mode, 1 day) to an Object Lock bucket.
aws s3api put-object-lock-configuration --bucket hl-m14-evidence-staging \
--object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":1}}}' \
--profile learn-sandbox --region us-east-1The time-travel syntax and the BytesScannedCutoffPerQuery option should be checked against aws athena create-work-group help and the Athena Iceberg documentation if a command is rejected ⚠ VERIFY.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.Athena;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.S3;
using Constructs;
namespace Harbourline.Architecture.Reporting;
public class ReportingEvidenceStack : Stack
{
public ReportingEvidenceStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var key = new Key(this, "EvidenceKey", new KeyProps
{
Alias = "alias/hl-m14-reports",
EnableKeyRotation = true,
});
var evidence = new Bucket(this, "EvidenceBucket", new BucketProps
{
Encryption = BucketEncryption.KMS,
EncryptionKey = key,
BucketKeyEnabled = true,
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
EnforceSSL = true,
Versioned = true,
ObjectLockEnabled = true,
// Learning shortcut: Governance + 1 day so the lab can be cleaned up.
ObjectLockDefaultRetention = ObjectLockRetention.Governance(Duration.Days(1)),
RemovalPolicy = RemovalPolicy.RETAIN,
});
new CfnWorkGroup(this, "ReportingWorkGroup", new CfnWorkGroupProps
{
Name = "hl-m14-reporting",
State = "ENABLED",
WorkGroupConfiguration = new CfnWorkGroup.WorkGroupConfigurationProperty
{
EnforceWorkGroupConfiguration = true,
BytesScannedCutoffPerQuery = 10_000_000_000d,
ResultConfiguration = new CfnWorkGroup.ResultConfigurationProperty
{
OutputLocation = $"s3://{evidence.BucketName}/athena-results/",
EncryptionConfiguration = new CfnWorkGroup.EncryptionConfigurationProperty
{
EncryptionOption = "SSE_KMS",
KmsKey = key.KeyArn,
},
},
},
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m14-l9");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "learner");
}
}Amazon.CDK.Tags.Of(this) avoids a naming clash with the stack's own Tags property (compiler error CS0176 otherwise).
Complete code
The tool below builds a report manifest and verifies it. It is the same logic you would run in the publish step: hash every input and the output, record who prepared and who approved, and refuse if the approver is the preparer or any file has changed. It has no AWS dependency, so you can run and test it locally.
// Run with: dotnet run ReportManifest.cs
// Builds a lineage manifest for a regulatory-report extract, then proves that
// tampering with an input or the output is detected.
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization.Metadata;
var work = Directory.CreateTempSubdirectory("report-demo").FullName;
var inputPath = Path.Combine(work, "transactions_2026-10.csv");
var reportPath = Path.Combine(work, "liquidity_return_2026-10.csv");
File.WriteAllText(inputPath, "account_id,amount\nACC-0000000001,100.00\nACC-0000000002,250.50\n");
File.WriteAllText(reportPath, "metric,value\ntotal_inflow,350.50\n");
var manifest = ReportManifest.Create(
reportId: "LIQ-2026-10",
reportVersion: 1,
queryText: "SELECT SUM(amount) FROM transactions WHERE booking_month = '2026-10'",
inputs: [inputPath],
output: reportPath,
preparedBy: "arn:aws:iam::111122223333:role/report-builder");
var signedOff = manifest with
{
SignedOffBy = "head.of.finance@example.com",
SignedOffAt = DateTimeOffset.Parse("2026-11-05T09:30:00Z"),
};
Console.WriteLine(JsonSerializer.Serialize(signedOff, new JsonSerializerOptions { WriteIndented = true, TypeInfoResolver = new DefaultJsonTypeInfoResolver() }));
Report("untouched files", ReportManifest.Verify(signedOff, [inputPath], reportPath));
File.AppendAllText(reportPath, "total_outflow,1.00\n"); // someone edits the output
Report("output edited after sign-off", ReportManifest.Verify(signedOff, [inputPath], reportPath));
var selfApproved = signedOff with { SignedOffBy = signedOff.PreparedBy };
Report("preparer approves own report", ReportManifest.Verify(selfApproved, [inputPath], reportPath));
Directory.Delete(work, recursive: true);
return 0;
static void Report(string label, IReadOnlyList<string> problems)
{
Console.WriteLine($"{label}: {(problems.Count == 0 ? "OK" : string.Join(" | ", problems))}");
}
public sealed record Manifest(
string ReportId,
int ReportVersion,
string QuerySha256,
Dictionary<string, string> InputSha256,
string OutputName,
string OutputSha256,
string PreparedBy,
DateTimeOffset PreparedAt,
string? SignedOffBy = null,
DateTimeOffset? SignedOffAt = null);
public static class ReportManifest
{
public static string Sha256File(string path) =>
Convert.ToHexString(SHA256.HashData(File.ReadAllBytes(path))).ToLowerInvariant();
public static string Sha256Text(string text) =>
Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(text))).ToLowerInvariant();
public static Manifest Create(string reportId, int reportVersion, string queryText,
IEnumerable<string> inputs, string output, string preparedBy) =>
new(reportId, reportVersion, Sha256Text(queryText),
inputs.ToDictionary(p => Path.GetFileName(p), Sha256File),
Path.GetFileName(output), Sha256File(output),
preparedBy, DateTimeOffset.UtcNow);
public static IReadOnlyList<string> Verify(Manifest m, IEnumerable<string> inputs, string output)
{
var problems = new List<string>();
foreach (var input in inputs)
{
var name = Path.GetFileName(input);
if (!m.InputSha256.TryGetValue(name, out var expected))
problems.Add($"input {name} not in manifest");
else if (Sha256File(input) != expected)
problems.Add($"input {name} changed");
}
if (Sha256File(output) != m.OutputSha256) problems.Add("output changed after manifest was written");
if (m.SignedOffBy is null) problems.Add("not signed off");
else if (m.SignedOffBy == m.PreparedBy) problems.Add("segregation of duties: preparer cannot sign off");
return problems;
}
}Expected output (hashes and the PreparedAt time vary):
untouched files: OK
output edited after sign-off: output changed after manifest was written
preparer approves own report: output changed after manifest was written | segregation of duties: preparer cannot sign offCode explained line by line
Manifestis an immutablerecord. Thewithexpression creates a modified copy, which models "the same manifest, now signed off" without mutating the original.InputSha256is keyed by file name so the verifier can find the expected hash for each input it is given.Sha256Filereads bytes and hashes them withSHA256.HashData. Lower-case hex is used so hashes compare as plain strings.Createhashes the query text too. In the real pipeline you would record the Git commit hash as well, since a hash of the SQL text proves identity but not provenance.Verifyreturns a list of problems rather than throwing on the first, so an auditor sees every issue at once.- The segregation-of-duties check compares approver and preparer identities. In production these are IAM Identity Center users or role session names, validated by the approval API, not typed strings.
- The
JsonSerializerOptionssetsTypeInfoResolverbecause .NET 10 file-based apps disable reflection-based JSON by default. A normal project does not need that line. - For a real submission, sign the manifest with an asymmetric KMS key (
kms:Sign) so tampering with the manifest itself is detectable ⚠ VERIFY the key spec and algorithm with your security team.
Security considerations
⚠ Learning shortcut — not for production
The lab uses Governance-mode Object Lock with one day of retention and keeps the evidence bucket in the same account as the pipeline. Production evidence normally lives in a separate, tightly controlled account, uses a retention period set by compliance, and may use Compliance mode, which nobody can shorten. Test retention periods with throw-away buckets first.
- Least privilege. The pipeline role has
s3:PutObjectands3:PutObjectRetentionon the evidence bucket and nothing else there. Nos3:DeleteObject*, nos3:BypassGovernanceRetention. - Encryption. KMS key per environment with a key policy that names the roles. Keep the key policy under change control; deleting the key makes evidence unreadable.
- Logging. CloudTrail data events for the evidence bucket, so every read of a report is attributable (CloudTrail for banks).
- Approval integrity. The approval API must derive the approver from the authenticated token, never from the request body.
- Personal data. Even aggregated reports can reveal individuals when a group is tiny. Ask the data-protection officer about minimum group sizes ⚠ VERIFY.
Banking use case
Harbourline's finance team currently builds the liquidity return in a spreadsheet from three exports. In the new design the monthly run starts on working day 2, waits for the cut-off marker, pins snapshots, builds the return, and emails the approver a link to a summary page. Two cycles run in parallel with the spreadsheet; differences are explained in a reconciliation log before the spreadsheet is retired. When finance finds a classification error on day 9, a resubmission run uses the same query commit plus a fix commit, produces version 2, and the manifest records supersedes: LIQ-2026-10 v1. The original evidence is untouched.
Common mistakes
- Reporting from the live table. Without a snapshot, a re-run months later disagrees. Pin snapshots.
- Approval by the preparer or by a shared account. Enforce difference of identities in code and prove it in logs.
- Putting data in Step Functions state. The 256 KiB limit is hard; pass S3 URIs.
- Setting retention by guesswork. Retention periods come from the retention schedule owned by compliance ⚠ VERIFY.
- Skipping the failed-run story. A run that stopped at the DQ gate is also evidence; keep its logs.
- Using Compliance-mode Object Lock in a lab. You cannot delete the objects until retention ends.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
InvalidBucketState when enabling Object Lock | Versioning not enabled, or bucket created without Object Lock | Enable versioning; create a new bucket with Object Lock enabled |
Athena HIVE_ACCESS_DENIED or Insufficient Lake Formation permission | Lake Formation grant missing for the workgroup role | Grant SELECT on the table to the role; check both IAM and Lake Formation |
| Query fails with data-scan limit exceeded | Workgroup cutoff reached, often a missing partition filter | Add the booking_month predicate or raise the limit deliberately |
KMS.AccessDeniedException on write | Key policy does not allow the role | Add the role to the key policy and check the encryption context |
| Task token never completes | Approval API called with a stale or truncated token | Store the token whole; tokens are long strings |
| Cannot delete an evidence object | Object Lock retention active | Wait, or use bypass permission in Governance mode only |
Cost and cleanup
This lesson is free if you only run the C# tool. The optional deploy uses an S3 bucket, a KMS key (about 1 USD per month, prorated ⚠ VERIFY), an Athena workgroup (pay per scan) and a few Step Functions transitions: well under 1 USD if torn down the same day. Object Lock retention delays deletion of objects by the retention period.
# Governance-mode objects need the bypass flag; the role needs s3:BypassGovernanceRetention.
aws s3api list-object-versions --bucket hl-m14-evidence-staging --profile learn-sandbox --region us-east-1
# Delete each version found above (repeat per version), then delete the bucket.
aws s3api delete-object --bucket hl-m14-evidence-staging --key <key> --version-id <id> \
--bypass-governance-retention --profile learn-sandbox --region us-east-1
aws s3api delete-bucket --bucket hl-m14-evidence-staging --profile learn-sandbox --region us-east-1
# Remove the workgroup and schedule the key for deletion (minimum waiting period is 7 days).
aws athena delete-work-group --work-group hl-m14-reporting --recursive-delete-option \
--profile learn-sandbox --region us-east-1
aws kms schedule-key-deletion --key-id alias/hl-m14-reports --pending-window-in-days 7 \
--profile learn-sandbox --region us-east-1
# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: run
ReportManifest.cs, then add a second input file and makeVerifyreport when an input is listed in the manifest but not supplied. - 🟡 Challenge: extend the manifest with
SupersedesReportVersionand a rule that a resubmission must reference an existing earlier version. Write three test cases. - 🔴 Live problem: finance reports that October's figure, re-run in December, is 0.4 percent different from the submitted value. Nothing was edited. List five possible causes in order of likelihood, the evidence that would confirm each, and the design change that prevents it.
Assignment
Add architecture/regulatory-reporting/ to your portfolio repository: the 15-section document for a report of your choice (fictional), a Mermaid pipeline diagram, a failure table with at least seven rows, an ADR choosing Athena over Redshift for this workload, and the manifest tool with unit tests.
Solutions
Loop over the manifest keys as well as the supplied inputs:
var supplied = inputs.Select(Path.GetFileName).ToHashSet();
foreach (var listed in m.InputSha256.Keys.Where(k => !supplied.Contains(k)))
problems.Add($"input {listed} is in the manifest but was not supplied");Why: verification must fail closed. If a verifier checks only the files it is handed, someone can drop an input and still pass.
Add int? SupersedesVersion to Manifest. In the publish step, if ReportVersion > 1 require SupersedesVersion == ReportVersion - 1 and check that the earlier evidence object exists. Tests: version 1 with no supersedes passes; version 2 superseding 1 passes; version 3 superseding 1 fails.
Why: a resubmission chain that can skip versions hides history.
Likely causes: (1) the source table was queried live and late data arrived (pin snapshots); (2) reference data such as exchange rates changed (version it); (3) the SQL was edited (commit hash); (4) a currency rounding rule changed in the engine or code (pin versions; compare with decimal rules); (5) time-zone handling of booking_date at month boundaries. Evidence: compare the snapshot IDs and commit hashes in the October manifest with today's; time-travel the table to the manifest snapshot and re-run. If the re-run matches the submission, the cause is data or logic drift, which pinning prevents.
Knowledge check
Answer: It cannot show which data version, which query and which approver produced it, and it can change when late data arrives. Evidence needs pinned inputs, a recorded query, a hash of the output, approval and immutable retention.
Answer: It requires bucket versioning. Governance mode can be overridden by principals with a bypass permission; Compliance mode cannot be shortened or removed by anyone until retention expires.
Answer: State input and output are limited to 256 KiB. Large payloads fail the execution, and data in state history is harder to control and encrypt.
Answer: By reading the exact table snapshot recorded in the manifest (Iceberg time travel) with the recorded query commit and reference-data versions.
Answer: In a separate account with write-only access for the pipeline, so a compromise of the reporting account cannot delete or alter evidence.
Interview questions
- How do you make a report reproducible? Pin data snapshots, query commit and reference-data versions, hash the output, store everything in an immutable manifest.
- How do you enforce four-eyes approval in a pipeline? A callback step whose approval API derives the approver from the authenticated identity and rejects the preparer, with both identities logged.
- Athena or Redshift for regulatory reporting? Depends on volume, concurrency and latency. For a monthly, modest-volume job Athena avoids a cluster; for heavy, concurrent analytics Redshift may win. Record the choice in an ADR.
- How do you handle a resubmission? A new version that links to the original; original evidence is never modified.
- Is a report built with this design compliant? That is not a claim engineering can make; regulatory reporting and compliance review the definitions, controls and retention.
Certification practice
A bank must retain monthly report packages so that no administrator can delete or alter them for seven years, while the reporting team still writes new packages. Which design meets this best? ⚠ VERIFY exact period locally.
A. A versioned S3 bucket in the reporting account with a bucket policy denying deletes. B. An S3 bucket in a separate account with Object Lock in Compliance mode, a retention period set by policy, and write-only access for the pipeline role. C. Daily snapshots of an EBS volume copied to another Region. D. Store the reports in DynamoDB with point-in-time recovery.
Answer: B. Compliance mode prevents shortening or deleting, and a separate account limits blast radius. A can be undone by a policy change. C and D are backups, not tamper-proof retention.
A monthly report queries 4 TB of CSV files in S3 with Athena and is slow and costly. What is the most effective change?
A. Increase the Athena workgroup concurrency. B. Convert the data to partitioned Parquet or Iceberg and filter on the partition column. C. Move the files to S3 Glacier. D. Use a larger EC2 instance for the client.
Answer: B. Athena cost and time track data scanned; columnar formats and partition pruning cut it. A does not reduce scanning, C makes access slower, D is irrelevant because the query runs in the service.
Which control best prevents a compromised pipeline role from erasing submitted reports?
A. Enable S3 server access logging.
B. Grant the pipeline only s3:PutObject, store evidence in a separate account with Object Lock, and deny delete actions with a service control policy.
C. Encrypt reports with SSE-S3.
D. Add a lifecycle rule to Glacier.
Answer: B. It removes the ability to delete or change retention. A detects but does not prevent. C protects confidentiality, not integrity. D moves storage class but does not stop deletion.
Completion checklist
- I can explain the four inputs that make a report reproducible
- I ran the manifest tool and saw all three verification results
- I wrote a failure table including late data and resubmission
- My compliance section says "requires review by" for every regulatory point
- I ran the cleanup steps if I deployed anything and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- A regulatory report is a versioned, hashed, approved, retained artefact, not a query result.
- Pin data, logic and reference data so a re-run reproduces the submission.
- Separate preparer from approver in code, and store evidence in a separate account with Object Lock.
- Pass references, not data, between workflow steps.
- Regulatory definitions, deadlines and retention periods need specialist review.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Customer notification systems: fan-out, preferences, retries and duplicate suppression
Design an event-driven notification platform for a bank: planning notifications from domain events, respecting preferences and quiet hours, fanning out to e-mail, SMS and push through per-channel queues, retrying safely, suppressing duplicates and recording delivery evidence.
Reconciliation architecture
Design an automated reconciliation of Harbourline's internal ledger against partner and scheme files: idempotent file ingestion, control totals, a tiered matching engine, break handling with an audit trail, and a runnable C# matcher.