Governance, lineage and cataloguing with Lake Formation
Govern Harbourline's data lake: register S3 locations with AWS Lake Formation, replace coarse IAM access with table, column, row and tag-based permissions, give a UK analyst only GBP rows without free-text columns, share safely across accounts, and record ownership and lineage.
Assumes: AWS Lake Formation · AWS Glue Data Catalog · Amazon Athena engine v3 · AWS SDK for .NET v4 (AWSSDK.LakeFormation 4.1.0.2) · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain why bucket-level IAM is too coarse for a bank's data lake, and what Lake Formation adds.
- Register an S3 location, move a table from "IAM decides" (
IAMAllowedPrincipals) to Lake Formation permissions, and use hybrid access mode to migrate safely. - Grant table and column permissions, create data cells filters for row- and cell-level security, and use LF-Tags for tag-based access control at scale.
- Share catalog resources with another account and explain what happens through AWS RAM.
- Define data ownership, classification and lineage, and know which AWS services record lineage.
Prerequisites
- Module 7 overview, Glue Data Catalog and crawlers, Validation and data quality
- Module 5: IAM in depth and Boundaries, SCPs and resource policies
- Tools: AWS CLI v2 (
learn-sandbox); the project stack deployed if you want to try it live
Why this matters in banking
Harbourline has three groups that want the same curated transactions table:
- UK retail analysts need GBP transactions to study card spend — but have no business reason to see counterparty names or payment references, which often contain people's names.
- EU subsidiary analysts must see only EUR transactions, because the subsidiary's data-sharing agreement covers only its own customers.
- Financial crime investigators need everything, including tokens they can resolve through a controlled process.
With only S3 bucket policies and IAM, access is all-or-nothing per object: anyone who can read the Parquet files can read every column and every row in them. You would end up copying the table three times with different columns and rows — tripling storage, pipelines and the places personal data lives. Regulators and auditors ask "who can see what, and why?"; the answer must be a short, reviewable list of grants, not a maze of copies.
The idea in plain language
AWS Lake Formation is a permissions layer on top of the Glue Data Catalog. Instead of granting access to S3 objects, you grant access to databases, tables, columns and rows — the way a DBA grants on SQL Server objects. When an integrated engine (Athena, Redshift Spectrum, Glue, EMR, Quick Sight through Athena) queries a governed table, Lake Formation checks the caller's permissions and hands the engine temporary credentials to read only the S3 data it needs; the engine applies the column and row filters before returning results.
Key words:
- Data lake administrator — a principal allowed to manage Lake Formation permissions.
- Registered location — an S3 path Lake Formation manages; it uses a role (the service-linked role, or one you provide) to read it on behalf of users.
IAMAllowedPrincipals— a special group that, by default, has full permission on new catalog resources, so access is decided only by IAM. Removing it from a table switches that table to Lake Formation permissions.- Hybrid access mode — a way to register a location so selected principals use Lake Formation while others keep IAM-based access, letting you migrate gradually.
- Data cells filter — a named filter on a table: a row filter expression, and included or excluded columns. Granting
SELECTon a filter gives row-, column- or cell-level access. - LF-Tags — key/value tags (for example
classification=confidential,domain=payments) on databases, tables and columns. You grant on tag expressions, and new tables with matching tags are covered automatically. This is tag-based access control (TBAC).
Analogy: S3 permissions are the key to the archive room — once inside you can open every box. Lake Formation is the archivist at the counter: you ask for a file, they check your clearance, and they hand you a copy with the lines you may not see blacked out.
What problem the AWS service solves
| Need | Lake Formation feature | Limitation |
|---|---|---|
| Column-level access | Column permissions or filters with included/excluded columns | Only through integrated engines; direct S3 access bypasses it if IAM still allows it |
| Row-level access | Data cells filter RowFilter expressions | Expressions support a subset of SQL ⚠ VERIFY supported operators; test with your engine |
| Scale to hundreds of tables | LF-Tags and tag expressions | Tag design needs governance; too many tags become unmanageable |
| Cross-account sharing | Grants to external accounts or organisation units, via AWS RAM | Recipient creates resource links; both sides need configuration |
| Audit | CloudTrail events for grants and data access (GetDataAccess) | Combine with Athena query logs for "who queried what" |
Lake Formation itself has no charge; you pay for the services that read data (Athena, Glue, Redshift) and for S3.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Microsoft Purview data map + catalog | Glue Data Catalog (+ SageMaker Catalog / Amazon DataZone for business catalogue) | Technical catalog and business catalog are separate products on AWS |
| ADLS Gen2 ACLs (POSIX) | S3 bucket/IAM policies | Object-level only; no column or row concept |
Synapse / SQL Server column-level GRANT | Lake Formation column permissions | Applied by the query engine using vended credentials |
| SQL Server row-level security (security predicates) | Data cells filter RowFilter | Defined per table filter, granted like an object |
| Purview access policies / classifications | LF-Tags (TBAC) | Tags drive grants directly |
| Azure Databricks Unity Catalog | Lake Formation + Glue Data Catalog (Iceberg tables) | Similar three-level model (catalog/database/table) |
| Purview lineage | DataZone / SageMaker Catalog lineage (OpenLineage-compatible) ⚠ VERIFY current naming | Lineage capture depends on the producing service |
Architecture
Security-boundary view. Analysts' IAM policies allow Athena, catalog reads and lakeformation:GetDataAccess — but no s3:GetObject on the curated bucket. They can only reach the data through an engine that Lake Formation trusts, which applies the filters. The Glue job role keeps direct S3 access (it is the producer) and is not affected by analyst filters. Administrators who manage grants are a separate role from analysts (segregation of duties).
Step-by-step implementation
Appoint data lake administrators
Add the platform team's role (in the lab, your SSO role) as a data lake administrator. Do this in the console once; scripts should not overwrite data lake settings.
Register the curated location
Register s3://hl-m07-curated-<account> with the service-linked role (or a custom role). Because the bucket uses SSE-KMS, that role needs kms:Decrypt on alias/hl-m07-data — add it to the key policy or a grant.
Switch the table to Lake Formation permissions
Revoke IAMAllowedPrincipals on hl_m07_curated.transactions (or use hybrid access mode while you migrate). From now on, IAM alone no longer grants table access.
Create filters for each audience
uk_gbp_no_free_text: row filter currency = 'GBP', exclude counterparty_name, reference, source_file. eu_eur_no_free_text: same with 'EUR'.
Grant
DESCRIBE on the database and SELECT on the filter, to each analyst role. Investigators get SELECT on the table.
Scale with LF-Tags
Define classification (public, internal, confidential) and domain (payments, cards, lending). Tag tables and sensitive columns. Grant analysts SELECT on classification in (public, internal) AND domain = payments; new tables tagged the same way are covered without new grants.
Verify and audit
Assume each role and run the same query; compare results. Find the grants and GetDataAccess calls in CloudTrail.
AWS Console instructions
- Lake Formation → Administration → Administrative roles and tasks → Data lake administrators → Add your SSO role.
- Data lake locations → Register location →
s3://hl-m07-curated-<account>, IAM role AWSServiceRoleForLakeFormationDataAccess, permission mode Lake Formation (or Hybrid access mode to migrate gradually). - Data catalog → Tables → transactions → Actions → View permissions → select
IAMAllowedPrincipals→ Revoke. - Data catalog → Data filters → Create new filter → name
uk_gbp_no_free_text, tablehl_m07_curated.transactions, Exclude columnscounterparty_name,reference,source_file, Filter rowscurrency = 'GBP'. - Permissions → Data lake permissions → Grant → principal
hl-m07-analyst-uk→ Named Data Catalog resources → databasehl_m07_curated, tabletransactions, data filteruk_gbp_no_free_text→ permission Select. - LF-Tags and permissions → LF-Tags → Add LF-Tag
classificationwith values; assign it under the table's Edit LF-Tags, and on columns via Edit schema.
AWS CLI
The lab script labs/m07-transaction-pipeline/infra/cli/lab75-lake-formation.sh runs these for you; the key commands:
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1 MSYS_NO_PATHCONV=1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
aws lakeformation register-resource --resource-arn "arn:aws:s3:::hl-m07-curated-${ACCOUNT}" --use-service-linked-role
aws lakeformation revoke-permissions --principal DataLakePrincipalIdentifier=IAM_ALLOWED_PRINCIPALS --permissions ALL \
--resource '{"Table":{"DatabaseName":"hl_m07_curated","Name":"transactions"}}'
aws lakeformation create-data-cells-filter --table-data '{
"TableCatalogId": "'"$ACCOUNT"'", "DatabaseName": "hl_m07_curated", "TableName": "transactions",
"Name": "uk_gbp_no_free_text",
"RowFilter": { "FilterExpression": "currency = '"'"'GBP'"'"'" },
"ColumnWildcard": { "ExcludedColumnNames": ["counterparty_name", "reference", "source_file"] } }'
aws lakeformation grant-permissions --principal DataLakePrincipalIdentifier="arn:aws:iam::${ACCOUNT}:role/hl-m07-analyst-uk" \
--permissions SELECT --resource '{"DataCellsFilter":{"TableCatalogId":"'"$ACCOUNT"'","DatabaseName":"hl_m07_curated","TableName":"transactions","Name":"uk_gbp_no_free_text"}}'
# LF-Tags: define, attach, grant by expression.
aws lakeformation create-lf-tag --tag-key classification --tag-values public internal confidential
aws lakeformation add-lf-tags-to-resource --resource '{"Table":{"DatabaseName":"hl_m07_curated","Name":"transactions"}}' \
--lf-tags '[{"TagKey":"classification","TagValues":["internal"]}]'
aws lakeformation add-lf-tags-to-resource \
--resource '{"TableWithColumns":{"DatabaseName":"hl_m07_curated","Name":"transactions","ColumnNames":["counterparty_name","reference"]}}' \
--lf-tags '[{"TagKey":"classification","TagValues":["confidential"]}]'
aws lakeformation grant-permissions --principal DataLakePrincipalIdentifier="arn:aws:iam::${ACCOUNT}:role/hl-m07-analyst-uk" \
--permissions SELECT DESCRIBE \
--resource '{"LFTagPolicy":{"ResourceType":"TABLE","Expression":[{"TagKey":"classification","TagValues":["public","internal"]}]}}'
# Who has what on the table?
aws lakeformation list-permissions --resource '{"Table":{"DatabaseName":"hl_m07_curated","Name":"transactions"}}'Infrastructure as Code
using Amazon.CDK.AWS.LakeFormation;
_ = new CfnResource(this, "CuratedLocation", new CfnResourceProps
{
ResourceArn = curated.BucketArn,
UseServiceLinkedRole = true,
});
var filter = new CfnDataCellsFilter(this, "UkFilter", new CfnDataCellsFilterProps
{
TableCatalogId = Account,
DatabaseName = "hl_m07_curated",
TableName = "transactions",
Name = "uk_gbp_no_free_text",
RowFilter = new CfnDataCellsFilter.RowFilterProperty { FilterExpression = "currency = 'GBP'" },
ColumnWildcard = new CfnDataCellsFilter.ColumnWildcardProperty
{
ExcludedColumnNames = ["counterparty_name", "reference", "source_file"],
},
});
var grant = new CfnPrincipalPermissions(this, "UkAnalystSelect", new CfnPrincipalPermissionsProps
{
Principal = new CfnPrincipalPermissions.DataLakePrincipalProperty { DataLakePrincipalIdentifier = ukAnalystRole.RoleArn },
Resource = new CfnPrincipalPermissions.ResourceProperty
{
DataCellsFilter = new CfnPrincipalPermissions.DataCellsFilterResourceProperty
{
TableCatalogId = Account, DatabaseName = "hl_m07_curated", TableName = "transactions", Name = "uk_gbp_no_free_text",
},
},
Permissions = ["SELECT"],
PermissionsWithGrantOption = [],
});
grant.AddDependency(filter);Complete code
An access review tool in C#: lists every Lake Formation grant on the curated database's tables and filters, so the data owner can review "who can see what" each quarter — a common bank control.
// dotnet add package AWSSDK.LakeFormation (4.1.0.2), plus AWSSDK.SSO and AWSSDK.SSOOIDC for the SSO profile
using Amazon.LakeFormation;
using Amazon.LakeFormation.Model;
const string database = "hl_m07_curated";
using var lf = new AmazonLakeFormationClient(); // AWS_PROFILE=learn-sandbox
var rows = new List<(string Principal, string Resource, string Permissions)>();
string? token = null;
do
{
// ListPermissions without a resource returns all grants the caller may see (admins see all).
var page = await lf.ListPermissionsAsync(new ListPermissionsRequest { NextToken = token, MaxResults = 100 });
foreach (var p in page.PrincipalResourcePermissions)
{
var resource = Describe(p.Resource);
if (resource is null)
{
continue; // not in our database
}
var perms = string.Join('+', p.Permissions.Select(x => x.Value).Order(StringComparer.Ordinal));
if (p.PermissionsWithGrantOption.Count > 0)
{
perms += " (grantable!)";
}
rows.Add((p.Principal.DataLakePrincipalIdentifier, resource, perms));
}
token = page.NextToken;
}
while (token is not null);
Console.WriteLine($"{"Principal",-70} {"Resource",-60} Permissions");
foreach (var r in rows.OrderBy(r => r.Resource, StringComparer.Ordinal).ThenBy(r => r.Principal, StringComparer.Ordinal))
{
Console.WriteLine($"{r.Principal,-70} {r.Resource,-60} {r.Permissions}");
}
// Flag the risky pattern: the table still open to IAM-only access.
if (rows.Any(r => r.Principal == "IAM_ALLOWED_PRINCIPALS"))
{
Console.WriteLine("WARNING: IAM_ALLOWED_PRINCIPALS still has access: Lake Formation filters are bypassed for IAM users.");
}
static string? Describe(Resource r) =>
r.Table is { DatabaseName: database } t ? $"table {t.DatabaseName}.{t.Name ?? "*"}"
: r.TableWithColumns is { DatabaseName: database } c ? $"columns {c.DatabaseName}.{c.Name} [{string.Join(',', c.ColumnNames ?? [])}]"
: r.DataCellsFilter is { DatabaseName: database } f ? $"filter {f.DatabaseName}.{f.TableName}/{f.Name}"
: r.Database is { Name: database } d ? $"database {d.Name}"
: null;Code explained line by line
ListPermissionsAsyncwithout a resource — returns grants across the catalog that the caller can see; data lake administrators see all. Paginate withNextToken.PrincipalResourcePermissions— each entry is one principal, one resource and a set of permissions.Describe(...)— a resource is one of several shapes (database, table, table with columns, data cells filter, LF-Tag policy…); the pattern-matching expression keeps only those in our database and prints them readably.PermissionsWithGrantOption— a principal who can grant to others is a governance risk; highlighted for the reviewer.- The
IAM_ALLOWED_PRINCIPALScheck — if it still has permissions on the table, anyone with IAM access to the catalog and S3 bypasses the filters. This is the most common Lake Formation misconfiguration. - Output sorted by resource then principal — the format a data owner can sign off.
Ownership, classification and lineage
Permissions are only as good as the decisions behind them. A governed lake also records:
- Ownership: every database or data product has a named data owner (business, accountable for who may use it) and a data steward/custodian (technical). Store this as table parameters or in a business catalog, and make the owner the approver for access requests.
- Classification: tag columns by sensitivity (
confidentialfor names and references;restrictedfor tokens linkable to people). Use Amazon Macie to discover sensitive data in S3 you did not expect (see Secure pipeline design). - Lineage: "where did this number come from?" In the lab, lineage is built in: every curated row carries
source_file(bucket, key, version) andingest_run_id, and the audit record links the run to the job. Managed options exist: Amazon DataZone and Amazon SageMaker Catalog (in SageMaker Unified Studio) can capture and display lineage, including OpenLineage-compatible events from Glue and other producers ⚠ VERIFY the current feature names and which producers emit lineage automatically.
Security considerations
⚠ Learning shortcut — not for production
The lab analyst role trusts the whole account (arn:aws:iam::<account>:root) so you can assume it from your SSO session, and the "region" is approximated by currency. In production, analysts reach roles through IAM Identity Center permission sets tied to their groups, row filters use a real business attribute (booking entity, branch region), and access requests go through an approval workflow with expiry.
- Remove direct S3 access from analyst roles; otherwise they can read Parquet files directly and bypass filters.
- Grant to roles and groups, not users. Review grants quarterly (the tool above) and after re-organisations.
- Separate administrators from consumers. Data lake administrators can grant themselves anything; keep that role small, MFA-protected and logged.
- Cross-account sharing uses AWS RAM; restrict sharing to your organisation and review RAM shares.
- Encryption: the registration role must be able to decrypt the bucket's KMS key; otherwise queries fail, not leak — but check before go-live.
Compliance review required
Removing names and references from analyst views reduces exposure, but tokenised and masked data can still be personal data under UK GDPR if it can be linked back to a person. The data-protection officer decides what each audience may see; Lake Formation enforces it.
Banking use case
Harbourline's curated transactions are registered with Lake Formation. Three filters exist: uk_gbp_no_free_text, eu_eur_no_free_text and fincrime_all. The EU subsidiary's analysts use a separate AWS account; the table is shared to that account with only the EUR filter. LF-Tags mark counterparty_name and reference as confidential, so any new table with tagged columns is protected automatically. Every quarter, the data owner signs the access review produced by the C# tool, and CloudTrail evidence of GetDataAccess calls is retained for audit.
Common mistakes
- Leaving
IAMAllowedPrincipalson the table → filters do nothing for IAM principals. Revoke it (or use hybrid mode deliberately). - Analysts with
s3:GetObjecton curated → filters bypassed by reading files directly. Remove it. - Registering an SSE-KMS bucket without key permissions →
AccessDeniedon every query. - One filter per person → unmanageable. Filters per audience, LF-Tags for scale.
- Treating the catalog as documentation-free → nobody knows the owner. Record ownership and classification.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Analyst sees all columns and rows | IAMAllowedPrincipals still granted, or analyst has direct S3 access | Revoke; remove S3 permissions |
Insufficient Lake Formation permission(s) | Missing DESCRIBE on database or SELECT on table/filter | Grant both; check the principal ARN exactly |
Query fails with KMS AccessDenied | Registration role cannot decrypt | Allow the Lake Formation role in the key policy |
| Row filter has no effect | Engine not integrated, or querying through a view/engine that bypasses LF | Use Athena/Redshift Spectrum/Glue with LF enabled |
| Cross-account user sees nothing | Missing resource link or RAM share not accepted | Accept the share; create a resource link in the recipient catalog |
create-data-cells-filter rejected | Invalid filter expression | Simplify to supported operators; quote string literals |
Cost and cleanup
Lake Formation permissions, filters and LF-Tags have no charge. Athena queries made to test the filters cost fractions of a cent on lab data.
./infra/cli/lab75-lake-formation.sh teardown # from labs/m07-transaction-pipeline
aws lakeformation delete-lf-tag --tag-key classification --profile learn-sandbox || trueExercises
- 🟢 Guided: Create the EU filter and an
hl-m07-analyst-eurole; prove with one query per role that each sees only its currency and noreferencecolumn. - 🟡 Challenge: Replace the two filters with an LF-Tag design that also covers a future
paymentstable: write the tags, the assignments and the grants. - 🔴 Live problem: An internal audit finding says "analysts can download full transaction files from S3". You checked, and Lake Formation filters are in place. Investigate how this can be true and write the remediation plan.
Assignment
Write an access matrix for the Harbourline lake (audiences × tables/columns/rows), the Lake Formation implementation for it (filters, tags, grants), and a quarterly access-review procedure using the C# tool. Add it to your portfolio.
Solutions
Create eu_eur_no_free_text with currency = 'EUR' and the same excluded columns; create the role exactly like the UK one; grant DESCRIBE on the database and SELECT on the filter. Test with aws sts assume-role (or a second CLI profile with role_arn and source_profile), then in Athena SELECT currency, count(*) FROM hl_m07_curated.transactions GROUP BY currency — the UK role returns only GBP, the EU role only EUR — and SELECT reference FROM ... LIMIT 1 fails with a column permission error for both.
Tags: domain (transactions, payments), classification (internal, confidential), entity (uk, eu). Assign domain and classification=internal at table level; classification=confidential on counterparty_name, reference, creditor_name. Grants: UK analysts SELECT on domain in (transactions,payments) AND classification = internal. Row restriction by entity is not a tag feature — keep a data cells filter per entity, or physically separate entity data into partitions/tables tagged entity=uk/eu. New tables tagged correctly are covered without new grants; that is the point of TBAC.
Likely causes: analyst roles still have s3:GetObject on the curated bucket (via a broad managed policy such as AmazonS3ReadOnlyAccess), or IAMAllowedPrincipals remains on some tables, or files were copied to another bucket. Plan: use IAM Access Analyzer and the policy simulator to find every principal with read on the bucket; remove direct S3 access from analyst roles; add a bucket policy that denies s3:GetObject except for the pipeline role and the Lake Formation registration role; revoke IAMAllowedPrincipals on all governed tables; check CloudTrail data events for past GetObject calls by analysts to size the exposure; report to the DPO if personal data was accessed beyond its purpose.
Knowledge check
Answer: Stops IAM permissions alone from granting table access, so Lake Formation grants (and filters) decide who can query it through integrated engines.
Answer: Create a data cells filter with a row filter and included/excluded columns, then grant SELECT on the filter.
Answer: To grant by classification or domain once and cover current and future tables and columns with matching tags, instead of granting table by table.
Answer: Not by itself. If their IAM policy allows s3:GetObject on the bucket, they can read files directly. Remove direct access and use bucket policies.
Interview questions
- "How would you implement column- and row-level security on a data lake?" — Lake Formation on the Glue catalog: register locations, revoke
IAMAllowedPrincipals, data cells filters per audience or LF-Tags, analysts without direct S3 access, Athena as the engine, CloudTrail for evidence. - "How do you share data with another account safely?" — Lake Formation cross-account grants via RAM on specific tables or filters; the consumer creates resource links; no bucket-wide access.
- "What is lineage and why does a bank care?" — The trace from a reported number back to source files and transformations; needed to explain numbers to auditors and regulators and to assess the impact of a bad source file.
- "What is the difference between a data owner and a data steward?" — Owner: business accountability and access approval. Steward: day-to-day quality and metadata.
Certification practice
Analysts in two regions query the same Glue table with Athena. Each region must see only its own rows, and no analyst may see the customer_name column. Which solution requires the least data duplication?
A. Create two copies of the table in separate S3 buckets with different columns.
B. Register the location in Lake Formation, create a data cells filter per region with a row filter and excluded customer_name, and grant SELECT on each filter to the region's role.
C. Use S3 bucket policies with prefix conditions.
D. Create Athena views and grant IAM access to the views only.
Answer: B. One table, filters per audience. A duplicates data. C cannot hide columns inside files. D: IAM cannot restrict access to a view while denying the underlying table in Athena without Lake Formation.
A data platform has 400 tables and adds 20 per month. Access depends on data sensitivity and business domain. What is the most scalable way to manage permissions?
A. Grant each role access table by table. B. Use Lake Formation LF-Tags on databases, tables and columns, and grant permissions on tag expressions. C. Use one IAM policy per table. D. Put all tables in one database and grant database-level access.
Answer: B. Tag-based access control covers new resources automatically. A and C do not scale. D is too coarse.
After enabling Lake Formation filters, a security engineer finds that some analysts can still read all columns. What is the most likely cause?
A. Lake Formation does not support Athena.
B. The table still grants permissions to IAMAllowedPrincipals, or analysts have direct S3 read access to the data location.
C. The KMS key is AWS managed.
D. The Glue crawler overwrote the filters.
Answer: B. Either keeps IAM-only access paths open. A is false. C affects encryption, not authorisation. D: crawlers do not remove filters.
Completion checklist
- I can explain registered locations,
IAMAllowedPrincipalsand hybrid access mode - I can create a data cells filter and grant
SELECTon it - I can design LF-Tags for classification and domain
- I know how analysts could bypass Lake Formation and how to prevent it
- I ran the teardown script and deleted LF-Tags I created
Key takeaways
- Lake Formation turns object-level S3 access into database/table/column/row permissions on the Glue catalog, enforced by integrated engines with vended credentials.
- Revoke
IAMAllowedPrincipals(or migrate with hybrid access) and remove analysts' direct S3 access, or the filters can be bypassed. - Data cells filters give row- and cell-level security; LF-Tags scale grants across many tables.
- Governance also means named owners, classification and lineage; the lab carries lineage in every row and audit record.
- Lake Formation is free; review grants regularly.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Validation and data quality: rules, quarantine and reconciliation
Build Harbourline's data-quality gate: row-level rules with reason codes in Python and C#, a Glue Data Quality (DQDL) ruleset, a quarantine zone with a reject report, control totals that prove nothing was lost, and an audit record for every run.
Secure pipeline design: encryption, least privilege, private networking and PII protection
Design a data pipeline that is safe by construction: a KMS key per data domain, one narrow role per job, private network paths, PII tokenised and masked inside the job, and an audit trail you can show a reviewer.