Skip to Content
Cloud Engineering Academy
7. Data Engineering & ETL Pipelines

Governance, lineage and cataloguing with Lake Formation

Govern Harbourline's data lake: register S3 locations with AWS Lake Formation, replace coarse IAM access with table, column, row and tag-based permissions, give a UK analyst only GBP rows without free-text columns, share safely across accounts, and record ownership and lineage.

advanced⏱ 1 h 50 min🟢 FreeDEA-C01SCS-C03SAA-C03

Assumes: AWS Lake Formation · AWS Glue Data Catalog · Amazon Athena engine v3 · AWS SDK for .NET v4 (AWSSDK.LakeFormation 4.1.0.2) · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain why bucket-level IAM is too coarse for a bank's data lake, and what Lake Formation adds.
  • Register an S3 location, move a table from "IAM decides" (IAMAllowedPrincipals) to Lake Formation permissions, and use hybrid access mode to migrate safely.
  • Grant table and column permissions, create data cells filters for row- and cell-level security, and use LF-Tags for tag-based access control at scale.
  • Share catalog resources with another account and explain what happens through AWS RAM.
  • Define data ownership, classification and lineage, and know which AWS services record lineage.

Prerequisites

Why this matters in banking

Harbourline has three groups that want the same curated transactions table:

  • UK retail analysts need GBP transactions to study card spend — but have no business reason to see counterparty names or payment references, which often contain people's names.
  • EU subsidiary analysts must see only EUR transactions, because the subsidiary's data-sharing agreement covers only its own customers.
  • Financial crime investigators need everything, including tokens they can resolve through a controlled process.

With only S3 bucket policies and IAM, access is all-or-nothing per object: anyone who can read the Parquet files can read every column and every row in them. You would end up copying the table three times with different columns and rows — tripling storage, pipelines and the places personal data lives. Regulators and auditors ask "who can see what, and why?"; the answer must be a short, reviewable list of grants, not a maze of copies.

The idea in plain language

AWS Lake Formation is a permissions layer on top of the Glue Data Catalog. Instead of granting access to S3 objects, you grant access to databases, tables, columns and rows — the way a DBA grants on SQL Server objects. When an integrated engine (Athena, Redshift Spectrum, Glue, EMR, Quick Sight through Athena) queries a governed table, Lake Formation checks the caller's permissions and hands the engine temporary credentials to read only the S3 data it needs; the engine applies the column and row filters before returning results.

Key words:

  • Data lake administrator — a principal allowed to manage Lake Formation permissions.
  • Registered location — an S3 path Lake Formation manages; it uses a role (the service-linked role, or one you provide) to read it on behalf of users.
  • IAMAllowedPrincipals — a special group that, by default, has full permission on new catalog resources, so access is decided only by IAM. Removing it from a table switches that table to Lake Formation permissions.
  • Hybrid access mode — a way to register a location so selected principals use Lake Formation while others keep IAM-based access, letting you migrate gradually.
  • Data cells filter — a named filter on a table: a row filter expression, and included or excluded columns. Granting SELECT on a filter gives row-, column- or cell-level access.
  • LF-Tags — key/value tags (for example classification=confidential, domain=payments) on databases, tables and columns. You grant on tag expressions, and new tables with matching tags are covered automatically. This is tag-based access control (TBAC).

Analogy: S3 permissions are the key to the archive room — once inside you can open every box. Lake Formation is the archivist at the counter: you ask for a file, they check your clearance, and they hand you a copy with the lines you may not see blacked out.

What problem the AWS service solves

NeedLake Formation featureLimitation
Column-level accessColumn permissions or filters with included/excluded columnsOnly through integrated engines; direct S3 access bypasses it if IAM still allows it
Row-level accessData cells filter RowFilter expressionsExpressions support a subset of SQL ⚠ VERIFY supported operators; test with your engine
Scale to hundreds of tablesLF-Tags and tag expressionsTag design needs governance; too many tags become unmanageable
Cross-account sharingGrants to external accounts or organisation units, via AWS RAMRecipient creates resource links; both sides need configuration
AuditCloudTrail events for grants and data access (GetDataAccess)Combine with Athena query logs for "who queried what"

Lake Formation itself has no charge; you pay for the services that read data (Athena, Glue, Redshift) and for S3.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Microsoft Purview data map + catalogGlue Data Catalog (+ SageMaker Catalog / Amazon DataZone for business catalogue)Technical catalog and business catalog are separate products on AWS
ADLS Gen2 ACLs (POSIX)S3 bucket/IAM policiesObject-level only; no column or row concept
Synapse / SQL Server column-level GRANTLake Formation column permissionsApplied by the query engine using vended credentials
SQL Server row-level security (security predicates)Data cells filter RowFilterDefined per table filter, granted like an object
Purview access policies / classificationsLF-Tags (TBAC)Tags drive grants directly
Azure Databricks Unity CatalogLake Formation + Glue Data Catalog (Iceberg tables)Similar three-level model (catalog/database/table)
Purview lineageDataZone / SageMaker Catalog lineage (OpenLineage-compatible) ⚠ VERIFY current namingLineage capture depends on the producing service

Architecture

Security-boundary view. Analysts' IAM policies allow Athena, catalog reads and lakeformation:GetDataAccess — but no s3:GetObject on the curated bucket. They can only reach the data through an engine that Lake Formation trusts, which applies the filters. The Glue job role keeps direct S3 access (it is the producer) and is not affected by analyst filters. Administrators who manage grants are a separate role from analysts (segregation of duties).

Step-by-step implementation

Appoint data lake administrators

Add the platform team's role (in the lab, your SSO role) as a data lake administrator. Do this in the console once; scripts should not overwrite data lake settings.

Register the curated location

Register s3://hl-m07-curated-<account> with the service-linked role (or a custom role). Because the bucket uses SSE-KMS, that role needs kms:Decrypt on alias/hl-m07-data — add it to the key policy or a grant.

Switch the table to Lake Formation permissions

Revoke IAMAllowedPrincipals on hl_m07_curated.transactions (or use hybrid access mode while you migrate). From now on, IAM alone no longer grants table access.

Create filters for each audience

uk_gbp_no_free_text: row filter currency = 'GBP', exclude counterparty_name, reference, source_file. eu_eur_no_free_text: same with 'EUR'.

Grant

DESCRIBE on the database and SELECT on the filter, to each analyst role. Investigators get SELECT on the table.

Scale with LF-Tags

Define classification (public, internal, confidential) and domain (payments, cards, lending). Tag tables and sensitive columns. Grant analysts SELECT on classification in (public, internal) AND domain = payments; new tables tagged the same way are covered without new grants.

Verify and audit

Assume each role and run the same query; compare results. Find the grants and GetDataAccess calls in CloudTrail.

AWS Console instructions

  1. Lake Formation → Administration → Administrative roles and tasks → Data lake administrators → Add your SSO role.
  2. Data lake locations → Register location → s3://hl-m07-curated-<account>, IAM role AWSServiceRoleForLakeFormationDataAccess, permission mode Lake Formation (or Hybrid access mode to migrate gradually).
  3. Data catalog → Tables → transactions → Actions → View permissions → select IAMAllowedPrincipals → Revoke.
  4. Data catalog → Data filters → Create new filter → name uk_gbp_no_free_text, table hl_m07_curated.transactions, Exclude columns counterparty_name, reference, source_file, Filter rows currency = 'GBP'.
  5. Permissions → Data lake permissions → Grant → principal hl-m07-analyst-uk → Named Data Catalog resources → database hl_m07_curated, table transactions, data filter uk_gbp_no_free_text → permission Select.
  6. LF-Tags and permissions → LF-Tags → Add LF-Tag classification with values; assign it under the table's Edit LF-Tags, and on columns via Edit schema.

AWS CLI

The lab script labs/m07-transaction-pipeline/infra/cli/lab75-lake-formation.sh runs these for you; the key commands:

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1 MSYS_NO_PATHCONV=1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

aws lakeformation register-resource --resource-arn "arn:aws:s3:::hl-m07-curated-${ACCOUNT}" --use-service-linked-role

aws lakeformation revoke-permissions --principal DataLakePrincipalIdentifier=IAM_ALLOWED_PRINCIPALS --permissions ALL \
  --resource '{"Table":{"DatabaseName":"hl_m07_curated","Name":"transactions"}}'

aws lakeformation create-data-cells-filter --table-data '{
  "TableCatalogId": "'"$ACCOUNT"'", "DatabaseName": "hl_m07_curated", "TableName": "transactions",
  "Name": "uk_gbp_no_free_text",
  "RowFilter": { "FilterExpression": "currency = '"'"'GBP'"'"'" },
  "ColumnWildcard": { "ExcludedColumnNames": ["counterparty_name", "reference", "source_file"] } }'

aws lakeformation grant-permissions --principal DataLakePrincipalIdentifier="arn:aws:iam::${ACCOUNT}:role/hl-m07-analyst-uk" \
  --permissions SELECT --resource '{"DataCellsFilter":{"TableCatalogId":"'"$ACCOUNT"'","DatabaseName":"hl_m07_curated","TableName":"transactions","Name":"uk_gbp_no_free_text"}}'

# LF-Tags: define, attach, grant by expression.
aws lakeformation create-lf-tag --tag-key classification --tag-values public internal confidential
aws lakeformation add-lf-tags-to-resource --resource '{"Table":{"DatabaseName":"hl_m07_curated","Name":"transactions"}}' \
  --lf-tags '[{"TagKey":"classification","TagValues":["internal"]}]'
aws lakeformation add-lf-tags-to-resource \
  --resource '{"TableWithColumns":{"DatabaseName":"hl_m07_curated","Name":"transactions","ColumnNames":["counterparty_name","reference"]}}' \
  --lf-tags '[{"TagKey":"classification","TagValues":["confidential"]}]'
aws lakeformation grant-permissions --principal DataLakePrincipalIdentifier="arn:aws:iam::${ACCOUNT}:role/hl-m07-analyst-uk" \
  --permissions SELECT DESCRIBE \
  --resource '{"LFTagPolicy":{"ResourceType":"TABLE","Expression":[{"TagKey":"classification","TagValues":["public","internal"]}]}}'

# Who has what on the table?
aws lakeformation list-permissions --resource '{"Table":{"DatabaseName":"hl_m07_curated","Name":"transactions"}}'

Infrastructure as Code

using Amazon.CDK.AWS.LakeFormation;

_ = new CfnResource(this, "CuratedLocation", new CfnResourceProps
{
    ResourceArn = curated.BucketArn,
    UseServiceLinkedRole = true,
});

var filter = new CfnDataCellsFilter(this, "UkFilter", new CfnDataCellsFilterProps
{
    TableCatalogId = Account,
    DatabaseName = "hl_m07_curated",
    TableName = "transactions",
    Name = "uk_gbp_no_free_text",
    RowFilter = new CfnDataCellsFilter.RowFilterProperty { FilterExpression = "currency = 'GBP'" },
    ColumnWildcard = new CfnDataCellsFilter.ColumnWildcardProperty
    {
        ExcludedColumnNames = ["counterparty_name", "reference", "source_file"],
    },
});

var grant = new CfnPrincipalPermissions(this, "UkAnalystSelect", new CfnPrincipalPermissionsProps
{
    Principal = new CfnPrincipalPermissions.DataLakePrincipalProperty { DataLakePrincipalIdentifier = ukAnalystRole.RoleArn },
    Resource = new CfnPrincipalPermissions.ResourceProperty
    {
        DataCellsFilter = new CfnPrincipalPermissions.DataCellsFilterResourceProperty
        {
            TableCatalogId = Account, DatabaseName = "hl_m07_curated", TableName = "transactions", Name = "uk_gbp_no_free_text",
        },
    },
    Permissions = ["SELECT"],
    PermissionsWithGrantOption = [],
});
grant.AddDependency(filter);

Complete code

An access review tool in C#: lists every Lake Formation grant on the curated database's tables and filters, so the data owner can review "who can see what" each quarter — a common bank control.

AccessReview.cs
// dotnet add package AWSSDK.LakeFormation (4.1.0.2), plus AWSSDK.SSO and AWSSDK.SSOOIDC for the SSO profile
using Amazon.LakeFormation;
using Amazon.LakeFormation.Model;

const string database = "hl_m07_curated";
using var lf = new AmazonLakeFormationClient(); // AWS_PROFILE=learn-sandbox

var rows = new List<(string Principal, string Resource, string Permissions)>();
string? token = null;
do
{
    // ListPermissions without a resource returns all grants the caller may see (admins see all).
    var page = await lf.ListPermissionsAsync(new ListPermissionsRequest { NextToken = token, MaxResults = 100 });
    foreach (var p in page.PrincipalResourcePermissions)
    {
        var resource = Describe(p.Resource);
        if (resource is null)
        {
            continue; // not in our database
        }

        var perms = string.Join('+', p.Permissions.Select(x => x.Value).Order(StringComparer.Ordinal));
        if (p.PermissionsWithGrantOption.Count > 0)
        {
            perms += " (grantable!)";
        }

        rows.Add((p.Principal.DataLakePrincipalIdentifier, resource, perms));
    }

    token = page.NextToken;
}
while (token is not null);

Console.WriteLine($"{"Principal",-70} {"Resource",-60} Permissions");
foreach (var r in rows.OrderBy(r => r.Resource, StringComparer.Ordinal).ThenBy(r => r.Principal, StringComparer.Ordinal))
{
    Console.WriteLine($"{r.Principal,-70} {r.Resource,-60} {r.Permissions}");
}

// Flag the risky pattern: the table still open to IAM-only access.
if (rows.Any(r => r.Principal == "IAM_ALLOWED_PRINCIPALS"))
{
    Console.WriteLine("WARNING: IAM_ALLOWED_PRINCIPALS still has access: Lake Formation filters are bypassed for IAM users.");
}

static string? Describe(Resource r) =>
    r.Table is { DatabaseName: database } t ? $"table {t.DatabaseName}.{t.Name ?? "*"}"
    : r.TableWithColumns is { DatabaseName: database } c ? $"columns {c.DatabaseName}.{c.Name} [{string.Join(',', c.ColumnNames ?? [])}]"
    : r.DataCellsFilter is { DatabaseName: database } f ? $"filter {f.DatabaseName}.{f.TableName}/{f.Name}"
    : r.Database is { Name: database } d ? $"database {d.Name}"
    : null;

Code explained line by line

  1. ListPermissionsAsync without a resource — returns grants across the catalog that the caller can see; data lake administrators see all. Paginate with NextToken.
  2. PrincipalResourcePermissions — each entry is one principal, one resource and a set of permissions.
  3. Describe(...) — a resource is one of several shapes (database, table, table with columns, data cells filter, LF-Tag policy…); the pattern-matching expression keeps only those in our database and prints them readably.
  4. PermissionsWithGrantOption — a principal who can grant to others is a governance risk; highlighted for the reviewer.
  5. The IAM_ALLOWED_PRINCIPALS check — if it still has permissions on the table, anyone with IAM access to the catalog and S3 bypasses the filters. This is the most common Lake Formation misconfiguration.
  6. Output sorted by resource then principal — the format a data owner can sign off.

Ownership, classification and lineage

Permissions are only as good as the decisions behind them. A governed lake also records:

  • Ownership: every database or data product has a named data owner (business, accountable for who may use it) and a data steward/custodian (technical). Store this as table parameters or in a business catalog, and make the owner the approver for access requests.
  • Classification: tag columns by sensitivity (confidential for names and references; restricted for tokens linkable to people). Use Amazon Macie to discover sensitive data in S3 you did not expect (see Secure pipeline design).
  • Lineage: "where did this number come from?" In the lab, lineage is built in: every curated row carries source_file (bucket, key, version) and ingest_run_id, and the audit record links the run to the job. Managed options exist: Amazon DataZone and Amazon SageMaker Catalog (in SageMaker Unified Studio) can capture and display lineage, including OpenLineage-compatible events from Glue and other producers ⚠ VERIFY the current feature names and which producers emit lineage automatically.

Security considerations

⚠ Learning shortcut — not for production

The lab analyst role trusts the whole account (arn:aws:iam::<account>:root) so you can assume it from your SSO session, and the "region" is approximated by currency. In production, analysts reach roles through IAM Identity Center permission sets tied to their groups, row filters use a real business attribute (booking entity, branch region), and access requests go through an approval workflow with expiry.

  • Remove direct S3 access from analyst roles; otherwise they can read Parquet files directly and bypass filters.
  • Grant to roles and groups, not users. Review grants quarterly (the tool above) and after re-organisations.
  • Separate administrators from consumers. Data lake administrators can grant themselves anything; keep that role small, MFA-protected and logged.
  • Cross-account sharing uses AWS RAM; restrict sharing to your organisation and review RAM shares.
  • Encryption: the registration role must be able to decrypt the bucket's KMS key; otherwise queries fail, not leak — but check before go-live.

Compliance review required

Removing names and references from analyst views reduces exposure, but tokenised and masked data can still be personal data under UK GDPR if it can be linked back to a person. The data-protection officer decides what each audience may see; Lake Formation enforces it.

Banking use case

Harbourline's curated transactions are registered with Lake Formation. Three filters exist: uk_gbp_no_free_text, eu_eur_no_free_text and fincrime_all. The EU subsidiary's analysts use a separate AWS account; the table is shared to that account with only the EUR filter. LF-Tags mark counterparty_name and reference as confidential, so any new table with tagged columns is protected automatically. Every quarter, the data owner signs the access review produced by the C# tool, and CloudTrail evidence of GetDataAccess calls is retained for audit.

Common mistakes

  • Leaving IAMAllowedPrincipals on the table → filters do nothing for IAM principals. Revoke it (or use hybrid mode deliberately).
  • Analysts with s3:GetObject on curated → filters bypassed by reading files directly. Remove it.
  • Registering an SSE-KMS bucket without key permissions → AccessDenied on every query.
  • One filter per person → unmanageable. Filters per audience, LF-Tags for scale.
  • Treating the catalog as documentation-free → nobody knows the owner. Record ownership and classification.

Troubleshooting

SymptomLikely causeFix
Analyst sees all columns and rowsIAMAllowedPrincipals still granted, or analyst has direct S3 accessRevoke; remove S3 permissions
Insufficient Lake Formation permission(s)Missing DESCRIBE on database or SELECT on table/filterGrant both; check the principal ARN exactly
Query fails with KMS AccessDeniedRegistration role cannot decryptAllow the Lake Formation role in the key policy
Row filter has no effectEngine not integrated, or querying through a view/engine that bypasses LFUse Athena/Redshift Spectrum/Glue with LF enabled
Cross-account user sees nothingMissing resource link or RAM share not acceptedAccept the share; create a resource link in the recipient catalog
create-data-cells-filter rejectedInvalid filter expressionSimplify to supported operators; quote string literals

Cost and cleanup

Lake Formation permissions, filters and LF-Tags have no charge. Athena queries made to test the filters cost fractions of a cent on lab data.

./infra/cli/lab75-lake-formation.sh teardown          # from labs/m07-transaction-pipeline
aws lakeformation delete-lf-tag --tag-key classification --profile learn-sandbox || true

Exercises

  • 🟢 Guided: Create the EU filter and an hl-m07-analyst-eu role; prove with one query per role that each sees only its currency and no reference column.
  • 🟡 Challenge: Replace the two filters with an LF-Tag design that also covers a future payments table: write the tags, the assignments and the grants.
  • 🔴 Live problem: An internal audit finding says "analysts can download full transaction files from S3". You checked, and Lake Formation filters are in place. Investigate how this can be true and write the remediation plan.

Assignment

Write an access matrix for the Harbourline lake (audiences × tables/columns/rows), the Lake Formation implementation for it (filters, tags, grants), and a quarterly access-review procedure using the C# tool. Add it to your portfolio.

Solutions

Knowledge check

Interview questions

  • "How would you implement column- and row-level security on a data lake?" — Lake Formation on the Glue catalog: register locations, revoke IAMAllowedPrincipals, data cells filters per audience or LF-Tags, analysts without direct S3 access, Athena as the engine, CloudTrail for evidence.
  • "How do you share data with another account safely?" — Lake Formation cross-account grants via RAM on specific tables or filters; the consumer creates resource links; no bucket-wide access.
  • "What is lineage and why does a bank care?" — The trace from a reported number back to source files and transformations; needed to explain numbers to auditors and regulators and to assess the impact of a bad source file.
  • "What is the difference between a data owner and a data steward?" — Owner: business accountability and access approval. Steward: day-to-day quality and metadata.

Certification practice

Completion checklist

  • I can explain registered locations, IAMAllowedPrincipals and hybrid access mode
  • I can create a data cells filter and grant SELECT on it
  • I can design LF-Tags for classification and domain
  • I know how analysts could bypass Lake Formation and how to prevent it
  • I ran the teardown script and deleted LF-Tags I created

Key takeaways

  • Lake Formation turns object-level S3 access into database/table/column/row permissions on the Glue catalog, enforced by integrated engines with vended credentials.
  • Revoke IAMAllowedPrincipals (or migrate with hybrid access) and remove analysts' direct S3 access, or the filters can be bypassed.
  • Data cells filters give row- and cell-level security; LF-Tags scale grants across many tables.
  • Governance also means named owners, classification and lineage; the lab carries lineage in every row and audit record.
  • Lake Formation is free; review grants regularly.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page