Data classification, masking and tokenisation
Classify banking data into four levels that drive real controls, find unclassified personal data with Macie, mask values by viewer role, and build a tokenisation vault for card numbers while knowing what it does and does not do for compliance scope.
Assumes: .NET 10 LTS · ASP.NET Core 10 · AWS SDK for .NET v4 · AWS Encryption SDK for .NET 5.0.0 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider 6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Define a four-level classification scheme for a bank and map each level to concrete engineering controls (key type, field-level encryption, logging, non-production use).
- Make classification enforceable in .NET: label every property of a data model, fail the build when one is unlabelled, and produce a data inventory from the code.
- Use Amazon Macie to find personal and financial data in S3, map its findings to classification levels, and detect drift between what a bucket is labelled and what it holds.
- Mask values per viewer role for the customer, support, fraud and audit views, and explain why masking is not protection.
- Build a tokenisation vault in C# for card numbers: validation, field-level encryption, format-preserving tokens, idempotency, restricted detokenisation and an audit trail.
- Compare masking, pseudonymisation, tokenisation, hashing and encryption, and explain the compliance caveats for PCI DSS and UK GDPR without claiming compliance.
Prerequisites
- Encryption at rest and in transit, and field-level encryption in .NET: this lesson reuses the Encryption SDK wrapper and the blind-index idea
- S3 protection: Block Public Access, bucket policies, Object Lock and Macie: Macie basics and a first scan
- PCI DSS on AWS: scope, the CDE and why tokens shrink it
- Tools: .NET 10 SDK (or Docker); AWS CLI v2 only for the optional Macie steps
- Lab code:
labs/m05-data-protection/
Why this matters in banking
Controls are only as good as the labels they hang on. Four stories:
- The log that knew too much. A developer adds
logger.LogInformation("Created {@Customer}", customer)to debug a bug. The structured logger serialises every property, including the IBAN and the date of birth, into a central log store that two hundred people can search. No encryption setting anywhere would have prevented it, because the application chose to write the data. - The bucket nobody classified. A team lands a nightly export in an S3 bucket tagged
Internal. Six months later a discovery scan finds card numbers in one file. The bucket's controls were set for Internal data, because that is what the tag said. - The test database. A copy of production goes to a developer environment "just for a week". The copy has real names and IBANs, the environment has weaker access control, and it is still there a year later.
- The card number in six systems. Because the payments API stored the PAN "for convenience", the analytics database, the support tool and the data lake all ended up inside the card data environment. The assessment now covers all of them.
The answer to all four is the same discipline: know what you hold, label it, and let the label decide what is allowed. Tokenisation then removes the most dangerous values from most of your estate.
Not legal or compliance advice
Whether tokenisation reduces PCI DSS scope, whether pseudonymised data is still personal data, and what your retention periods are: your QSA, data protection officer and counsel decide. This lesson teaches the engineering. All data is fictional, and card numbers are official test numbers.
The idea in plain language
Classification is a label that says how bad it would be if a piece of data leaked. A bank keeps the scheme small enough that people use it. Harbourline has four levels:
| Level | Meaning | Example |
|---|---|---|
| Public | Safe to publish | Product names, branch opening hours |
| Internal | Business data with low harm if leaked | Account status, internal customer id, transaction type |
| Confidential | Personal or commercially sensitive | Name, email, IBAN, balances |
| Restricted | Highest harm if leaked | Full card number (PAN), date of birth, credentials, keys |
The label is only useful if it triggers controls. The control matrix the lab encodes (your data protection officer owns the real one):
| Level | Customer managed key required | Field-level encryption | May appear in logs | Copy to non-production |
|---|---|---|---|---|
| Public | No | No | Yes | Yes |
| Internal | No | No | Yes | Yes |
| Confidential | Yes | No | Masked only | No (use pseudonymised data) |
| Restricted | Yes | Yes | Never | No |
Then four techniques reduce what people and systems see:
- Masking hides part of a value for display:
GB82**************5432. One-way and lossy. For recognition, not use. - Pseudonymisation replaces identifiers with consistent substitutes (for test data), keeping joins working. Whoever holds the key can re-identify, so under UK GDPR it is still personal data.
- Tokenisation replaces a value with a random surrogate (a token) and keeps the real value in a protected vault. The token is useless without the vault.
- Encryption makes a value unreadable without a key and reversible with it.
Analogy: A cheque book has a counterfoil. Masking is the bank clerk reading out only the last four digits of your account. Tokenisation is the clerk giving you a numbered receipt and locking the account details in the vault: your receipt gets you served, but a thief who steals it learns nothing about the account. Encryption is a sealed envelope, which anyone can carry but only the key holder can open.
What problem the AWS services solve
AWS gives you building blocks, not a classification product:
- Tags and tag policies. A tag such as
data-classification=restrictedon a bucket, table or volume is the label. AWS Organizations tag policies can restrict the allowed values, and AWS Config rules (for example the required-tags managed rule) can flag resources with no label. ⚠ VERIFY the managed rule names when you configure them. - Amazon Macie. Discovers sensitive data in S3 only using managed data identifiers (card numbers, IBANs for more than 50 countries, credentials, personal data) and your custom identifiers, plus bucket inventory and policy findings. Status check: the Macie user guide's document history I read on 2026-11-18 shows its latest entry in July 2025 and no maintenance or closure notice, but AWS has closed or restricted several services recently, so ⚠ VERIFY current availability, pricing and the free trial before you plan around it. Limits: S3 only, probabilistic detection (false positives and negatives), findings kept 90 days (export them if you need longer), and it needs
kms:Decrypton customer managed keys to read encrypted objects. - KMS and the Encryption SDK for the vault and for field-level encryption (previous lesson).
- DynamoDB is a good home for a token vault: single-item lookups by token, a customer managed key, point-in-time recovery and conditional writes for uniqueness.
- Lake Formation and AWS Glue can restrict columns and mask or detect sensitive fields in analytics pipelines ⚠ VERIFY current features. Macie cannot scan RDS or DynamoDB directly; a common pattern is to export a snapshot or table to S3 and scan the export.
- Macie does not decide the label. It produces evidence that the label may be wrong. A person (the data owner) changes it.
What AWS does not provide: a card-network-approved token service. Real card tokenisation is usually done by your payment provider or acquirer, who issue network tokens or their own tokens, or by an assessed vault you build and operate.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Microsoft Purview sensitivity labels and data map | Resource tags + Amazon Macie findings | Macie discovers; it does not apply labels for you. The label is a tag or a column in your inventory |
| Purview sensitive information types | Macie managed and custom data identifiers | S3 only |
| SQL Server Dynamic Data Masking | Application-layer masking (this lesson), Redshift dynamic data masking (⚠ VERIFY), Lake Formation column filters | RDS PostgreSQL and MySQL have no built-in equivalent: mask in the API |
| Always Encrypted | AWS Encryption SDK field-level encryption | In code, not in the driver |
| Azure Policy tag enforcement | Organizations tag policies, Config rules, SCPs on tagging | Tag policies are separate from permissions |
| Microsoft.Extensions.Compliance.Redaction | The same package works on AWS | It plugs the same classification idea into ILogger; the lab shows the underlying approach |
| Entra ID access reviews for who sees what | IAM Identity Center + permission sets, application roles | Masking by role is your code, driven by claims |
Architecture
Security-boundary view. The vault is the only place the full card number exists, and it has its own key and its own narrow role:
Step-by-step implementation
Write the classification scheme and the control matrix
Four levels, one page, an owner. For each level state the controls in the table above. Agree it with the data protection officer, and put it in the repository next to the code.
Label the data model in code
Every property of every data type that crosses a trust boundary gets a [Classified(level, kind)] attribute. Mark the type [DataModel]. Then a unit test scans the assembly and fails if any property has no label. A new property cannot ship unlabelled.
Make the labels drive behaviour
Use the same labels to project objects for responses and for logs. Fail safe: an unlabelled property prints [UNCLASSIFIED], not its value.
Label the stores and scan them
Tag buckets and tables with the level of the most sensitive data they may hold. Enable Macie, add a custom identifier for the bank's customer id, run a one-time job on landing buckets, and send SensitiveData: findings to a queue. A small job compares each finding with the bucket's label.
Remove the PAN from most of the estate
Capture the card number in one small component (or a provider's hosted field), tokenise it immediately, and pass only {token, last4} onward. Restrict detokenisation to an allow-list of callers and audit every call.
Prepare non-production data
Never copy production. Use synthetic data, or pseudonymised data from a keyed process whose key is not available in the target environment. Treat pseudonymised data as personal data in your risk assessment.
Choosing the right technique
| Technique | Reversible? | Keeps format? | Good for | Watch out |
|---|---|---|---|---|
| Masking | No | Partly | Screens, logs, reports | Not protection: the original still exists in the store |
| Hashing (plain) | No | No | Almost nothing for low-entropy data | Brute-forceable for card numbers, dates of birth, phone numbers |
| Keyed hash (HMAC) | No | No | Fingerprints, duplicate detection, blind indexes | Key custody; equal inputs give equal outputs |
| Encryption | With the key | No | Storing a value you must read back | The key is the new crown jewel; search and sort are lost |
| Tokenisation (vault) | Via the vault only | Yes, if designed to | Card numbers, bank account numbers | The vault and its audit become the in-scope system |
| Format-preserving encryption (no vault) | With the key | Yes | Legacy systems that need a PAN-shaped value | Algorithm choice is a cryptography decision: do not improvise, and ⚠ VERIFY which modes current standards accept |
| Pseudonymisation | With the key or mapping | Often | Test data, analytics | Still personal data under UK GDPR |
| Anonymisation | No | n/a | Published statistics | Very hard: quasi-identifiers (postcode plus date of birth) re-identify people |
AWS Console instructions
Tag a bucket with its classification. Console → S3 → bucket → Properties → Tags → Edit → key data-classification, value confidential.
Enable Macie and add a custom identifier. Console → Amazon Macie → Get started → Enable Macie. Then Custom data identifiers → Create → name hl-customer-id, regex CUS-[0-9]{8}, keywords customer_id, customer, maximum match distance 50 → Test with customer_id: CUS-00010001 → Submit.
Run a one-time job. Macie → Jobs → Create job → choose the bucket → One-time job → managed identifiers Recommended plus your custom identifier → Submit. Findings appear under Findings when the job finishes.
Route findings. Console → Amazon EventBridge → Rules → Create rule → event pattern source aws.macie, detail type Macie Finding, with a SensitiveData: prefix filter on detail.type → target an SQS queue (SQS-managed encryption: EventBridge cannot send to a queue encrypted with the AWS managed aws/sqs key).
AWS CLI
# Tag a bucket with its classification (the label that drives controls)
aws s3api put-bucket-tagging --bucket "$BUCKET" \
--tagging 'TagSet=[{Key=data-classification,Value=confidential}]' --profile learn-sandbox
# Enable Macie in this Region (starts the trial if you are eligible; check pricing first)
aws macie2 enable-macie --profile learn-sandbox
# Create the custom data identifier for Harbourline customer ids
aws macie2 create-custom-data-identifier --name hl-customer-id \
--regex 'CUS-[0-9]{8}' --keywords customer_id customer --maximum-match-distance 50 \
--profile learn-sandbox
# Test the regex before you rely on it
aws macie2 test-custom-data-identifier --regex 'CUS-[0-9]{8}' \
--sample-text 'customer_id: CUS-00010001' --profile learn-sandbox
# List sensitive-data findings (classification findings, not policy findings)
aws macie2 list-findings \
--finding-criteria '{"criterion":{"category":{"eq":["CLASSIFICATION"]}}}' --profile learn-sandbox
# Find buckets with no classification tag, using the Resource Groups Tagging API
aws resourcegroupstaggingapi get-resources --resource-type-filters s3 \
--query "ResourceTagMappingList[?!(Tags[?Key=='data-classification'])].ResourceARN" \
--profile learn-sandbox
# Always disable Macie after the lab (see cleanup)
aws macie2 disable-macie --profile learn-sandboxThe tagging query only lists buckets that have at least one tag; a bucket with no tags at all is missing from the results. Use AWS Config or a Resource Explorer search to find the untagged ones. ⚠ VERIFY the Tagging API behaviour for resources with no tags.
Infrastructure as Code
The lab's ClassificationStack creates a vault table with its own key, enables Macie, defines the custom identifier and queues sensitive-data findings. It compiles with Amazon.CDK.Lib 2.277.0.
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.Macie;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.DataProtection.Cdk;
public sealed class ClassificationStack : Stack
{
public ClassificationStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
// The vault gets a key of its own: only the tokenisation service's role will be a key user.
var vaultKey = new Key(this, "VaultKey", new KeyProps
{
Alias = "alias/hl-m05-token-vault",
EnableKeyRotation = true,
PendingWindow = Duration.Days(7),
RemovalPolicy = RemovalPolicy.DESTROY,
});
_ = new Table(this, "TokenVault", new TableProps
{
TableName = "hl-m05-token-vault",
// Attribute is ambiguous with System.Attribute, so the type is fully qualified.
PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "token", Type = AttributeType.STRING },
Encryption = TableEncryption.CUSTOMER_MANAGED,
EncryptionKey = vaultKey,
BillingMode = BillingMode.PAY_PER_REQUEST,
PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification { PointInTimeRecoveryEnabled = true },
RemovalPolicy = RemovalPolicy.DESTROY,
});
_ = new CfnSession(this, "Macie", new CfnSessionProps
{
Status = "ENABLED",
FindingPublishingFrequency = "FIFTEEN_MINUTES",
});
_ = new CfnCustomDataIdentifier(this, "CustomerIdIdentifier", new CfnCustomDataIdentifierProps
{
Name = "hl-customer-id",
Description = "Harbourline customer id, CUS- followed by eight digits",
Regex = "CUS-[0-9]{8}",
Keywords = ["customer_id", "customer"],
MaximumMatchDistance = 50,
});
var queue = new Queue(this, "FindingsQueue", new QueueProps
{
// EventBridge cannot publish to a queue encrypted with the AWS managed key.
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
RetentionPeriod = Duration.Days(4),
});
// Amazon.CDK.AWS.Events.Rule is fully qualified to avoid any clash with other Rule types.
_ = new Amazon.CDK.AWS.Events.Rule(this, "MacieSensitiveFindings", new RuleProps
{
EventPattern = new EventPattern
{
Source = ["aws.macie"],
DetailType = ["Macie Finding"],
Detail = new Dictionary<string, object>
{
["type"] = new Dictionary<string, object> { ["prefix"] = "SensitiveData:" },
},
},
Targets = [new SqsQueue(queue)],
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m05-data-protection");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
}
}Complete code
The lab project labs/m05-data-protection has all of it; the tests run without an AWS account. Four pieces carry the lesson.
1. Labels that fail the build
public enum DataClass { Public = 0, Internal = 1, Confidential = 2, Restricted = 3 }
public enum FieldKind { None = 0, Name, Email, Iban, Pan, DateOfBirth }
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct)]
public sealed class DataModelAttribute : Attribute;
[AttributeUsage(AttributeTargets.Property)]
public sealed class ClassifiedAttribute(DataClass level, FieldKind kind = FieldKind.None) : Attribute
{
public DataClass Level { get; } = level;
public FieldKind Kind { get; } = kind;
}
public static class DataInventory
{
public static IReadOnlyList<InventoryRow> Scan(Assembly assembly)
{
var rows = new List<InventoryRow>();
foreach (var type in assembly.GetTypes().Where(t => t.GetCustomAttribute<DataModelAttribute>() is not null))
{
foreach (var property in type.GetProperties(BindingFlags.Public | BindingFlags.Instance))
{
var label = property.GetCustomAttribute<ClassifiedAttribute>();
rows.Add(new InventoryRow(type.Name, property.Name, label?.Level, label?.Kind ?? FieldKind.None));
}
}
return rows;
}
public static IReadOnlyList<InventoryRow> Unclassified(Assembly assembly) =>
Scan(assembly).Where(r => r.Level is null).ToList();
}A labelled model looks like this (a positional record needs [property: ...] so the attribute lands on the property, not the constructor parameter):
[DataModel]
public sealed record CustomerView(
[property: Classified(DataClass.Internal)] string CustomerId,
[property: Classified(DataClass.Confidential, FieldKind.Name)] string FullName,
[property: Classified(DataClass.Confidential, FieldKind.Email)] string Email,
[property: Classified(DataClass.Confidential, FieldKind.Iban)] string Iban,
[property: Classified(DataClass.Restricted, FieldKind.Pan)] string CardNumber,
[property: Classified(DataClass.Restricted, FieldKind.DateOfBirth)] string DateOfBirth);2. Masking by viewer role
public enum ViewerRole { Customer, SupportAgent, FraudAnalyst, Auditor, Service }
public static class Masker
{
public const string Redacted = "[REDACTED]";
public const string Masked = "[MASKED]";
public const string Unclassified = "[UNCLASSIFIED]";
public static string Mask(FieldKind kind, DataClass level, string value, ViewerRole role) => kind switch
{
FieldKind.Pan => MaskPan(value, role),
FieldKind.Iban => MaskIbanFor(value, role),
FieldKind.Email => role == ViewerRole.Customer ? value : MaskEmail(value),
FieldKind.Name => role is ViewerRole.Customer or ViewerRole.SupportAgent or ViewerRole.FraudAnalyst ? value : Initials(value),
FieldKind.DateOfBirth => role == ViewerRole.Customer ? value : Redacted,
_ => level switch
{
DataClass.Public or DataClass.Internal => value,
DataClass.Confidential => role == ViewerRole.Customer ? value : Masked,
_ => Redacted,
},
};
public static IReadOnlyDictionary<string, string> Project(object value, ViewerRole role)
{
var result = new Dictionary<string, string>(StringComparer.Ordinal);
foreach (var property in value.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance))
{
var label = property.GetCustomAttribute<ClassifiedAttribute>();
var raw = property.GetValue(value)?.ToString() ?? string.Empty;
result[property.Name] = label is null ? Unclassified : Mask(label.Kind, label.Level, raw, role);
}
return result;
}
// PCI DSS allows people with a business need to see at most the first six and last four digits
// (⚠ VERIFY the current wording of requirement 3.4.1). We show less by default.
private static string MaskPan(string pan, ViewerRole role)
{
var digits = new string(pan.Where(char.IsAsciiDigit).ToArray());
if (digits.Length < 12) return new string('*', digits.Length);
var head = role == ViewerRole.FraudAnalyst ? digits[..6] : string.Empty;
return head + new string('*', digits.Length - head.Length - 4) + digits[^4..];
}
}The ASP.NET Core endpoint uses it in one line: Results.Json(Masker.Project(customer, role)). The same projection with ViewerRole.Service is the log-safe view.
3. The tokenisation vault
public sealed class TokenVault(
EsdkFieldEncryptor encryptor,
byte[] fingerprintKey,
ITokenStore store,
IReadOnlySet<string> allowedDetokenisers,
TimeProvider clock,
bool luhnValidTokens = false)
{
public TokenResult Tokenise(string pan)
{
var digits = new string(pan.Where(char.IsAsciiDigit).ToArray());
if (digits.Length is < 13 or > 19 || !Luhn.IsValid(digits))
throw new ArgumentException("Not a valid card number.", nameof(pan));
// Keyed fingerprint: the same card always maps to the same token, and a thief with the
// table cannot test guesses without the key.
var fingerprint = Convert.ToHexStringLower(HMACSHA256.HashData(fingerprintKey, Encoding.UTF8.GetBytes(digits)));
if (store.FindByFingerprint(fingerprint) is { } existing)
return new TokenResult(existing.Token, existing.Last4);
while (true)
{
var token = NewToken(digits.Length, digits[^4..]);
var ciphertext = encryptor.Encrypt(digits, VaultContext(token));
var record = new TokenRecord(token, fingerprint, ciphertext, digits[^4..], clock.GetUtcNow());
if (store.TryAdd(record)) return new TokenResult(token, record.Last4);
}
}
public string Detokenise(string token, string caller, string purpose)
{
var allowed = allowedDetokenisers.Contains(caller);
lock (_auditGate) { _audit.Add(new DetokeniseAuditEntry(clock.GetUtcNow(), caller, purpose, token, allowed)); }
if (!allowed) throw new UnauthorizedAccessException($"'{caller}' may not detokenise.");
var record = store.FindByToken(token) ?? throw new KeyNotFoundException("Unknown token.");
return encryptor.Decrypt(record.Ciphertext, VaultContext(token));
}
// Binding the ciphertext to its own token stops an attacker swapping two rows' ciphertexts.
private static Dictionary<string, string> VaultContext(string token) =>
new(StringComparer.Ordinal) { ["purpose"] = "pan-vault", ["token"] = token };
}NewToken draws random digits for the middle, keeps the last four for display, starts every token with 9 (a prefix you agree with your acquirer and QSA so tokens are recognisable) and solves one digit so the Luhn check does or does not pass, as configured. A Luhn-invalid token can never be mistaken for a real card number; a Luhn-valid one passes legacy validation. Which to choose is a design decision with your payment partners. ⚠ VERIFY any rules your card scheme or acquirer sets for token formats.
4. Macie findings to classification levels
public static class MacieFindingMapper
{
public static MacieFinding? Parse(string json)
{
using var document = JsonDocument.Parse(json);
var root = document.RootElement;
// EventBridge wraps the finding in "detail"; get-findings returns it bare.
var finding = root.TryGetProperty("detail", out var detail) ? detail : root;
// ... reads type, severity.description and resourcesAffected.s3Bucket.name
}
// Finding type names: ⚠ VERIFY against the current Macie documentation.
public static DataClass? ImpliedClass(string findingType) => findingType switch
{
"SensitiveData:S3Object/Credentials" => DataClass.Restricted,
"SensitiveData:S3Object/Financial" => DataClass.Restricted,
"SensitiveData:S3Object/Multiple" => DataClass.Restricted,
"SensitiveData:S3Object/Personal" => DataClass.Confidential,
"SensitiveData:S3Object/CustomIdentifier" => DataClass.Confidential,
_ => null, // policy findings are not classification evidence
};
public static bool IsDrift(DataClass declared, MacieFinding finding) =>
ImpliedClass(finding.Type) is { } implied && implied > declared;
}The tests
The lab tests for this lesson include:
- every property of every
[DataModel]type is labelled, and a deliberately sloppy type is caught; - the control matrix gets stricter at each level;
Masker.Projectprints[UNCLASSIFIED]for an unlabelled property and never contains the card number for theServicerole;- Macie JSON parses and drift is detected (bucket labelled Internal, finding says Financial);
- tokens keep the length and last four, start with
9, are Luhn-invalid by default and Luhn-valid on request; - the same card gets the same token, and a bad number is refused;
- only allow-listed callers detokenise, every attempt is audited and no audit entry contains the card number;
- 500 distinct cards give 500 distinct tokens;
- the encryption-posture evaluator flags unencrypted stores and weak keys for sensitive data.
Code explained line by line
Classification
DataModelAttributemarks a type as "everything on me must be labelled". The scanner only checks marked types, so you opt types in and cannot accidentally fail on framework classes.ClassifiedAttribute(level, kind)carries two facts: how sensitive (level) and what it is (kind, which picks the masking rule). A positional record needs[property: Classified(...)]or the attribute attaches to the constructor parameter and the scan finds nothing.Unclassifiedreturns the rows with no label. The test asserts it is empty for the real model.
Masker
Maskswitches on kind first, then on level. The kind rules are specific (PAN, IBAN), the level rules are the fallback.- Restricted values with no special kind default to
[REDACTED]for everyone, including the customer. The customer sees their own date of birth only becauseDateOfBirthhas a rule that says so. Projecthandles an unlabelled property by returning[UNCLASSIFIED]: fail safe, not fail open.MaskPanshows nothing but the last four digits to most roles and the first six plus last four to the fraud role. Showing fewer digits is always safer.
TokenVault
- The PAN is validated with the Luhn algorithm before anything is stored, so typos never enter the vault.
- The fingerprint is an HMAC. A plain hash of a 16-digit card number can be brute-forced; the key prevents that. The same card gives the same fingerprint, so tokenising the same card twice returns the same token (idempotent).
- The
while (true)loop retries if a randomly generated token already exists. The store'sTryAddis atomic. In DynamoDB this is a conditional put (attribute_not_exists(token)). - The vault encrypts the digits with the Encryption SDK. The context holds
purposeand the token, so a ciphertext copied under another token's row fails to decrypt. Detokenisewrites the audit entry before it checks the decision, so refused attempts are recorded too. The entry has the caller, purpose and token but never the card number.
Security considerations
⚠ Learning shortcut: not for production
TokenVault is a teaching vault with an in-memory store and a raw AES keyring in the tests. A production vault runs in its own account and network segment, keeps keys in KMS or CloudHSM, rate-limits detokenisation, alarms on unusual volumes, authenticates callers with IAM or mTLS rather than a string name, and is assessed by your QSA. The demo API takes the viewer role from a request header; production takes it from a validated token.
- Masking is presentation, not protection. The raw value still exists in the database and in memory. Mask at the last moment, and keep Restricted fields encrypted or tokenised underneath.
- Hashing is not anonymising. A plain SHA-256 of a card number, a phone number or a date of birth can be reversed by brute force. Use a keyed hash and keep the key outside the data.
- Pseudonymised data is personal data. Under UK GDPR, data that can be re-identified with additional information stays in scope. Anonymisation needs far more than replacing names. Your data protection officer decides whether a test data set is acceptable.
- Tokenisation does not remove PCI scope by itself. What changes is which systems store, process or transmit card data. Systems that only handle tokens may fall out of scope if segmentation is sound, but that is a QSA decision, and the vault stays in scope. ⚠ VERIFY the current PCI SSC guidance on tokenisation before relying on any scope argument. See PCI DSS on AWS.
- Never store the card verification code after authorisation and never log a PAN, even masked in the wrong format. PCI DSS has specific rules; ⚠ VERIFY them with your QSA.
- Classify the logs too. Log groups hold derived data. Apply the matrix: Confidential data is masked, Restricted is never logged, and log access is limited. See audit logging and evidence collection.
- Erasure and tokens. Deleting a customer's vault entry and their wrapped data keys makes copies of their ciphertext unreadable, but backups complicate this. See UK GDPR and data residency for the erasure discussion; counsel decides what is acceptable.
- Macie reads your data. Its role needs
kms:Decrypton the keys of the buckets it scans. Grant it narrowly and review the finding output, because samples of sensitive data can appear in results if you enable the reveal feature.
Banking use case
Harbourline's payments team wants a card-on-file feature. The design applies every idea in the lesson:
- A hosted field captures the card number and sends it straight to the tokenisation service. The payments API never sees it.
- The vault tokenises it: validation, encryption with its own key, a random token with the last four preserved, and an HMAC fingerprint to avoid duplicates.
- The payments API stores
{token, last4, brand}. The data lake receives the same. Neither can reach the vault table. - The authorisation adapter is the only caller on the detokenise allow-list. Every call is audited with its purpose, and a CloudWatch alarm fires on a spike.
- The customer service tool shows
************4242to agents and nothing else. The fraud analyst sees the first six and last four. - Landing buckets are tagged with a level. Macie scans them weekly and sends
SensitiveData:findings to a review queue; a finding that exceeds a bucket's label raises a ticket for the bucket owner. - Non-production environments get synthetic data. A migration that needs realistic joins uses keyed pseudonymisation with a key that never leaves production, and the DPO signs off.
Common mistakes
- Classification as a document nobody reads → it triggers no controls → tie each level to checks in code, Config and pipelines.
- Labels only on stores, not on fields → a Restricted value in a Confidential table is under-protected → label the data model.
- Masking in the UI only → the API returns the full value to anyone who calls it → mask in the API, driven by role.
- A client-controlled role header → anyone can claim to be an auditor → take roles from the validated token.
- Plain hashes as "tokens" → reversible by brute force and not random → use a vault token or a keyed construction with a reviewed design.
- Detokenising in the wrong places → the vault boundary leaks → allow-list callers, audit each call, and review the list quarterly.
- Assuming Macie covers every store → it reads S3 only → export RDS and DynamoDB data to S3 for scanning, or use other discovery tools.
- Copying production to test → real data in a weak environment → synthetic or pseudonymised data, with approval.
- Treating pseudonymised data as anonymous → UK GDPR still applies → keep it under the personal data controls.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Inventory test finds no properties on a positional record | Attribute attached to the constructor parameter | Write [property: Classified(...)] |
Every property prints [UNCLASSIFIED] | Type not labelled, or labels on a different type than the one serialised | Label the actual response type |
| Macie finds nothing in an encrypted bucket | Macie cannot decrypt with the customer managed key | Add a key-policy statement for the Macie service-linked role |
| Macie reports objects as unclassifiable | Unsupported file type, encryption type or object size | Check the classification coverage page and convert or export the data |
| EventBridge rule matches but the queue stays empty | Queue policy missing or queue uses the AWS managed key | Allow events.amazonaws.com for the rule ARN and use SQS-managed encryption |
Detokenise throws UnauthorizedAccessException for a legitimate service | Caller name not on the allow-list | Add it through change control, not by editing the vault in place |
| Tokenising the same card twice gives two tokens | The fingerprint key differs between instances | Load the same key from Secrets Manager in every instance |
terraform apply fails on the Macie job | Macie not yet enabled, or the bucket is not visible to it | Keep depends_on on the account resource and check the bucket policy |
Cost and cleanup
Free: all unit tests, dotnet build of the CDK project, terraform validate, and the masking and token code.
Credits (deploy, scan, destroy in one session):
- Macie: a trial may be available for new accounts, and afterwards you pay for bucket inventory and per-GB of data inspected ⚠ VERIFY on the pricing page. A one-time job over a few kilobytes of fictional data costs well under a cent, but leaving Macie enabled across Regions or with automated discovery can cost real money.
- Customer managed key for the vault: USD 1 per key per month, prorated hourly ⚠ VERIFY.
- DynamoDB on demand with a handful of items, SQS and EventBridge: negligible.
Design-only alternative: skip the deployment. Run the tests, read the CDK and Terraform, and write the control matrix and bucket labels for your own system.
# Disable Macie (stops charges; findings are discarded after 90 days)
aws macie2 disable-macie --profile learn-sandbox
# CDK: destroy the classification stack
cd labs/m05-data-protection/infra/cdk
cdk destroy Hl-M05-Classification --profile learn-sandbox
# Terraform: destroy everything the root module created
cd labs/m05-data-protection/infra/terraform
terraform destroy -var "db_subnet_group_name=..." -var "alb_arn=..." -var "certificate_arn=..."
# Confirm no custom data identifiers, jobs or Macie session remain
aws macie2 get-macie-session --profile learn-sandboxget-macie-session should return an error saying Macie is not enabled. Check the billing console the next day.
Exercises
- 🟢 Guided: Run the lab tests. In
DemoHost.csadd a new propertyPhoneNumbertoCustomerViewwithout a[Classified]attribute. Run the tests, read the failure, then add the label (Confidential) and a masking rule that shows only the last three digits to support agents. - 🟡 Challenge: Extend
TokenVaultso detokenisation is limited to 5 calls per caller per minute, with the limit enforced in code and refused attempts audited. Write tests with a manualTimeProvider. - 🔴 Live problem: A Macie scan raises
SensitiveData:S3Object/Financialon a bucket taggedinternal, owned by the analytics team, that feeds a dashboard used by 300 people. Write the response: immediate containment, how you find where the data came from, what changes in the pipeline, how you correct the label, and what you tell the data protection officer.
Assignment
Add a data classification policy and an inventory test to your portfolio repository: a one-page control matrix, a [Classified] attribute on one real data model (it may be the Account API), a CI test that fails on an unlabelled property, and a README section showing the masked output for three viewer roles.
Solutions
The inventory test fails with the row CustomerView.PhoneNumber unclassified, and Masker.Project returns [UNCLASSIFIED] for it in the meantime, so the value cannot leak while the test is red. Add [property: Classified(DataClass.Confidential, FieldKind.Phone)] after adding Phone to the FieldKind enum, and in Mask return the last three digits for SupportAgent, the full number for Customer, and [MASKED] for other roles. Because masking runs in the API, the rule applies to every endpoint that projects the type.
Keep a dictionary from caller to a queue of timestamps. On each Detokenise, drop timestamps older than a minute using clock.GetUtcNow(), then refuse if five remain. Write the audit entry first (as now) with Allowed = false for a refused call, so abuse is visible. Tests: five calls pass, the sixth throws, the audit shows six entries, and after advancing a manual TimeProvider by 61 seconds the next call passes. Remember that a per-process limit is bypassed by running many instances; production needs a shared counter (DynamoDB conditional updates or a gateway limit) and an alarm.
Knowledge check
Answer: Because a label drives controls. An unlabelled property would get the default behaviour (probably "log it and return it"), and new fields appear constantly. Forcing a label at review time catches the field before it ships.
Answer: Pseudonymisation replaces identifiers with consistent substitutes, usually so that data remains usable and joinable for analytics or testing; the mapping or key can re-identify. Tokenisation replaces a value with a random surrogate and keeps the real value in a protected vault, usually to keep a value like a card number out of most systems. Both reduce exposure; neither makes the data non-personal by itself.
Answer: Card numbers have a small effective search space (known BIN ranges plus the Luhn check), so an attacker can compute hashes for every candidate and look up matches. A keyed HMAC prevents that if the key is protected, and a random vault token avoids the problem entirely.
Answer: The label may be wrong, so the controls applied to the bucket may be too weak. It is evidence, not a verdict: the data owner confirms, then either moves the data, removes it or raises the bucket's label and controls.
Answer: Amazon S3 only. For data in RDS or DynamoDB, export it to S3 and scan the export, or use other discovery tools.
Answer: Refused attempts are the most interesting security signal. If only successful calls were logged, an attacker probing for a way in would leave no trace.
Interview questions
- "How would you stop sensitive fields ending up in logs?" Classify the data model in code, project objects through a classification-aware redactor with a fail-safe default for unlabelled properties, never log Restricted values, and add a test that no property is unlabelled. Mention the compliance redaction package for
ILoggeras a standard way to apply it, and log review as a detective control. - "Explain tokenisation to a product owner." Instead of keeping the card number everywhere, we keep a harmless stand-in everywhere except one tightly controlled vault. If a system is breached, the attacker gets stand-ins. Whether this reduces the audit scope is decided by the assessor, so we design for it but do not promise it.
- "Masking, encryption or tokenisation for a support screen?" Masking for display, encryption underneath for storage, and tokenisation where the real value is needed only by one system. Support sees the last four digits and never the full number.
- "How do you find personal data you did not know you had?" Macie for S3 (and exports of databases), a data inventory built from labelled models, tag policies and Config rules for unlabelled stores, and a review process for drift.
- "Is pseudonymised data safe to give a vendor?" It is still personal data under UK GDPR, so it needs a lawful basis, a contract, minimisation and security. Ask the data protection officer.
Certification practice
A company stores customer exports in several S3 buckets and suspects some contain payment card numbers although the buckets are tagged for internal use. The security team needs to discover which objects contain card numbers and be notified of new occurrences. What is the BEST solution?
A. Enable Amazon Macie, run sensitive data discovery jobs or automated discovery on the buckets, and send findings to Amazon EventBridge for notification. B. Enable S3 server access logging and search the logs for card numbers. C. Enable S3 Object Lock on the buckets. D. Use AWS Config to detect objects containing card numbers.
Answer: A. Macie inspects object contents with managed data identifiers and emits findings through EventBridge. B logs requests, not contents. C stops deletion and edits, not discovery. D evaluates resource configuration, not file contents.
An application must let a payments service charge stored cards, while the analytics and customer service systems must never be able to obtain full card numbers. Which design BEST meets this?
A. Store card numbers in each system, encrypted with the same KMS key. B. Store the card number only in a tokenisation vault with its own KMS key and a narrow role; give other systems tokens and the last four digits. C. Hash the card numbers with SHA-256 and share the hashes. D. Mask the card numbers in the user interface of each system.
Answer: B. The full number exists in one place and only the authorisation path can reach it. A gives every system the ability to decrypt. C is reversible by brute force and cannot be charged. D hides the value only on screen; the full number is still stored and returned by the API.
A bank wants developers to use realistic data in test environments without exposing customers. Joins between tables must keep working. Which approach is MOST appropriate, subject to approval by the data protection officer?
A. Copy production and rely on the test environment's network isolation. B. Replace identifiers with keyed deterministic pseudonyms generated with a key that is not available in the test environment, and treat the result as personal data. C. Delete all foreign keys. D. Encrypt the copy with the production key and give developers the key.
Answer: B. Consistent pseudonyms preserve joins, and keeping the key out of the test environment prevents re-identification there. A leaves real data in a weaker environment, C breaks the data model, and D gives developers the means to read everything.
A customer wants to know which AWS service can automatically discover sensitive data such as personal information stored in Amazon S3. Which service should be recommended?
A. Amazon Macie B. AWS Shield C. Amazon Inspector D. AWS Artifact
Answer: A. Macie discovers sensitive data in S3. Shield protects against DDoS, Inspector scans workloads for vulnerabilities and Artifact provides compliance reports.
Completion checklist
- I can define the four classification levels and say what each one obliges the engineering team to do
- I ran the lab tests and saw the inventory test fail for an unlabelled property
- I can explain the difference between masking, hashing, pseudonymisation, encryption and tokenisation
- I can say why tokenisation does not by itself reduce PCI DSS scope, and who decides
- I completed the guided exercise and the assignment
- I disabled Macie, ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- A classification label is only worth the controls attached to it: make the label drive keys, logging, masking and non-production rules, and enforce it in code.
- Masking is for display; encryption and tokenisation are for storage. Never rely on masking alone.
- Tokenisation keeps the most dangerous values in one tightly controlled place and gives everything else a harmless stand-in. Scope decisions belong to your QSA.
- Macie finds evidence of misclassification in S3; people correct the label.
- Pseudonymised data is still personal data, and plain hashes of small value spaces are not protection.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
S3 protection: Block Public Access, bucket policies, Object Lock and Macie
Lock an S3 bucket down so it cannot be made public, accepts only TLS and the bank's KMS key, keeps WORM copies of records with Object Lock, and is scanned for personal data by Amazon Macie.
Segmentation, private endpoints and egress filtering
Layer security groups, NACLs, VPC endpoint policies, DNS Firewall and AWS Network Firewall so a banking workload can reach only the services and partners it genuinely needs.