Release management and audit evidence: from ticket to commit to deployment
Turn change approval, segregation of duties and traceability into pipeline code: a release record that links ticket, pull request, commit, image digest, approvals and scans, a release policy that blocks bad releases, and tamper-resistant evidence storage.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3) · xUnit v3 · Terraform AWS provider 6.x · GitHub Actions · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what a change record, a release and a deployment are, and how to link them so an auditor can follow one chain from ticket to running task.
- Design a release evidence record and write a release policy as code that fails a production release which breaks segregation of duties.
- Collect approvals from GitHub environments and CodePipeline manual approvals, and correlate them with AWS CloudTrail.
- Store evidence in S3 with versioning, KMS encryption and Object Lock, and choose between GOVERNANCE and COMPLIANCE retention modes.
- Generate release notes and apply semantic versioning without guessing.
- Measure delivery performance with the DORA metrics without turning them into targets.
Prerequisites
- Git and GitHub for regulated teams — protected branches, CODEOWNERS, required reviews.
- Environment separation — per-environment approvals and roles.
- ECS deployment strategies — how a release actually reaches customers, and
PAUSEhooks. - Audit logging and evidence collection and Federation and segregation of duties in Module 5.
- Tools: AWS CLI v2, .NET 10 SDK (or Docker), the lab in
labs/m09-delivery-pipeline.
Why this matters in banking
A change-advisory board (CAB) asks the same questions every week: What is changing? Why? Who wrote it, who reviewed it, who approved it for production? What did we test? What happens if it goes wrong? Many banks answer with screenshots and spreadsheets assembled by hand after the fact, which are slow, incomplete and easy to dispute.
Three situations show why this lesson matters:
- The audit sample. An internal auditor picks release 4 of last quarter at random and asks you to prove the person who wrote the code did not also approve it for production. Without records you spend a week in chat history; with a record you open one JSON file.
- The incident. At 02:10 a payment bug appears. You must say which commit is running, which build produced it, whether it was scanned and who approved it. If your running service cannot tell you its own commit, you are guessing.
- The "emergency fix". An engineer pushes a change straight to production because it is urgent. Segregation of duties is a rule that matters most when people are in a hurry, so it must be enforced by the pipeline and not by memory.
This lesson does not give you a compliance programme. Whether a given record satisfies your internal policy, PCI DSS or any regulator is for your risk, compliance and audit teams to decide. What you can do is produce complete, consistent and tamper-resistant evidence automatically.
The idea in plain language
Three words are often mixed up:
| Term | Meaning | Example |
|---|---|---|
| Change | A business decision to alter a system, tracked in your ITSM tool | CHG-104233: "Raise the daily transfer limit warning" |
| Release | A specific, versioned, immutable build that implements one or more changes | accounts-api-2026.11.18-1, built from commit 3f9c2ab…, image …@sha256:4b1f… |
| Deployment | The act of putting a release into one environment at a point in time | The release reached prod at 10:05 UTC after approval |
Traceability means you can walk the chain in either direction:
Analogy: a bank draft. The draft names the payer, the payee, the amount, the date and the authorising signatures, and it carries a serial number. If anyone questions it years later, you can pull the record by serial number and see every signature. A release record is the serial-numbered draft for a software change.
What problem the AWS services and tools solve
| Piece | What it gives you | Limits and sharp edges |
|---|---|---|
| GitHub pull requests, reviews, merge history | Who proposed, reviewed and merged | Approvals can be dismissed or become stale after new commits; configure branch rules to dismiss stale approvals |
| GitHub environment protection | Required reviewers, "prevent self-review", branch restriction; approval history per run through the API | Repository admins can change settings; protect the settings with CODEOWNERS on workflow files and org-level rulesets ⚠ VERIFY current admin-bypass options |
| AWS CodePipeline manual approval | An approval action, with comments, recorded in the pipeline state and history; SNS notification | Approval permissions are IAM permissions on codepipeline:PutApprovalResult; scope them to the approver role |
| AWS CloudTrail | Independent record of who called which AWS API, including UpdateService and PutApprovalResult | Management events are on by default (90-day event history); keep a trail to S3 for longer retention. See Audit logging and evidence collection |
| Amazon S3 with Object Lock | Write-once-read-many storage: versions cannot be deleted or overwritten during retention | Object Lock needs versioning; COMPLIANCE mode cannot be shortened even by the root user, so test with GOVERNANCE first |
| AWS KMS | Encryption with key policies and audit | Keys cost money; keep key policy and the evidence role aligned |
| AWS Systems Manager Change Manager | A change-request workflow | In maintenance (see the module overview callout); model approval with GitHub environments and your ITSM tool instead |
| Amazon ECR image scanning and Amazon Inspector | Findings tied to an image digest | Keep the scan result in the evidence pack; findings change over time |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS / GitHub equivalent | What's different |
|---|---|---|
| Azure Boards work item linked to PR and build | Jira/ServiceNow ticket id in the branch name, PR title and commit trailer | The link is by convention plus a pipeline check, not native |
| Azure DevOps release approvals and pre-deployment gates | GitHub environment required reviewers, CodePipeline manual approval | Approver identity must be a person (not a service account) and ideally not the author |
| Azure DevOps release history and deployment records | Workflow run history plus the evidence record you generate | Run history is retained according to GitHub settings; copy what you need into your own store |
| Azure Activity Log | AWS CloudTrail | Both are independent, platform-side records of who did what |
| Immutable blob storage (WORM) with time-based retention | S3 Object Lock (GOVERNANCE or COMPLIANCE) | Same idea; Object Lock is per object version and needs bucket versioning |
| Change Management in Azure DevOps with ServiceNow | Same ITSM tool, integrated through APIs or webhooks | Keep the ticket id in the evidence; do not duplicate the whole ticket |
Architecture
Security-boundary view
The pipeline can only add records. It cannot read other releases, list the bucket or delete anything. Auditors can read but not write.
Step-by-step implementation
Agree the policy first
Sit with change management, risk and audit and write the rules in plain sentences: every production release references an approved change; the author cannot be the only reviewer; the production approver is not the author; no critical vulnerabilities without a documented risk acceptance; the image is referenced by digest. These sentences become the release policy code. If you cannot get agreement, do not automate yet.
Put the ticket id into the commit and pull request
Use a branch name such as feature/CHG-104233-transfer-limit-warning, require the id in the PR title with a check, and add a commit trailer. A pipeline check can reject a PR without a matching id.
Make the running service report what it is
The lab API exposes GET /version with version, commit, environment and release id, set at build and deploy time. Anyone can then map a running task to a release record.
Assemble the release record in the production job
Collect: ticket, repository, commit SHA, commit author, PR number, merged-by, reviewers, image digest, test summary, scan summaries, approvals (from the GitHub run approvals API), run URL and time. The lab's release.yml does this with gh api and jq.
Run the release policy before deploying
The policy returns a list of violations. Any violation fails the job before any AWS deploy call. The failed attempt is also valuable evidence, so record it too.
Deploy by digest, then store the evidence
Deploy using the digest that passed test. After the deployment finishes (success or failure), write the record to the evidence bucket with a conditional write that refuses to overwrite.
Cross-check with CloudTrail
On a schedule, a small job compares release records with CloudTrail UpdateService and RegisterTaskDefinition events in production. A deployment in CloudTrail without a release record is an unrecorded change, which is exactly what auditors want to know about.
AWS Console instructions
Create the evidence bucket (learning version): Console → S3 → Create bucket. Name hl-m09-release-evidence-<account-id>; Region us-east-1. Under Object Ownership keep ACLs disabled. Block all public access: on. Bucket Versioning: Enable. Under Advanced settings, Object Lock: Enable. After creation open Properties → Object Lock → Edit and set a default retention: GOVERNANCE for 1 day in the lab (COMPLIANCE for the retention period your records policy states in production). Default encryption: SSE-KMS with a customer managed key.
Review CloudTrail: Console → CloudTrail → Event history, filter Event name = UpdateService, read userIdentity.arn (it should show your OIDC deploy role session, named like gh-<run-id>-prod), eventTime and requestParameters.taskDefinition.
CodePipeline approvals: Console → CodePipeline → your pipeline → Release changes, find the Approval stage, and History → execution → the approval action shows approver and comment.
GitHub: Repository → Actions → the run → Review deployments shows who approved the prod environment and when.
AWS CLI
P=learn-sandbox
BUCKET=hl-m09-release-evidence-$(aws sts get-caller-identity --query Account --output text --profile $P)
# 1. List release records for one service and year.
aws s3api list-objects-v2 --bucket $BUCKET --prefix releases/hl-m09-accounts-api/2026/ \
--query 'Contents[].[Key,LastModified,Size]' --output table --profile $P
# 2. Read one record and confirm it has not changed: the pipeline stored its SHA-256 as object metadata.
aws s3api head-object --bucket $BUCKET --key releases/hl-m09-accounts-api/2026/accounts-api-2026.11.18-1.json \
--query '{Metadata:Metadata,ObjectLockMode:ObjectLockMode,RetainUntil:ObjectLockRetainUntilDate,VersionId:VersionId}' --profile $P
# 3. Show the Object Lock configuration of the bucket.
aws s3api get-object-lock-configuration --bucket $BUCKET --profile $P
# 4. A second write of the same key must fail (conditional write): this returns an error such as PreconditionFailed.
aws s3api put-object --bucket $BUCKET --key releases/hl-m09-accounts-api/2026/accounts-api-2026.11.18-1.json \
--body samples/release-evidence.json --if-none-match '*' --profile $P
# 5. Who changed the production service in the last day? (CloudTrail management events)
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService \
--start-time "$(date -u -d '-1 day' +%Y-%m-%dT%H:%M:%SZ)" \
--query 'Events[].[EventTime,Username]' --output table --profile $P
# 6. CodePipeline: who approved? get-pipeline-state shows the latest approval status and summary.
aws codepipeline get-pipeline-state --name hl-m09-accounts-api --profile $P \
--query "stageStates[?stageName=='Approval'].actionStates[].latestExecution"
# 7. Approve a pending CodePipeline approval (done by the approver role, never by the pipeline itself).
aws codepipeline put-approval-result --pipeline-name hl-m09-accounts-api --stage-name Approval \
--action-name ProdApproval --token <token-from-get-pipeline-state> \
--result summary="CHG-104233 reviewed; rollback tested in staging",status=Approved --profile $PGitHub's side of approvals can be read with the GitHub CLI inside the job, using the run's own token:
gh api repos/OWNER/REPO/actions/runs/RUN_ID/approvals --jq '[.[] | {approver: .user.login, state: .state, environments: [.environments[].name]}]'Infrastructure as Code
The evidence bucket is the part worth codifying. The Terraform version is in the lab (infra/terraform/evidence.tf); here are both tools.
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.S3;
using Constructs;
namespace Harbourline.Delivery.Infra;
public sealed class EvidenceBucket : Construct
{
public Bucket Bucket { get; }
public EvidenceBucket(Construct scope, string id, IRole prodDeployRole) : base(scope, id)
{
var key = new Key(this, "Key", new KeyProps
{
EnableKeyRotation = true,
Alias = "alias/hl-release-evidence",
RemovalPolicy = RemovalPolicy.RETAIN,
});
Bucket = new Bucket(this, "Evidence", new BucketProps
{
BucketName = $"hl-m09-release-evidence-{Stack.Of(this).Account}",
Versioned = true,
ObjectLockEnabled = true,
// Learning shortcut: GOVERNANCE 1 day. Production: COMPLIANCE for the records-policy period.
ObjectLockDefaultRetention = ObjectLockRetention.Governance(Duration.Days(1)),
Encryption = BucketEncryption.KMS,
EncryptionKey = key,
BucketKeyEnabled = true,
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
EnforceSSL = true,
RemovalPolicy = RemovalPolicy.RETAIN, // evidence outlives the stack
});
// The prod deploy role may only ADD records under releases/. No read, no list, no delete.
prodDeployRole.AddToPrincipalPolicy(new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.ALLOW,
Actions = ["s3:PutObject"],
Resources = [Bucket.ArnForObjects("releases/*")],
}));
key.GrantEncrypt(prodDeployRole);
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m09-delivery-pipeline");
}
}Property names such as ObjectLockDefaultRetention come from the CDK Bucket construct; confirm them against your Amazon.CDK.Lib version ⚠ VERIFY.
GOVERNANCE or COMPLIANCE? In GOVERNANCE mode, principals with the s3:BypassGovernanceRetention permission can remove protection; it suits training and processes where an authorised override must exist. In COMPLIANCE mode nobody, including the account root user, can shorten or remove the retention until it expires, so a typo in a retention period is a long-lived cost. Choose with records management and legal; prototype in GOVERNANCE.
Complete code
The lab already contains the evidence model, the release policy and the store. Read them in src/Harbourline.Delivery/Evidence/. Here is how they fit together in a small program, including a release-notes generator that is not part of the lab and that you can add:
using System.Text;
using System.Text.RegularExpressions;
namespace Harbourline.Delivery.Evidence;
public enum Bump { None, Patch, Minor, Major }
public sealed record CommitInfo(string Sha, string Subject, string Author);
public static partial class ReleaseNotes
{
// Conventional-commit style: feat: ..., fix: ..., feat!: ..., or "BREAKING CHANGE" in the subject.
[GeneratedRegex(@"^(?<type>feat|fix|perf|refactor|docs|chore|test|build|ci)(\((?<scope>[^)]+)\))?(?<bang>!)?:\s*(?<text>.+)$")]
private static partial Regex Conventional();
[GeneratedRegex(@"CHG-\d{5,}")]
private static partial Regex Ticket();
public static Bump Classify(IEnumerable<CommitInfo> commits)
{
var bump = Bump.None;
foreach (var c in commits)
{
var m = Conventional().Match(c.Subject);
if (!m.Success) { bump = Max(bump, Bump.Patch); continue; }
if (m.Groups["bang"].Success || c.Subject.Contains("BREAKING CHANGE", StringComparison.Ordinal))
{
return Bump.Major;
}
bump = m.Groups["type"].Value switch
{
"feat" => Max(bump, Bump.Minor),
"fix" or "perf" => Max(bump, Bump.Patch),
_ => bump,
};
}
return bump;
}
public static string Render(string version, string changeTicket, IReadOnlyList<CommitInfo> commits)
{
var sb = new StringBuilder();
sb.AppendLine($"# Release {version}");
sb.AppendLine($"Change: {changeTicket}");
sb.AppendLine();
foreach (var group in commits.GroupBy(c => Conventional().Match(c.Subject) is { Success: true } m ? m.Groups["type"].Value : "other"))
{
sb.AppendLine($"## {group.Key}");
foreach (var c in group)
{
var ticket = Ticket().Match(c.Subject);
sb.AppendLine($"- {c.Subject} ({c.Sha[..7]}, {c.Author}){(ticket.Success ? "" : " [no ticket]")}");
}
sb.AppendLine();
}
return sb.ToString();
}
private static Bump Max(Bump a, Bump b) => (Bump)Math.Max((int)a, (int)b);
}And a short program that loads the sample record, applies the policy and prints the verdict (the lab's CLI evidence command does the same):
using Harbourline.Delivery.Evidence;
var evidence = ReleaseEvidence.FromJson(File.ReadAllText("samples/release-evidence.json"));
var violations = ReleasePolicy.Violations(evidence);
if (violations.Count == 0)
{
Console.WriteLine($"{evidence.ReleaseId}: release policy satisfied. Record hash {evidence.Sha256()[..12]}…");
return 0;
}
foreach (var v in violations) Console.WriteLine($"VIOLATION: {v}");
return 1;The tests that prove the policy (author cannot approve their own release, no tests means no evidence, critical findings need documented acceptance) are in tests/Harbourline.Delivery.Tests/EvidenceStoreTests.cs and ImageAndGateTests.cs. Run them with dotnet test --solution DeliveryPipeline.slnx from the lab root. The full production pipeline that creates the record is on the module project page.
Code explained line by line
ReleaseEvidence(lab) is an immutable C#record: ticket, repository, commit SHA, author, merger, reviewers, image reference, test summary, scans, approvals, run URL and creation time. Required properties mean a missing field is a compile or deserialisation error, not a silent blank.Sha256()hashes the canonical JSON;EvidenceStore.SaveAsyncstores it as object metadata, so a later reader can detect changes even before Object Lock is considered.EvidenceStore.SaveAsyncsetsIfNoneMatch = "*": the write succeeds only if the key does not exist. A rerun of the same release cannot overwrite the first record.ReleasePolicy.Violationsreturns all problems, not only the first, so the engineer fixes them in one go. It checks: ticket present, 40-character SHA, digest-pinned image, no failed tests, at least one test ran, critical findings covered by accepted-risk ids, a production approval exists, at least one reviewer who is not the author, and the production approver is not the author.Same(a, b)compares identities case-insensitively and trimmed, becausePriya.Shahandpriya.shahare the same person. Real systems should compare immutable ids, not display names ⚠ VERIFY which identifier your ITSM and GitHub integration exposes.- In
ReleaseNotes.Classify, a!after the type or the wordsBREAKING CHANGEreturnsMajorimmediately;featraises toMinor;fixandperftoPatch. Commits that do not follow the convention default toPatch, so the pipeline never under-reports a change. [GeneratedRegex]compiles the regular expressions at build time, which is the modern .NET approach.Rendergroups commits by type and flags any commit with no change ticket. A pipeline policy can turn that flag into a failure.- In
CheckRelease.csthe exit code is the contract with the pipeline: 0 means proceed, 1 means block.
Security considerations
⚠ Learning shortcut — not for production
The lab uses GOVERNANCE mode for one day, a single bucket in the learning account, a placeholder tests summary in the workflow and trusts the commit author name that GitHub reports. Production: COMPLIANCE mode with the retention your records policy requires, the evidence bucket in a separate security or log-archive account, real test counts read from the JUnit artifacts, immutable identity ids instead of display names, and cross-account replication for resilience.
- The pipeline is a witness, not the judge. Evidence generated by the same system that deploys can be wrong or forged by whoever controls the pipeline. Independent records (CloudTrail, GitHub audit log, ITSM history) let you cross-check.
- Segregation of duties needs more than a check. The release policy blocks the obvious case. Also protect the configuration that defines it: CODEOWNERS on
.github/workflows/, rulesets that prevent admins bypassing reviews for the default branch, and "prevent self-review" on the production environment. See Federation and segregation of duties. - Least privilege on the bucket. The deploy role writes only under
releases/*. Readers cannot write. Nobody deletes during retention. Add a bucket policy denyings3:DeleteObjectVersionands3:PutBucketObjectLockConfigurationto everyone except a break-glass security role ⚠ VERIFY exact actions to deny. - Personal data. Records name people (author, reviewers, approvers). Under UK GDPR and GDPR keep only what is needed, define a retention period, and ask your data protection officer whether names or user ids are appropriate in evidence.
- Do not store secrets or full logs in evidence. Keep references (run URL, scan summary) rather than raw logs that may contain tokens.
- Compliance. This is evidence engineering. It supports your control narrative; it does not satisfy PCI DSS, SOX-style controls or any regulation by itself.
Banking use case
Harbourline's rule for the Accounts API: standard changes go to production only through the pipeline. A release manager opens CHG-104233 in the ITSM tool, the engineer references it in the PR, a second engineer reviews, and on merge the pipeline builds, tests, scans and deploys to test. To release to production, the release manager starts the release workflow with the ticket id. The prod environment requires a reviewer from the risk and operations group, never the author, and the workflow runs the release policy. After the deployment the record lands in the evidence bucket. Each month the internal audit team samples five releases: for each they open the record, check the PR and approvals in GitHub, and look up the matching CloudTrail UpdateService event. An emergency fix follows the same path with an expedited approval and a ticket of type "emergency", reviewed after the event by the CAB. The pipeline does not make the process faster by skipping controls; it makes the controls cheap enough to apply always.
Common mistakes
- Evidence assembled by hand afterwards. It is incomplete, inconsistent and not trusted. Generate it in the pipeline.
- Approval by the author. Enable "prevent self-review" and enforce it again in policy code.
- Image referenced by tag. A tag can move. Record and deploy the digest.
- Keeping only the build log. Logs expire and are noisy. Keep a compact structured record and link to the run.
- COMPLIANCE mode too early. A mistaken long retention cannot be undone. Test in GOVERNANCE and agree the period first.
- Ticket ids that are not checked. If
CHG-???is free text, it will be wrong sometimes. Validate the format and, where possible, the status of the ticket through the ITSM API. - Metrics as targets. Turning deployment frequency into a performance target makes teams split releases to look good.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Release policy fails: "No production approval recorded" | The approvals API returned nothing because the job ran before approval, or the token lacks actions: read | Collect approvals in the job after the environment gate; grant actions: read |
gh api .../approvals returns 404 | Wrong run id or no environment with required reviewers on that run | Use github.run_id; confirm the job declares an environment with reviewers |
PreconditionFailed writing evidence | The key already exists (a re-run of the same release) | Intended; use a new version or a -retry-1 suffix and explain the retry in the record |
AccessDenied writing to the bucket | Deploy role lacks s3:PutObject on releases/* or kms:GenerateDataKey on the key | Add both; check the bucket policy |
| Cannot delete bucket at teardown | Locked object versions within retention | Wait for retention or use a role with s3:BypassGovernanceRetention and --bypass-governance-retention (GOVERNANCE only) |
commitAuthor is a bot or a squash committer | Squash merge replaced author or GitHub reports the merger | Prefer the PR author from the PR API, record both, and compare identities with a mapping to staff ids |
CloudTrail shows UpdateService with no release record | Deployment outside the pipeline | Treat as an incident of process; alert, investigate and reconcile |
Record hash differs from the stored sha256 metadata | Serialisation changed or object was altered | Re-serialise with the same options; if not explained, escalate |
Cost and cleanup
Cost: free. The tests and policy run locally. Storing a few small JSON files in S3 and one KMS key cost a few cents a month (a customer managed KMS key is billed monthly, about 1 USD, plus requests ⚠ VERIFY). If you deploy the bucket for the lab, destroy it afterwards.
P=learn-sandbox
cd labs/m09-delivery-pipeline/infra/terraform
terraform destroy
# Objects under Object Lock (GOVERNANCE, 1 day) cannot be deleted until retention passes, unless bypassed:
BUCKET=hl-m09-release-evidence-$(aws sts get-caller-identity --query Account --output text --profile $P)
aws s3api list-object-versions --bucket $BUCKET --profile $P
# For each version (only with s3:BypassGovernanceRetention permission, GOVERNANCE mode only):
aws s3api delete-object --bucket $BUCKET --key <key> --version-id <id> --bypass-governance-retention --profile $P
# Confirm nothing tagged for the lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m09-delivery-pipeline --profile $P --region us-east-1Free alternative: run everything against the sample files. dotnet run --project src/Harbourline.Delivery -- evidence samples/release-evidence.json prints "Release policy satisfied"; copy the file, set commitAuthor equal to the approver and watch the policy fail.
Exercises
- 🟢 Guided: Run the lab's evidence command on
samples/release-evidence.json. Then copy it, change the approver topriya.shah(the commit author) and run it again. Write down which rule fired and why that rule exists. - 🟡 Challenge: Extend
ReleaseEvidenceandReleasePolicywith aRollbackPlanUrlfield and a rule that production releases must include one. Add xUnit tests for the pass and fail cases, and update the sample JSON. - 🔴 Live problem: Ticket CHG-104412: "Internal audit sampled release
accounts-api-2026.10.07-2. The release record names the same person as commit author and prod approver. The pipeline allowed it. Explain how, fix it, and tell audit what to do about the existing release." Include what you would check in GitHub settings, how you would find other affected releases, and how you handle the finding.
Assignment
Add to your portfolio repository: (1) a docs/release-process.md with the chain of ticket, PR, commit, digest, deployment and who approves what; (2) a release-policy test suite for at least six rules; (3) a sample evidence pack folder containing one release record, the SBOM and the scan report. Describe in one paragraph what independent record you would compare it with.
Solutions
The original sample passes (Release policy satisfied): author priya.shah, merged by tom.okafor, reviewer tom.okafor, prod approver grace.adeyemi. After changing the approver to priya.shah, the policy reports "The commit author approved their own production release." The rule exists because the person who wrote a change must not also be the independent control that releases it; one person could otherwise introduce and release a defect or a fraud without a second pair of eyes.
Add public required string RollbackPlanUrl { get; init; } to ReleaseEvidence, and in ReleasePolicy.Violations add:
if (string.IsNullOrWhiteSpace(e.RollbackPlanUrl))
{
v.Add("No rollback plan: production releases must link a rehearsed rollback procedure.");
}Tests: a record with a URL yields no violation for that rule; a record with an empty string yields exactly that message. Update samples/release-evidence.json with a rollbackPlanUrl, otherwise deserialisation fails because the property is required. The lesson in design: required makes the missing field loud.
Likely causes: the production environment lacked "prevent self-review", the policy step was skipped or continue-on-error was set, the approver identity mapping compared different display names for the same person, or the release ran from a branch that did not contain the policy step. Check the environment settings, the workflow history for removed steps (CODEOWNERS should have required review for workflow changes), and the job log. Fix: enable prevent self-review, make the policy step required and non-bypassable, compare immutable ids, and protect workflow files. Find other releases by scanning the evidence bucket for records where author equals approver, and cross-check with GitHub approvals. Tell audit exactly what you found, do not rewrite the record (Object Lock forbids it anyway); add a dated finding record linked to the release, and follow your bank's incident and exception process.
Knowledge check
Answer: A release is a specific immutable build (version, commit, image digest). A deployment is putting a release into one environment at a time. One release can have several deployments (test, staging, prod).
Answer: The write succeeds only if the key does not exist, so a re-run cannot overwrite the original record. It complements Object Lock, which protects stored versions.
Answer: COMPLIANCE. In GOVERNANCE, principals with the bypass permission can remove protection.
Answer: A digest identifies exact bytes; a tag can be moved to different content, which would make the record misleading.
Answer: It is an independent, platform-side record. A production UpdateService with no matching release record reveals an unrecorded change, even if the pipeline's own data is wrong or missing.
Interview questions
- How do you enforce segregation of duties in a pipeline? Branch rules requiring reviews by someone other than the author, environment approvals with prevent self-review, and a release policy in code that fails the job; plus protecting the configuration and the workflow files and cross-checking with independent logs.
- What would you put in a release evidence record? Ticket, repo, commit, author, reviewers, merger, image digest, tests, scans, approvals, run URL, timestamps, and the hash of the record.
- How do you keep evidence tamper-resistant? Append-only writes, S3 versioning with Object Lock, KMS, restricted roles, independent cross-checks.
- GOVERNANCE vs COMPLIANCE? Overridable by authorised principals vs not shortenable by anyone until expiry; pick with records management.
- What are the DORA metrics and how can they mislead? Deployment frequency, lead time for changes, change failure rate and time to restore; they mislead when used as targets or compared across unlike teams.
Certification practice
A bank must retain release records for seven years and ensure nobody, including administrators, can alter or delete them during that time. Which solution meets the requirement?
A. An S3 bucket with versioning and a lifecycle rule to Glacier. B. An S3 bucket with Object Lock in COMPLIANCE mode and a seven-year default retention. C. An S3 bucket with Object Lock in GOVERNANCE mode and seven-year retention. D. A DynamoDB table with point-in-time recovery.
Answer: B. COMPLIANCE mode prevents any principal from shortening retention or deleting locked versions. Versioning alone (A) lets authorised users delete versions; GOVERNANCE (C) can be bypassed with the right permission; PITR (D) is for recovery, not immutability.
An auditor asks for proof that no production deployment happened outside the approved pipeline last month. Which data source gives an independent record?
A. The pipeline's own log files.
B. AWS CloudTrail events for UpdateService and RegisterTaskDefinition, correlated with release records.
C. ECS task CPU metrics.
D. ECR image scan results.
Answer: B. CloudTrail is recorded by the platform and shows the identity that made each call; any call whose session is not the pipeline role, or has no release record, is a finding. Pipeline logs (A) come from the system being audited; C and D do not show who deployed.
A pipeline approves production releases through a manual approval action. The company wants only the on-call lead role to approve and wants approvals recorded. What is the best configuration?
A. Allow codepipeline:PutApprovalResult for all developers.
B. Allow codepipeline:PutApprovalResult only for the on-call lead role, rely on the approval history and CloudTrail for the record.
C. Disable the approval action and use Slack messages.
D. Give the pipeline role permission to approve its own action.
Answer: B. Scoping the IAM permission limits who can approve; the approval history and CloudTrail record who did. A broadens access, C leaves no reliable record, D defeats the control.
Completion checklist
- I can explain change, release and deployment and trace one through the chain
- I ran the release policy on the sample and saw a violation I created
- I can say when to use GOVERNANCE and when COMPLIANCE
- I know which independent record I would cross-check against
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Evidence is a by-product of the pipeline, generated the same way every time.
- A release record links ticket, PR, commit, digest, approvals and scans, and the running service can report its own commit.
- Segregation of duties is enforced by settings and by policy code, and protected against change.
- Object Lock, versioning, conditional writes and narrow roles make records tamper-resistant; cross-checks with CloudTrail make them credible.
- DORA metrics describe, they do not grade.
Further reading
- Locking objects with S3 Object Lock
- Conditional writes in Amazon S3
- Manage approval actions in CodePipeline
- Reviewing deployments (GitHub Docs) ⚠ VERIFY URL
- DORA research and metrics ⚠ VERIFY the current metric set
- Next: Module project: complete CI/CD pipeline
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Infrastructure automation in the pipeline: CDK and Terraform with approval gates
Run CDK and Terraform from GitHub Actions the way a bank needs: plan on every pull request with a read-only role, apply only from main after a protected-environment approval, apply exactly the plan that was reviewed, and detect drift.
Module project: complete CI/CD for an ASP.NET Core API
Build, scan and sign off a release of the Harbourline Accounts API end to end: OIDC everywhere, one image built once and deployed by digest to test and then, after a human approval and a release-policy check, to production with ECS blue/green and alarm rollback, ending in an append-only evidence pack.