Automated testing strategy for pipelines: unit, integration, contract, end-to-end and load
Decide which tests run at which pipeline stage, write fast unit tests, in-process integration tests, Testcontainers tests against PostgreSQL and LocalStack, a consumer contract test, a post-deploy smoke test and a k6 load test, and keep the whole suite trustworthy and quick.
Assumes: .NET 10 LTS · xUnit v3 4.0.3 on Microsoft.Testing.Platform · Microsoft.AspNetCore.Mvc.Testing 10.0.15 · Testcontainers.LocalStack 4.17.0 · Testcontainers.PostgreSql 4.17.0 · k6 · AWS SDK for .NET v4 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Place unit, integration, contract, end-to-end and load tests at the pipeline stage where they give the fastest useful feedback.
- Write xUnit v3 unit tests and in-process integration tests with
WebApplicationFactorythat need no network and no AWS account. - Use Testcontainers to start a real PostgreSQL database and a LocalStack S3 in a test, and know when emulation is not good enough.
- Write a lightweight consumer contract test that turns an accidental breaking change into a red build.
- Add a post-deployment smoke test and a k6 load test with pass/fail thresholds.
- Publish JUnit test results from CI and manage flaky tests without hiding them.
Prerequisites
- CI/CD concepts on AWS — stages and gates.
- Docker and Amazon ECR in pipelines — you build the image the later tests exercise.
- Tools: Docker Desktop, and either the .NET 10 SDK or the
mcr.microsoft.com/dotnet/sdk:10.0image. No AWS account is required.
Why this matters in banking
A rounding rule changes, a JSON field is renamed from balance to amount, a package update changes how dates serialise. None of these crashes the service. The API still returns 200 OK. The mobile app, however, shows a blank balance to 40,000 customers.
Tests are the pipeline's evidence that behaviour did not change by accident. For a bank they also serve a second purpose: the test result is a record. When an auditor asks how you know a release was tested, a retained JUnit report tied to the commit and the run is a better answer than "the team said so".
There is a trap in the other direction. A suite that takes 90 minutes, fails randomly and is rerun until green teaches engineers to ignore red. A good strategy therefore balances coverage (what is proven) against speed and trust (how soon, and how reliably).
The idea in plain language
Think of tests as concentric rings of confidence, each slower and more expensive than the last:
- Unit tests check one piece of logic (an account-number validator). Milliseconds, thousands of them.
- Integration tests check that pieces work together: the HTTP layer, serialisation and a real database. Seconds.
- Contract tests check that the shape of what you send or receive matches what another team depends on. Seconds.
- End-to-end (E2E) and smoke tests check that a deployed system works in a real environment. Minutes.
- Load tests check that the system keeps its promises under traffic. Many minutes.
The well-known picture is a pyramid: many fast tests at the bottom, few slow ones at the top. The reason is cost, not fashion. A failing unit test points at a function; a failing E2E test points at "something, somewhere".
Analogy: testing a payment-card machine. You check each chip on the bench (unit), then the assembled machine with a test bank (integration), then check the receipt layout matches what the till software expects (contract), then run a card through one real terminal in the shop (smoke), and finally rush 500 customers through (load). You would not rush 500 customers through to find a loose wire.
What each test type is for
| Type | Proves | Needs | Typical time | Runs |
|---|---|---|---|---|
| Unit | A piece of logic is correct | Nothing | Under 1 second each | Every commit and pull request |
| In-process integration | HTTP, routing, serialisation, validation and DI work together | Nothing external | Seconds | Every commit and pull request |
| Container integration (Testcontainers) | Real PostgreSQL behaviour, real S3 API shape (emulated) | Docker | 10 to 60 seconds | Pull request or main |
| Contract | Response shape matches the published consumer contract | Nothing external | Seconds | Every commit and pull request |
| Smoke (post-deploy) | The deployed service is the right version and basic paths work | A deployed environment | Under 1 minute | After every deployment |
| End-to-end | A business journey works across services | A full environment | Minutes | After test deployment, nightly |
| Load / performance | Latency and error rate under traffic | A prod-like environment | 10+ minutes | Before major releases, scheduled |
What problem the tools solve
| Tool | What it solves | Limits and sharp edges |
|---|---|---|
| xUnit v3 on Microsoft.Testing.Platform (MTP) | A modern test framework; tests are executables, no VSTest adapter | Command-line options differ from VSTest ⚠ VERIFY the dotnet test syntax for your SDK; the lab uses dotnet test --solution |
WebApplicationFactory<Program> | Runs the whole ASP.NET Core pipeline in memory | Your app needs a public Program (a partial class or InternalsVisibleTo) |
| Testcontainers for .NET | Starts real dependencies (PostgreSQL, LocalStack) in disposable containers | Needs a Docker daemon; slower than in-memory; images must be pulled |
| LocalStack | Emulates many AWS APIs locally | An emulator is not AWS: IAM enforcement, eventual consistency, limits and newer features can differ. Recent image releases need an auth token ⚠ VERIFY current licensing |
| NSubstitute | Fakes an AWS SDK client (IAmazonS3) in a unit test | A fake only proves your code calls the SDK the way you expect |
| k6 (open source) | Scripted load tests with thresholds that fail the run | You must run it from somewhere with network access to the target |
| JUnit XML reports | A machine-readable result that CI, CodeBuild and auditors can display | Needs a reporter extension in MTP; the lab passes --report-junit |
Be honest about emulation. LocalStack is excellent for checking that your code calls S3 correctly, that the request is well-formed and that your error handling works. It cannot tell you whether the real IAM policy permits the call. Use it for the first and an AWS test account (or the sandbox) for the second.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS or open-source equivalent | What is different |
|---|---|---|
| MSTest/NUnit/xUnit with VSTest | xUnit v3 on Microsoft.Testing.Platform | Tests are executables; same C# you already write |
WebApplicationFactory | Identical | It is an ASP.NET Core feature, not AWS-specific |
| Azurite (Azure Storage emulator) | LocalStack, or Testcontainers modules | LocalStack covers many services; Azurite covers Storage |
| Azure Test Plans | Not an AWS service; tests live in code and CI | There is no AWS-native test-management product ⚠ VERIFY |
| Azure Load Testing | k6, NBomber, or Distributed Load Testing on AWS (a solution template) ⚠ VERIFY current name | Run it as a pipeline step or a scheduled job |
| Azure Pipelines test results tab | Actions run summary / artefacts, CodeBuild report groups | Publish JUnit XML so results are viewable |
| Pact / consumer-driven contracts | Same tools; or a simple schema test | A lightweight test is a good start; a broker is optional |
Architecture
Which test runs where
Security-boundary view
Tests that run on pull requests never use real credentials. LocalStack accepts any key pair, so tests use the obvious fake test/test. Tests that touch a real AWS account use a separate, narrowly scoped test identity and run only after merge.
Step-by-step implementation
Start from the lab's test layout
labs/m09-delivery-pipeline has two test projects: Harbourline.Accounts.Api.Tests (unit, in-process integration and contract tests) and Harbourline.Delivery.Tests (policy, image-tag, health-gate and evidence tests, including the LocalStack test). Both use xUnit v3 4.0.3 on Microsoft.Testing.Platform.
Write unit tests for pure logic
Pure logic is the cheapest thing to test. Use [Theory] with [InlineData] for edge cases. Include the bad inputs: null, whitespace, wrong case, wrong length.
Add in-process integration tests
Use WebApplicationFactory<Program> to call real endpoints through the real middleware. Override configuration per test with WithWebHostBuilder and UseSetting.
Add a contract test for every consumer-facing response
List the fields and JSON value kinds that consumers depend on, and assert them. This catches renames and type changes that compile cleanly.
Add Testcontainers for what an in-memory fake cannot prove
Use a real PostgreSQL container for SQL behaviour (constraints, transactions, collations) and LocalStack for S3 API shape. Gate the tests behind an environment variable so developers without Docker are not blocked.
Publish the results
Run tests with --report-junit, upload the XML as an artefact, and keep it for the evidence window your policy sets.
Add post-deploy smoke and scheduled load tests
The smoke test checks /health/ready and /version against the deployed environment. The load test runs on a schedule against staging, not on every commit.
AWS Console instructions
Most of this lesson is code, but two console views help:
- CodeBuild test reports (if you run tests in CodeBuild): Console → CodeBuild → Report groups. Each build that declares a
reports:section in the buildspec produces a report with pass and fail counts and test case durations. - CloudWatch for load tests: Console → CloudWatch → Dashboards. During a load test watch
HTTPCode_Target_5XX_CountandTargetResponseTimefor the load balancer, and CPU and memory for the ECS service.
For GitHub Actions, open a workflow run and look at Summary → Artifacts for the uploaded test-results.
AWS CLI
The test code itself does not need the AWS CLI. These commands help when tests run in CodeBuild or when a load test runs against an ALB.
# List CodeBuild report groups (test reports published by builds).
aws codebuild list-report-groups --profile learn-sandbox --region us-east-1
# Show the latest reports in a group (replace the ARN with yours).
aws codebuild list-reports-for-report-group \
--report-group-arn arn:aws:codebuild:us-east-1:111122223333:report-group/hl-m09-tests \
--profile learn-sandbox --region us-east-1
# Fetch the 5xx count for a load balancer over the last 15 minutes (replace the dimension value).
aws cloudwatch get-metric-statistics \
--namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
--dimensions Name=LoadBalancer,Value=app/hl-m09-alb/1234567890abcdef \
--start-time "$(date -u -d '-15 minutes' +%Y-%m-%dT%H:%M:%SZ)" \
--end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--period 60 --statistics Sum \
--profile learn-sandbox --region us-east-1Infrastructure as Code
The one piece of infrastructure tests need is a place to publish results when you use CodeBuild: a report group. It is optional if you run tests only in GitHub Actions.
using Amazon.CDK;
using Amazon.CDK.AWS.CodeBuild;
using Constructs;
namespace Harbourline.Pipelines;
public class TestReportsStack : Stack
{
public TestReportsStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
// A report group collects JUnit results from CodeBuild builds. RemovalPolicy.DESTROY is a learning shortcut.
_ = new ReportGroup(this, "Tests", new ReportGroupProps
{
ReportGroupName = "hl-m09-tests",
RemovalPolicy = RemovalPolicy.DESTROY,
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m09-testing");
}
}Complete code
The lab already contains tested code; read it in this order. Everything below comes from labs/m09-delivery-pipeline and passes without an AWS account (74 tests passed, 1 skipped when last run).
1. Unit tests: pure logic
public class IdentifierTests
{
[Theory]
[InlineData("ACC-0000000001", true)]
[InlineData("ACC-000000001", false)]
[InlineData("acc-0000000001", false)]
[InlineData("ACC-0000000001 ", false)]
[InlineData(null, false)]
public void Account_ids_follow_the_canonical_format(string? id, bool valid) =>
Assert.Equal(valid, Identifiers.IsAccountId(id));
}2. In-process integration tests
public class ApiIntegrationTests(WebApplicationFactory<Program> factory) : IClassFixture<WebApplicationFactory<Program>>
{
[Fact]
public async Task Forced_unready_returns_503_for_the_rollback_drill()
{
var client = factory.WithWebHostBuilder(b => b.UseSetting("Health:ForceUnready", "true")).CreateClient();
var ready = await client.GetAsync("/health/ready", TestContext.Current.CancellationToken);
var live = await client.GetAsync("/health/live", TestContext.Current.CancellationToken);
Assert.Equal(HttpStatusCode.ServiceUnavailable, ready.StatusCode);
Assert.Equal(HttpStatusCode.OK, live.StatusCode); // liveness must not fail, or ECS would restart-loop
}
}3. Consumer contract test
public class AccountContractTests(WebApplicationFactory<Program> factory) : IClassFixture<WebApplicationFactory<Program>>
{
private static readonly Dictionary<string, JsonValueKind> Contract = new()
{
["accountId"] = JsonValueKind.String,
["customerId"] = JsonValueKind.String,
["iban"] = JsonValueKind.String,
["currency"] = JsonValueKind.String,
["product"] = JsonValueKind.String,
["status"] = JsonValueKind.String,
["balance"] = JsonValueKind.Number,
["openedAt"] = JsonValueKind.String,
};
[Fact]
public async Task Account_response_honours_the_published_contract()
{
var json = await factory.CreateClient().GetStringAsync("/v1/accounts/ACC-0000000001", TestContext.Current.CancellationToken);
using var doc = JsonDocument.Parse(json);
foreach (var (field, kind) in Contract)
{
Assert.True(doc.RootElement.TryGetProperty(field, out var value), $"Contract field '{field}' is missing.");
Assert.Equal(kind, value.ValueKind);
}
}
}4. Unit test with a faked AWS client, and a LocalStack integration test
The unit test substitutes IAmazonS3; the integration test starts LocalStack with Testcontainers, and is skipped unless you opt in:
[Fact]
public async Task Round_trips_evidence_through_s3_and_refuses_overwrite()
{
Assert.SkipUnless(Environment.GetEnvironmentVariable("RUN_LOCALSTACK_TESTS") == "1",
"Set RUN_LOCALSTACK_TESTS=1 (and have Docker running) to run LocalStack integration tests.");
var ct = TestContext.Current.CancellationToken;
var image = Environment.GetEnvironmentVariable("LOCALSTACK_IMAGE") ?? "localstack/localstack:4.14";
var builder = new LocalStackBuilder(image);
var token = Environment.GetEnvironmentVariable("LOCALSTACK_AUTH_TOKEN");
if (!string.IsNullOrEmpty(token))
{
builder = builder.WithEnvironment("LOCALSTACK_AUTH_TOKEN", token);
}
await using var localstack = builder.Build();
await localstack.StartAsync(ct);
using var s3 = new AmazonS3Client(
new BasicAWSCredentials("test", "test"), // LocalStack accepts any credentials; never real ones
new AmazonS3Config { ServiceURL = localstack.GetConnectionString(), ForcePathStyle = true, AuthenticationRegion = "us-east-1" });
// ... create bucket, save, load, and assert a second save is refused
}5. A PostgreSQL Testcontainers test
This pattern is for a service with a database. It is not part of the lab (the lab API uses in-memory sample data), so treat it as a template ⚠ VERIFY by running it once in your own project.
using Npgsql;
using Testcontainers.PostgreSql;
namespace Harbourline.Accounts.Tests;
// IAsyncLifetime starts one container for the whole class and disposes it afterwards.
public class AccountRepositoryTests : IAsyncLifetime
{
private readonly PostgreSqlContainer _db = new PostgreSqlBuilder("postgres:17-alpine")
.WithDatabase("accounts")
.WithUsername("test")
.WithPassword("test-only-password") // a throw-away container; never a real secret
.Build();
public async ValueTask InitializeAsync() => await _db.StartAsync();
public async ValueTask DisposeAsync() => await _db.DisposeAsync();
[Fact]
public async Task Balance_column_rejects_nulls_and_keeps_two_decimals()
{
await using var conn = new NpgsqlConnection(_db.GetConnectionString());
await conn.OpenAsync(TestContext.Current.CancellationToken);
await using (var create = new NpgsqlCommand(
"CREATE TABLE account (id text PRIMARY KEY, balance numeric(18,2) NOT NULL)", conn))
{
await create.ExecuteNonQueryAsync(TestContext.Current.CancellationToken);
}
await using var insert = new NpgsqlCommand("INSERT INTO account VALUES ('ACC-0000000001', 1520.455)", conn);
await insert.ExecuteNonQueryAsync(TestContext.Current.CancellationToken);
await using var read = new NpgsqlCommand("SELECT balance FROM account WHERE id = 'ACC-0000000001'", conn);
var balance = (decimal)(await read.ExecuteScalarAsync(TestContext.Current.CancellationToken))!;
Assert.Equal(1520.46m, balance); // PostgreSQL rounds half away from zero for numeric
}
}The builder constructor that takes an image name is the current Testcontainers style; older tutorials use a parameterless builder with WithImage ⚠ VERIFY for your package version.
6. A post-deployment smoke test
The VerifyDeployed.cs program from CI/CD concepts is the smoke test. In a workflow it looks like this (the SHA is the commit the pipeline built):
- name: Smoke test the deployed version
env:
SMOKE_URL: ${{ vars.SMOKE_URL }}
EXPECTED_SHA: ${{ github.sha }}
run: |
dotnet run VerifyDeployed.cs -- "$SMOKE_URL" "$EXPECTED_SHA" 10 6
curl -fsS "$SMOKE_URL/health/ready"7. A k6 load test with thresholds
import http from 'k6/http';
import { check, sleep } from 'k6';
// Ramp to 50 virtual users, hold, ramp down. The run FAILS if a threshold is missed.
export const options = {
stages: [
{ duration: '1m', target: 50 },
{ duration: '5m', target: 50 },
{ duration: '1m', target: 0 },
],
thresholds: {
http_req_failed: ['rate<0.01'], // fewer than 1% errors
http_req_duration: ['p(95)<300'], // 95% of requests under 300 ms
},
};
const BASE = __ENV.BASE_URL; // for example https://test.accounts.example.internal
export default function () {
const res = http.get(`${BASE}/v1/accounts/ACC-0000000001`);
check(res, {
'status is 200': (r) => r.status === 200,
'has a balance': (r) => r.json('balance') !== undefined,
});
sleep(1);
}Run it locally against the lab container, or in a pipeline job after deployment to a test environment:
docker run --rm -i -e BASE_URL=http://host.docker.internal:8080 grafana/k6 run - < load/accounts.jsThe image name and the run - form are the documented pattern ⚠ VERIFY against the current k6 installation page.
8. Running the suite
# From labs/m09-delivery-pipeline. Unit, in-process integration and contract tests: no AWS, no Docker.
dotnet test --solution DeliveryPipeline.slnx
# Include the LocalStack integration test (needs Docker, and possibly a LocalStack auth token).
RUN_LOCALSTACK_TESTS=1 dotnet test --solution DeliveryPipeline.slnx
# Produce JUnit XML for the pipeline's evidence pack.
dotnet test --solution DeliveryPipeline.slnx -c Release --report-junit --results-directory ./artifacts/test-resultsCode explained line by line
Contract test
Contract = new() { ["balance"] = JsonValueKind.Number ... }— the contract is data: field name to JSON kind. A consumer team can review this table without reading controller code.JsonDocument.Parse(json)— the test parses the wire format, not the C# type. Renaming a C# property while keeping[JsonPropertyName]would pass; changing the wire name would not. That is the point.TryGetProperty(field, out var value)— fails with a message naming the missing field, so the red build explains itself.Assert.Equal(kind, value.ValueKind)— catches"balance": "1520.45"(a string) where a number is expected.
Faked AWS client versus LocalStack
Assert.SkipUnless(...)— xUnit v3 can skip dynamically with a reason; the skip appears in the report as "skipped", not as a silent pass.LocalStackBuilder(image)— Testcontainers starts a throw-away container and waits until it is ready.BasicAWSCredentials("test", "test")— LocalStack ignores credentials; using obviously fake ones means the test can never touch real AWS by accident.ForcePathStyle = true— S3 URLs becomehost/bucket/key, which works with a plainlocalhostendpoint (virtual-hosted style would need a wildcard DNS name).await using var localstack— the container is removed when the test ends, even on failure.
PostgreSQL test
IAsyncLifetime— xUnit v3'sInitializeAsyncandDisposeAsyncreturnValueTask.PostgreSqlBuilder("postgres:17-alpine")— pin the same major version you run in production, so the test catches version-specific behaviour.numeric(18,2)with1520.455— demonstrates a real database rounding rule that an in-memory fake would not reproduce.
k6 script
stages— a ramp avoids measuring cold-start effects as steady state.thresholds— turn the load test into a gate: a missed threshold makes the run exit non-zero.__ENV.BASE_URL— the target comes from the environment, so the same script runs against test and staging, and a pull request cannot point it at production.
Security considerations
⚠ Learning shortcut — not for production
The lab uses fake credentials (test/test) and an in-container PostgreSQL password written in the test. That is fine for a throw-away container on a developer or CI machine. It is not acceptable to reuse these patterns against real databases. Never put a real credential in a test file, and never run load tests against production without an explicit, approved plan.
- Test data is fictional. Use generated or synthetic data. Real customer records in a test database are a GDPR and UK GDPR risk and usually a policy breach. Card numbers in tests come only from the payment brands' published test ranges.
- Load tests can look like attacks. Notify your security team and the platform owner, and only target environments you own. AWS has a policy for load testing your own resources ⚠ VERIFY the current "penetration testing and load testing" policy page before running tests at high volume.
- LocalStack tokens are secrets. If you use
LOCALSTACK_AUTH_TOKEN, store it as a GitHub environment secret and run the job only onmain, never on fork pull requests (the lab'sci.ymldoes exactly this). - Do not log secrets or personal data from test failures. Failing assertions print values; keep sensitive values out of assertions.
- Tests are code under review. A test that is weakened to make a release pass is a control bypass. CODEOWNERS can require a second reviewer for changes under
tests/that remove assertions.
Banking use case
Harbourline's statements service and the mobile app are owned by different teams. The statements team once renamed balance to amount in the Accounts API response and merged after unit tests passed. The contract test now lives in the Accounts API repository and is reviewed by both teams. The next time someone changes a field, the build fails with Contract field 'balance' is missing, and the conversation happens before release instead of after.
Their pipeline runs unit, in-process, contract and PostgreSQL integration tests on every pull request in under six minutes. After deployment to test, the smoke test and a ten-minute end-to-end journey run (open account, post a fictional transaction, read the statement). A staging load test runs every night, and its p95 latency is charted against the service-level objective.
Common mistakes
- Testing implementation, not behaviour → tests break on every refactor → assert observable outputs (HTTP responses, database state).
- Sharing mutable state between tests → order-dependent failures → create fresh data per test or per class fixture, and let containers be disposable.
- Using
Thread.Sleepto wait → slow and flaky → poll with a timeout (VerifyDeployed.csdoes). - Treating LocalStack as AWS → green locally,
AccessDeniedin AWS → verify IAM behaviour in a real test account. - Rerunning until green → flaky tests accumulate → quarantine with an owner and a deadline, or fix.
- Running all tests on every commit → a 45-minute feedback loop → keep the pull-request path fast, and move slow suites to after-merge or scheduled runs.
- No retention of results → no evidence → upload JUnit XML as an artefact with a defined retention.
- Load testing from a laptop over Wi-Fi → you measure your network → run from a runner or load generator near the target.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Cannot connect to the Docker daemon in a Testcontainers test | Docker not running, or runner without Docker | Start Docker Desktop; GitHub-hosted Ubuntu runners include Docker; CodeBuild needs privileged mode |
LocalStack test errors mentioning LOCALSTACK_AUTH_TOKEN | Newer LocalStack image requires a token | Set the token from a secret, or pin LOCALSTACK_IMAGE to an older tag ⚠ VERIFY |
WebApplicationFactory<Program> does not compile: Program is inaccessible | Top-level Program is internal | Add public partial class Program { } or InternalsVisibleTo |
| Tests pass locally, fail in CI only | Time zone, culture, file path case or Docker availability | Set invariant culture in tests; avoid absolute paths; run in the same Docker image locally |
| Test discovery finds zero tests | Wrong dotnet test syntax for Microsoft.Testing.Platform | Use dotnet test --solution X.slnx or --project ⚠ VERIFY; ensure global.json opts in to MTP |
| JUnit file is not produced | Reporter not enabled | Pass --report-junit and a --results-directory |
k6 run exits 99 | A threshold was crossed | Read the threshold summary; it is a failed test, not a tool error |
| Smoke test flaps after deploy | Deployment still shifting traffic | Retry with a timeout; verify the version before the health check |
Cost and cleanup
Cost: free. Everything here runs locally or on a GitHub-hosted runner. Testcontainers images (LocalStack, PostgreSQL) are downloaded once. A k6 load test against a real environment can cost money: Fargate tasks, load-balancer capacity units and data transfer. Estimate before running against AWS, and run for minutes, not hours.
Cleanup:
# Testcontainers removes its containers automatically; list anything left behind.
docker ps -a --filter "label=org.testcontainers=true"
# Remove stopped Testcontainers containers and unused images when you are done.
docker container prune -f --filter "label=org.testcontainers=true"
docker image rm localstack/localstack:4.14 postgres:17-alpine 2>/dev/null
# Remove build output from the lab.
find labs/m09-delivery-pipeline \( -name bin -o -name obj \) -type d -prune -exec rm -rf {} +Exercises
- 🟢 Guided: From the lab folder, run
dotnet test --solution DeliveryPipeline.slnx(or the Docker form from the lab README). Then rename theBalanceproperty of the account record inAccounts.cstoAmount(fix any compile errors the rename causes), run the tests again, and read the failure message. Revert the change. - 🟡 Challenge: Add a contract test for
GET /v1/customers/{id}/accounts: the response is a JSON array, each element has the same required fields as the single-account response. Also assert that an unknown customer returns an empty array, not a 404. - 🔴 Live problem: Your pull-request pipeline now takes 28 minutes and fails about one run in ten with no code change. Product wants releases daily. Propose a plan: how to find where the time goes, how to treat the flaky tests, which tests to move to after-merge, and how you would prove to the team that the faster pipeline is still safe.
Assignment
In your portfolio repository, add tests/ to a small ASP.NET Core API with: one unit test class, one in-process integration test, one contract test with an explicit Contract dictionary, and one Testcontainers test gated by an environment variable. Add a workflow that runs them with --report-junit and uploads the XML. Include a TESTING.md of ten lines saying which test type runs at which stage.
Solutions
After renaming the property, the contract test fails with Contract field 'balance' is missing. and the unit tests still pass. That demonstrates why contract tests exist: the code compiles, the API returns 200 OK, and only the wire format changed. Revert the change and the suite returns to 74 passed, 1 skipped.
[Fact]
public async Task Customer_accounts_response_honours_the_contract()
{
var json = await factory.CreateClient().GetStringAsync("/v1/customers/CUS-00012345/accounts", TestContext.Current.CancellationToken);
using var doc = JsonDocument.Parse(json);
Assert.Equal(JsonValueKind.Array, doc.RootElement.ValueKind);
foreach (var element in doc.RootElement.EnumerateArray())
{
foreach (var (field, kind) in AccountContractTests.Contract)
{
Assert.True(element.TryGetProperty(field, out var value), $"Contract field '{field}' is missing.");
Assert.Equal(kind, value.ValueKind);
}
}
}
[Fact]
public async Task Unknown_customer_returns_an_empty_array()
{
var response = await factory.CreateClient().GetAsync("/v1/customers/CUS-99999999/accounts", TestContext.Current.CancellationToken);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync(TestContext.Current.CancellationToken));
Assert.Equal(0, doc.RootElement.GetArrayLength());
}Make Contract internal static so both classes can use it. The reason to assert an empty array rather than 404: consumers treat "no accounts" as a normal state, so returning an error would break their screen. Use a customer id that the sample data does not contain.
- Measure. Export per-test and per-step durations (the JUnit XML has them). Sort by total time. Typically a handful of tests and the image build dominate.
- Flaky tests. Identify them from rerun history, quarantine each with an owner and a due date (tracked, visible, not silently skipped), and fix root causes such as shared state, real clocks and sleeps. Do not add automatic retries as a permanent fix; retries hide real defects.
- Restructure. Keep unit, in-process, contract and one fast container test on pull requests. Move end-to-end and load tests to after-merge or scheduled runs. Run independent jobs in parallel; cache NuGet packages and Docker layers.
- Prove safety. Show that the contract and smoke tests still guard the highest-risk behaviours, track change failure rate and escaped defects for a month, and keep a post-deployment verification plus automatic rollback so a gap in tests does not become an outage.
Why: a fast pipeline that nobody trusts is worse than a slow one, so the plan must improve speed and reliability together, and it must be measured.
Knowledge check
Answer: On every pull request, because they are fast and in-process, and a breaking change to a response shape is cheapest to catch before merge. Running them only after deployment would let the break reach an environment first.
Answer: LocalStack ignores credentials, and fake values mean the test cannot accidentally authenticate to real AWS. Using real credentials in a test is both unnecessary and a leak risk.
Answer: LocalStack does not enforce your real IAM policies, so it cannot detect a missing permission. Verify permissions with a test in a real AWS account or with IAM Policy Simulator and Access Analyzer.
Answer: It hides flaky tests and real intermittent defects (races, timeouts), makes pipelines slower, and teaches the team to distrust failures. Quarantine and fix instead.
Answer: Thresholds make a load test a pass-or-fail gate (for example p95 below 300 ms and under 1% errors) rather than a graph someone must interpret, so the pipeline can act on it.
Interview questions
- "What is the test pyramid and when would you break it?" Many fast unit tests, fewer integration tests, few end-to-end tests, because cost and diagnosis difficulty rise with scope. You might add more integration tests where the risk is in the integration itself (SQL, serialisation), as in banking data layers.
- "How do you stop a breaking API change reaching consumers?" Consumer contract tests in the provider's pipeline, reviewed by consumers, plus versioned endpoints.
- "Emulator or real AWS for tests?" Emulator for fast feedback about request shape and error handling; a real test account for IAM, quotas and service behaviour. State what each cannot tell you.
- "A test is flaky. What do you do?" Reproduce, find the shared state or timing cause, quarantine with an owner and deadline if blocking, fix; never normalise reruns.
- "What test evidence would you give an auditor?" The JUnit report tied to the commit and run, retained for the policy period, referenced from the release record.
Certification practice
A CodeBuild project runs tests for a .NET service, and the team wants pass and fail counts visible per build without opening logs. What should they configure?
A. A CloudWatch dashboard that parses build logs.
B. A reports: section in the buildspec that publishes JUnit XML to a CodeBuild report group.
C. An S3 event that triggers a Lambda function to count lines containing "passed".
D. A manual approval action after the build.
Answer: B. CodeBuild report groups ingest JUnit and other formats and show test case results per build. A and C are fragile log parsing. D adds a human step but no visibility.
A developer wants integration tests for code that uses Amazon S3 to run on every pull request without an AWS account and without cost. Which approach fits?
A. Run the tests against the production bucket with read-only access. B. Use a local S3-compatible emulator started in a container, with fake credentials. C. Skip integration tests and rely on unit tests only. D. Create a new S3 bucket per pull request in the production account.
Answer: B. An emulator in a container (for example LocalStack through Testcontainers) tests request shape and error handling at no cost. A touches production, C removes the check, D creates cost and production-account risk.
After deployment to the test environment, the pipeline must confirm that the running service is the build that was just released. Which check is most reliable?
A. Confirm the deployment API call returned HTTP 200.
B. Call a /version endpoint that reports the commit SHA baked into the image and compare it to the pipeline's commit.
C. Check that the ECS service desired count is greater than zero.
D. Wait five minutes and assume success.
Answer: B. The running software states what it is, and the pipeline compares it to what it built. A only proves the API accepted the request, C says nothing about the version, and D is a guess.
Completion checklist
- I can explain which test type runs at which pipeline stage and why
- I ran the lab test suite and saw the contract test fail after renaming a field
- I completed the guided exercise and reverted the change
- I can say what LocalStack cannot prove
- I removed build output and Testcontainers leftovers
Key takeaways
- Put fast, deterministic tests on the pull-request path; move slow ones after merge or onto a schedule.
- A contract test turns a silent breaking change into a red build before consumers feel it.
- Testcontainers gives you real dependencies in disposable containers; LocalStack proves request shape, not IAM.
- A smoke test that checks the running version closes the loop between build and deployment.
- Retain JUnit results with the commit; flaky tests must be fixed or quarantined with an owner, not retried into silence.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Docker and Amazon ECR in pipelines
Build a small, reproducible, non-root .NET 10 container image, tag it so every running container can be traced to a commit, push it to Amazon ECR through a narrowly scoped role, scan it before and after the push, and deploy it by digest.
Lab 9.1: a CI workflow with Testcontainers and LocalStack
Run the Accounts API test suite locally in Docker, read the ci.yml workflow job by job, add a LocalStack-backed integration test with Testcontainers, and download JUnit results as pipeline evidence, all without an AWS account or any AWS credentials.