Microservices communication and service discovery
Draw bounded contexts for a bank, choose between a query, a command and an event for every interaction, make synchronous calls safe with timeouts and retries, and let services find each other with Amazon ECS Service Connect and AWS Cloud Map.
Assumes: .NET 10 LTS · Microsoft.Extensions.Http.Resilience 10.11.0 · AWSSDK.ServiceDiscovery 4.1.0.2 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Split a banking domain into bounded contexts, say which data each context owns, and explain why a shared database defeats the split.
- Classify any interaction as a query, a command or an event, and choose a synchronous call, an SQS queue or an EventBridge event for it.
- Make a synchronous HTTP call safe in .NET with a timeout budget, jittered retries and a circuit breaker (
AddStandardResilienceHandler). - Explain how AWS Cloud Map and Amazon ECS Service Connect let one service reach another by name, and how that differs from plain DNS.
- Deploy two Fargate services that call each other as
http://accounts:8080, with the infrastructure written in CDK (C#) and Terraform. - Version events and APIs so that one team can deploy without breaking another.
Prerequisites
- Asynchronous communication and event-driven architecture: the decision tree for queues, topics, buses and streams.
- Retry strategies: backoff, jitter and retry storms.
- Delivery guarantees and ordering and Idempotency in .NET.
- The outbox pattern with EF Core and Distributed transactions and the saga pattern: how a service publishes events and coordinates a multi-service flow.
- Module 3: .NET on AWS: containers, ECR and Fargate. This lesson reuses that vocabulary (task definition, service, cluster).
- Tools: .NET 10 SDK, AWS CLI v2 with the
learn-sandboxprofile. The AWS deployment is optional; everything in the code section runs on your laptop.
Why this matters in banking
Harbourline's payments team wants to release twice a week. Their code lives in a modular monolith that also contains the accounts logic, the notification templates and the sanctions rules. Every release needs a regression run across all four, and a bug in the notification templates once delayed a payments fix by nine days.
So the bank splits the monolith into services. Six months later a different problem appears. The payment service calls the accounts service, which calls the limits service, which calls the customer-profile service, all synchronously, to authorise one transfer. If each service is available 99.9% of the time, a chain of four gives roughly 99.6% for the whole request (0.999 to the power of 4), about 35 hours of failure a year instead of about 9. One slow profile lookup makes every payment slow. Worse, a retry at each hop multiplies load: three retries at four layers can turn one request into 81 calls against the struggling service at the bottom.
Two things went wrong. The boundaries were drawn around technical layers instead of business capabilities, and every interaction was a synchronous query, even the ones that were really announcements ("a payment settled"). This lesson gives you a way to draw the boundaries, a rule for choosing the communication style, and the AWS plumbing (service discovery, timeouts, retries) that makes the synchronous calls you do keep behave.
Design note for banks
Service boundaries are also data boundaries. Which service may hold which personal data, and which service calls are logged for audit, are decisions for your data-protection and security teams. This lesson shows mechanisms, not a compliance position.
The idea in plain language
A microservice is a small program that owns one business capability and its data, deploys on its own, and talks to others only over the network. The hard part is not the code. It is deciding where to cut and how the pieces talk.
Bounded context. A term from domain-driven design (DDD): an area of the business in which every word has one precise meaning. "Account" means something different to the Accounts team (a ledger with a balance and a status) than to the Notifications team (a place to send a message about). Each context has its own model, its own database and its own team. They share identifiers (ACC-0000000001, CUS-00012345), not tables.
Three kinds of message cross a boundary. Naming them correctly decides almost every design question:
| Kind | Meaning | Example | Direction of knowledge |
|---|---|---|---|
| Query | "Tell me something." No side effect; safe to repeat. | Does account ACC-0000000001 have 120.00 GBP available? | Caller knows the callee |
| Command | "Do this." One addressed receiver, may fail, has a side effect. | Send an SMS for transaction T-123. | Sender knows the receiver |
| Event | "This happened." Past tense, a fact, many or zero receivers. | PaymentSettled | Publisher does not know who listens |
Analogy: In a bank branch, a query is you leaning over to a colleague and asking "is this account frozen?". You wait for the answer, and if they are away you are stuck. A command is a task slip dropped in the mortgage team's tray: it will be handled, you may or may not hear back. An event is the announcement on the branch intercom: "the vault has been opened". Anyone who cares reacts; the announcer does not know or wait for them.
The rule of thumb that follows: ask with a synchronous call only when you cannot proceed without the answer, tell with a queue when one specific service must act, announce with an event when the fact matters to others. Everything that is not a query should lean asynchronous, because asynchronous messaging decouples the availability and speed of services, not just their code.
What problem the AWS services solve
Splitting a system moves function calls onto the network, and a network call raises questions a function call never did:
- Where is the other service right now? Containers start, stop and move. Their IP addresses change. Hard-coding addresses, or asking an operator to update a config file, does not work. This is service discovery.
- What if it is slow or down? You need timeouts, retries, circuit breakers and a fallback. This is resilience, partly in your code and partly in the network layer.
- How do we change it without breaking callers? Contracts need versioning, and consumers must tolerate unknown fields. This is evolvability.
AWS offers several building blocks. You choose by where the caller and callee run and how much you want managed:
| Need | AWS service | Notes |
|---|---|---|
| Name → address for ECS services | Amazon ECS Service Connect | Configuration on the ECS service; a managed proxy sidecar does round-robin load balancing, outlier detection and retries |
| Name → address for anything (EC2, ECS, Lambda, DynamoDB tables, SQS queues, on-premises) | AWS Cloud Map | A registry you query with DiscoverInstances or with DNS; you register and deregister |
| Public or internal HTTP entry point with path routing and health checks | Application Load Balancer | Good in front of a service that many clients (and non-ECS clients) call |
| Managed API front door with auth, throttling and usage plans | Amazon API Gateway | Covered in Module 4 |
| Cross-VPC and cross-account service-to-service networking, policy at the edge | Amazon VPC Lattice | A different, more general application-networking service. Not covered here ⚠ VERIFY current feature set before choosing it |
| Decoupled "tell" and "announce" | SQS, SNS, EventBridge | The rest of this module |
Limits to know about Service Connect (from the ECS documentation, checked 2026-11-18):
- Only ECS services are supported, not standalone tasks. Windows containers, HTTP/1.0 and FIPS mode are not supported.
- A service belongs to one namespace, and the namespace must be in the same Region. Each endpoint and each discovery name must be unique within the namespace.
- Existing tasks do not see new endpoints. A client must be redeployed before it can resolve an endpoint added after its last deployment. That is a deliberate design: the client team decides when its configuration changes.
- Tasks need a task-level memory limit. AWS recommends adding about 256 CPU units and 64 MiB of memory for the proxy sidecar on top of your own sizing. ⚠ VERIFY the figures in the current ECS guide before capacity planning.
- There is no extra charge for Service Connect or for the Cloud Map usage it generates; you pay for the compute the sidecar uses. TLS encryption with AWS Private CA is billed separately.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Container Apps with Dapr service invocation, or built-in app names | ECS Service Connect | Name-based calls with retries and metrics, but configured on the ECS service, with no Dapr runtime to operate |
AKS ClusterIP Service + CoreDNS names (http://accounts) | Service Connect, or ECS service discovery (Cloud Map DNS) | Service Connect resolves names through its proxy and does not depend on VPC DNS; Cloud Map DNS is plain DNS with caching |
| Consul / Eureka / Steeltoe discovery clients | AWS Cloud Map DiscoverInstances | A managed registry with an IAM-protected API; instances carry attributes |
Polly (Microsoft.Extensions.Http.Polly) | Microsoft.Extensions.Http.Resilience | The same ideas (retry, circuit breaker, timeout) packaged as one "standard handler" on the HttpClient builder |
| Azure API Management in front of internal services | API Gateway or an internal ALB | AWS puts auth and throttling in API Gateway; internal service-to-service traffic usually skips it |
| Service Bus queue (command) and Event Grid topic (event) | SQS (command), EventBridge or SNS (event) | The same split between "tell one" and "announce to many" |
| Shared SQL Server across several apps | One database per service, e.g. Aurora, DynamoDB | The hardest habit to break. Shared tables couple deployments |
Architecture
The Harbourline context map. Solid arrows are synchronous calls; dashed arrows are asynchronous.
Every context has its own data store, and no arrow goes into another context's database. The Payments service asks Accounts a query because it cannot decide without the answer. It sends the scheme a command through a queue so a slow scheme does not hold a customer request open. It announces PaymentSettled so that Notifications, Fraud and the data platform can react without Payments knowing they exist.
Security-boundary view. Inside the VPC, tasks run in private subnets (the lab uses public subnets to avoid the cost of a NAT gateway, flagged below). A security group allows Payments tasks to reach Accounts tasks on the container port, and nothing else. IAM task roles decide which AWS APIs each service may call. Service Connect is a naming and routing feature. It does not authenticate callers by itself, so add TLS (optionally Service Connect TLS) and application-level authorisation when the data warrants it.
Drawing the boundaries: a quick method
- List the business capabilities (open account, authorise payment, screen sanctions, notify customer) rather than technical layers.
- For each, list the nouns it owns and changes (balance, payment, screening result, delivery attempt). The context that changes a noun owns it.
- Check each pair for the words that mean different things. If "customer" is a profile in one place and a legal entity in another, you have found a boundary.
- Prefer fewer, larger services at first. A "distributed monolith" (many services that must be deployed together) is worse than a monolith.
- Write the interactions down as query, command or event. A context that needs ten synchronous queries to do its job is probably cut in the wrong place.
| Context | Owns | Publishes | Consumes | Store |
|---|---|---|---|---|
| Accounts | Accounts, balances, holds | AccountFrozen, PostingBooked | PaymentSettled (to post) | Aurora PostgreSQL |
| Payments | Payment lifecycle, scheme submissions | PaymentSubmitted, PaymentSettled, PaymentRejected | ScreeningCleared | Aurora PostgreSQL |
| Screening | Sanctions screening decisions | ScreeningCleared, ScreeningHit | PaymentSubmitted | DynamoDB |
| Notifications | Delivery attempts, templates, opt-ins | NotificationSent | PaymentSettled, PostingBooked | DynamoDB |
Step-by-step implementation
Classify the interaction
For every arrow in the context map, ask three questions. Can the caller continue without the answer? Is there exactly one receiver? Is it a fact or a request? The answers choose the style:
| Can the caller proceed without the answer? | Receivers | Style |
|---|---|---|
| No | One | Synchronous query (HTTP) with strict timeouts |
| Yes | One | Command on a queue (SQS) |
| Yes | Zero to many | Event on a bus or topic (EventBridge, SNS) |
In this lesson, "does the debtor have funds?" is a query. "Submit to the scheme" is a command. "Payment settled" is an event.
Build the callee: a query with a narrow answer
The Accounts service exposes a funds check that returns only what Payments needs: yes or no, and a reason. It does not return the balance. That keeps the contract small and keeps personal data inside its context.
Build the caller with a resilience pipeline
The Payments service uses a typed HttpClient with the standard resilience handler. You set a timeout for each attempt, a ceiling for all attempts together, a small number of jittered retries and a circuit breaker.
Run both locally
Start both services with dotnet run and point Services:Accounts:BaseUrl at http://localhost:5080. Nothing in the code mentions AWS yet.
Deploy to ECS with Service Connect
Put the same images in two ECS services in one namespace. Accounts publishes the endpoint accounts:8080; Payments is a client. The only change to the Payments configuration is the base URL: http://accounts:8080.
Add the asynchronous edges
Add the queue for the command and the event for the announcement. The outbox pattern makes the publish reliable, and AWS.Messaging for .NET shows the framework version of publishers and handlers.
AWS Console instructions
To see Service Connect after deployment:
- Console → Amazon ECS → Clusters →
hl-m08-services→ Services →accounts. Open the Configuration and networking tab and find Service Connect: the namespaceharbourline.internal, the discovery nameaccountsand the client aliasaccounts:8080. - Open the Tasks tab. Each task lists two containers: yours and the Service Connect proxy sidecar that ECS added.
- Open the Logs tab for the service's log group, and the Metrics tab for the Service Connect request count, error counts and latency. ⚠ VERIFY exact tab and metric names; the console changes often.
- Console → AWS Cloud Map → Namespaces →
harbourline.internalto see the HTTP namespace and the services that ECS created inside it. Look, but do not edit: the ECS documentation warns that manually changing Cloud Map resources that Service Connect manages can break routing and later deployments.
AWS CLI
# List namespaces. Service Connect created an HTTP namespace with the name you chose.
aws servicediscovery list-namespaces --profile learn-sandbox
# Ask Cloud Map for the instances behind the "accounts" service (works for HTTP namespaces).
# This is the same API a non-ECS client (a Lambda function, an EC2 job) would call.
aws servicediscovery discover-instances \
--namespace-name harbourline.internal \
--service-name accounts \
--profile learn-sandbox
# Show Service Connect settings and recent deployments for one ECS service.
aws ecs describe-services --cluster hl-m08-services --services accounts \
--query "services[0].{sc:deployments[0].serviceConnectConfiguration,status:deployments[0].rolloutState}" \
--profile learn-sandbox
# Tail the payments container logs while you call it.
aws logs tail /hl-m08/services --follow --filter-pattern payments --profile learn-sandbox
# Exec into the payments task and call the callee by name (needs ECS Exec enabled; see Module 3).
# aws ecs execute-command ... --command "wget -qO- 'http://accounts:8080/accounts/ACC-0000000001/funds-check?amount=10¤cy=GBP'"Infrastructure as Code
Both versions create a VPC (CDK only), a cluster whose default Service Connect namespace is harbourline.internal, two Fargate services, a log group and a security rule that lets Payments reach Accounts on port 8080. Replace the image placeholders with images you pushed to your own ECR repository (Module 3 shows how).
using Amazon.CDK;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.Logs;
using Constructs;
var app = new App();
_ = new ServicesStack(app, "HlM08-Services", new StackProps());
app.Synth();
public sealed class ServicesStack : Stack
{
public ServicesStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m08-microservices");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
var vpc = new Vpc(this, "Vpc", new VpcProps
{
MaxAzs = 2,
NatGateways = 0, // lab only: tasks get public IPs. Production: private subnets + VPC endpoints or NAT.
SubnetConfiguration = [new SubnetConfiguration { Name = "public", SubnetType = SubnetType.PUBLIC, CidrMask = 24 }],
});
// The cluster's default namespace: Service Connect services join it automatically.
var cluster = new Cluster(this, "Cluster", new ClusterProps
{
ClusterName = "hl-m08-services",
Vpc = vpc,
DefaultCloudMapNamespace = new CloudMapNamespaceOptions
{
Name = "harbourline.internal",
Type = Amazon.CDK.AWS.ServiceDiscovery.NamespaceType.HTTP,
},
});
var logs = new LogGroup(this, "Logs", new LogGroupProps
{
LogGroupName = "/hl-m08/services",
Retention = RetentionDays.ONE_WEEK,
RemovalPolicy = RemovalPolicy.DESTROY,
});
var accountsTask = new FargateTaskDefinition(this, "AccountsTask", new FargateTaskDefinitionProps
{
Cpu = 256,
MemoryLimitMiB = 512,
RuntimePlatform = new RuntimePlatform { CpuArchitecture = CpuArchitecture.ARM64, OperatingSystemFamily = OperatingSystemFamily.LINUX },
});
accountsTask.AddContainer("accounts", new ContainerDefinitionOptions
{
Image = ContainerImage.FromRegistry("REPLACE_WITH_YOUR_ECR_ACCOUNTS_IMAGE"),
Essential = true,
// The port NAME is what Service Connect refers to. appProtocol gives HTTP metrics.
PortMappings = [new PortMapping { Name = "http", ContainerPort = 8080, AppProtocol = AppProtocol.Http }],
Logging = LogDriver.AwsLogs(new AwsLogDriverProps { StreamPrefix = "accounts", LogGroup = logs }),
HealthCheck = new Amazon.CDK.AWS.ECS.HealthCheck
{
Command = ["CMD-SHELL", "wget -qO- http://localhost:8080/healthz || exit 1"],
Interval = Duration.Seconds(15),
Retries = 3,
},
});
var accounts = new FargateService(this, "AccountsService", new FargateServiceProps
{
Cluster = cluster,
ServiceName = "accounts",
TaskDefinition = accountsTask,
DesiredCount = 2,
AssignPublicIp = true,
ServiceConnectConfiguration = new ServiceConnectProps
{
Services =
[
new ServiceConnectService
{
PortMappingName = "http",
DnsName = "accounts",
Port = 8080,
PerRequestTimeout = Duration.Seconds(5),
},
],
},
});
var paymentsTask = new FargateTaskDefinition(this, "PaymentsTask", new FargateTaskDefinitionProps
{
Cpu = 256,
MemoryLimitMiB = 512,
RuntimePlatform = new RuntimePlatform { CpuArchitecture = CpuArchitecture.ARM64, OperatingSystemFamily = OperatingSystemFamily.LINUX },
});
paymentsTask.AddContainer("payments", new ContainerDefinitionOptions
{
Image = ContainerImage.FromRegistry("REPLACE_WITH_YOUR_ECR_PAYMENTS_IMAGE"),
Essential = true,
PortMappings = [new PortMapping { Name = "http", ContainerPort = 8080, AppProtocol = AppProtocol.Http }],
Environment = new Dictionary<string, string> { ["Services__Accounts__BaseUrl"] = "http://accounts:8080" },
Logging = LogDriver.AwsLogs(new AwsLogDriverProps { StreamPrefix = "payments", LogGroup = logs }),
});
var payments = new FargateService(this, "PaymentsService", new FargateServiceProps
{
Cluster = cluster,
ServiceName = "payments",
TaskDefinition = paymentsTask,
DesiredCount = 1,
AssignPublicIp = true,
// A client-only service: no Services list, so it only resolves other endpoints.
ServiceConnectConfiguration = new ServiceConnectProps(),
});
// Security group: payments tasks may reach accounts tasks on the container port, nothing else.
accounts.Connections.AllowFrom(payments, Port.Tcp(8080), "payments to accounts via Service Connect");
}
}Both versions compiled (CDK) and validated (terraform validate with the v6 AWS provider) on 2026-11-18. Neither was deployed; cdk deploy and terraform apply need your own images and execution role. In the CDK version the execution role and the VPC are created for you. In the Terraform version, supply an existing VPC, subnets and task execution role through variables.
Complete code
Two small ASP.NET Core minimal APIs and the shared contract. They use fictional data only. Create two projects (dotnet new web -n Accounts and dotnet new web -n Payments), and add Microsoft.Extensions.Http.Resilience to Payments.
using System.Collections.Concurrent;
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
// Fictional data. In a real service this is the Accounts context's own database.
var accounts = new ConcurrentDictionary<string, AccountView>
{
["ACC-0000000001"] = new("ACC-0000000001", "GBP", 2_500.00m, "open"),
["ACC-0000000002"] = new("ACC-0000000002", "GBP", 15.00m, "frozen"),
};
app.MapGet("/healthz", () => Results.Ok("ok"));
// A QUERY: safe to repeat, no side effect. Returns only what the caller needs.
app.MapGet("/accounts/{accountId}/funds-check", (string accountId, decimal amount, string currency) =>
{
if (!accounts.TryGetValue(accountId, out var account))
{
return Results.Problem(statusCode: 404, title: "Account not found");
}
var ok = account.Status == "open" && account.Currency == currency && account.Balance >= amount;
return Results.Ok(new FundsCheck(accountId, ok, ok ? null : "insufficient funds or account not open"));
});
app.Run();
public sealed record AccountView(string AccountId, string Currency, decimal Balance, string Status);
public sealed record FundsCheck(string AccountId, bool Sufficient, string? Reason);using System.Globalization;
using System.Net.Http.Json;
using Microsoft.Extensions.Http.Resilience;
var builder = WebApplication.CreateBuilder(args);
// One setting changes between laptop and AWS:
// laptop: Services:Accounts:BaseUrl = http://localhost:5080
// ECS Service Connect: Services:Accounts:BaseUrl = http://accounts:8080
builder.Services.AddHttpClient<IAccountsClient, AccountsClient>(client =>
{
client.BaseAddress = new Uri(builder.Configuration["Services:Accounts:BaseUrl"]
?? throw new InvalidOperationException("Services:Accounts:BaseUrl is not set."));
})
.AddStandardResilienceHandler(options =>
{
// Queries are safe to retry. Keep the whole budget (all attempts) well under the caller's own timeout.
options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(6);
options.Retry.MaxRetryAttempts = 2;
options.Retry.UseJitter = true;
options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30); // must be at least twice the attempt timeout
});
var app = builder.Build();
app.MapPost("/payments", async (PaymentRequest request, IAccountsClient accounts, CancellationToken ct) =>
{
FundsCheck? check;
try
{
check = await accounts.CheckFundsAsync(request.DebtorAccountId, request.Amount, request.Currency, ct);
}
catch (HttpRequestException)
{
// Accounts is unavailable after retries: fail fast with a clear, retryable status. Do not guess.
return Results.Problem(statusCode: 503, title: "Accounts service unavailable", detail: "Try again shortly.");
}
return check is { Sufficient: true }
? Results.Accepted($"/payments/{request.PaymentId}", new { request.PaymentId, status = "received" })
: Results.Problem(statusCode: 422, title: "Payment rejected", detail: check?.Reason);
});
app.Run();
public sealed record PaymentRequest(string PaymentId, string DebtorAccountId, decimal Amount, string Currency);
public sealed record FundsCheck(string AccountId, bool Sufficient, string? Reason);
public interface IAccountsClient
{
Task<FundsCheck?> CheckFundsAsync(string accountId, decimal amount, string currency, CancellationToken ct);
}
public sealed class AccountsClient(HttpClient http) : IAccountsClient
{
public Task<FundsCheck?> CheckFundsAsync(string accountId, decimal amount, string currency, CancellationToken ct)
{
var url = $"accounts/{Uri.EscapeDataString(accountId)}/funds-check"
+ $"?amount={amount.ToString(CultureInfo.InvariantCulture)}¤cy={currency}";
return http.GetFromJsonAsync<FundsCheck>(url, ct);
}
}Run it locally:
# Terminal 1
dotnet run --project Accounts --urls http://localhost:5080
# Terminal 2
dotnet run --project Payments --urls http://localhost:5090 -- --Services:Accounts:BaseUrl=http://localhost:5080
# Terminal 3: enough funds, then too much, then a frozen account
curl -s -X POST http://localhost:5090/payments -H "Content-Type: application/json" \
-d '{"paymentId":"PAY-1","debtorAccountId":"ACC-0000000001","amount":120.00,"currency":"GBP"}'
curl -s -X POST http://localhost:5090/payments -H "Content-Type: application/json" \
-d '{"paymentId":"PAY-2","debtorAccountId":"ACC-0000000001","amount":9000.00,"currency":"GBP"}'
curl -s -X POST http://localhost:5090/payments -H "Content-Type: application/json" \
-d '{"paymentId":"PAY-3","debtorAccountId":"ACC-0000000002","amount":10.00,"currency":"GBP"}'
# Now stop Accounts (Ctrl+C) and send PAY-1 again: after retries you get a 503, not a hang.Calling from a client that is not an ECS service with Service Connect (for example, a Lambda function) uses the Cloud Map API instead of a name that the proxy resolves:
using Amazon.ServiceDiscovery;
using Amazon.ServiceDiscovery.Model;
public sealed class CloudMapResolver(IAmazonServiceDiscovery cloudMap)
{
/// <summary>Returns "ip:port" of one healthy instance. Cache the result for a few seconds, not per request.</summary>
public async Task<string> ResolveAsync(string namespaceName, string serviceName, CancellationToken ct)
{
var response = await cloudMap.DiscoverInstancesAsync(new DiscoverInstancesRequest
{
NamespaceName = namespaceName,
ServiceName = serviceName,
HealthStatus = HealthStatusFilter.HEALTHY_OR_ELSE_ALL,
}, ct);
var instances = response.Instances;
if (instances.Count == 0)
{
throw new InvalidOperationException($"No instances registered for {serviceName} in {namespaceName}.");
}
var chosen = instances[Random.Shared.Next(instances.Count)];
return $"{chosen.Attributes["AWS_INSTANCE_IPV4"]}:{chosen.Attributes["AWS_INSTANCE_PORT"]}";
}
}The package is AWSSDK.ServiceDiscovery 4.1.0.2. The attribute names AWS_INSTANCE_IPV4 and AWS_INSTANCE_PORT are Cloud Map's standard keys for instances registered by ECS service discovery ⚠ VERIFY against the instances you actually register, and note that Service Connect instances are meant to be reached by name through the proxy, not resolved by hand.
Code explained line by line
MapGet("/accounts/{accountId}/funds-check", ...): a query endpoint. It usesGET, has no side effects and is therefore safe for the caller to retry. A command such as "post this payment" would usePOSTand need an idempotency key (Idempotency in .NET).return Results.Ok(new FundsCheck(accountId, ok, ...)): the answer is a narrow view, not the account. Payments never sees the balance. If Accounts later changes how balances work, the contract does not change.AddHttpClient<IAccountsClient, AccountsClient>(...): a typed client managed byIHttpClientFactory. It reuses connections and lets you attach handlers to the pipeline.BaseAddressread from configuration: the only thing that differs between laptop and cluster. Code does not know about Cloud Map or Service Connect. This is why name-based discovery is pleasant: local, test and production all use a URL..AddStandardResilienceHandler(...): adds a pipeline of rate limiter, total timeout, retry, circuit breaker and per-attempt timeout in one call. The options below tune it.AttemptTimeout2 s,TotalRequestTimeout6 s,MaxRetryAttempts2: three tries of up to 2 s each fit inside 6 s. Your timeout budget must be smaller than your caller's timeout, which must be smaller than its caller's. In a chain, the outermost timeout is the customer's patience; every inner one shrinks.UseJitter = true: randomises retry delays so that many Payments tasks do not retry in lockstep (Retry strategies).CircuitBreaker.SamplingDuration: the library validates that this is at least double the attempt timeout. The breaker opens after repeated failures, so the next calls fail instantly instead of queueing behind a dead service.catch (HttpRequestException)→503: when the callee is down after retries, Payments tells its own caller "unavailable, try again". It does not invent an answer about the customer's money. A circuit-breaker rejection surfaces as a different exception type (BrokenCircuitExceptionfrom the underlying Polly library) so a real service should catch that as well; the lab keeps one catch for readability ⚠ VERIFY the exact exception your version throws by stopping Accounts and running the test above.Services__Accounts__BaseUrlin the CDK and Terraform code: ASP.NET Core maps a double underscore in an environment variable to the:configuration separator, so this setsServices:Accounts:BaseUrl.PortMapping { Name = "http", AppProtocol = Http }: Service Connect addresses a port by its name;AppProtocoltells the proxy the traffic is HTTP, which unlocks HTTP-level metrics such as 5xx counts.ServiceConnectService { DnsName = "accounts", Port = 8080 }: the client alias. Clients in the namespace callhttp://accounts:8080; the proxy picks one of the two Accounts tasks (round robin) and retries a failed connection on another task. ECS's documentation says the proxy retries twice, avoiding the host that just failed ⚠ VERIFY if you rely on the exact number.PerRequestTimeout = 5 s: the proxy gives up on a request after five seconds even if your client waits longer. The ECS default is 15 seconds. Keep the proxy's timeout below the client's total budget so that the layers do not fight.new ServiceConnectProps()on Payments: a client-only configuration. NoServicesmeans it publishes no endpoint, but it joins the namespace and can resolveaccounts.accounts.Connections.AllowFrom(payments, Port.Tcp(8080), ...): the security-group rule that lets the Payments tasks reach the Accounts proxy port. Service Connect routes traffic; you still open the network path.
Security considerations
⚠ Learning shortcut — not for production
The lab runs tasks in public subnets with public IPs and NatGateways = 0 so that the demo avoids a NAT gateway bill and extra setup. Production runs tasks in private subnets, reaches ECR, CloudWatch Logs and S3 through VPC endpoints (or a NAT gateway), and never gives tasks public IPs. The security group in the lab allows only Payments → Accounts on 8080; keep that least-privilege shape in production.
- Service Connect is not authentication. It resolves names and balances load. A service that must restrict callers needs its own check: mutual TLS (Service Connect can encrypt traffic with certificates from AWS Private CA, which is billed separately), signed tokens, or IAM-based authorisation at an API Gateway or ALB edge. Decide per data class.
- Least-privilege task roles. Accounts and Payments get separate task roles. Payments may publish to its bus and queue; it has no access to the Accounts database or table.
- No shared database. If Payments reads Accounts' tables directly, you bypass the contract, the audit log and the owner's ability to change the schema. In banking, the owning service is also the single place that enforces posting rules.
- Keep personal data out of events and logs. Events carry identifiers and facts (
paymentId,amount,currency), not names or full IBANs. Log correlation IDs and outcomes, not request bodies. This follows the same pattern as the module project. - Don't edit Cloud Map by hand. The ECS documentation warns that manual changes to the resources Service Connect creates can break traffic and later deployments. Control who has
servicediscovery:*write permissions. - Compliance. Which services may exchange which personal data, in which Region, and what the audit trail must show, needs review by your legal, data-protection and security teams. Nothing here is a compliance claim.
Banking use case
Harbourline authorises a Faster Payments transfer in three moves. Payments asks Accounts a query ("funds and status?") with a 2-second attempt timeout and a 6-second budget, because the customer is waiting. If the answer is yes, Payments writes the payment and an outbox row in one transaction, and returns 202 Accepted. A relay publishes PaymentSubmitted to EventBridge; the Screening context reacts and emits ScreeningCleared. A command goes to the scheme-submission queue; a worker with idempotent handling submits to the scheme. When the scheme confirms, PaymentSettled is announced, and three contexts react without Payments knowing about them: Notifications texts the customer, Accounts posts the debit, the data platform stores the fact.
Only one of those five interactions is synchronous, and it is the only one the customer is waiting on. A Notifications outage, a Screening backlog or a slow scheme never blocks a customer request, and each can be redeployed on its own schedule.
Versioning the contracts
| Change | Safe for consumers? | How |
|---|---|---|
| Add an optional field to an event | Yes | Consumers ignore unknown fields (set JsonSerializerOptions to tolerate them) |
| Rename or remove a field, change a type | No | Publish a new event type (PaymentSettled.v2) next to the old one, migrate consumers, then retire the old one |
| Change the meaning of a field | No | Same as a breaking change, even though the shape is identical |
| Add a new query parameter with a default | Yes | Keep old behaviour when it is absent |
| Change an endpoint's response shape | No | Version the path or media type; run both while consumers migrate |
Put the version in the event's detail-type or in the message type identifier (hl.payment.settled.v1), the same convention AWS.Messaging uses. Add consumer-driven contract tests: each consumer keeps a small test with the JSON it depends on, and the publisher's build runs them.
Common mistakes
- Cutting by technical layer (a "database service", a "validation service") → every business change touches all of them → cut along business capabilities and test with "can one team ship this alone?".
- A synchronous chain of four services → availability multiplies down and latency adds up → collapse the chain, cache read-only data, or turn the non-blocking parts into events.
- Retries at every layer → a single failure becomes dozens of calls (a retry storm) → retry in one layer, usually the outermost caller, and cap the budget (Retry strategies).
- Inner timeout longer than outer timeout → the caller gives up while the callee is still working, and the work is wasted or duplicated → budgets must shrink inward.
- Retrying a command without an idempotency key → duplicate postings → keys on every money-moving call.
- Sharing a database "just for reporting" → a hidden contract that nobody owns → publish events to the data platform instead.
- Assuming a new endpoint is visible immediately → in Service Connect, running clients do not see endpoints added after their last deployment → redeploy the clients, in the order callee first, then callers.
- Hand-editing Cloud Map entries that ECS manages → traffic breaks or later deployments fail → change the ECS service, not the registry.
- Event with a command's meaning ("SendSms") published to a bus → hidden coupling and surprise consumers → commands go to one queue; events are facts in the past tense.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Name or service not known for accounts from Payments | Payments tasks started before Accounts published the endpoint, or Payments has no Service Connect configuration | Force a new Payments deployment (aws ecs update-service --force-new-deployment) after Accounts is running; confirm both are in the same namespace |
| Connection times out between services | Security group has no rule from Payments to the Accounts container port | Allow ingress on 8080 from the Payments security group |
| Calls fail with 5xx from the proxy right after a deploy | Target tasks not yet healthy | Add a container health check on the essential container; Service Connect waits for health before routing |
| Every call takes 15 s, then fails | Proxy default per-request timeout while the app hangs | Fix the callee; set PerRequestTimeout deliberately; keep client timeouts shorter |
ResourceNotFoundException or empty list from discover-instances | Service name or namespace wrong, or the namespace is an HTTP namespace and you queried DNS | Check the discovery name, query by namespace name with the API, and use Service Connect names inside ECS |
| Circuit breaker opens and stays open in tests | Sampling duration too short, or minimum throughput too low | Raise SamplingDuration; test with a realistic request rate |
| Stack deletion fails or leaves a namespace | Service Connect does not delete namespaces with the cluster | Delete the Cloud Map namespace explicitly (the Terraform and CDK code above manage it for you) |
| Old consumers break after a publisher change | A breaking field change shipped under the same version | Add a new versioned event type and keep the old one until every consumer moves |
Cost and cleanup
Cost label: 🟡 credits. Running on Fargate for an hour costs a few cents (three tasks; an estimate follows). The laptop-only run in the code section is free.
| Item | Estimate for 1 hour (us-east-1) | Notes |
|---|---|---|
| 3 Fargate tasks, 0.25 vCPU / 0.5 GB, ARM64 | about USD 0.03 | ⚠ VERIFY on the Fargate pricing page; rates change |
| 3 public IPv4 addresses | about USD 0.015 | Public IPv4 addresses are billed per hour; verify the current rate |
| CloudWatch Logs, Cloud Map, Service Connect | about USD 0 at this volume | No extra charge for Service Connect or its Cloud Map usage |
Teardown is the same whichever tool you used:
# CDK
cd infra/cdk && cdk destroy HlM08-Services --profile learn-sandbox
# Terraform
cd infra/terraform && terraform destroy -auto-approve
# Check that the namespace and tagged resources are gone (tag data can lag a few minutes)
aws servicediscovery list-namespaces --profile learn-sandbox
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m08-microservices \
--query "ResourceTagMappingList[].ResourceARN" --output text --profile learn-sandboxIf harbourline.internal is still listed, delete it in the Cloud Map console or with aws servicediscovery delete-namespace --id <ns-id> after deleting its services.
Free alternative: run the two projects locally as shown. Or do the design-only exercises below.
Exercises
- 🟢 Guided: Run Accounts and Payments locally. Send the three payments, then stop Accounts and send PAY-1 again. Read the log lines and note how many attempts you see and how long the request took before the
503. - 🟡 Challenge: Add a
GET /accounts/{id}/summaryendpoint that returns the balance, and notice that Payments does not need it. Then write a small contract test in Payments that deserialises a storedFundsCheckJSON sample and fails if a required field is removed. Decide what a breaking change toFundsChecklooks like and how you would roll it out. - 🔴 Live problem: The Notifications team reports that after Accounts deployed yesterday, new notification requests succeed but old Payments tasks still call a removed endpoint and fail for about 40 minutes each deploy. Draw the timeline, explain which Service Connect behaviour is involved, and propose a deployment order and a contract rule so it cannot happen again.
Assignment
Write a two-page context map for Harbourline's notification flow and add it to your portfolio repository. Page one: a diagram like the one above with every arrow labelled query, command or event, and a sentence on why. Page two: a table of event contracts (name, version, publisher, consumers, required and optional fields, breaking-change policy), plus the timeout budget from the customer's screen down to the deepest synchronous call.
Solutions
With three attempts of up to 2 s and jittered delays between them, you should see the request return a 503 after roughly 2 to 6 seconds, never longer than the 6-second total budget. If the connection is refused immediately (nothing is listening), each attempt fails quickly, so the whole request is shorter than the worst case. That is the point of budgets: the worst case is bounded, and the caller gets a clear, retryable answer instead of a hung request. If you restart Accounts and send a request after the breaker has opened, the first calls may still fail quickly until the breaker half-opens and probes again.
Payments only needs a yes or no, so the summary endpoint is for other consumers (a mobile app, perhaps). Adding it to Accounts is a non-breaking change: nothing in FundsCheck moves. Typical breaking changes to FundsCheck: renaming Sufficient, changing it to a string, or making Reason required. A safe rollout: add the new field next to the old one; deploy Accounts first (it now returns both); update Payments to read the new field; wait until every Payments version in service reads it; then remove the old one in a later release. The contract test makes "which fields do I depend on?" explicit and runs in the publisher's pipeline.
The ECS documentation states that existing tasks cannot resolve or connect to an endpoint added after their last deployment, and that a service that changes its configuration creates a new deployment. The timeline: Accounts deploys and drops the old endpoint (or path); old Payments tasks continue to use the old configuration; until Payments redeploys, those tasks call something that is gone. Rules that prevent it: (1) expand, then contract: add the new endpoint or path first, keep the old one until all callers have moved; (2) deploy callee first, then callers, and never remove a contract in the same release that adds its replacement; (3) a consumer-driven contract test in the publisher's pipeline; (4) alarm on 5xx rate in the Service Connect metrics during deployments. Mention that this is a coordination problem, not a tooling one.
Knowledge check
Answer: A synchronous query, with a tight timeout budget. The caller cannot continue without the answer. A queue would add delay and complicate the customer's request; an event does not return a result. Keep the query narrow and idempotent so retries are safe.
Answer: Availabilities multiply when every service must succeed: 0.999 to the power of 4 is about 0.996. Each synchronous hop adds failure probability and latency. Fixes: fewer hops, caching, or asynchronous design where the caller does not need the answer.
Answer: A hidden, unowned contract. The owning team can no longer change the schema or enforce its rules without breaking the other service, and releases become coupled. Publish events or expose a query API instead.
Answer: Tasks take their endpoint list when they start, so endpoints added after their last deployment are not visible to them. Redeploy the client services (for example with a forced new deployment). The design lets each client team decide when its configuration changes.
Answer: SendSms is a command: it addresses one receiver and asks for an action. PaymentSettled is an event: a past-tense fact, published without knowing the listeners. Naming commands as events hides coupling.
Answer: Service Connect names resolve only for tasks in services configured for the namespace. A Lambda function is not one, so use the Cloud Map DiscoverInstances API (or expose the service through an internal load balancer) rather than expecting the short name accounts to resolve.
Interview questions
- "Where do you draw service boundaries in a bank?" — Around business capabilities and the data they change: accounts, payments, screening, notifications. The service that changes a piece of data owns it. I check with "can this team ship alone?" and "does the same word mean different things on either side?". I start with fewer, larger services.
- "When do you use a synchronous call, a queue or an event?" — Synchronous only when the caller cannot proceed without the answer and the call is a safe query. A queue when one specific service must act and the caller need not wait. An event when a fact matters to others and the publisher should not know who listens.
- "How do you make a synchronous call safe?" — A timeout per attempt and a budget for all attempts, smaller than the caller's own timeout; jittered retries on queries only (commands need idempotency keys); a circuit breaker; and a clear failure response instead of guessing.
- "How do services find each other on AWS?" — On ECS, Service Connect: names configured on the service, a managed proxy that balances load and retries. For other compute, Cloud Map's API or an internal load balancer. Inside the code, the address is just configuration.
- "How do you change a contract without breaking consumers?" — Expand, then contract: add the new field or version beside the old, migrate consumers, then retire the old. Tolerant readers, versioned event types and consumer-driven contract tests.
Certification practice
An application consists of an order-intake service that synchronously calls a stock service, a pricing service and an email service in turn. Intake fails whenever the email service is slow. The business only needs the email to be sent eventually. Which change MOST improves resilience with the LEAST change to the business flow?
A. Increase the timeout on the email call to 60 seconds. B. Publish an event (for example to an SNS topic or EventBridge bus) and let the email service consume it from its own SQS queue. C. Run the email service on larger instances. D. Add a second synchronous call to a backup email service.
Answer: B. The email is a side effect the caller does not need to wait for, so decouple it. A makes intake slower when email is slow. C does not remove the dependency. D doubles synchronous dependencies. Stock and pricing remain synchronous because intake needs their answers.
A developer runs two ECS services on Fargate. The orders service must call the inventory service by a stable name without managing IP addresses, with retries across inventory tasks and request metrics. Which approach requires the LEAST operational work?
A. Store the task IP addresses in a DynamoDB table and update them from a scheduled Lambda function. B. Configure Amazon ECS Service Connect with a shared namespace and a named port mapping for inventory. C. Put the inventory service behind a public load balancer and use its DNS name. D. Use an EC2 instance running a custom reverse proxy.
Answer: B. Service Connect provides naming, round-robin load balancing, outlier detection, retries and metrics as ECS configuration. A and D are custom machinery. C works but exposes the service publicly and adds a load balancer to manage; an internal load balancer would be the closer alternative.
A service calls a downstream API with three retries, and the downstream is also called by an upstream service that also retries three times. During a brief downstream slowdown, the downstream's load multiplies and it cannot recover. What is the BEST corrective action?
A. Increase the retry count so more requests eventually succeed. B. Retry in only one layer, add jitter and a circuit breaker, and make inner timeouts shorter than outer ones. C. Remove all timeouts so requests are not cancelled. D. Move all retries into the downstream API.
Answer: B. Multiplied retries create a retry storm. Limit retries to one layer, spread them with jitter, stop calling a failing dependency with a circuit breaker, and shrink timeouts inward. A and C make it worse. D cannot work, because the downstream does not control its callers' retries.
Completion checklist
- I can define a bounded context and name the data each Harbourline context owns
- I can classify an interaction as query, command or event and choose the matching AWS mechanism
- I ran the two services locally and saw the
503after the retry budget - I can explain what Service Connect does, what it does not do (authenticate callers) and why running clients must be redeployed to see new endpoints
- I read both the CDK and Terraform versions and can say what each setting does
- If I deployed, I destroyed the stack and confirmed the Cloud Map namespace and tagged resources were gone
Key takeaways
- Cut by business capability, own your data. A shared database turns microservices back into a distributed monolith.
- Ask, tell, announce. Queries are synchronous, commands go to one queue, events are past-tense facts for anyone who cares. Keep the customer-facing path synchronous only where the answer is needed.
- Make synchronous calls safe: timeouts that shrink inward, jittered retries in one layer, a circuit breaker, and a clear failure instead of a guess.
- On ECS, Service Connect is the default for names; Cloud Map is the general registry. Neither authenticates callers, and running clients see new endpoints only after redeployment.
- Evolve contracts by expanding, then contracting, with versioned events and contract tests.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Amazon MQ: bridging legacy JMS and AMQP systems
Know when a managed ActiveMQ or RabbitMQ broker is the right tool: keep a JMS/AMQP system running on Amazon MQ, bridge it to SQS, SNS and EventBridge with a Lambda event source mapping and a translator, handle poison messages, and plan the migration, with .NET 10 clients, CDK and Terraform.
AWS.Messaging for .NET: publishers, handlers and envelopes
Publish and handle SQS, SNS and EventBridge messages with the AWS Message Processing Framework for .NET: typed publishers, IMessageHandler classes, the CloudEvents envelope, middleware and error handlers, hosted in a Worker Service or Lambda, with idempotency added on top.