Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

Microservices communication and service discovery

Draw bounded contexts for a bank, choose between a query, a command and an event for every interaction, make synchronous calls safe with timeouts and retries, and let services find each other with Amazon ECS Service Connect and AWS Cloud Map.

advanced⏱ 2 h 🟡 Low costDVA-C03SAA-C03

Assumes: .NET 10 LTS · Microsoft.Extensions.Http.Resilience 10.11.0 · AWSSDK.ServiceDiscovery 4.1.0.2 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Split a banking domain into bounded contexts, say which data each context owns, and explain why a shared database defeats the split.
  • Classify any interaction as a query, a command or an event, and choose a synchronous call, an SQS queue or an EventBridge event for it.
  • Make a synchronous HTTP call safe in .NET with a timeout budget, jittered retries and a circuit breaker (AddStandardResilienceHandler).
  • Explain how AWS Cloud Map and Amazon ECS Service Connect let one service reach another by name, and how that differs from plain DNS.
  • Deploy two Fargate services that call each other as http://accounts:8080, with the infrastructure written in CDK (C#) and Terraform.
  • Version events and APIs so that one team can deploy without breaking another.

Prerequisites

Why this matters in banking

Harbourline's payments team wants to release twice a week. Their code lives in a modular monolith that also contains the accounts logic, the notification templates and the sanctions rules. Every release needs a regression run across all four, and a bug in the notification templates once delayed a payments fix by nine days.

So the bank splits the monolith into services. Six months later a different problem appears. The payment service calls the accounts service, which calls the limits service, which calls the customer-profile service, all synchronously, to authorise one transfer. If each service is available 99.9% of the time, a chain of four gives roughly 99.6% for the whole request (0.999 to the power of 4), about 35 hours of failure a year instead of about 9. One slow profile lookup makes every payment slow. Worse, a retry at each hop multiplies load: three retries at four layers can turn one request into 81 calls against the struggling service at the bottom.

Two things went wrong. The boundaries were drawn around technical layers instead of business capabilities, and every interaction was a synchronous query, even the ones that were really announcements ("a payment settled"). This lesson gives you a way to draw the boundaries, a rule for choosing the communication style, and the AWS plumbing (service discovery, timeouts, retries) that makes the synchronous calls you do keep behave.

Design note for banks

Service boundaries are also data boundaries. Which service may hold which personal data, and which service calls are logged for audit, are decisions for your data-protection and security teams. This lesson shows mechanisms, not a compliance position.

The idea in plain language

A microservice is a small program that owns one business capability and its data, deploys on its own, and talks to others only over the network. The hard part is not the code. It is deciding where to cut and how the pieces talk.

Bounded context. A term from domain-driven design (DDD): an area of the business in which every word has one precise meaning. "Account" means something different to the Accounts team (a ledger with a balance and a status) than to the Notifications team (a place to send a message about). Each context has its own model, its own database and its own team. They share identifiers (ACC-0000000001, CUS-00012345), not tables.

Three kinds of message cross a boundary. Naming them correctly decides almost every design question:

KindMeaningExampleDirection of knowledge
Query"Tell me something." No side effect; safe to repeat.Does account ACC-0000000001 have 120.00 GBP available?Caller knows the callee
Command"Do this." One addressed receiver, may fail, has a side effect.Send an SMS for transaction T-123.Sender knows the receiver
Event"This happened." Past tense, a fact, many or zero receivers.PaymentSettledPublisher does not know who listens

Analogy: In a bank branch, a query is you leaning over to a colleague and asking "is this account frozen?". You wait for the answer, and if they are away you are stuck. A command is a task slip dropped in the mortgage team's tray: it will be handled, you may or may not hear back. An event is the announcement on the branch intercom: "the vault has been opened". Anyone who cares reacts; the announcer does not know or wait for them.

The rule of thumb that follows: ask with a synchronous call only when you cannot proceed without the answer, tell with a queue when one specific service must act, announce with an event when the fact matters to others. Everything that is not a query should lean asynchronous, because asynchronous messaging decouples the availability and speed of services, not just their code.

What problem the AWS services solve

Splitting a system moves function calls onto the network, and a network call raises questions a function call never did:

  1. Where is the other service right now? Containers start, stop and move. Their IP addresses change. Hard-coding addresses, or asking an operator to update a config file, does not work. This is service discovery.
  2. What if it is slow or down? You need timeouts, retries, circuit breakers and a fallback. This is resilience, partly in your code and partly in the network layer.
  3. How do we change it without breaking callers? Contracts need versioning, and consumers must tolerate unknown fields. This is evolvability.

AWS offers several building blocks. You choose by where the caller and callee run and how much you want managed:

NeedAWS serviceNotes
Name → address for ECS servicesAmazon ECS Service ConnectConfiguration on the ECS service; a managed proxy sidecar does round-robin load balancing, outlier detection and retries
Name → address for anything (EC2, ECS, Lambda, DynamoDB tables, SQS queues, on-premises)AWS Cloud MapA registry you query with DiscoverInstances or with DNS; you register and deregister
Public or internal HTTP entry point with path routing and health checksApplication Load BalancerGood in front of a service that many clients (and non-ECS clients) call
Managed API front door with auth, throttling and usage plansAmazon API GatewayCovered in Module 4
Cross-VPC and cross-account service-to-service networking, policy at the edgeAmazon VPC LatticeA different, more general application-networking service. Not covered here ⚠ VERIFY current feature set before choosing it
Decoupled "tell" and "announce"SQS, SNS, EventBridgeThe rest of this module

Limits to know about Service Connect (from the ECS documentation, checked 2026-11-18):

  • Only ECS services are supported, not standalone tasks. Windows containers, HTTP/1.0 and FIPS mode are not supported.
  • A service belongs to one namespace, and the namespace must be in the same Region. Each endpoint and each discovery name must be unique within the namespace.
  • Existing tasks do not see new endpoints. A client must be redeployed before it can resolve an endpoint added after its last deployment. That is a deliberate design: the client team decides when its configuration changes.
  • Tasks need a task-level memory limit. AWS recommends adding about 256 CPU units and 64 MiB of memory for the proxy sidecar on top of your own sizing. ⚠ VERIFY the figures in the current ECS guide before capacity planning.
  • There is no extra charge for Service Connect or for the Cloud Map usage it generates; you pay for the compute the sidecar uses. TLS encryption with AWS Private CA is billed separately.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Container Apps with Dapr service invocation, or built-in app namesECS Service ConnectName-based calls with retries and metrics, but configured on the ECS service, with no Dapr runtime to operate
AKS ClusterIP Service + CoreDNS names (http://accounts)Service Connect, or ECS service discovery (Cloud Map DNS)Service Connect resolves names through its proxy and does not depend on VPC DNS; Cloud Map DNS is plain DNS with caching
Consul / Eureka / Steeltoe discovery clientsAWS Cloud Map DiscoverInstancesA managed registry with an IAM-protected API; instances carry attributes
Polly (Microsoft.Extensions.Http.Polly)Microsoft.Extensions.Http.ResilienceThe same ideas (retry, circuit breaker, timeout) packaged as one "standard handler" on the HttpClient builder
Azure API Management in front of internal servicesAPI Gateway or an internal ALBAWS puts auth and throttling in API Gateway; internal service-to-service traffic usually skips it
Service Bus queue (command) and Event Grid topic (event)SQS (command), EventBridge or SNS (event)The same split between "tell one" and "announce to many"
Shared SQL Server across several appsOne database per service, e.g. Aurora, DynamoDBThe hardest habit to break. Shared tables couple deployments

Architecture

The Harbourline context map. Solid arrows are synchronous calls; dashed arrows are asynchronous.

Every context has its own data store, and no arrow goes into another context's database. The Payments service asks Accounts a query because it cannot decide without the answer. It sends the scheme a command through a queue so a slow scheme does not hold a customer request open. It announces PaymentSettled so that Notifications, Fraud and the data platform can react without Payments knowing they exist.

Security-boundary view. Inside the VPC, tasks run in private subnets (the lab uses public subnets to avoid the cost of a NAT gateway, flagged below). A security group allows Payments tasks to reach Accounts tasks on the container port, and nothing else. IAM task roles decide which AWS APIs each service may call. Service Connect is a naming and routing feature. It does not authenticate callers by itself, so add TLS (optionally Service Connect TLS) and application-level authorisation when the data warrants it.

Drawing the boundaries: a quick method

  1. List the business capabilities (open account, authorise payment, screen sanctions, notify customer) rather than technical layers.
  2. For each, list the nouns it owns and changes (balance, payment, screening result, delivery attempt). The context that changes a noun owns it.
  3. Check each pair for the words that mean different things. If "customer" is a profile in one place and a legal entity in another, you have found a boundary.
  4. Prefer fewer, larger services at first. A "distributed monolith" (many services that must be deployed together) is worse than a monolith.
  5. Write the interactions down as query, command or event. A context that needs ten synchronous queries to do its job is probably cut in the wrong place.
ContextOwnsPublishesConsumesStore
AccountsAccounts, balances, holdsAccountFrozen, PostingBookedPaymentSettled (to post)Aurora PostgreSQL
PaymentsPayment lifecycle, scheme submissionsPaymentSubmitted, PaymentSettled, PaymentRejectedScreeningClearedAurora PostgreSQL
ScreeningSanctions screening decisionsScreeningCleared, ScreeningHitPaymentSubmittedDynamoDB
NotificationsDelivery attempts, templates, opt-insNotificationSentPaymentSettled, PostingBookedDynamoDB

Step-by-step implementation

Classify the interaction

For every arrow in the context map, ask three questions. Can the caller continue without the answer? Is there exactly one receiver? Is it a fact or a request? The answers choose the style:

Can the caller proceed without the answer?ReceiversStyle
NoOneSynchronous query (HTTP) with strict timeouts
YesOneCommand on a queue (SQS)
YesZero to manyEvent on a bus or topic (EventBridge, SNS)

In this lesson, "does the debtor have funds?" is a query. "Submit to the scheme" is a command. "Payment settled" is an event.

Build the callee: a query with a narrow answer

The Accounts service exposes a funds check that returns only what Payments needs: yes or no, and a reason. It does not return the balance. That keeps the contract small and keeps personal data inside its context.

Build the caller with a resilience pipeline

The Payments service uses a typed HttpClient with the standard resilience handler. You set a timeout for each attempt, a ceiling for all attempts together, a small number of jittered retries and a circuit breaker.

Run both locally

Start both services with dotnet run and point Services:Accounts:BaseUrl at http://localhost:5080. Nothing in the code mentions AWS yet.

Deploy to ECS with Service Connect

Put the same images in two ECS services in one namespace. Accounts publishes the endpoint accounts:8080; Payments is a client. The only change to the Payments configuration is the base URL: http://accounts:8080.

Add the asynchronous edges

Add the queue for the command and the event for the announcement. The outbox pattern makes the publish reliable, and AWS.Messaging for .NET shows the framework version of publishers and handlers.

AWS Console instructions

To see Service Connect after deployment:

  1. Console → Amazon ECS → Clusters → hl-m08-services → Services → accounts. Open the Configuration and networking tab and find Service Connect: the namespace harbourline.internal, the discovery name accounts and the client alias accounts:8080.
  2. Open the Tasks tab. Each task lists two containers: yours and the Service Connect proxy sidecar that ECS added.
  3. Open the Logs tab for the service's log group, and the Metrics tab for the Service Connect request count, error counts and latency. ⚠ VERIFY exact tab and metric names; the console changes often.
  4. Console → AWS Cloud Map → Namespaces → harbourline.internal to see the HTTP namespace and the services that ECS created inside it. Look, but do not edit: the ECS documentation warns that manually changing Cloud Map resources that Service Connect manages can break routing and later deployments.

AWS CLI

# List namespaces. Service Connect created an HTTP namespace with the name you chose.
aws servicediscovery list-namespaces --profile learn-sandbox

# Ask Cloud Map for the instances behind the "accounts" service (works for HTTP namespaces).
# This is the same API a non-ECS client (a Lambda function, an EC2 job) would call.
aws servicediscovery discover-instances \
  --namespace-name harbourline.internal \
  --service-name accounts \
  --profile learn-sandbox

# Show Service Connect settings and recent deployments for one ECS service.
aws ecs describe-services --cluster hl-m08-services --services accounts \
  --query "services[0].{sc:deployments[0].serviceConnectConfiguration,status:deployments[0].rolloutState}" \
  --profile learn-sandbox

# Tail the payments container logs while you call it.
aws logs tail /hl-m08/services --follow --filter-pattern payments --profile learn-sandbox

# Exec into the payments task and call the callee by name (needs ECS Exec enabled; see Module 3).
# aws ecs execute-command ... --command "wget -qO- 'http://accounts:8080/accounts/ACC-0000000001/funds-check?amount=10&currency=GBP'"

Infrastructure as Code

Both versions create a VPC (CDK only), a cluster whose default Service Connect namespace is harbourline.internal, two Fargate services, a log group and a security rule that lets Payments reach Accounts on port 8080. Replace the image placeholders with images you pushed to your own ECR repository (Module 3 shows how).

infra/cdk/Program.cs
using Amazon.CDK;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.Logs;
using Constructs;

var app = new App();
_ = new ServicesStack(app, "HlM08-Services", new StackProps());
app.Synth();

public sealed class ServicesStack : Stack
{
    public ServicesStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m08-microservices");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");

        var vpc = new Vpc(this, "Vpc", new VpcProps
        {
            MaxAzs = 2,
            NatGateways = 0, // lab only: tasks get public IPs. Production: private subnets + VPC endpoints or NAT.
            SubnetConfiguration = [new SubnetConfiguration { Name = "public", SubnetType = SubnetType.PUBLIC, CidrMask = 24 }],
        });

        // The cluster's default namespace: Service Connect services join it automatically.
        var cluster = new Cluster(this, "Cluster", new ClusterProps
        {
            ClusterName = "hl-m08-services",
            Vpc = vpc,
            DefaultCloudMapNamespace = new CloudMapNamespaceOptions
            {
                Name = "harbourline.internal",
                Type = Amazon.CDK.AWS.ServiceDiscovery.NamespaceType.HTTP,
            },
        });

        var logs = new LogGroup(this, "Logs", new LogGroupProps
        {
            LogGroupName = "/hl-m08/services",
            Retention = RetentionDays.ONE_WEEK,
            RemovalPolicy = RemovalPolicy.DESTROY,
        });

        var accountsTask = new FargateTaskDefinition(this, "AccountsTask", new FargateTaskDefinitionProps
        {
            Cpu = 256,
            MemoryLimitMiB = 512,
            RuntimePlatform = new RuntimePlatform { CpuArchitecture = CpuArchitecture.ARM64, OperatingSystemFamily = OperatingSystemFamily.LINUX },
        });
        accountsTask.AddContainer("accounts", new ContainerDefinitionOptions
        {
            Image = ContainerImage.FromRegistry("REPLACE_WITH_YOUR_ECR_ACCOUNTS_IMAGE"),
            Essential = true,
            // The port NAME is what Service Connect refers to. appProtocol gives HTTP metrics.
            PortMappings = [new PortMapping { Name = "http", ContainerPort = 8080, AppProtocol = AppProtocol.Http }],
            Logging = LogDriver.AwsLogs(new AwsLogDriverProps { StreamPrefix = "accounts", LogGroup = logs }),
            HealthCheck = new Amazon.CDK.AWS.ECS.HealthCheck
            {
                Command = ["CMD-SHELL", "wget -qO- http://localhost:8080/healthz || exit 1"],
                Interval = Duration.Seconds(15),
                Retries = 3,
            },
        });

        var accounts = new FargateService(this, "AccountsService", new FargateServiceProps
        {
            Cluster = cluster,
            ServiceName = "accounts",
            TaskDefinition = accountsTask,
            DesiredCount = 2,
            AssignPublicIp = true,
            ServiceConnectConfiguration = new ServiceConnectProps
            {
                Services =
                [
                    new ServiceConnectService
                    {
                        PortMappingName = "http",
                        DnsName = "accounts",
                        Port = 8080,
                        PerRequestTimeout = Duration.Seconds(5),
                    },
                ],
            },
        });

        var paymentsTask = new FargateTaskDefinition(this, "PaymentsTask", new FargateTaskDefinitionProps
        {
            Cpu = 256,
            MemoryLimitMiB = 512,
            RuntimePlatform = new RuntimePlatform { CpuArchitecture = CpuArchitecture.ARM64, OperatingSystemFamily = OperatingSystemFamily.LINUX },
        });
        paymentsTask.AddContainer("payments", new ContainerDefinitionOptions
        {
            Image = ContainerImage.FromRegistry("REPLACE_WITH_YOUR_ECR_PAYMENTS_IMAGE"),
            Essential = true,
            PortMappings = [new PortMapping { Name = "http", ContainerPort = 8080, AppProtocol = AppProtocol.Http }],
            Environment = new Dictionary<string, string> { ["Services__Accounts__BaseUrl"] = "http://accounts:8080" },
            Logging = LogDriver.AwsLogs(new AwsLogDriverProps { StreamPrefix = "payments", LogGroup = logs }),
        });

        var payments = new FargateService(this, "PaymentsService", new FargateServiceProps
        {
            Cluster = cluster,
            ServiceName = "payments",
            TaskDefinition = paymentsTask,
            DesiredCount = 1,
            AssignPublicIp = true,
            // A client-only service: no Services list, so it only resolves other endpoints.
            ServiceConnectConfiguration = new ServiceConnectProps(),
        });

        // Security group: payments tasks may reach accounts tasks on the container port, nothing else.
        accounts.Connections.AllowFrom(payments, Port.Tcp(8080), "payments to accounts via Service Connect");
    }
}

Both versions compiled (CDK) and validated (terraform validate with the v6 AWS provider) on 2026-11-18. Neither was deployed; cdk deploy and terraform apply need your own images and execution role. In the CDK version the execution role and the VPC are created for you. In the Terraform version, supply an existing VPC, subnets and task execution role through variables.

Complete code

Two small ASP.NET Core minimal APIs and the shared contract. They use fictional data only. Create two projects (dotnet new web -n Accounts and dotnet new web -n Payments), and add Microsoft.Extensions.Http.Resilience to Payments.

Accounts/Program.cs
using System.Collections.Concurrent;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

// Fictional data. In a real service this is the Accounts context's own database.
var accounts = new ConcurrentDictionary<string, AccountView>
{
    ["ACC-0000000001"] = new("ACC-0000000001", "GBP", 2_500.00m, "open"),
    ["ACC-0000000002"] = new("ACC-0000000002", "GBP", 15.00m, "frozen"),
};

app.MapGet("/healthz", () => Results.Ok("ok"));

// A QUERY: safe to repeat, no side effect. Returns only what the caller needs.
app.MapGet("/accounts/{accountId}/funds-check", (string accountId, decimal amount, string currency) =>
{
    if (!accounts.TryGetValue(accountId, out var account))
    {
        return Results.Problem(statusCode: 404, title: "Account not found");
    }

    var ok = account.Status == "open" && account.Currency == currency && account.Balance >= amount;
    return Results.Ok(new FundsCheck(accountId, ok, ok ? null : "insufficient funds or account not open"));
});

app.Run();

public sealed record AccountView(string AccountId, string Currency, decimal Balance, string Status);
public sealed record FundsCheck(string AccountId, bool Sufficient, string? Reason);
Payments/Program.cs
using System.Globalization;
using System.Net.Http.Json;
using Microsoft.Extensions.Http.Resilience;

var builder = WebApplication.CreateBuilder(args);

// One setting changes between laptop and AWS:
//   laptop:               Services:Accounts:BaseUrl = http://localhost:5080
//   ECS Service Connect:  Services:Accounts:BaseUrl = http://accounts:8080
builder.Services.AddHttpClient<IAccountsClient, AccountsClient>(client =>
{
    client.BaseAddress = new Uri(builder.Configuration["Services:Accounts:BaseUrl"]
        ?? throw new InvalidOperationException("Services:Accounts:BaseUrl is not set."));
})
.AddStandardResilienceHandler(options =>
{
    // Queries are safe to retry. Keep the whole budget (all attempts) well under the caller's own timeout.
    options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
    options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(6);
    options.Retry.MaxRetryAttempts = 2;
    options.Retry.UseJitter = true;
    options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30); // must be at least twice the attempt timeout
});

var app = builder.Build();

app.MapPost("/payments", async (PaymentRequest request, IAccountsClient accounts, CancellationToken ct) =>
{
    FundsCheck? check;
    try
    {
        check = await accounts.CheckFundsAsync(request.DebtorAccountId, request.Amount, request.Currency, ct);
    }
    catch (HttpRequestException)
    {
        // Accounts is unavailable after retries: fail fast with a clear, retryable status. Do not guess.
        return Results.Problem(statusCode: 503, title: "Accounts service unavailable", detail: "Try again shortly.");
    }

    return check is { Sufficient: true }
        ? Results.Accepted($"/payments/{request.PaymentId}", new { request.PaymentId, status = "received" })
        : Results.Problem(statusCode: 422, title: "Payment rejected", detail: check?.Reason);
});

app.Run();

public sealed record PaymentRequest(string PaymentId, string DebtorAccountId, decimal Amount, string Currency);
public sealed record FundsCheck(string AccountId, bool Sufficient, string? Reason);

public interface IAccountsClient
{
    Task<FundsCheck?> CheckFundsAsync(string accountId, decimal amount, string currency, CancellationToken ct);
}

public sealed class AccountsClient(HttpClient http) : IAccountsClient
{
    public Task<FundsCheck?> CheckFundsAsync(string accountId, decimal amount, string currency, CancellationToken ct)
    {
        var url = $"accounts/{Uri.EscapeDataString(accountId)}/funds-check"
            + $"?amount={amount.ToString(CultureInfo.InvariantCulture)}&currency={currency}";
        return http.GetFromJsonAsync<FundsCheck>(url, ct);
    }
}

Run it locally:

# Terminal 1
dotnet run --project Accounts --urls http://localhost:5080
# Terminal 2
dotnet run --project Payments --urls http://localhost:5090 -- --Services:Accounts:BaseUrl=http://localhost:5080
# Terminal 3: enough funds, then too much, then a frozen account
curl -s -X POST http://localhost:5090/payments -H "Content-Type: application/json" \
  -d '{"paymentId":"PAY-1","debtorAccountId":"ACC-0000000001","amount":120.00,"currency":"GBP"}'
curl -s -X POST http://localhost:5090/payments -H "Content-Type: application/json" \
  -d '{"paymentId":"PAY-2","debtorAccountId":"ACC-0000000001","amount":9000.00,"currency":"GBP"}'
curl -s -X POST http://localhost:5090/payments -H "Content-Type: application/json" \
  -d '{"paymentId":"PAY-3","debtorAccountId":"ACC-0000000002","amount":10.00,"currency":"GBP"}'
# Now stop Accounts (Ctrl+C) and send PAY-1 again: after retries you get a 503, not a hang.

Calling from a client that is not an ECS service with Service Connect (for example, a Lambda function) uses the Cloud Map API instead of a name that the proxy resolves:

CloudMapResolver.cs
using Amazon.ServiceDiscovery;
using Amazon.ServiceDiscovery.Model;

public sealed class CloudMapResolver(IAmazonServiceDiscovery cloudMap)
{
    /// <summary>Returns "ip:port" of one healthy instance. Cache the result for a few seconds, not per request.</summary>
    public async Task<string> ResolveAsync(string namespaceName, string serviceName, CancellationToken ct)
    {
        var response = await cloudMap.DiscoverInstancesAsync(new DiscoverInstancesRequest
        {
            NamespaceName = namespaceName,
            ServiceName = serviceName,
            HealthStatus = HealthStatusFilter.HEALTHY_OR_ELSE_ALL,
        }, ct);

        var instances = response.Instances;
        if (instances.Count == 0)
        {
            throw new InvalidOperationException($"No instances registered for {serviceName} in {namespaceName}.");
        }

        var chosen = instances[Random.Shared.Next(instances.Count)];
        return $"{chosen.Attributes["AWS_INSTANCE_IPV4"]}:{chosen.Attributes["AWS_INSTANCE_PORT"]}";
    }
}

The package is AWSSDK.ServiceDiscovery 4.1.0.2. The attribute names AWS_INSTANCE_IPV4 and AWS_INSTANCE_PORT are Cloud Map's standard keys for instances registered by ECS service discovery ⚠ VERIFY against the instances you actually register, and note that Service Connect instances are meant to be reached by name through the proxy, not resolved by hand.

Code explained line by line

  1. MapGet("/accounts/{accountId}/funds-check", ...): a query endpoint. It uses GET, has no side effects and is therefore safe for the caller to retry. A command such as "post this payment" would use POST and need an idempotency key (Idempotency in .NET).
  2. return Results.Ok(new FundsCheck(accountId, ok, ...)): the answer is a narrow view, not the account. Payments never sees the balance. If Accounts later changes how balances work, the contract does not change.
  3. AddHttpClient<IAccountsClient, AccountsClient>(...): a typed client managed by IHttpClientFactory. It reuses connections and lets you attach handlers to the pipeline.
  4. BaseAddress read from configuration: the only thing that differs between laptop and cluster. Code does not know about Cloud Map or Service Connect. This is why name-based discovery is pleasant: local, test and production all use a URL.
  5. .AddStandardResilienceHandler(...): adds a pipeline of rate limiter, total timeout, retry, circuit breaker and per-attempt timeout in one call. The options below tune it.
  6. AttemptTimeout 2 s, TotalRequestTimeout 6 s, MaxRetryAttempts 2: three tries of up to 2 s each fit inside 6 s. Your timeout budget must be smaller than your caller's timeout, which must be smaller than its caller's. In a chain, the outermost timeout is the customer's patience; every inner one shrinks.
  7. UseJitter = true: randomises retry delays so that many Payments tasks do not retry in lockstep (Retry strategies).
  8. CircuitBreaker.SamplingDuration: the library validates that this is at least double the attempt timeout. The breaker opens after repeated failures, so the next calls fail instantly instead of queueing behind a dead service.
  9. catch (HttpRequestException) → 503: when the callee is down after retries, Payments tells its own caller "unavailable, try again". It does not invent an answer about the customer's money. A circuit-breaker rejection surfaces as a different exception type (BrokenCircuitException from the underlying Polly library) so a real service should catch that as well; the lab keeps one catch for readability ⚠ VERIFY the exact exception your version throws by stopping Accounts and running the test above.
  10. Services__Accounts__BaseUrl in the CDK and Terraform code: ASP.NET Core maps a double underscore in an environment variable to the : configuration separator, so this sets Services:Accounts:BaseUrl.
  11. PortMapping { Name = "http", AppProtocol = Http }: Service Connect addresses a port by its name; AppProtocol tells the proxy the traffic is HTTP, which unlocks HTTP-level metrics such as 5xx counts.
  12. ServiceConnectService { DnsName = "accounts", Port = 8080 }: the client alias. Clients in the namespace call http://accounts:8080; the proxy picks one of the two Accounts tasks (round robin) and retries a failed connection on another task. ECS's documentation says the proxy retries twice, avoiding the host that just failed ⚠ VERIFY if you rely on the exact number.
  13. PerRequestTimeout = 5 s: the proxy gives up on a request after five seconds even if your client waits longer. The ECS default is 15 seconds. Keep the proxy's timeout below the client's total budget so that the layers do not fight.
  14. new ServiceConnectProps() on Payments: a client-only configuration. No Services means it publishes no endpoint, but it joins the namespace and can resolve accounts.
  15. accounts.Connections.AllowFrom(payments, Port.Tcp(8080), ...): the security-group rule that lets the Payments tasks reach the Accounts proxy port. Service Connect routes traffic; you still open the network path.

Security considerations

⚠ Learning shortcut — not for production

The lab runs tasks in public subnets with public IPs and NatGateways = 0 so that the demo avoids a NAT gateway bill and extra setup. Production runs tasks in private subnets, reaches ECR, CloudWatch Logs and S3 through VPC endpoints (or a NAT gateway), and never gives tasks public IPs. The security group in the lab allows only Payments → Accounts on 8080; keep that least-privilege shape in production.

  • Service Connect is not authentication. It resolves names and balances load. A service that must restrict callers needs its own check: mutual TLS (Service Connect can encrypt traffic with certificates from AWS Private CA, which is billed separately), signed tokens, or IAM-based authorisation at an API Gateway or ALB edge. Decide per data class.
  • Least-privilege task roles. Accounts and Payments get separate task roles. Payments may publish to its bus and queue; it has no access to the Accounts database or table.
  • No shared database. If Payments reads Accounts' tables directly, you bypass the contract, the audit log and the owner's ability to change the schema. In banking, the owning service is also the single place that enforces posting rules.
  • Keep personal data out of events and logs. Events carry identifiers and facts (paymentId, amount, currency), not names or full IBANs. Log correlation IDs and outcomes, not request bodies. This follows the same pattern as the module project.
  • Don't edit Cloud Map by hand. The ECS documentation warns that manual changes to the resources Service Connect creates can break traffic and later deployments. Control who has servicediscovery:* write permissions.
  • Compliance. Which services may exchange which personal data, in which Region, and what the audit trail must show, needs review by your legal, data-protection and security teams. Nothing here is a compliance claim.

Banking use case

Harbourline authorises a Faster Payments transfer in three moves. Payments asks Accounts a query ("funds and status?") with a 2-second attempt timeout and a 6-second budget, because the customer is waiting. If the answer is yes, Payments writes the payment and an outbox row in one transaction, and returns 202 Accepted. A relay publishes PaymentSubmitted to EventBridge; the Screening context reacts and emits ScreeningCleared. A command goes to the scheme-submission queue; a worker with idempotent handling submits to the scheme. When the scheme confirms, PaymentSettled is announced, and three contexts react without Payments knowing about them: Notifications texts the customer, Accounts posts the debit, the data platform stores the fact.

Only one of those five interactions is synchronous, and it is the only one the customer is waiting on. A Notifications outage, a Screening backlog or a slow scheme never blocks a customer request, and each can be redeployed on its own schedule.

Versioning the contracts

ChangeSafe for consumers?How
Add an optional field to an eventYesConsumers ignore unknown fields (set JsonSerializerOptions to tolerate them)
Rename or remove a field, change a typeNoPublish a new event type (PaymentSettled.v2) next to the old one, migrate consumers, then retire the old one
Change the meaning of a fieldNoSame as a breaking change, even though the shape is identical
Add a new query parameter with a defaultYesKeep old behaviour when it is absent
Change an endpoint's response shapeNoVersion the path or media type; run both while consumers migrate

Put the version in the event's detail-type or in the message type identifier (hl.payment.settled.v1), the same convention AWS.Messaging uses. Add consumer-driven contract tests: each consumer keeps a small test with the JSON it depends on, and the publisher's build runs them.

Common mistakes

  • Cutting by technical layer (a "database service", a "validation service") → every business change touches all of them → cut along business capabilities and test with "can one team ship this alone?".
  • A synchronous chain of four services → availability multiplies down and latency adds up → collapse the chain, cache read-only data, or turn the non-blocking parts into events.
  • Retries at every layer → a single failure becomes dozens of calls (a retry storm) → retry in one layer, usually the outermost caller, and cap the budget (Retry strategies).
  • Inner timeout longer than outer timeout → the caller gives up while the callee is still working, and the work is wasted or duplicated → budgets must shrink inward.
  • Retrying a command without an idempotency key → duplicate postings → keys on every money-moving call.
  • Sharing a database "just for reporting" → a hidden contract that nobody owns → publish events to the data platform instead.
  • Assuming a new endpoint is visible immediately → in Service Connect, running clients do not see endpoints added after their last deployment → redeploy the clients, in the order callee first, then callers.
  • Hand-editing Cloud Map entries that ECS manages → traffic breaks or later deployments fail → change the ECS service, not the registry.
  • Event with a command's meaning ("SendSms") published to a bus → hidden coupling and surprise consumers → commands go to one queue; events are facts in the past tense.

Troubleshooting

SymptomLikely causeFix
Name or service not known for accounts from PaymentsPayments tasks started before Accounts published the endpoint, or Payments has no Service Connect configurationForce a new Payments deployment (aws ecs update-service --force-new-deployment) after Accounts is running; confirm both are in the same namespace
Connection times out between servicesSecurity group has no rule from Payments to the Accounts container portAllow ingress on 8080 from the Payments security group
Calls fail with 5xx from the proxy right after a deployTarget tasks not yet healthyAdd a container health check on the essential container; Service Connect waits for health before routing
Every call takes 15 s, then failsProxy default per-request timeout while the app hangsFix the callee; set PerRequestTimeout deliberately; keep client timeouts shorter
ResourceNotFoundException or empty list from discover-instancesService name or namespace wrong, or the namespace is an HTTP namespace and you queried DNSCheck the discovery name, query by namespace name with the API, and use Service Connect names inside ECS
Circuit breaker opens and stays open in testsSampling duration too short, or minimum throughput too lowRaise SamplingDuration; test with a realistic request rate
Stack deletion fails or leaves a namespaceService Connect does not delete namespaces with the clusterDelete the Cloud Map namespace explicitly (the Terraform and CDK code above manage it for you)
Old consumers break after a publisher changeA breaking field change shipped under the same versionAdd a new versioned event type and keep the old one until every consumer moves

Cost and cleanup

Cost label: 🟡 credits. Running on Fargate for an hour costs a few cents (three tasks; an estimate follows). The laptop-only run in the code section is free.

ItemEstimate for 1 hour (us-east-1)Notes
3 Fargate tasks, 0.25 vCPU / 0.5 GB, ARM64about USD 0.03⚠ VERIFY on the Fargate pricing page; rates change
3 public IPv4 addressesabout USD 0.015Public IPv4 addresses are billed per hour; verify the current rate
CloudWatch Logs, Cloud Map, Service Connectabout USD 0 at this volumeNo extra charge for Service Connect or its Cloud Map usage

Teardown is the same whichever tool you used:

# CDK
cd infra/cdk && cdk destroy HlM08-Services --profile learn-sandbox

# Terraform
cd infra/terraform && terraform destroy -auto-approve

# Check that the namespace and tagged resources are gone (tag data can lag a few minutes)
aws servicediscovery list-namespaces --profile learn-sandbox
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m08-microservices \
  --query "ResourceTagMappingList[].ResourceARN" --output text --profile learn-sandbox

If harbourline.internal is still listed, delete it in the Cloud Map console or with aws servicediscovery delete-namespace --id <ns-id> after deleting its services.

Free alternative: run the two projects locally as shown. Or do the design-only exercises below.

Exercises

  • 🟢 Guided: Run Accounts and Payments locally. Send the three payments, then stop Accounts and send PAY-1 again. Read the log lines and note how many attempts you see and how long the request took before the 503.
  • 🟡 Challenge: Add a GET /accounts/{id}/summary endpoint that returns the balance, and notice that Payments does not need it. Then write a small contract test in Payments that deserialises a stored FundsCheck JSON sample and fails if a required field is removed. Decide what a breaking change to FundsCheck looks like and how you would roll it out.
  • 🔴 Live problem: The Notifications team reports that after Accounts deployed yesterday, new notification requests succeed but old Payments tasks still call a removed endpoint and fail for about 40 minutes each deploy. Draw the timeline, explain which Service Connect behaviour is involved, and propose a deployment order and a contract rule so it cannot happen again.

Assignment

Write a two-page context map for Harbourline's notification flow and add it to your portfolio repository. Page one: a diagram like the one above with every arrow labelled query, command or event, and a sentence on why. Page two: a table of event contracts (name, version, publisher, consumers, required and optional fields, breaking-change policy), plus the timeout budget from the customer's screen down to the deepest synchronous call.

Solutions

Knowledge check

Interview questions

  • "Where do you draw service boundaries in a bank?" — Around business capabilities and the data they change: accounts, payments, screening, notifications. The service that changes a piece of data owns it. I check with "can this team ship alone?" and "does the same word mean different things on either side?". I start with fewer, larger services.
  • "When do you use a synchronous call, a queue or an event?" — Synchronous only when the caller cannot proceed without the answer and the call is a safe query. A queue when one specific service must act and the caller need not wait. An event when a fact matters to others and the publisher should not know who listens.
  • "How do you make a synchronous call safe?" — A timeout per attempt and a budget for all attempts, smaller than the caller's own timeout; jittered retries on queries only (commands need idempotency keys); a circuit breaker; and a clear failure response instead of guessing.
  • "How do services find each other on AWS?" — On ECS, Service Connect: names configured on the service, a managed proxy that balances load and retries. For other compute, Cloud Map's API or an internal load balancer. Inside the code, the address is just configuration.
  • "How do you change a contract without breaking consumers?" — Expand, then contract: add the new field or version beside the old, migrate consumers, then retire the old. Tolerant readers, versioned event types and consumer-driven contract tests.

Certification practice

Completion checklist

  • I can define a bounded context and name the data each Harbourline context owns
  • I can classify an interaction as query, command or event and choose the matching AWS mechanism
  • I ran the two services locally and saw the 503 after the retry budget
  • I can explain what Service Connect does, what it does not do (authenticate callers) and why running clients must be redeployed to see new endpoints
  • I read both the CDK and Terraform versions and can say what each setting does
  • If I deployed, I destroyed the stack and confirmed the Cloud Map namespace and tagged resources were gone

Key takeaways

  • Cut by business capability, own your data. A shared database turns microservices back into a distributed monolith.
  • Ask, tell, announce. Queries are synchronous, commands go to one queue, events are past-tense facts for anyone who cares. Keep the customer-facing path synchronous only where the answer is needed.
  • Make synchronous calls safe: timeouts that shrink inward, jittered retries in one layer, a circuit breaker, and a clear failure instead of a guess.
  • On ECS, Service Connect is the default for names; Cloud Map is the general registry. Neither authenticates callers, and running clients see new endpoints only after redeployment.
  • Evolve contracts by expanding, then contracting, with versioned events and contract tests.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page