AWS SDK for .NET v4 in depth
Use the AWS SDK for .NET v4 the way production code should: explicit Regions, the credential provider chain, SSO profiles for local work, retries and deadlines, paginators, error handling and the v4 breaking changes that bite during upgrades.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.Core 4.1.x, AWSSDK.S3 4.1.1, AWSSDK.SecurityToken 4.1.0.2) · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain how the SDK for .NET v4 is packaged and what a service client does for every call (endpoint resolution, SigV4 signing, retries, response parsing).
- Describe the credential provider chain and the Region resolution order, and predict which credentials a process will use on your laptop, in a container and on EC2.
- Run a C# console app against your sandbox using the
learn-sandboxIAM Identity Center (SSO) profile, without a single access key on disk. - Configure retries, connect timeouts and per-call deadlines with
CancellationToken, and explain why "retry everything" is dangerous for money-moving calls. - Read paginated results with paginators (
IAsyncEnumerable) and with an explicit loop, and say when to use each. - Handle
AmazonServiceExceptionandAmazonClientExceptioncorrectly and log the request ID. - List the v4 breaking changes that matter most when upgrading a v3 code base: nullable value types, null collections, Standard retry mode, IMDSv2 only and Region-strict S3 clients.
Prerequisites
- Configuring the AWS CLI with SSO profiles — you have a working
learn-sandboxprofile - IAM fundamentals — roles, policies and temporary credentials
- Tools: .NET 10 SDK (or Docker), AWS CLI v2, an editor
Why this matters in banking
Harbourline's reconciliation service started failing at 02:00 one Sunday. Nothing had changed in the code. The cause took six hours to find: a developer had once copied an access key into appsettings.Production.json "just to test", the key was rotated by the security team on Saturday, and the SDK had been quietly picking up that stale key instead of the ECS task role, because explicit configuration beats the default chain.
Three other incidents in the same quarter had the same root cause — engineers did not know how the SDK decides who it is, where it calls and what it does when a call fails:
- An S3 client created with no Region on a build agent defaulted to whatever the agent's profile said, and wrote a statement file to a bucket in the wrong Region — a data-residency question for the EU subsidiary.
- A payment-status poller used the default retry settings plus its own Polly retry loop, so one throttled call became up to twelve HTTP requests, which made the throttling worse.
- A migration from SDK v3 to v4 compiled cleanly and then threw
NullReferenceExceptionin production, because empty lists in responses are nownull.
Regulators will not ask you about SDK internals, but they will ask why a payment was submitted twice, why a customer file landed in another jurisdiction or why a credential was hard-coded. Every one of those answers starts with understanding the SDK.
The idea in plain language
The AWS SDK for .NET is a set of NuGet packages that turn C# method calls into signed HTTPS requests to AWS service endpoints. You never build URLs or signatures yourself. You create a client for a service (for example AmazonS3Client), give it a Region and credentials (or let it find them), and call async methods such as ListObjectsV2Async.
Each call goes through the same pipeline: resolve the endpoint for the Region, sign the request with Signature Version 4 (SigV4) using temporary credentials, send it, retry if the failure is transient, then parse the JSON or XML response into C# objects.
Analogy: a service client is like a branch courier with a sealed pouch. The courier knows the address of head office for each region (endpoint resolution), carries a signed authority letter that expires at the end of the shift (temporary credentials and SigV4), tries again if the door is busy (retries) and hands you back a typed receipt (the response object). Your job is to tell the courier which region and whose authority to use — or to trust the courier to find the right letter in the right drawer (the credential chain).
What problem the SDK solves
Without the SDK you would have to:
- canonicalise every request and compute SigV4 signatures with HMAC-SHA256;
- refresh temporary credentials before they expire;
- know each service's endpoint pattern per Region and partition;
- implement exponential back-off with jitter and decide which errors are retryable;
- parse dozens of response formats.
The SDK does all of that. v4 (generally available since 2025) is a major version with breaking changes. The most important are covered below. It targets .NET Framework 4.7.2, .NET Standard 2.0 and .NET 8 or later, so it runs everywhere from legacy Windows services to .NET 10 containers.
Packaging. AWSSDK.Core holds the runtime: credentials, signing, retries, HTTP. Each service has its own package: AWSSDK.S3, AWSSDK.DynamoDBv2, AWSSDK.SecurityToken, and so on. You reference only the services you use. All service packages in one application should come from the same 4.x line, so they agree on the AWSSDK.Core version.
Honest limitations:
- The SDK cannot fix a wrong permission.
AccessDeniedis a policy problem, not an SDK problem. - The default retry settings are tuned for "most apps". A user-facing banking API usually wants fewer retries and a tighter deadline than the defaults.
- Some high-level helpers live in separate packages and evolve on their own schedule. Examples are the S3 Transfer Utility, the DynamoDB object persistence model and the Secrets Manager caching client.
- Mocking paginators in unit tests is awkward. You will see a workaround in this lesson.
The v4 changes that matter most
| Change in v4 | What breaks | What to do |
|---|---|---|
Value-type properties on requests and responses are nullable (int?, bool?, long?, DateTime?) | if (response.IsTruncated) no longer compiles; arithmetic on obj.Size needs a null check | Compare with == true, use ?? 0 |
Collections in requests and responses default to null | response.Items.Count throws NullReferenceException when nothing came back | Use response.Items ?? [] or is { Count: > 0 }. AWSConfigs.InitializeCollections = true restores v3 behaviour as a stop-gap |
Default RetryMode is Standard, and the Legacy mode is removed | Retry counts and back-off differ from v3 | State your retry settings explicitly per client |
DefaultConfigurationMode defaults to Standard | Timeouts and other defaults differ from v3 | Set ConnectTimeout and per-call deadlines yourself |
| Only IMDSv2 is used on EC2 | Nothing, if your instances require IMDSv2 (they should) | Enforce IMDSv2 on every launch template |
| STS-based credential providers always call the regional STS endpoint | Firewall rules that only allowed sts.amazonaws.com | Allow the regional endpoint, or use a VPC endpoint for STS |
An S3 client configured for us-east-1 can no longer reach buckets in other Regions | Code that relied on the global endpoint redirect | One S3 client per Region, configured for the bucket's Region |
ECSTaskCredentials was removed | Code that constructed it explicitly | Use the default chain; GenericContainerCredentials covers ECS and EKS Pod Identity |
log4net integration moved out; Microsoft.Extensions.Logging integration is an adaptor package | No SDK logs in your ILogger output | Add AWSSDK.Extensions.Logging.ILoggerAdaptor and call ConfigureAWSSDKLogging() |
| Timestamps parsed from responses are UTC | Code that assumed local time | Treat SDK DateTime values as UTC (which you should do in banking anyway) |
SSO credentials no longer open a login flow by default (SupportsGettingNewToken = false) | A desktop tool that relied on the SDK launching a browser | Run aws sso login first, or opt in explicitly for interactive tools |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
Azure SDK client libraries (Azure.Storage.Blobs, Azure.Messaging.ServiceBus) | AWSSDK.S3, AWSSDK.SQS and the other service packages | One package per service on both sides. AWS method names mirror the HTTP API operation names (PutObject, ListObjectsV2) |
DefaultAzureCredential | The SDK's default credential provider chain | Both try environment, developer login and managed identity in turn. AWS has no "Visual Studio credential"; you use a CLI SSO profile |
| Managed identity | IAM role for the EC2 instance, ECS task or Lambda function | The SDK fetches and refreshes the role's temporary credentials automatically |
RetryOptions on ClientOptions | RetryMode and MaxErrorRetry on the client config | AWS Standard mode adds a client-side retry quota (token bucket) so retries cannot snowball |
Pageable<T> / AsyncPageable<T> | client.Paginators.<Operation>(request) returning IAsyncEnumerable | Paginators are single-use. Create a new one each time |
RequestFailedException (Status, ErrorCode) | AmazonServiceException (StatusCode, ErrorCode, RequestId) plus service-specific subclasses | Always log the RequestId: AWS Support asks for it |
CancellationToken on every async method | The same | Use it as your per-call deadline |
| Azure Resource Manager endpoints per cloud | Endpoint resolution per Region and partition (aws, aws-cn, aws-us-gov) | The Region decides the endpoint; you rarely set ServiceURL except for local emulators |
Architecture
The diagram shows what happens inside one SDK call.
And this is how the default credential chain decides who you are. The first source that produces credentials wins.
Security-boundary view. Your code never sees long-term secrets in a well-built system. On a laptop, temporary credentials come from the SSO cache that aws sso login fills. In AWS, they come from the platform's metadata or container endpoint for the role attached to the compute. Explicit keys in configuration (the top of the chain) override the role. That is exactly why they are dangerous: one forgotten key silently replaces a well-scoped role.
Region resolution follows a similar order when you do not set RegionEndpoint on the client:
- Application configuration files (
app.config/web.configon .NET Framework). AWS_REGION, thenAWS_DEFAULT_REGION.- The
regionin the profile (fromAWSConfigs.AWSProfileName,AWS_PROFILEordefault). - EC2 instance metadata, on EC2.
If nothing is found, the client throws when it is created. Inside ECS and Lambda, AWS sets AWS_REGION for you.
Step-by-step implementation
Sign in with your SSO profile
aws sso login --profile learn-sandbox
aws sts get-caller-identity --profile learn-sandboxThe second command should print your sandbox account ID and an ARN containing AWSReservedSSO_. This is your proof that the profile works before any C# is involved.
Create the console project
dotnet new console -n Harbourline.SdkTour -f net10.0
cd Harbourline.SdkTour
dotnet add package AWSSDK.S3 --version 4.1.1
dotnet add package AWSSDK.SecurityToken --version 4.1.0.2
dotnet add package AWSSDK.SSO --version 4.1.0.2
dotnet add package AWSSDK.SSOOIDC --version 4.1.0.2AWSSDK.SSO and AWSSDK.SSOOIDC are not called directly by your code. The core runtime loads them when it meets an SSO profile. Without them you get a confusing "assembly not found" style error the first time you run with learn-sandbox.
Point the process at the profile, not the code
Set AWS_PROFILE in the environment rather than hard-coding a profile name. In Properties/launchSettings.json that looks like this:
{
"profiles": {
"SdkTour": {
"commandName": "Project",
"environmentVariables": {
"AWS_PROFILE": "learn-sandbox",
"AWS_REGION": "us-east-1"
}
}
}
}From a terminal, use $env:AWS_PROFILE = "learn-sandbox" in PowerShell or export AWS_PROFILE=learn-sandbox in bash. The same binary then uses the ECS task role in AWS with no code change, because there will be no profile there.
Replace Program.cs and run
Paste the complete code from the next sections, then run:
dotnet run
dotnet run -- --bucket hl-m03-statements-yourname # optional, if you created a bucket earlierExpected output (your values differ):
Account : 111122223333
Caller : arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_SandboxDeveloper_0123456789abcdef/alex
Region : us-east-1
bucket hl-m00-budget-reports (created 2026-11-02)
1 bucket(s) in us-east-1.Break it on purpose
Learn the failure modes while they are cheap:
- Run
aws sso logout, thendotnet run. You get anAmazonClientExceptionabout the SSO token. The code prints the hint to log in again. - Set
$env:AWS_REGION = "eu-west-2"and run again. The bucket list now shows only buckets whose Region iseu-west-2, because the request filters onBucketRegion. - Run with
--bucketand a bucket name that is not yours. You getAccessDenied(HTTP 403) with a request ID. That is a service error, not a client error.
AWS Console instructions
The SDK has no console of its own, but two console views help you check what your code did:
- Console → CloudTrail → Event history. Filter by Event name =
ListBuckets. Open the event and look atuserIdentity.arn(your SSO role session) anduserAgent, which names the .NET SDK and its version ⚠ VERIFY (exact user-agent format). This is how auditors link an action to a person and a tool. - Console → IAM Identity Center → AWS accounts → (your sandbox) → Permission sets. Check which permission set backs
learn-sandbox. If a call fails withAccessDenied, this is where the missing permission lives.
AWS CLI
# 1. Log in (opens a browser once; caches a token under ~/.aws/sso/cache).
aws sso login --profile learn-sandbox
# 2. See which credentials and Region the CLI (and, by the same rules, the SDK) will use, and where each came from.
aws configure list --profile learn-sandbox
# 3. Who am I? The same call the C# code makes first.
aws sts get-caller-identity --profile learn-sandbox
# 4. List buckets in one Region (the same filter the C# code uses).
aws s3api list-buckets --bucket-region us-east-1 --max-buckets 100 --profile learn-sandbox
# 5. Short-lived credentials as environment variables, for a tool that cannot read SSO profiles.
# They expire with the SSO role session. Never paste them into config files.
aws configure export-credentials --profile learn-sandbox --format powershellInfrastructure as Code
The console app needs very little permission. In a real team the permission set (or, in AWS, the task role) would be defined in code. Here is a least-privilege customer managed policy for this lesson, scoped to the statements bucket prefix used later in the module.
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Constructs;
namespace Harbourline.Learn.Infra;
public sealed class SdkTourPolicyStack : Stack
{
public SdkTourPolicyStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
const string bucket = "hl-m03-statements-uk";
var policy = new ManagedPolicy(this, "SdkTourPolicy", new ManagedPolicyProps
{
ManagedPolicyName = "hl-m03-sdk-tour",
Statements =
[
// ListBuckets works only on "*": it is an account-level action.
new PolicyStatement(new PolicyStatementProps
{
Actions = ["s3:ListAllMyBuckets"],
Resources = ["*"],
}),
// Listing keys is limited to the statements/ prefix of one bucket.
new PolicyStatement(new PolicyStatementProps
{
Actions = ["s3:ListBucket"],
Resources = [$"arn:aws:s3:::{bucket}"],
Conditions = new Dictionary<string, object>
{
["StringLike"] = new Dictionary<string, object> { ["s3:prefix"] = new[] { "statements/*" } },
},
}),
],
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m03-sdk");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
_ = new CfnOutput(this, "PolicyArn", new CfnOutputProps { Value = policy.ManagedPolicyArn });
}
}sts:GetCallerIdentity needs no permission: it always works for any valid credentials, which is why it is the standard "who am I?" check.
Complete code
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.SdkTour</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.S3" Version="4.1.1" />
<PackageReference Include="AWSSDK.SecurityToken" Version="4.1.0.2" />
<!-- Required at run time to resolve an IAM Identity Center (SSO) profile. -->
<PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
</ItemGroup>
</Project>using Amazon;
using Amazon.Runtime;
using Amazon.Runtime.CredentialManagement;
using Amazon.S3;
using Amazon.S3.Model;
using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;
// Usage:
// dotnet run -> default credential chain (set AWS_PROFILE=learn-sandbox first)
// dotnet run -- --profile NAME -> explicit named profile (local development only)
// dotnet run -- --bucket NAME -> also list up to 50 object keys under statements/
string? profileName = ArgValue(args, "--profile");
string? bucketName = ArgValue(args, "--bucket");
// 1. Region: explicit beats implicit. Fall back to us-east-1 for this lab.
var region = RegionEndpoint.GetBySystemName(Environment.GetEnvironmentVariable("AWS_REGION") ?? "us-east-1");
// 2. Credentials: null means "let the SDK walk the default chain".
AWSCredentials? credentials = null;
if (profileName is not null)
{
var chain = new CredentialProfileStoreChain();
if (!chain.TryGetAWSCredentials(profileName, out credentials))
{
Console.Error.WriteLine($"Profile '{profileName}' was not found in ~/.aws/config or ~/.aws/credentials.");
return 2;
}
}
// 3. One config object per client: Region, retry behaviour and a connect timeout.
var stsConfig = new AmazonSecurityTokenServiceConfig
{
RegionEndpoint = region,
RetryMode = RequestRetryMode.Standard, // the v4 default, stated for readers
MaxErrorRetry = 3, // first attempt + up to 3 retries
ConnectTimeout = TimeSpan.FromSeconds(3),
};
var s3Config = new AmazonS3Config
{
RegionEndpoint = region,
RetryMode = RequestRetryMode.Standard,
MaxErrorRetry = 3,
ConnectTimeout = TimeSpan.FromSeconds(3),
};
// 4. Clients are thread-safe and expensive to build: create once, reuse, dispose at the end.
using var sts = credentials is null
? new AmazonSecurityTokenServiceClient(stsConfig)
: new AmazonSecurityTokenServiceClient(credentials, stsConfig);
using var s3 = credentials is null
? new AmazonS3Client(s3Config)
: new AmazonS3Client(credentials, s3Config);
try
{
// 5. Every async call takes a CancellationToken: this is your per-call deadline.
using var callTimeout = new CancellationTokenSource(TimeSpan.FromSeconds(10));
var me = await sts.GetCallerIdentityAsync(new GetCallerIdentityRequest(), callTimeout.Token);
Console.WriteLine($"Account : {me.Account}");
Console.WriteLine($"Caller : {me.Arn}");
Console.WriteLine($"Region : {region.SystemName}");
// 6. Paginator: the SDK follows ContinuationToken for you. Never reuse a paginator object.
using var listTimeout = new CancellationTokenSource(TimeSpan.FromSeconds(30));
var buckets = s3.Paginators.ListBuckets(new ListBucketsRequest { BucketRegion = region.SystemName, MaxBuckets = 100 });
var count = 0;
await foreach (var bucket in buckets.Buckets.WithCancellation(listTimeout.Token))
{
Console.WriteLine($" bucket {bucket.BucketName} (created {bucket.CreationDate:yyyy-MM-dd})");
count++;
}
Console.WriteLine($"{count} bucket(s) in {region.SystemName}.");
if (bucketName is not null)
{
var request = new ListObjectsV2Request { BucketName = bucketName, Prefix = "statements/", MaxKeys = 50 };
var page = await s3.ListObjectsV2Async(request, listTimeout.Token);
// SDK v4: S3Objects is null (not empty) when nothing matches.
foreach (var obj in page.S3Objects ?? [])
{
Console.WriteLine($" {obj.Key,-60} {obj.Size,10:N0} bytes");
}
Console.WriteLine(page.IsTruncated == true ? " ...more keys exist (use a paginator to read them all)" : " (end of list)");
}
return 0;
}
catch (AmazonServiceException ex)
{
// The service answered with an error: AccessDenied, NoSuchBucket, ThrottlingException...
Console.Error.WriteLine($"AWS error {ex.ErrorCode} (HTTP {(int)ex.StatusCode}), request id {ex.RequestId}: {ex.Message}");
return 1;
}
catch (AmazonClientException ex)
{
// Nothing reached AWS: no credentials, expired SSO token, no Region, DNS/network failure.
Console.Error.WriteLine($"SDK client error: {ex.Message}");
Console.Error.WriteLine("If you use SSO, run: aws sso login --profile learn-sandbox");
return 1;
}
catch (OperationCanceledException)
{
Console.Error.WriteLine("The call did not finish before its deadline.");
return 1;
}
static string? ArgValue(string[] args, string name)
{
var i = Array.IndexOf(args, name);
return i >= 0 && i + 1 < args.Length ? args[i + 1] : null;
}This code was compiled against the package versions above with TreatWarningsAsErrors on.
Code explained line by line
ArgValue(args, "--profile")reads optional switches. Production code would useSystem.CommandLineor the generic host. Plain parsing keeps the SDK in focus here.RegionEndpoint.GetBySystemName(...)turns"us-east-1"into the SDK's Region object. Setting the Region explicitly on every client is the habit that prevents "it wrote to the wrong Region" incidents. Reading it fromAWS_REGIONkeeps it configurable without code changes.AWSCredentials? credentials = null. Anullhere means "do not pass credentials". The client then walks the default chain, which is what you want in AWS.CredentialProfileStoreChain.TryGetAWSCredentialslooks up a named profile in~/.aws/configand~/.aws/credentials. For an SSO profile it returns credentials that read the SSO token cache and exchange it for role credentials. Use this only for local tools that must switch profiles at run time. Services should rely on the chain.AmazonSecurityTokenServiceConfig/AmazonS3Configare per-service subclasses ofClientConfig. Configuration is fixed when the client is created and cannot change afterwards.RetryMode = RequestRetryMode.Standardis already the v4 default. Writing it out documents intent for reviewers. Standard mode retries throttling errors, 5xx errors and some network errors with exponential back-off and jitter. It also keeps a retry quota: when many calls are failing, the client stops retrying so it does not pile more load on a struggling service.MaxErrorRetry = 3means one attempt plus up to three retries. The documented Standard-mode default is 2 retries for most services and 10 for DynamoDB. Choose a number that fits your latency budget, not "as many as possible".ConnectTimeoutbounds the TCP and TLS connect phase. A dead endpoint then fails in 3 seconds instead of waiting for the operating system's default.using var sts = .... Clients implementIDisposableand own anHttpClientconnection pool. In a console app, create them once and dispose at exit. In ASP.NET Core you register them as singletons instead (next lesson).new CancellationTokenSource(TimeSpan.FromSeconds(10))is the per-call deadline. The SDK's guide recommends theCancellationTokenfor timeouts on async calls. It covers the whole operation, including retries. When it fires you getOperationCanceledException(or its subclassTaskCanceledException).GetCallerIdentityAsyncreturns the account, user ID and ARN of whoever the credentials belong to. It needs no IAM permission.s3.Paginators.ListBuckets(...)returns a paginator..Bucketsis the "key result": anIAsyncEnumerableofS3Bucket. The paginator fetches the next page withContinuationTokenonly when you iterate past the current one..Responsesgives you whole response objects instead. A paginator object can be enumerated only once.BucketRegion = region.SystemNameasks S3 to return only buckets in that Region.MaxBucketssets the page size. Both arrived whenListBucketsbecame paginated.WithCancellation(listTimeout.Token)passes the deadline into the async enumeration.page.S3Objects ?? []is the v4 null-collection rule in one line. In v3 this was an empty list; in v4 it isnullwhen nothing matched.page.IsTruncated == true.IsTruncatedisbool?in v4. Comparing withtruetreats "unknown" as "no".catch (AmazonServiceException ex). The request reached AWS and AWS said no.ErrorCodeis the machine-readable code (AccessDenied,NoSuchBucket).StatusCodeis the HTTP status.RequestIdis what AWS Support needs to trace the call. Service packages add subclasses such asAmazonS3ExceptionandNoSuchBucketExceptionthat you can catch more specifically.catch (AmazonClientException ex). The SDK could not even make the call: no credentials, an expired SSO token, no Region or a network failure. Treat these as configuration or infrastructure problems, not business errors.catch (OperationCanceledException). Your deadline fired. In an API this becomes a 503 or 504 with a ProblemDetails body. A long-running worker logs it and moves on.
Retries and money: a warning
Retries are safe for reads and for writes that are idempotent. The SDK retries PutObject because writing the same object twice gives the same result. It also retries a DynamoDB PutItem with a condition, because the condition makes a duplicate harmless.
Your own business operation may not be idempotent. If your code calls a partner's payment API after an SQS message arrives, and you wrap the whole handler in another retry loop, you can submit the payment twice. Rules for Harbourline:
- Do not stack your own retry loop (for example Polly) on top of SDK retries for the same AWS call. Pick one layer: usually the SDK for AWS calls, and Polly for non-AWS HTTP calls.
- Make every money-moving write idempotent with a client-supplied key (
Idempotency-Key, DynamoDB conditional writes). See Module 4. - Use Adaptive retry mode only when you understand it. It adds client-side rate limiting after throttling, which suits batch jobs hammering one table more than user-facing APIs.
Paginator or explicit loop?
| Use a paginator when… | Use an explicit loop when… |
|---|---|
| You want every item and the code is not unit-tested against a mock | You want to unit-test with NSubstitute or Moq. Mocking ListObjectsV2Async is easy; mocking IS3PaginatorFactory and its async enumerables is not |
| The volume is bounded (dozens of pages at most) | You need to stop early, return one page to an API caller with a nextToken, or checkpoint progress |
| Readability matters most | You need to control the page size per request |
The lab code in labs/m03-accounts-api/src/Harbourline.Accounts.Api/Data/DynamoDbAccountRepository.cs uses the explicit loop for exactly the testing reason.
SDK logging without leaking data
To see SDK activity in your normal ILogger output, add the AWSSDK.Extensions.Logging.ILoggerAdaptor package and call ConfigureAWSSDKLogging() on the ILoggerFactory at startup. Keep the AWSSDK category at Warning in production. Never turn on response logging (LogResponse / AWSConfigs.LoggingConfig.LogResponses) in an environment that handles customer data. Responses contain account items, statement keys and sometimes secrets.
Security considerations
⚠ Learning shortcut — not for production
The console app accepts --profile and reads profiles from your user folder. That is fine on a developer laptop. Services in AWS must never ship a profile name or a credentials file: they get credentials from the role attached to the EC2 instance, ECS task or Lambda function. Keep AWS:Profile only in appsettings.Development.json (next lesson), or use the AWS_PROFILE environment variable locally.
- No long-term access keys. Harbourline's rule is that IAM users with access keys do not exist outside break-glass procedures. Locally you use SSO; in AWS you use roles; in CI you use OIDC federation (Module 9).
- Explicit credentials override roles. An
AWS_ACCESS_KEY_IDenvironment variable left in a container definition beats the task role. Scan task definitions and Beanstalk environment properties for it. - Enforce IMDSv2. The v4 SDK only uses IMDSv2. Requiring IMDSv2 on instances (hop limit 1 unless containers need more) also blocks the server-side request forgery pattern that stole credentials through IMDSv1.
- Pin the Region for data residency. For the EU subsidiary, the client's Region is part of your data-residency control. Do not leave it to whatever profile a build agent has.
- Log request IDs, not payloads. Request IDs are safe to log and essential for investigations. Payloads may contain personal data under UK GDPR.
- Keep packages patched. The SDK is a security-relevant dependency. Let Dependabot or Renovate raise 4.x patch updates weekly.
Banking use case
Harbourline's Statements Service produces monthly PDF statements for 1.2 million fictional current accounts. A nightly .NET 10 worker on ECS Fargate:
- Lists the statement objects already generated under
statements/<account-id>/with an explicit pagination loop, so it can checkpoint after each page. - Writes new PDFs with
PutObjectAsyncand server-side encryption with a KMS key. The SDK retries these safely because each object key is deterministic. - Uses the task role, which allows only
s3:PutObjectands3:ListBucketon the statements bucket andkms:GenerateDataKeyon one key. - Sets
MaxErrorRetry = 3and a 15-second per-call deadline. A slow S3 call fails fast, the worker records the account in a retry queue and carries on, and the batch finishes inside its window. - Logs one structured line per page with the S3 request ID and the number of objects, never the customer's name.
The EU subsidiary's run uses a second client pinned to eu-west-1 and a different bucket. Whether that arrangement meets the subsidiary's data-residency obligations is a question for Harbourline's legal and compliance teams. The SDK settings only enforce the decision they make.
Common mistakes
- Creating a new client per request → it happens because clients look cheap → each one builds its own
HttpClient, so you exhaust sockets and repeat credential lookups → create once and reuse (singleton). - Leaving out the Region → works on your laptop because your profile has a Region → fails or writes elsewhere on a build agent → set
RegionEndpointorAWS:Regionexplicitly. .Resultor.Wait()on SDK calls → copied from old samples → thread-pool starvation under load in ASP.NET Core →awaiteverywhere.- Treating
nullas "the API returned nothing" after upgrading to v4 →NullReferenceException→?? []and== truepatterns, or turn onAWSConfigs.InitializeCollectionstemporarily while you fix the code. - Reusing a paginator → the second enumeration yields nothing or throws → create a new paginator each time.
- Catching
Exceptionand retrying everything → you retryAccessDeniedand validation errors forever → catchAmazonServiceException, checkErrorCode, and retry only transient errors (the SDK already did that for you). - Missing
AWSSDK.SSO/AWSSDK.SSOOIDC→ works for colleagues with old access keys, fails for you with SSO → add both packages to any app you run locally with SSO. - Using an S3 client in
us-east-1for a bucket ineu-west-2→ v3 used to follow the redirect; v4 does not → one client per Region.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
AmazonClientException: Failed to resolve AWS credentials listing several providers | No profile, environment variables or role reachable | Set AWS_PROFILE locally; attach a role in AWS. Read each provider's message in the exception: it tells you what was tried |
| Error mentioning the SSO token being expired or missing | The SSO session ended (typically after 8–12 hours, depending on your organisation's settings) | aws sso login --profile learn-sandbox |
Could not load file or assembly 'AWSSDK.SSOOIDC' (or similar) with an SSO profile | SSO packages not referenced | Add AWSSDK.SSO and AWSSDK.SSOOIDC |
No RegionEndpoint or ServiceURL configured | No Region anywhere in the resolution order | Set RegionEndpoint, AWS_REGION or the profile's region |
AccessDenied (403) with a request ID | IAM policy, bucket policy, SCP or KMS key policy denies the call | Check the caller ARN with get-caller-identity, then the policies. CloudTrail shows the denied event |
PermanentRedirect or AuthorizationHeaderMalformed from S3 | Client Region differs from the bucket's Region | Create the client for the bucket's Region |
NullReferenceException after upgrading to v4 | A response collection is now null | Null-coalesce; see the v4 changes table |
TaskCanceledException after exactly your deadline | Your CancellationToken fired: network path, throttling or a slow service | Check retries in SDK logs (AWSSDK category at Debug locally); check NAT/VPC endpoint routing in AWS |
ThrottlingException / SlowDown surfacing to callers | Sustained request rate above the service limit; retries exhausted | Reduce concurrency, add jittered back-off in the caller, request a quota increase or redesign the access pattern |
SignatureDoesNotMatch or clock-skew errors | Machine clock is wrong | Fix NTP; the SDK corrects small skews automatically |
Cost and cleanup
This lesson is free:
sts:GetCallerIdentityhas no charge.- S3
ListBucketsandListObjectsV2are LIST requests, billed at a fraction of a cent per thousand requests inus-east-1. A few runs cost effectively nothing. - No resources are created, unless you deploy the optional IaC policy, and IAM policies are free.
Cleanup:
# If you deployed the optional policy with Terraform:
terraform destroy
# or with CDK:
cdk destroy SdkTourPolicyStack --profile learn-sandbox
# End your SSO session when you finish for the day.
aws sso logout
# Confirm nothing tagged for this lesson remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m03-sdk \
--region us-east-1 --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"IAM policies are global and do not appear in the Regional tagging API. Check IAM → Policies → Customer managed for hl-m03-sdk-tour if you created it.
Exercises
- 🟢 Guided: Run the console app with
AWS_PROFILE=learn-sandbox, then with--profile learn-sandbox, then with neither (expect the credential error). For each run, write down which link in the credential chain was used. - 🟡 Challenge: Change the bucket listing to use
.Responsesinstead of.Buckets, and print the page number, the number of buckets per page and theContinuationTokenfor each page. SetMaxBuckets = 1to force several pages. - 🔴 Live problem: A colleague reports: "After we moved the reconciliation job to v4, it started failing only in the EU subsidiary account, with
PermanentRedirectonGetObject. The UK account is fine." The job creates oneAmazonS3Clientwith no Region, and the EU bucket is ineu-west-1. Diagnose and propose a fix that also makes the Region an auditable setting.
Assignment
Add a docs/sdk-conventions.md page to your portfolio repository that a new Harbourline developer could follow. It should cover: how to authenticate locally (SSO profile, AWS_PROFILE), the client lifetime rule, the default retry and deadline values for user-facing and batch code, how to handle AmazonServiceException, the paginator-versus-loop rule, and the v4 null rules. Include the console app as a working sample.
Solutions
| Run | Chain link used | Evidence |
|---|---|---|
AWS_PROFILE=learn-sandbox | Profile named by AWS_PROFILE (an SSO profile, read from the SSO token cache) | ARN shows AWSReservedSSO_... |
--profile learn-sandbox | None: credentials passed explicitly to the client constructor | Same ARN; the chain was skipped |
| Neither | Nothing found (no environment variables, no default profile, no container endpoint or IMDS on a laptop) | AmazonClientException: Failed to resolve AWS credentials |
Why: the same binary behaves differently only because of its environment. That is the property you want: in ECS the "neither" case finds the container credentials endpoint and uses the task role.
var pages = s3.Paginators.ListBuckets(new ListBucketsRequest { BucketRegion = region.SystemName, MaxBuckets = 1 });
var pageNumber = 0;
await foreach (var response in pages.Responses.WithCancellation(listTimeout.Token))
{
pageNumber++;
var names = (response.Buckets ?? []).Select(b => b.BucketName);
Console.WriteLine($"page {pageNumber}: {string.Join(", ", names)}; next token: {response.ContinuationToken ?? "(none)"}");
}Why: .Responses exposes each whole ListBucketsResponse, including paging metadata. Note the ?? []: a page can legitimately contain no buckets. A new paginator object is created because the earlier one was already enumerated.
- Diagnosis: with no Region on the client, the SDK resolved one from the environment, almost certainly
us-east-1from the job's profile orAWS_REGION. In v3, an S3 client inus-east-1could follow redirects to buckets in other Regions. In v4 that behaviour was removed, soGetObjectagainst theeu-west-1bucket fails withPermanentRedirect. The UK account works because its bucket happens to be in the Region the client resolved. - Fix: create one client per Region, each configured explicitly. With DI that is
AddKeyedAWSService<IAmazonS3>("eu", configuration.GetAWSOptions("AwsEu"))(next lesson). The Region lives in configuration (AwsEu:Region = eu-west-1) that is versioned and reviewed. - Auditability: log the bucket and client Region at startup. Add a startup check that calls
GetBucketLocation(orHeadBucket) and fails fast if the bucket's Region differs from the configured Region. - Regression test: a unit test that resolves the keyed client and asserts
Config.RegionEndpoint == RegionEndpoint.EUWest1.
Knowledge check
Answer: The environment variables. They come before the container credentials provider in the default chain. That is why stray keys are dangerous: they silently replace a well-scoped role.
Answer: null, not an empty list. Use response.Items ?? [] or pattern matching such as is { Count: > 0 }. AWSConfigs.InitializeCollections = true restores the v3 behaviour globally as a temporary migration aid.
Answer: The token bounds the total time of the operation, including all retries and back-off. Retry counts bound attempts, not time. A user-facing API needs a time budget it can promise to its own callers.
Answer: A paginator keeps its own paging state and can be enumerated only once. Re-enumerating it does not start again from the first page.
Answer: AmazonClientException (for example: no credentials, no Region, an expired SSO token or a network failure). AmazonServiceException means AWS received the request and returned an error, with an ErrorCode, StatusCode and RequestId.
Interview questions
- "How does your .NET service authenticate to AWS?" — It has no keys. The SDK's default credential chain finds the IAM role attached to the compute: the container credentials endpoint on ECS, IMDSv2 on EC2, environment variables injected by Lambda. Locally, developers use an IAM Identity Center profile through
AWS_PROFILE. Explicit keys are forbidden because they override roles. - "What changed in SDK v4 that you had to handle?" — Nullable value types and null collections (we added null-coalescing and fixed tests), Standard retry mode by default, Region-strict S3 clients, IMDSv2 only, and the move of logging integrations into adaptor packages.
- "How do you configure retries for a payments API?" — Few SDK retries (two or three) with a tight per-call deadline via
CancellationToken. No second retry layer around AWS calls. Idempotency keys or conditional writes for anything that moves money. - "A call fails intermittently with throttling. What do you do?" — Confirm with CloudWatch metrics and SDK logs. Check for retry amplification (several retry layers). Reduce concurrency or batch requests. Consider Adaptive retry mode for batch jobs. Request a quota increase only after the access pattern is sane.
Certification practice
A developer's .NET application on an EC2 instance must read objects from one S3 bucket. The current code loads an access key ID and secret access key from appsettings.json. The security team requires removal of long-term credentials with the fewest code changes. What should the developer do?
A. Store the keys in AWS Secrets Manager and read them at startup.
B. Attach an IAM role with read access to the bucket to the instance through an instance profile, and remove the keys from configuration so the SDK uses the default credential chain.
C. Encrypt appsettings.json with AWS KMS.
D. Create a new IAM user per instance and rotate its keys every 90 days.
Answer: B. The default chain falls through to instance metadata and uses the role's temporary credentials. Removing the keys is essential, because explicit or environment credentials take precedence. A and C still use long-term keys, only stored differently. D multiplies long-term keys.
A .NET application must process every object under a prefix that contains about 30,000 objects. The current code calls ListObjectsV2Async once and processes only 1,000 objects. Which change processes all objects with the least custom code?
A. Set MaxKeys = 30000 on the request.
B. Use s3Client.Paginators.ListObjectsV2(request) and iterate its S3Objects with await foreach.
C. Call ListObjectsAsync (version 1) instead.
D. Increase the client's Timeout.
Answer: B. The paginator follows continuation tokens automatically. S3 returns at most 1,000 keys per call, whatever MaxKeys says (A). Version 1 of the API has the same page limit (C). Timeouts are unrelated to paging (D).
A .NET service on ECS calls DynamoDB. Under peak load, some requests fail with ProvisionedThroughputExceededException and p99 latency spikes. The code wraps every SDK call in a Polly policy that retries five times. What is the BEST first step?
A. Increase the SDK's MaxErrorRetry to 20.
B. Remove the extra Polly retry layer around the SDK calls and rely on the SDK's Standard retry mode with a per-call deadline, then review the table's capacity mode.
C. Create a new AmazonDynamoDBClient for each request.
D. Switch to synchronous SDK calls.
Answer: B. Two retry layers multiply attempts and amplify throttling. Standard mode already retries throttling errors with jittered back-off and a retry quota. Capacity (on-demand, or more provisioned capacity) addresses the root cause. A makes amplification worse. C wastes connections. D blocks threads.
Completion checklist
- I ran the console app with my SSO profile and saw my account and role ARN
- I can draw the credential chain from memory and say which link wins in ECS, on EC2 and on my laptop
- I set retries, a connect timeout and a per-call deadline, and can justify the values
- I used a paginator and an explicit loop, and know when each is appropriate
- I know the five v4 changes most likely to break an upgrade
- I logged out of SSO and removed any optional policy I created
Key takeaways
- A service client handles endpoints, SigV4 signing, retries and parsing. Create it once, configure it explicitly (Region, retries, connect timeout) and reuse it.
- The default credential chain lets the same binary use SSO locally and roles in AWS. Anything explicit (constructor or environment keys) overrides the role.
- Use a
CancellationTokenas your deadline. Keep SDK retries modest and never stack a second retry layer on AWS calls that move money. - Paginators are convenient and single-use. Explicit loops are easier to test and to stop early.
- v4 returns
nullcollections and nullable values. Most upgrade bugs come from that.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Module 3: .NET Application Development on AWS
Build, containerise, configure and deploy ASP.NET Core applications on AWS the way a bank would run them. You compare every hosting option (EC2, Elastic Beanstalk, ECS on Fargate, Lambda and more) and finish by deploying the same Web API three different ways.
AWS clients and dependency injection
Register AWS SDK v4 clients in ASP.NET Core with AWSSDK.Extensions.NETCore.Setup: default options from configuration, per-client tuning, keyed clients for a second Region, correct lifetimes, per-environment settings, and tests that run without an AWS account.