Playbook: AccessDenied and IAM policy evaluation
Turn an AccessDenied error into a located cause in minutes: read the message, confirm who is really calling, find the denying layer with CloudTrail, the policy simulator and Access Analyzer, and fix it with the smallest safe change.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · IAM · AWS CloudTrail · IAM Access Analyzer · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Read an
AccessDeniedmessage and name the layer that caused it (identity policy, resource policy, permissions boundary, SCP, RCP, session policy, trust policy or VPC endpoint policy). - Confirm which principal is really calling, including the common case where the workload is not using the role you think it is.
- Find the exact denied call in CloudTrail and extract the facts you need from it.
- Use the IAM policy simulator and IAM Access Analyzer correctly, and say what they cannot tell you.
- Make the smallest safe fix, deliver it through code review and IaC, and write the audit record.
- Avoid the three bad fixes: adding
*, asking for an exemption from a guardrail, and "temporarily" attaching administrator access.
Prerequisites
- IAM fundamentals and IAM in depth
- Policy evaluation logic, the full theory behind this playbook (the error-phrase table lives there)
- CloudTrail for operations and Least privilege and Access Analyzer
- Tools: AWS CLI v2 with the
learn-sandboxprofile; .NET 10 SDK for the triage tool
Why this matters in banking
It is change night. The new statements service is deployed to production at 22:15 and the first run fails: AccessDeniedException ... not authorized to perform: dynamodb:GetItem. The release manager asks "can you just give it admin for tonight?" Three bad things can follow from saying yes: a broad permission that never gets removed, an audit finding for breaking least privilege and segregation of duties, and a real security exposure if the service has a vulnerability. The senior move is to find the denying layer in ten minutes and add the one action that is actually missing, so the release proceeds and the audit trail stays clean.
Many AccessDenied errors are not really permission problems. They are identity problems (the workload is a different principal than you assumed), resource policy problems (the other side said no), or guardrail problems (an organisation control did exactly what it was designed to do). Fixing the wrong layer is how engineers end up with a role that has * and still does not work.
The idea in plain language
AWS decides each request by asking in a fixed order: is there an explicit deny anywhere? Do all the caps allow it? Is there an explicit allow from an identity policy or a resource policy? If nothing allows it, the answer is no. Your job during an incident is to find out which question gave the wrong answer.
Analogy: You are turned away at a bank's vault door. The guard's note might say "your badge is not on the list" (no identity allow), "the vault owner has not listed your department" (no resource allow), "head office has locked this door for everyone today" (SCP deny), or "you are using a visitor badge instead of your own" (wrong principal). The note usually tells you which. Reading it carefully is half the work.
The five facts every AccessDenied gives you (or lets you find)
| Fact | Where it is | Why you need it |
|---|---|---|
| Principal | User: arn:aws:sts::...:assumed-role/ROLE/SESSION in the message | Confirms which role actually called, not which role you meant |
| Action | is not authorized to perform: dynamodb:GetItem | The exact IAM action, which is not always the API name you called |
| Resource | on resource: arn:aws:... | The exact ARN, including its account and Region |
| Reason | because no identity-based policy allows... or with an explicit deny in a service control policy | The layer. Not every service includes it, so CloudTrail is your backup |
| Time and request ID | The log line, the SDK exception's RequestId | Lets you find the call in CloudTrail |
What problem the AWS services solve
- The enhanced error message names the layer for many services. When it does, you can skip straight to the right policy.
- AWS CloudTrail records the call with the caller, action, resource, source IP and the error. Management events are in Event history for 90 days at no cost. Data events (such as S3 object-level and DynamoDB item-level calls) are recorded only if you enabled them on a trail (⚠ VERIFY which data-plane services you have enabled).
- The IAM policy simulator (
simulate-principal-policy) evaluates identity policies and reports decisions for the permissions boundary and Organizations policies as separate fields. It does not see every real-world factor (network conditions, session policy of a live session, some resource-policy interactions), so a simulator "allowed" is a hint, not proof. - IAM Access Analyzer validates policies, finds unintended external access and can generate a least-privilege policy from CloudTrail activity. Unused-access analysis is a paid feature (⚠ VERIFY current pricing).
- IAM last accessed information shows which services and actions a role actually used, which you need for right-sizing afterwards.
Honest limits: IAM is eventually consistent, so a fix can take seconds to apply; some services return a bare Access Denied with no reason; and CloudTrail events arrive with a delay, usually minutes (⚠ VERIFY current delivery time), so the denied call may not be searchable the second after it happens.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
403 AuthorizationFailed with the role assignment scope in the message | AccessDenied message naming action, resource and often the denying policy type | AWS evaluates several policy types together, and an explicit deny in any of them wins |
| Azure Activity Log filtered on failed operations | CloudTrail event history with errorCode | Event history shows management events for 90 days; data events need a trail |
| "What if" tool for role assignments, or the Entra access reviews | IAM policy simulator and Access Analyzer | The simulator answers "would this principal be allowed this action on this resource" for given context |
| Managed identity token fetched from IMDS | Role session credentials from the ECS task role endpoint or Lambda environment | The SDK credential chain can pick a different source first (environment variables, profile), a classic cause of "wrong identity" |
| Azure Policy deny effect | SCP or RCP explicit deny | Not visible in the identity's own policies. Look in Organizations |
| Key Vault access policy denial | KMS key policy | A key's own policy can deny even when IAM allows. See the KMS playbook |
| Conditional Access block | IAM condition keys (aws:SourceVpc, aws:MultiFactorAuthPresent, aws:RequestedRegion) | Conditions sit inside policy statements, so the Allow can exist and still not match |
Architecture
The ten-minute decision flow
What can say no
A deny at any step stops the request, and no step except an explicit Allow can grant. Caps (SCP, RCP, boundary, session policy, endpoint policy) only remove permissions.
Step-by-step implementation
Minutes 0 to 3: capture the evidence before changing anything
Copy the full error text, the timestamp and the request ID (the SDK exception has RequestId; the .NET AmazonServiceException also exposes ErrorCode and StatusCode). Note the environment (production, account ID, Region) and what changed recently: a deployment, a new role, a policy edit, an SCP change.
Minutes 3 to 8: confirm the caller really is who you think
From inside the failing workload (ECS Exec, Lambda test event, or the code itself) run aws sts get-caller-identity, or call GetCallerIdentity in code. The ARN must match the role you configured. Typical reasons it does not:
- an
AWS_ACCESS_KEY_IDorAWS_PROFILEenvironment variable overrides the task role or execution role; - a profile on a developer laptop is being used instead of the container role (the SDK credential chain order);
- the service uses the task execution role (for pulling images and secrets) where it should use the task role (for application calls), or the reverse;
- an Identity Center permission set differs from the role you inspected.
Minutes 8 to 15: locate the layer
Follow the table below, then go to the CLI section for the exact commands.
| Message says | Look at | Typical fix |
|---|---|---|
because no identity-based policy allows the X action | Policies attached to the session issuer role (managed and inline) | Add the action, scoped to the specific resource |
with an explicit deny in an identity-based policy | The same policies: find the Deny | Remove or narrow the deny, with the policy owner's approval |
because no resource-based policy allows ... | The resource's own policy (bucket, KMS key, secret, queue) | Add the principal to the resource policy. Cross-account needs both sides |
with an explicit deny in a service control policy | SCPs on the account's OU path | A guardrail. Fix by working inside it, or request a reviewed change from the platform team. Never ask for a blanket exemption |
with an explicit deny in a resource control policy | RCPs on the resource account's path | Same as SCP, on the resource side |
because no permissions boundary allows | The boundary on the role | Add to the boundary only if the access is legitimate |
because no session policy allows | Policy passed when the session was created | Fix the code or permission set that created the session |
because no VPC endpoint policy allows | The endpoint policy of the endpoint used | Allow the principal and action in the endpoint policy |
AccessDenied with no reason | CloudTrail, then the simulator | See the next step |
The full explanation of each phrase is in Policy evaluation logic.
Minutes 15 to 25: find the call in CloudTrail when the message is not enough
Search for the event by name and time, then read userIdentity, errorCode, errorMessage, requestParameters and sourceIPAddress. The sourceIPAddress and userAgent (for example aws-sdk-dotnet-...) confirm whether the call came from your service or from something else. If the denied call was a data-plane call and you have no data events enabled, it will not be there, and the application's own logs are your evidence.
Minutes 25 to 40: fix the smallest thing
Rules for the change:
- Name the action and resource exactly.
dynamodb:GetItemon the specific table ARN, notdynamodb:*on*. - Put it in code. The fix goes into the CDK or Terraform that defines the role, through the normal pull request, not in the console. Console fixes get overwritten by the next deployment and are invisible to reviewers.
- Respect the guardrail. If an SCP caused the denial, the fix is a design change or a reviewed SCP change by its owner.
- Test with the simulator, then for real. Wait a short time for IAM propagation, then retry the actual call.
- If it is production and urgent, follow the emergency change process: time-boxed, approved, recorded, and reverted or made permanent in code within an agreed time.
After the incident: prevent and record
Add an automated policy check in the pipeline (Access Analyzer validate-policy), a unit or integration test that calls the real action under the real role in a non-production account, and a short record: what was denied, the layer, the change, the approver and the verification.
AWS Console instructions
- Console → CloudTrail → Event history. Set Lookup attributes to Event name (for example
GetItem) or User name (the role session name) and choose the time range. Open the event and read Event record (JSON). - Console → IAM → Roles → (the role). On Permissions, expand each policy. Check Permissions boundary at the bottom, and open Trust relationships for assume-role failures.
- Console → IAM → Policy simulator (the simulator is linked from the IAM console help menu, or use the CLI). Choose the role, the service and action, set the resource ARN and Run simulation. Read the Permissions boundary and Organizations SCPs columns.
- Console → IAM Access Analyzer → Policy validation to check a draft policy, and Roles → Generate policy on the role's Last accessed tab (or the Access Analyzer policy generation feature) to propose a least-privilege policy from CloudTrail.
- Console → AWS Organizations → Policies → Service control policies in the management account (or a delegated administrator) to see which SCPs attach to the account's OU path.
AWS CLI
# 1. Who am I, from the failing workload? (run via ECS Exec or in a test Lambda that calls STS)
aws sts get-caller-identity --profile learn-sandbox
# 2. Which role does a session belong to? (assumed-role/<ROLE>/<SESSION>: inspect <ROLE>)
aws iam get-role --role-name hl-statements-task-role \
--query 'Role.{Arn:Arn,Boundary:PermissionsBoundary.PermissionsBoundaryArn,Trust:AssumeRolePolicyDocument}' \
--profile learn-sandbox
# 3. List every policy attached to the role: managed policies and inline policies.
aws iam list-attached-role-policies --role-name hl-statements-task-role --profile learn-sandbox
aws iam list-role-policies --role-name hl-statements-task-role --profile learn-sandbox
aws iam get-role-policy --role-name hl-statements-task-role --policy-name InlineAccess --profile learn-sandbox
# 4. Simulate the failing call against the specific resource.
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam::111122223333:role/hl-statements-task-role \
--action-names dynamodb:GetItem \
--resource-arns arn:aws:dynamodb:us-east-1:111122223333:table/hl-accounts \
--query 'EvaluationResults[].{Action:EvalActionName,Decision:EvalDecision,Matched:MatchedStatements[].SourcePolicyId,Org:OrganizationsDecisionDetail.AllowedByOrganizations,Boundary:PermissionsBoundaryDecisionDetail.AllowedByPermissionsBoundary}' \
--profile learn-sandbox
# 5. Find the denied call in CloudTrail event history (management events, last 90 days).
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=Username,AttributeValue=<session-name-from-the-ARN> \
--start-time 2026-11-13T22:00:00Z --end-time 2026-11-13T22:30:00Z \
--query 'Events[].CloudTrailEvent' --output text --profile learn-sandbox
# 6. Which SCPs apply to the account? (run in the management account or as a delegated administrator)
aws organizations list-policies-for-target --target-id 111122223333 \
--filter SERVICE_CONTROL_POLICY --profile learn-management
aws organizations describe-policy --policy-id p-examplepolicy --profile learn-management
# 7. Validate a proposed policy document before deploying it.
aws accessanalyzer validate-policy --policy-type IDENTITY_POLICY \
--policy-document file://proposed-policy.json --profile learn-sandbox
# 8. Decode an encoded authorization failure message (some services return one; needs sts:DecodeAuthorizationMessage).
aws sts decode-authorization-message --encoded-message <encoded-text> --profile learn-sandboxA Logs Insights query for your own application logs, which should record the SDK error code, action and request ID on failures:
fields @timestamp, correlationId, awsErrorCode, awsRequestId, message
| filter awsErrorCode in ["AccessDenied", "AccessDeniedException", "UnauthorizedOperation"]
| stats count(*) as denials by bin(5m), message
| sort @timestamp descInfrastructure as Code
The fix for a missing identity permission is a precisely scoped statement on the role in code.
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Constructs;
public class StatementsRoleStack : Stack
{
public StatementsRoleStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
var role = new Role(this, "StatementsTaskRole", new RoleProps
{
RoleName = "hl-statements-task-role",
AssumedBy = new ServicePrincipal("ecs-tasks.amazonaws.com")
});
// The ONE missing permission, scoped to the specific table and index.
role.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Sid = "ReadAccountsTable",
Effect = Effect.ALLOW,
Actions = new[] { "dynamodb:GetItem", "dynamodb:Query" },
Resources = new[]
{
$"arn:aws:dynamodb:{Region}:{Account}:table/hl-accounts",
$"arn:aws:dynamodb:{Region}:{Account}:table/hl-accounts/index/*"
}
}));
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-accessdenied");
}
}Complete code
A small triage tool: paste a CloudTrail event (the JSON from event history, or the CloudTrailEvent string from lookup-events) and it prints the facts and the layer to look at.
using System.Text.Json;
namespace Harbourline.Triage;
public sealed record DenialFacts(
string PrincipalType, string PrincipalArn, string? SessionIssuerArn,
string EventSource, string EventName, string ErrorCode, string? ErrorMessage,
string SourceIp, string UserAgent, string Layer, string LookNext);
public static class CloudTrailTriage
{
// Order matters: the more specific phrases come first.
private static readonly (string Phrase, string Layer, string LookNext)[] Phrases =
[
("explicit deny in a service control policy", "SCP (explicit deny)", "Organizations > SCPs on the account's OU path. A guardrail: fix the design or request a reviewed SCP change."),
("explicit deny in a resource control policy", "RCP (explicit deny)", "Organizations > RCPs for the resource account. A guardrail."),
("because no service control policy allows", "SCP (no allow)", "An allow-list SCP is missing this action, or FullAWSAccess is detached."),
("explicit deny in a permissions boundary", "Permissions boundary (explicit deny)", "Boundary policy on the role."),
("because no permissions boundary allows", "Permissions boundary (no allow)", "Boundary policy on the role: add the action only if legitimate."),
("explicit deny in a session policy", "Session policy (explicit deny)", "Policy passed on AssumeRole or defined in the permission set."),
("because no session policy allows", "Session policy (no allow)", "Policy passed on AssumeRole or defined in the permission set."),
("because no VPC endpoint policy allows", "VPC endpoint policy", "Endpoint policy of the endpoint used for the call."),
("explicit deny in an identity-based policy", "Identity policy (explicit deny)", "Find the Deny statement in the role's managed and inline policies."),
("because no identity-based policy allows", "Identity policy (no allow)", "Role policies: missing action, or resource ARN does not match."),
("explicit deny in a resource-based policy", "Resource policy (explicit deny)", "Bucket, key, queue or secret policy."),
("because no resource-based policy allows", "Resource policy (no allow)", "Resource policy: add the principal. Cross-account needs both sides to allow."),
];
public static DenialFacts Parse(string cloudTrailEventJson)
{
using var doc = JsonDocument.Parse(cloudTrailEventJson);
var root = doc.RootElement;
string Str(JsonElement e, params string[] path)
{
foreach (var p in path)
{
if (e.ValueKind != JsonValueKind.Object || !e.TryGetProperty(p, out e)) return "";
}
return e.ValueKind == JsonValueKind.String ? e.GetString() ?? "" : e.ToString();
}
var errorCode = Str(root, "errorCode");
if (string.IsNullOrEmpty(errorCode))
throw new ArgumentException("This event has no errorCode: it was not a failed call.");
var message = Str(root, "errorMessage");
var (layer, next) = Classify(errorCode, message);
return new DenialFacts(
PrincipalType: Str(root, "userIdentity", "type"),
PrincipalArn: Str(root, "userIdentity", "arn"),
SessionIssuerArn: NullIfEmpty(Str(root, "userIdentity", "sessionContext", "sessionIssuer", "arn")),
EventSource: Str(root, "eventSource"),
EventName: Str(root, "eventName"),
ErrorCode: errorCode,
ErrorMessage: NullIfEmpty(message),
SourceIp: Str(root, "sourceIPAddress"),
UserAgent: Str(root, "userAgent"),
Layer: layer,
LookNext: next);
}
private static (string Layer, string Next) Classify(string errorCode, string message)
{
foreach (var (phrase, layer, next) in Phrases)
if (message.Contains(phrase, StringComparison.OrdinalIgnoreCase)) return (layer, next);
if (!errorCode.Contains("Denied", StringComparison.OrdinalIgnoreCase) &&
!errorCode.Contains("Unauthorized", StringComparison.OrdinalIgnoreCase))
return ("Not an authorisation error", $"errorCode is '{errorCode}': this is probably not an IAM problem.");
return ("Unknown (no layer named)",
"Confirm the principal, run the policy simulator, check the resource policy, then the SCPs.");
}
private static string? NullIfEmpty(string s) => string.IsNullOrWhiteSpace(s) ? null : s;
}using Harbourline.Triage;
using Xunit;
public class CloudTrailTriageTests
{
private const string DynamoDenied = """
{
"eventVersion": "1.10",
"userIdentity": {
"type": "AssumedRole",
"arn": "arn:aws:sts::111122223333:assumed-role/hl-statements-task-role/0f3c9a",
"accountId": "111122223333",
"sessionContext": { "sessionIssuer": { "type": "Role", "arn": "arn:aws:iam::111122223333:role/hl-statements-task-role" } }
},
"eventSource": "dynamodb.amazonaws.com",
"eventName": "GetItem",
"sourceIPAddress": "10.0.12.34",
"userAgent": "aws-sdk-dotnet-coreclr/4.0.0",
"errorCode": "AccessDeniedException",
"errorMessage": "User: arn:aws:sts::111122223333:assumed-role/hl-statements-task-role/0f3c9a is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-east-1:111122223333:table/hl-accounts because no identity-based policy allows the dynamodb:GetItem action"
}
""";
[Fact]
public void Missing_identity_allow_is_classified_and_role_is_extracted()
{
var f = CloudTrailTriage.Parse(DynamoDenied);
Assert.Equal("Identity policy (no allow)", f.Layer);
Assert.Equal("arn:aws:iam::111122223333:role/hl-statements-task-role", f.SessionIssuerArn);
Assert.Equal("GetItem", f.EventName);
}
[Fact]
public void Scp_deny_is_classified_as_a_guardrail()
{
var scp = DynamoDenied.Replace(
"because no identity-based policy allows the dynamodb:GetItem action",
"with an explicit deny in a service control policy");
Assert.Equal("SCP (explicit deny)", CloudTrailTriage.Parse(scp).Layer);
}
[Fact]
public void A_successful_event_is_rejected()
{
Assert.Throws<ArgumentException>(() => CloudTrailTriage.Parse("""{ "eventName": "GetItem" }"""));
}
}Code explained line by line
Phrasesis an ordered list of message fragments to layers. Specific phrases come first because the messageno resource-based policyandno identity-based policyare different layers and must not be confused.Str(...)walks a JSON path safely and returns an empty string if any step is missing, because CloudTrail records differ between services and not every field is present.errorCodeempty means the call succeeded, so the tool rejects it rather than printing a misleading diagnosis.SessionIssuerArnis the role behind the session (assumed-role/NAME/SESSIONbecomesrole/NAME), which is the object whose policies you actually inspect. The session ARN is not an IAM object you can open.Classifychecks the phrases against the message. If none match, anerrorCodethat is not an authorisation error (for exampleThrottlingException) is reported as "not an IAM problem". That saves time when someone calls any 4xx "AccessDenied".- The tests use a fictional record (account
111122223333) and show the SCP case by replacing only the reason phrase, which is how you would add a regression case from a real incident.
Security considerations
⚠ Learning shortcut — not for production
The CLI commands above use a single learn-sandbox profile with broad permissions so you can read policies and search CloudTrail. In a real bank, incident engineers use read-only, audited roles for investigation (for example a role with IAM read and CloudTrail read, plus a separate, approval-gated role to make changes), and lookup-events is replaced by queries on the central, protected log archive. Never investigate in production with a role that can also change policies.
- Never fix a denial with
Action: "*"orResource: "*". It removes the evidence of what was needed and expands the blast radius of a compromise. - Do not seek an SCP exemption to unblock a release. SCPs and RCPs are the organisation's preventive controls. If the guardrail is wrong, the platform team changes it through its own review.
- Segregation of duties. The person who approves the permission should not be the person who requested it. Use the pull request and ticket as the evidence.
- Protect the evidence. CloudTrail logs are audit evidence. Do not delete or modify them, and keep access to the archive tightly controlled.
- Do not paste policies containing real account IDs or ARNs into public tools or AI assistants. Use the fictional IDs in this course for practice.
- Whether a given control satisfies an auditor is a question for your compliance function. This playbook describes engineering practice only.
Banking use case
Harbourline's platform team runs an organisation with an SCP that denies actions outside two approved Regions and denies disabling CloudTrail. When a developer in the cards team reports "AccessDenied creating a secret in eu-central-1", the message contains with an explicit deny in a service control policy. Instead of arguing, the engineer reads the policy name in the message, sees the Region guardrail and explains that the cards workload must stay in approved Regions, because the data-residency design depends on it. The fix is to use us-east-1 (in the course) or the bank's approved Region, not an exemption. In a second case, a new reconciliation service gets because no resource-based policy allows the kms:Decrypt action reading an encrypted bucket owned by another account: the fix is a key-policy change by the key owner, covered in the KMS playbook.
Common mistakes
- Editing the wrong role. The error names the session role; people edit the role they think the service uses (for example the execution role instead of the task role).
- Fixing only the identity side of a cross-account call. Both sides must allow. If the message says
no resource-based policy, no identity edit will help. - Resource ARN mismatches. Table ARN versus index ARN (
table/x/index/*), bucket versus object ARN (bucketforListBucket,bucket/*forGetObject), wrong account or Region. - Assuming the simulator is the truth. It does not model every layer of a real request.
- Not waiting for propagation. Retrying within a second of a change can look like the fix failed. Allow a short time and retry.
- Console-only fixes. They vanish on the next deployment and no reviewer saw them.
- Treating
ThrottlingExceptionas access denied. Check theerrorCode. - Forgetting conditions. An Allow with
aws:SourceVpc,aws:RequestedRegionor MFA conditions silently does not match.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
get-caller-identity shows an unexpected role | Environment variables or profile override the container role | Remove AWS_ACCESS_KEY_ID and related variables from the task or function, set the task role correctly |
| Policy simulator says allowed, real call is denied | A layer the simulator does not model: resource policy, session policy, VPC endpoint policy or a condition on network context | Read the real error phrase, check the resource policy and conditions |
| Works in one account, denied in another | Different role policies, different SCPs per OU, or cross-account trust missing | Compare role and OU policies. Check the trust policy |
AccessDenied on s3:GetObject but the identity policy allows it | Bucket policy deny, SSE-KMS key policy, or the object owned by another account | Check the bucket policy, then the KMS key policy. See the KMS playbook |
AccessDenied on ListBucket for a missing key shows as 403 instead of 404 | The caller lacks s3:ListBucket, so S3 hides whether the key exists | Add s3:ListBucket on the bucket ARN if the caller should see the 404 |
not authorized to perform: sts:AssumeRole | Trust policy does not name the caller, or the caller lacks sts:AssumeRole | Fix the trust policy and the caller's identity policy (both are needed across accounts) |
| Denied only from a particular network or IP | A condition on source VPC or IP, or a VPC endpoint policy | Check conditions and the endpoint policy |
| Event not found in CloudTrail | Data event not enabled, wrong Region, or delivery delay | Enable data events for the future, check the Region, retry after a few minutes |
| Fix applied but still denied | Eventual consistency or a cached credentials session with an old session policy | Wait briefly. Start a new session if a session policy is involved |
Cost and cleanup
Cost label: free. IAM and the simulator cost nothing. CloudTrail event history is free for management events. If you create a role for the IaC example in the sandbox, there is no charge, but remove it afterwards.
# CDK
cdk destroy StatementsRoleStack --profile learn-sandbox
# Or by hand (inline policy first, then the role)
aws iam delete-role-policy --role-name hl-statements-task-role --policy-name ReadAccountsTable --profile learn-sandbox
aws iam delete-role --role-name hl-statements-task-role --profile learn-sandbox
# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-accessdenied --profile learn-sandboxExercises
- 🟢 Guided: Add
CloudTrailTriageand its tests to a console project and rundotnet test. Then add a fourth test forbecause no resource-based policy allows the kms:Decrypt actionand assert the layer. - 🟡 Challenge: Make the tool also extract the action and the resource ARN from the message with a regular expression, and print "Add
<action>on<resource>to the policies of<session issuer role>" when the layer is an identity-policy gap. - 🔴 Live problem: 22:15, change night. The statements service gets
AccessDeniedreading S3. Its role hass3:GetObjectonarn:aws:s3:::hl-statements/*. The bucket is in another account, encrypted with SSE-KMS. The release manager says "give it admin". Write what you check, in order, what you refuse to do and what you say to the release manager.
Assignment
Add an AccessDenied triage card to your portfolio: one page with the five facts, the phrase-to-layer table, your ten-minute flow and three worked examples with fictional messages. Commit it as playbooks/access-denied-triage.md, and include your CloudTrailTriage tool with tests. Mark clearly that all IDs are fictional.
Solutions
The new test message ... because no resource-based policy allows the kms:Decrypt action matches the phrase because no resource-based policy allows and returns Resource policy (no allow). Assert on f.Layer and that LookNext mentions the cross-account requirement. Why: adding a test per real incident turns your experience into a regression suite for the triage logic.
Use Regex.Match(message, @"perform: (?<action>[\w:*]+) on resource: (?<res>\S+)"). Trim the trailing quotes from the resource. If the layer is Identity policy (no allow) and both groups matched, build the sentence from action, res and SessionIssuerArn. Keep the output as a suggestion only; a human reviews and a pull request changes the role. Why: the tool speeds up reading the error but must not edit policies itself, which would bypass review.
Order: (1) read the full error and check the reason phrase; (2) get-caller-identity from the workload to confirm the role; (3) check the bucket policy in the other account for this role (cross-account needs both sides); (4) check the KMS key policy for kms:Decrypt for this role, since SSE-KMS adds a second authorisation; (5) CloudTrail for both the S3 call and the KMS call to see which one failed. Refuse: administrator access, * actions, an SCP exemption or editing policies in the console. Say to the release manager: "It is not a missing permission on our role; the bucket and key are in another account and need to allow us. I've asked the data-platform team to add this role to the bucket policy and key policy; here is the change request with the exact statements. I expect 30 minutes. Giving admin would not fix it anyway, because a cross-account resource policy still has to allow." Why: it identifies that two resource-side authorisations are involved, protects least privilege, and gives the manager a time and an action.
Knowledge check
Answer: The KMS key's own policy (and the caller's identity policy must also allow, for cross-account use). Editing only the caller's role will not help.
Answer: The SDK credential chain: environment variables, profile settings, and whether the execution role was used instead of the task role. The first source in the chain that supplies credentials wins.
Answer: A resource policy, VPC endpoint policy or session policy denies it, or a condition on network or request context (source VPC, Region, MFA) that the simulation did not include.
Answer: It grants far more than needed, may never be removed, defeats least privilege and segregation of duties, and removes the evidence of what was actually required. Add the specific missing action and resource.
Answer: The owner of the organisation's guardrails (platform or cloud governance team), through their review process, or the workload design changes to comply. An account-level policy edit cannot override an SCP deny.
Interview questions
- How do you troubleshoot an
AccessDenied? Read the message for principal, action, resource and reason; confirm the real caller withget-caller-identity; find the layer; use CloudTrail and the simulator if unclear; fix the smallest thing through IaC and review. - What is the difference between implicit and explicit deny? Implicit deny is the default when nothing allows the request. Explicit deny is a
Denystatement and always wins over any Allow. - When does IAM alone not decide access? When a resource policy, SCP, RCP, permissions boundary, session policy or VPC endpoint policy also applies. Cross-account requests need both accounts to allow.
- Someone asks for administrator access to unblock a release. What do you do? Decline, find the missing action and resource, and make a scoped, reviewed change. Offer a timeline.
- How would you reduce
AccessDeniedincidents? Test roles in lower environments with the real calls, validate policies with Access Analyzer in the pipeline, generate least-privilege policies from CloudTrail activity, and keep the guardrail documentation visible to developers.
Certification practice
An application running with an IAM role in Account A receives AccessDenied reading an S3 object in Account B. The role has an identity policy that allows s3:GetObject on the object's ARN. The message says because no resource-based policy allows the s3:GetObject action. What is the most appropriate fix?
A. Attach AdministratorAccess to the role in Account A.
B. Add a statement to the bucket policy in Account B that allows the role's principal to perform s3:GetObject.
C. Remove the permissions boundary from the role in Account A.
D. Enable versioning on the bucket.
Answer: B. Cross-account access needs an allow on both sides, and the message says the resource side is missing. A is excessive and does not change the bucket policy. C is not mentioned in the message. D is unrelated.
A developer cannot disable AWS CloudTrail in a member account and receives with an explicit deny in a service control policy. What does this mean?
A. The IAM role in the account lacks the cloudtrail:StopLogging permission, which can be added to the role.
B. An organisation-level guardrail denies the action for the account, and an account-level IAM change cannot override it.
C. CloudTrail is not available in the Region.
D. The developer's session expired.
Answer: B. An SCP explicit deny applies to every principal in the account and cannot be overridden by identity policies. A would be an identity policy message. C and D would produce different errors.
A .NET service on Amazon ECS calls DynamoDB and receives AccessDeniedException. aws sts get-caller-identity run inside the container shows a different role than the task role. Which is the most likely cause?
A. The DynamoDB table is in the wrong Region. B. Credentials supplied by environment variables or a profile are taking precedence over the task role in the SDK credential chain. C. The table needs a global secondary index. D. The security group blocks DynamoDB.
Answer: B. The SDK uses the first credentials source in its chain, so stray environment variables or profile settings override the task role. A would give a not-found error, C is unrelated to authorisation and D would produce a timeout, not an authorisation error.
Completion checklist
- I can name the five facts in an AccessDenied and find each
- I can map the reason phrase to the layer without looking it up
- I know three reasons the caller is not the role I expected
- I ran the triage tool tests and added one new case
- I can explain why admin, wildcards and SCP exemptions are not fixes
- I ran the cleanup steps and confirmed nothing tagged
m17-playbook-accessdeniedremains
Key takeaways
- Read the message first: principal, action, resource, reason.
- Confirm the real caller before editing any policy.
- Match the reason phrase to the layer; cross-account needs both sides to allow.
- The simulator and Access Analyzer help but do not replace the real error and CloudTrail.
- Fix the smallest thing through code and review, then prevent recurrence and record it.
Further reading
- Policy evaluation logic (AWS IAM User Guide)
- Troubleshooting access denied error messages
- IAM Access Analyzer policy validation
- Previous playbook: Partner API timeouts
- Next playbook: KMS AccessDeniedException
- Related: Module 17 overview
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Playbook: partner API timeouts
Diagnose a partner API that times out by splitting the call into DNS, connect, TLS and response time, checking the egress path and the partner's limits, and choosing timeouts, retries and a circuit breaker that fail safe for a bank.
Playbook: KMS AccessDeniedException
Diagnose a KMS AccessDeniedException by checking the key policy, IAM, grants, encryption context, key state and cross-account rules in order, and tell service-principal and SSE-KMS failures apart from ordinary IAM gaps.