Skip to Content
Cloud Engineering Academy
17. Daily-Work Playbooks & Career

Playbook: AccessDenied and IAM policy evaluation

Turn an AccessDenied error into a located cause in minutes: read the message, confirm who is really calling, find the denying layer with CloudTrail, the policy simulator and Access Analyzer, and fix it with the smallest safe change.

intermediate⏱ 1 h 20 min🟢 FreeSCS-C03SAA-C03DVA-C03SOA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · IAM · AWS CloudTrail · IAM Access Analyzer · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this playbook you can:

  • Read an AccessDenied message and name the layer that caused it (identity policy, resource policy, permissions boundary, SCP, RCP, session policy, trust policy or VPC endpoint policy).
  • Confirm which principal is really calling, including the common case where the workload is not using the role you think it is.
  • Find the exact denied call in CloudTrail and extract the facts you need from it.
  • Use the IAM policy simulator and IAM Access Analyzer correctly, and say what they cannot tell you.
  • Make the smallest safe fix, deliver it through code review and IaC, and write the audit record.
  • Avoid the three bad fixes: adding *, asking for an exemption from a guardrail, and "temporarily" attaching administrator access.

Prerequisites

Why this matters in banking

It is change night. The new statements service is deployed to production at 22:15 and the first run fails: AccessDeniedException ... not authorized to perform: dynamodb:GetItem. The release manager asks "can you just give it admin for tonight?" Three bad things can follow from saying yes: a broad permission that never gets removed, an audit finding for breaking least privilege and segregation of duties, and a real security exposure if the service has a vulnerability. The senior move is to find the denying layer in ten minutes and add the one action that is actually missing, so the release proceeds and the audit trail stays clean.

Many AccessDenied errors are not really permission problems. They are identity problems (the workload is a different principal than you assumed), resource policy problems (the other side said no), or guardrail problems (an organisation control did exactly what it was designed to do). Fixing the wrong layer is how engineers end up with a role that has * and still does not work.

The idea in plain language

AWS decides each request by asking in a fixed order: is there an explicit deny anywhere? Do all the caps allow it? Is there an explicit allow from an identity policy or a resource policy? If nothing allows it, the answer is no. Your job during an incident is to find out which question gave the wrong answer.

Analogy: You are turned away at a bank's vault door. The guard's note might say "your badge is not on the list" (no identity allow), "the vault owner has not listed your department" (no resource allow), "head office has locked this door for everyone today" (SCP deny), or "you are using a visitor badge instead of your own" (wrong principal). The note usually tells you which. Reading it carefully is half the work.

The five facts every AccessDenied gives you (or lets you find)

FactWhere it isWhy you need it
PrincipalUser: arn:aws:sts::...:assumed-role/ROLE/SESSION in the messageConfirms which role actually called, not which role you meant
Actionis not authorized to perform: dynamodb:GetItemThe exact IAM action, which is not always the API name you called
Resourceon resource: arn:aws:...The exact ARN, including its account and Region
Reasonbecause no identity-based policy allows... or with an explicit deny in a service control policyThe layer. Not every service includes it, so CloudTrail is your backup
Time and request IDThe log line, the SDK exception's RequestIdLets you find the call in CloudTrail

What problem the AWS services solve

  • The enhanced error message names the layer for many services. When it does, you can skip straight to the right policy.
  • AWS CloudTrail records the call with the caller, action, resource, source IP and the error. Management events are in Event history for 90 days at no cost. Data events (such as S3 object-level and DynamoDB item-level calls) are recorded only if you enabled them on a trail (⚠ VERIFY which data-plane services you have enabled).
  • The IAM policy simulator (simulate-principal-policy) evaluates identity policies and reports decisions for the permissions boundary and Organizations policies as separate fields. It does not see every real-world factor (network conditions, session policy of a live session, some resource-policy interactions), so a simulator "allowed" is a hint, not proof.
  • IAM Access Analyzer validates policies, finds unintended external access and can generate a least-privilege policy from CloudTrail activity. Unused-access analysis is a paid feature (⚠ VERIFY current pricing).
  • IAM last accessed information shows which services and actions a role actually used, which you need for right-sizing afterwards.

Honest limits: IAM is eventually consistent, so a fix can take seconds to apply; some services return a bare Access Denied with no reason; and CloudTrail events arrive with a delay, usually minutes (⚠ VERIFY current delivery time), so the denied call may not be searchable the second after it happens.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
403 AuthorizationFailed with the role assignment scope in the messageAccessDenied message naming action, resource and often the denying policy typeAWS evaluates several policy types together, and an explicit deny in any of them wins
Azure Activity Log filtered on failed operationsCloudTrail event history with errorCodeEvent history shows management events for 90 days; data events need a trail
"What if" tool for role assignments, or the Entra access reviewsIAM policy simulator and Access AnalyzerThe simulator answers "would this principal be allowed this action on this resource" for given context
Managed identity token fetched from IMDSRole session credentials from the ECS task role endpoint or Lambda environmentThe SDK credential chain can pick a different source first (environment variables, profile), a classic cause of "wrong identity"
Azure Policy deny effectSCP or RCP explicit denyNot visible in the identity's own policies. Look in Organizations
Key Vault access policy denialKMS key policyA key's own policy can deny even when IAM allows. See the KMS playbook
Conditional Access blockIAM condition keys (aws:SourceVpc, aws:MultiFactorAuthPresent, aws:RequestedRegion)Conditions sit inside policy statements, so the Allow can exist and still not match

Architecture

The ten-minute decision flow

What can say no

A deny at any step stops the request, and no step except an explicit Allow can grant. Caps (SCP, RCP, boundary, session policy, endpoint policy) only remove permissions.

Step-by-step implementation

Minutes 0 to 3: capture the evidence before changing anything

Copy the full error text, the timestamp and the request ID (the SDK exception has RequestId; the .NET AmazonServiceException also exposes ErrorCode and StatusCode). Note the environment (production, account ID, Region) and what changed recently: a deployment, a new role, a policy edit, an SCP change.

Minutes 3 to 8: confirm the caller really is who you think

From inside the failing workload (ECS Exec, Lambda test event, or the code itself) run aws sts get-caller-identity, or call GetCallerIdentity in code. The ARN must match the role you configured. Typical reasons it does not:

  • an AWS_ACCESS_KEY_ID or AWS_PROFILE environment variable overrides the task role or execution role;
  • a profile on a developer laptop is being used instead of the container role (the SDK credential chain order);
  • the service uses the task execution role (for pulling images and secrets) where it should use the task role (for application calls), or the reverse;
  • an Identity Center permission set differs from the role you inspected.

Minutes 8 to 15: locate the layer

Follow the table below, then go to the CLI section for the exact commands.

Message saysLook atTypical fix
because no identity-based policy allows the X actionPolicies attached to the session issuer role (managed and inline)Add the action, scoped to the specific resource
with an explicit deny in an identity-based policyThe same policies: find the DenyRemove or narrow the deny, with the policy owner's approval
because no resource-based policy allows ...The resource's own policy (bucket, KMS key, secret, queue)Add the principal to the resource policy. Cross-account needs both sides
with an explicit deny in a service control policySCPs on the account's OU pathA guardrail. Fix by working inside it, or request a reviewed change from the platform team. Never ask for a blanket exemption
with an explicit deny in a resource control policyRCPs on the resource account's pathSame as SCP, on the resource side
because no permissions boundary allowsThe boundary on the roleAdd to the boundary only if the access is legitimate
because no session policy allowsPolicy passed when the session was createdFix the code or permission set that created the session
because no VPC endpoint policy allowsThe endpoint policy of the endpoint usedAllow the principal and action in the endpoint policy
AccessDenied with no reasonCloudTrail, then the simulatorSee the next step

The full explanation of each phrase is in Policy evaluation logic.

Minutes 15 to 25: find the call in CloudTrail when the message is not enough

Search for the event by name and time, then read userIdentity, errorCode, errorMessage, requestParameters and sourceIPAddress. The sourceIPAddress and userAgent (for example aws-sdk-dotnet-...) confirm whether the call came from your service or from something else. If the denied call was a data-plane call and you have no data events enabled, it will not be there, and the application's own logs are your evidence.

Minutes 25 to 40: fix the smallest thing

Rules for the change:

  1. Name the action and resource exactly. dynamodb:GetItem on the specific table ARN, not dynamodb:* on *.
  2. Put it in code. The fix goes into the CDK or Terraform that defines the role, through the normal pull request, not in the console. Console fixes get overwritten by the next deployment and are invisible to reviewers.
  3. Respect the guardrail. If an SCP caused the denial, the fix is a design change or a reviewed SCP change by its owner.
  4. Test with the simulator, then for real. Wait a short time for IAM propagation, then retry the actual call.
  5. If it is production and urgent, follow the emergency change process: time-boxed, approved, recorded, and reverted or made permanent in code within an agreed time.

After the incident: prevent and record

Add an automated policy check in the pipeline (Access Analyzer validate-policy), a unit or integration test that calls the real action under the real role in a non-production account, and a short record: what was denied, the layer, the change, the approver and the verification.

AWS Console instructions

  1. Console → CloudTrail → Event history. Set Lookup attributes to Event name (for example GetItem) or User name (the role session name) and choose the time range. Open the event and read Event record (JSON).
  2. Console → IAM → Roles → (the role). On Permissions, expand each policy. Check Permissions boundary at the bottom, and open Trust relationships for assume-role failures.
  3. Console → IAM → Policy simulator (the simulator is linked from the IAM console help menu, or use the CLI). Choose the role, the service and action, set the resource ARN and Run simulation. Read the Permissions boundary and Organizations SCPs columns.
  4. Console → IAM Access Analyzer → Policy validation to check a draft policy, and Roles → Generate policy on the role's Last accessed tab (or the Access Analyzer policy generation feature) to propose a least-privilege policy from CloudTrail.
  5. Console → AWS Organizations → Policies → Service control policies in the management account (or a delegated administrator) to see which SCPs attach to the account's OU path.

AWS CLI

# 1. Who am I, from the failing workload? (run via ECS Exec or in a test Lambda that calls STS)
aws sts get-caller-identity --profile learn-sandbox

# 2. Which role does a session belong to? (assumed-role/<ROLE>/<SESSION>: inspect <ROLE>)
aws iam get-role --role-name hl-statements-task-role \
  --query 'Role.{Arn:Arn,Boundary:PermissionsBoundary.PermissionsBoundaryArn,Trust:AssumeRolePolicyDocument}' \
  --profile learn-sandbox

# 3. List every policy attached to the role: managed policies and inline policies.
aws iam list-attached-role-policies --role-name hl-statements-task-role --profile learn-sandbox
aws iam list-role-policies --role-name hl-statements-task-role --profile learn-sandbox
aws iam get-role-policy --role-name hl-statements-task-role --policy-name InlineAccess --profile learn-sandbox

# 4. Simulate the failing call against the specific resource.
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::111122223333:role/hl-statements-task-role \
  --action-names dynamodb:GetItem \
  --resource-arns arn:aws:dynamodb:us-east-1:111122223333:table/hl-accounts \
  --query 'EvaluationResults[].{Action:EvalActionName,Decision:EvalDecision,Matched:MatchedStatements[].SourcePolicyId,Org:OrganizationsDecisionDetail.AllowedByOrganizations,Boundary:PermissionsBoundaryDecisionDetail.AllowedByPermissionsBoundary}' \
  --profile learn-sandbox

# 5. Find the denied call in CloudTrail event history (management events, last 90 days).
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=<session-name-from-the-ARN> \
  --start-time 2026-11-13T22:00:00Z --end-time 2026-11-13T22:30:00Z \
  --query 'Events[].CloudTrailEvent' --output text --profile learn-sandbox

# 6. Which SCPs apply to the account? (run in the management account or as a delegated administrator)
aws organizations list-policies-for-target --target-id 111122223333 \
  --filter SERVICE_CONTROL_POLICY --profile learn-management
aws organizations describe-policy --policy-id p-examplepolicy --profile learn-management

# 7. Validate a proposed policy document before deploying it.
aws accessanalyzer validate-policy --policy-type IDENTITY_POLICY \
  --policy-document file://proposed-policy.json --profile learn-sandbox

# 8. Decode an encoded authorization failure message (some services return one; needs sts:DecodeAuthorizationMessage).
aws sts decode-authorization-message --encoded-message <encoded-text> --profile learn-sandbox

A Logs Insights query for your own application logs, which should record the SDK error code, action and request ID on failures:

fields @timestamp, correlationId, awsErrorCode, awsRequestId, message
| filter awsErrorCode in ["AccessDenied", "AccessDeniedException", "UnauthorizedOperation"]
| stats count(*) as denials by bin(5m), message
| sort @timestamp desc

Infrastructure as Code

The fix for a missing identity permission is a precisely scoped statement on the role in code.

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Constructs;

public class StatementsRoleStack : Stack
{
    public StatementsRoleStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        var role = new Role(this, "StatementsTaskRole", new RoleProps
        {
            RoleName = "hl-statements-task-role",
            AssumedBy = new ServicePrincipal("ecs-tasks.amazonaws.com")
        });

        // The ONE missing permission, scoped to the specific table and index.
        role.AddToPolicy(new PolicyStatement(new PolicyStatementProps
        {
            Sid = "ReadAccountsTable",
            Effect = Effect.ALLOW,
            Actions = new[] { "dynamodb:GetItem", "dynamodb:Query" },
            Resources = new[]
            {
                $"arn:aws:dynamodb:{Region}:{Account}:table/hl-accounts",
                $"arn:aws:dynamodb:{Region}:{Account}:table/hl-accounts/index/*"
            }
        }));

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-accessdenied");
    }
}

Complete code

A small triage tool: paste a CloudTrail event (the JSON from event history, or the CloudTrailEvent string from lookup-events) and it prints the facts and the layer to look at.

CloudTrailTriage.cs
using System.Text.Json;

namespace Harbourline.Triage;

public sealed record DenialFacts(
    string PrincipalType, string PrincipalArn, string? SessionIssuerArn,
    string EventSource, string EventName, string ErrorCode, string? ErrorMessage,
    string SourceIp, string UserAgent, string Layer, string LookNext);

public static class CloudTrailTriage
{
    // Order matters: the more specific phrases come first.
    private static readonly (string Phrase, string Layer, string LookNext)[] Phrases =
    [
        ("explicit deny in a service control policy", "SCP (explicit deny)", "Organizations > SCPs on the account's OU path. A guardrail: fix the design or request a reviewed SCP change."),
        ("explicit deny in a resource control policy", "RCP (explicit deny)", "Organizations > RCPs for the resource account. A guardrail."),
        ("because no service control policy allows", "SCP (no allow)", "An allow-list SCP is missing this action, or FullAWSAccess is detached."),
        ("explicit deny in a permissions boundary", "Permissions boundary (explicit deny)", "Boundary policy on the role."),
        ("because no permissions boundary allows", "Permissions boundary (no allow)", "Boundary policy on the role: add the action only if legitimate."),
        ("explicit deny in a session policy", "Session policy (explicit deny)", "Policy passed on AssumeRole or defined in the permission set."),
        ("because no session policy allows", "Session policy (no allow)", "Policy passed on AssumeRole or defined in the permission set."),
        ("because no VPC endpoint policy allows", "VPC endpoint policy", "Endpoint policy of the endpoint used for the call."),
        ("explicit deny in an identity-based policy", "Identity policy (explicit deny)", "Find the Deny statement in the role's managed and inline policies."),
        ("because no identity-based policy allows", "Identity policy (no allow)", "Role policies: missing action, or resource ARN does not match."),
        ("explicit deny in a resource-based policy", "Resource policy (explicit deny)", "Bucket, key, queue or secret policy."),
        ("because no resource-based policy allows", "Resource policy (no allow)", "Resource policy: add the principal. Cross-account needs both sides to allow."),
    ];

    public static DenialFacts Parse(string cloudTrailEventJson)
    {
        using var doc = JsonDocument.Parse(cloudTrailEventJson);
        var root = doc.RootElement;

        string Str(JsonElement e, params string[] path)
        {
            foreach (var p in path)
            {
                if (e.ValueKind != JsonValueKind.Object || !e.TryGetProperty(p, out e)) return "";
            }
            return e.ValueKind == JsonValueKind.String ? e.GetString() ?? "" : e.ToString();
        }

        var errorCode = Str(root, "errorCode");
        if (string.IsNullOrEmpty(errorCode))
            throw new ArgumentException("This event has no errorCode: it was not a failed call.");

        var message = Str(root, "errorMessage");
        var (layer, next) = Classify(errorCode, message);

        return new DenialFacts(
            PrincipalType: Str(root, "userIdentity", "type"),
            PrincipalArn: Str(root, "userIdentity", "arn"),
            SessionIssuerArn: NullIfEmpty(Str(root, "userIdentity", "sessionContext", "sessionIssuer", "arn")),
            EventSource: Str(root, "eventSource"),
            EventName: Str(root, "eventName"),
            ErrorCode: errorCode,
            ErrorMessage: NullIfEmpty(message),
            SourceIp: Str(root, "sourceIPAddress"),
            UserAgent: Str(root, "userAgent"),
            Layer: layer,
            LookNext: next);
    }

    private static (string Layer, string Next) Classify(string errorCode, string message)
    {
        foreach (var (phrase, layer, next) in Phrases)
            if (message.Contains(phrase, StringComparison.OrdinalIgnoreCase)) return (layer, next);

        if (!errorCode.Contains("Denied", StringComparison.OrdinalIgnoreCase) &&
            !errorCode.Contains("Unauthorized", StringComparison.OrdinalIgnoreCase))
            return ("Not an authorisation error", $"errorCode is '{errorCode}': this is probably not an IAM problem.");

        return ("Unknown (no layer named)",
                "Confirm the principal, run the policy simulator, check the resource policy, then the SCPs.");
    }

    private static string? NullIfEmpty(string s) => string.IsNullOrWhiteSpace(s) ? null : s;
}
CloudTrailTriageTests.cs
using Harbourline.Triage;
using Xunit;

public class CloudTrailTriageTests
{
    private const string DynamoDenied = """
    {
      "eventVersion": "1.10",
      "userIdentity": {
        "type": "AssumedRole",
        "arn": "arn:aws:sts::111122223333:assumed-role/hl-statements-task-role/0f3c9a",
        "accountId": "111122223333",
        "sessionContext": { "sessionIssuer": { "type": "Role", "arn": "arn:aws:iam::111122223333:role/hl-statements-task-role" } }
      },
      "eventSource": "dynamodb.amazonaws.com",
      "eventName": "GetItem",
      "sourceIPAddress": "10.0.12.34",
      "userAgent": "aws-sdk-dotnet-coreclr/4.0.0",
      "errorCode": "AccessDeniedException",
      "errorMessage": "User: arn:aws:sts::111122223333:assumed-role/hl-statements-task-role/0f3c9a is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-east-1:111122223333:table/hl-accounts because no identity-based policy allows the dynamodb:GetItem action"
    }
    """;

    [Fact]
    public void Missing_identity_allow_is_classified_and_role_is_extracted()
    {
        var f = CloudTrailTriage.Parse(DynamoDenied);

        Assert.Equal("Identity policy (no allow)", f.Layer);
        Assert.Equal("arn:aws:iam::111122223333:role/hl-statements-task-role", f.SessionIssuerArn);
        Assert.Equal("GetItem", f.EventName);
    }

    [Fact]
    public void Scp_deny_is_classified_as_a_guardrail()
    {
        var scp = DynamoDenied.Replace(
            "because no identity-based policy allows the dynamodb:GetItem action",
            "with an explicit deny in a service control policy");
        Assert.Equal("SCP (explicit deny)", CloudTrailTriage.Parse(scp).Layer);
    }

    [Fact]
    public void A_successful_event_is_rejected()
    {
        Assert.Throws<ArgumentException>(() => CloudTrailTriage.Parse("""{ "eventName": "GetItem" }"""));
    }
}

Code explained line by line

  1. Phrases is an ordered list of message fragments to layers. Specific phrases come first because the message no resource-based policy and no identity-based policy are different layers and must not be confused.
  2. Str(...) walks a JSON path safely and returns an empty string if any step is missing, because CloudTrail records differ between services and not every field is present.
  3. errorCode empty means the call succeeded, so the tool rejects it rather than printing a misleading diagnosis.
  4. SessionIssuerArn is the role behind the session (assumed-role/NAME/SESSION becomes role/NAME), which is the object whose policies you actually inspect. The session ARN is not an IAM object you can open.
  5. Classify checks the phrases against the message. If none match, an errorCode that is not an authorisation error (for example ThrottlingException) is reported as "not an IAM problem". That saves time when someone calls any 4xx "AccessDenied".
  6. The tests use a fictional record (account 111122223333) and show the SCP case by replacing only the reason phrase, which is how you would add a regression case from a real incident.

Security considerations

⚠ Learning shortcut — not for production

The CLI commands above use a single learn-sandbox profile with broad permissions so you can read policies and search CloudTrail. In a real bank, incident engineers use read-only, audited roles for investigation (for example a role with IAM read and CloudTrail read, plus a separate, approval-gated role to make changes), and lookup-events is replaced by queries on the central, protected log archive. Never investigate in production with a role that can also change policies.

  • Never fix a denial with Action: "*" or Resource: "*". It removes the evidence of what was needed and expands the blast radius of a compromise.
  • Do not seek an SCP exemption to unblock a release. SCPs and RCPs are the organisation's preventive controls. If the guardrail is wrong, the platform team changes it through its own review.
  • Segregation of duties. The person who approves the permission should not be the person who requested it. Use the pull request and ticket as the evidence.
  • Protect the evidence. CloudTrail logs are audit evidence. Do not delete or modify them, and keep access to the archive tightly controlled.
  • Do not paste policies containing real account IDs or ARNs into public tools or AI assistants. Use the fictional IDs in this course for practice.
  • Whether a given control satisfies an auditor is a question for your compliance function. This playbook describes engineering practice only.

Banking use case

Harbourline's platform team runs an organisation with an SCP that denies actions outside two approved Regions and denies disabling CloudTrail. When a developer in the cards team reports "AccessDenied creating a secret in eu-central-1", the message contains with an explicit deny in a service control policy. Instead of arguing, the engineer reads the policy name in the message, sees the Region guardrail and explains that the cards workload must stay in approved Regions, because the data-residency design depends on it. The fix is to use us-east-1 (in the course) or the bank's approved Region, not an exemption. In a second case, a new reconciliation service gets because no resource-based policy allows the kms:Decrypt action reading an encrypted bucket owned by another account: the fix is a key-policy change by the key owner, covered in the KMS playbook.

Common mistakes

  • Editing the wrong role. The error names the session role; people edit the role they think the service uses (for example the execution role instead of the task role).
  • Fixing only the identity side of a cross-account call. Both sides must allow. If the message says no resource-based policy, no identity edit will help.
  • Resource ARN mismatches. Table ARN versus index ARN (table/x/index/*), bucket versus object ARN (bucket for ListBucket, bucket/* for GetObject), wrong account or Region.
  • Assuming the simulator is the truth. It does not model every layer of a real request.
  • Not waiting for propagation. Retrying within a second of a change can look like the fix failed. Allow a short time and retry.
  • Console-only fixes. They vanish on the next deployment and no reviewer saw them.
  • Treating ThrottlingException as access denied. Check the errorCode.
  • Forgetting conditions. An Allow with aws:SourceVpc, aws:RequestedRegion or MFA conditions silently does not match.

Troubleshooting

SymptomLikely causeFix
get-caller-identity shows an unexpected roleEnvironment variables or profile override the container roleRemove AWS_ACCESS_KEY_ID and related variables from the task or function, set the task role correctly
Policy simulator says allowed, real call is deniedA layer the simulator does not model: resource policy, session policy, VPC endpoint policy or a condition on network contextRead the real error phrase, check the resource policy and conditions
Works in one account, denied in anotherDifferent role policies, different SCPs per OU, or cross-account trust missingCompare role and OU policies. Check the trust policy
AccessDenied on s3:GetObject but the identity policy allows itBucket policy deny, SSE-KMS key policy, or the object owned by another accountCheck the bucket policy, then the KMS key policy. See the KMS playbook
AccessDenied on ListBucket for a missing key shows as 403 instead of 404The caller lacks s3:ListBucket, so S3 hides whether the key existsAdd s3:ListBucket on the bucket ARN if the caller should see the 404
not authorized to perform: sts:AssumeRoleTrust policy does not name the caller, or the caller lacks sts:AssumeRoleFix the trust policy and the caller's identity policy (both are needed across accounts)
Denied only from a particular network or IPA condition on source VPC or IP, or a VPC endpoint policyCheck conditions and the endpoint policy
Event not found in CloudTrailData event not enabled, wrong Region, or delivery delayEnable data events for the future, check the Region, retry after a few minutes
Fix applied but still deniedEventual consistency or a cached credentials session with an old session policyWait briefly. Start a new session if a session policy is involved

Cost and cleanup

Cost label: free. IAM and the simulator cost nothing. CloudTrail event history is free for management events. If you create a role for the IaC example in the sandbox, there is no charge, but remove it afterwards.

# CDK
cdk destroy StatementsRoleStack --profile learn-sandbox

# Or by hand (inline policy first, then the role)
aws iam delete-role-policy --role-name hl-statements-task-role --policy-name ReadAccountsTable --profile learn-sandbox
aws iam delete-role --role-name hl-statements-task-role --profile learn-sandbox

# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-accessdenied --profile learn-sandbox

Exercises

  • 🟢 Guided: Add CloudTrailTriage and its tests to a console project and run dotnet test. Then add a fourth test for because no resource-based policy allows the kms:Decrypt action and assert the layer.
  • 🟡 Challenge: Make the tool also extract the action and the resource ARN from the message with a regular expression, and print "Add <action> on <resource> to the policies of <session issuer role>" when the layer is an identity-policy gap.
  • 🔴 Live problem: 22:15, change night. The statements service gets AccessDenied reading S3. Its role has s3:GetObject on arn:aws:s3:::hl-statements/*. The bucket is in another account, encrypted with SSE-KMS. The release manager says "give it admin". Write what you check, in order, what you refuse to do and what you say to the release manager.

Assignment

Add an AccessDenied triage card to your portfolio: one page with the five facts, the phrase-to-layer table, your ten-minute flow and three worked examples with fictional messages. Commit it as playbooks/access-denied-triage.md, and include your CloudTrailTriage tool with tests. Mark clearly that all IDs are fictional.

Solutions

Knowledge check

Interview questions

  • How do you troubleshoot an AccessDenied? Read the message for principal, action, resource and reason; confirm the real caller with get-caller-identity; find the layer; use CloudTrail and the simulator if unclear; fix the smallest thing through IaC and review.
  • What is the difference between implicit and explicit deny? Implicit deny is the default when nothing allows the request. Explicit deny is a Deny statement and always wins over any Allow.
  • When does IAM alone not decide access? When a resource policy, SCP, RCP, permissions boundary, session policy or VPC endpoint policy also applies. Cross-account requests need both accounts to allow.
  • Someone asks for administrator access to unblock a release. What do you do? Decline, find the missing action and resource, and make a scoped, reviewed change. Offer a timeline.
  • How would you reduce AccessDenied incidents? Test roles in lower environments with the real calls, validate policies with Access Analyzer in the pipeline, generate least-privilege policies from CloudTrail activity, and keep the guardrail documentation visible to developers.

Certification practice

Completion checklist

  • I can name the five facts in an AccessDenied and find each
  • I can map the reason phrase to the layer without looking it up
  • I know three reasons the caller is not the role I expected
  • I ran the triage tool tests and added one new case
  • I can explain why admin, wildcards and SCP exemptions are not fixes
  • I ran the cleanup steps and confirmed nothing tagged m17-playbook-accessdenied remains

Key takeaways

  • Read the message first: principal, action, resource, reason.
  • Confirm the real caller before editing any policy.
  • Match the reason phrase to the layer; cross-account needs both sides to allow.
  • The simulator and Access Analyzer help but do not replace the real error and CloudTrail.
  • Fix the smallest thing through code and review, then prevent recurrence and record it.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page