Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

AWS CloudTrail for operations: who changed what, and when

Use CloudTrail event history, a trail delivered to CloudWatch Logs and EventBridge rules to answer 'who changed this security group?' in minutes, alert on risky API calls, and keep audit trails separate from application logs.

intermediate⏱ 1 h 30 min🟢 FreeSOA-C03DOP-C02SCS-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.CloudTrail) · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what CloudTrail records, and separate management events, data events, Insights events and network activity events.
  • Find who made a change, from where and with which credentials, using event history in the console, the CLI and a small C# program.
  • Read a CloudTrail record field by field, including the userIdentity block for roles, IAM Identity Center and federated users.
  • Send a trail to CloudWatch Logs and build a metric filter, alarm and EventBridge rule for risky operations such as a security group change.
  • Explain how CloudTrail differs from application logging, and why a bank needs both.

Prerequisites

  • CloudTrail for banks builds the organisation-wide, immutable audit trail. This lesson is the operations view: using CloudTrail to troubleshoot and alert. Read that one for integrity validation, Object Lock and the log-archive account.
  • CloudWatch Logs for .NET and Alarms and composite alarms, because CloudTrail events are routed into log groups and alarmed on.
  • Tools: AWS CLI v2 with the learn-sandbox SSO profile, .NET 10 SDK.

Why this matters in banking

It is 14:10 on a Wednesday. The payments API starts timing out talking to its database. Nothing was deployed. After twenty minutes someone notices that the database security group no longer allows traffic from the application subnets. The obvious next question is not "how do we fix it" (you can fix it in a minute) but "who changed it, with what credentials, from which address, and was it an accident or an attack?".

Without CloudTrail you cannot answer. With it, the answer takes one query, and the same event records are the evidence an auditor or an incident reviewer will ask for. Operational teams use CloudTrail to find the cause of a fault. Security teams use the same data to detect misuse. This lesson is about the first use; Module 5 covers the second in depth.

The idea in plain language

Every call to an AWS API is a request: "create this bucket", "change this security group", "assume this role". CloudTrail writes down each request as a small JSON document: who asked, what they asked for, when, from which IP address and tool, whether it worked, and what came back.

Two ideas matter most:

  • It records control-plane calls by default. Actions that manage resources (create, modify, delete, assume a role, sign in) are management events. You get the last 90 days of them for free in event history with no setup.
  • It does not record what happens inside your application. Reading an S3 object or invoking a Lambda function is a data event, and those are off by default and cost money when you switch them on. What your own code did with a customer record is not in CloudTrail at all; that is your application log.

Analogy: CloudTrail is the bank's building access log: who badged into which room and at what time. Application logs are the CCTV of what people did once inside. The access log cannot tell you what they did in the vault, but it tells you which door to ask about.

What problem the AWS service solves

CloudTrail answers three questions that otherwise rely on memory and guesswork:

  1. Who did it? The principal, session, MFA status and source address.
  2. What exactly changed? The API name and request parameters.
  3. Did it work? The response elements, and the errorCode for failures, which also reveals denied attempts.

Event types, with honest cost notes:

Event typeWhat it recordsDefaultCost note
Management eventsControl-plane API calls (create, modify, delete, sign-in, STS)On, 90 days in event historyThe first copy delivered by a trail to S3 is free; extra copies are priced per 100,000 events ⚠ VERIFY current rate
Data eventsData-plane calls: S3 object operations, Lambda invokes, DynamoDB item operations and moreOffPriced per 100,000 events; can be very large for busy buckets
Insights eventsUnusual call volume or error rates for management (and optionally data) eventsOffPriced per events analysed
Network activity eventsNetwork-level API activity through VPC endpoints for supported servicesOffPriced per events delivered ⚠ VERIFY

At the time of checking, the pricing page lists management events as free for the first copy per trail and $2.00 per 100,000 for additional copies, data events at $0.10 per 100,000, and Insights at $0.35 per 100,000 management events analysed per insight type. ⚠ VERIFY all prices before you build a budget on them.

Limits that bite:

  • Event history shows management events only, for the past 90 days, one Region at a time, with a single attribute filter plus a time range. It cannot show data events.
  • A trail is separate from event history. To keep events longer than 90 days or to query them in your own tools, create a trail.
  • Delivery is not instant. CloudTrail typically delivers log files to S3 within minutes rather than seconds, so it is a forensic and near-real-time tool, not a transaction log. ⚠ VERIFY current typical delay for your Region.
  • CloudTrail Lake is closed to new customers since 2026-05-31. This lesson uses event history, trails with CloudWatch Logs and Athena instead. AWS points to CloudWatch's unified log management as the replacement for analytics.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Activity LogCloudTrail management events and event historyEvent history is 90 days and per Region; Activity Log is per subscription
Diagnostic settings for data-plane logsCloudTrail data events, plus service logs (S3 server access logs, VPC Flow Logs)Data events are opt-in and billed per event
Entra ID sign-in logs and audit logsCloudTrail ConsoleLogin and IAM events, IAM Identity Center eventsIdentity events are part of the same trail
Activity Log alertsEventBridge rules on CloudTrail events; metric filters on the trail's log groupTwo common patterns; EventBridge is event-driven, metric filters feed alarms
Log Analytics query over Activity LogAthena over trail logs in S3, or Logs Insights over the trail's log groupPick by retention and cost
Microsoft Defender for Cloud recommendationsSecurity Hub, GuardDuty (read CloudTrail)Detection services consume the same events

Architecture

The security-boundary view: the trail's S3 bucket and log group hold evidence, so write access must belong only to the CloudTrail service principal. The operations team gets read access; nobody gets delete access. In a real bank the trail lives in a separate log-archive account with Object Lock, as described in CloudTrail for banks.

Step-by-step implementation

Look at event history first

Before building anything, open event history and find your own last action. This proves the free path works and teaches you the record shape.

Read one record carefully

Check userIdentity, eventSource, eventName, sourceIPAddress, userAgent, requestParameters and errorCode. See the field guide below.

Answer a real question

In your sandbox, change a security group rule on purpose, then find your own change with the console, the CLI and the C# tool. This is Lab 11.5.

Create a trail with CloudWatch Logs delivery

Management events, all Regions where you work, a log group with 7-day retention and a KMS-encrypted bucket.

Add a metric filter, alarm and EventBridge rule

The metric filter and alarm give you a counted, thresholded signal. The EventBridge rule gives you an immediate notification with the full event attached.

Field guide to a CloudTrail record

FieldWhat it tells youWatch for
eventTimeUTC time of the callConvert from UTC before comparing with local logs
eventSource / eventNameThe service and API, such as ec2.amazonaws.com / AuthorizeSecurityGroupIngressConsole actions map to API names that are not always obvious
awsRegionWhere it happenedGlobal services such as IAM report us-east-1
sourceIPAddressCaller IP, or an AWS service name when the call came from another serviceA console action through a service can show a service name
userAgentTool: console, aws-cli, Boto3, aws-sdk-dotnet, Terraform, CDKQuickly separates humans from pipelines
userIdentity.typeRoot, IAMUser, AssumedRole, FederatedUser, IdentityCenterUser, AWSService and othersAny Root event in a bank is an incident until proven otherwise
userIdentity.arnPrincipal that made the call; for roles the ARN includes the session nameSession name is how you find the person behind a shared role
userIdentity.sessionContext.sessionIssuerWhich role the temporary credentials came fromUse with principalId to follow a session
userIdentity.sessionContext.attributes.mfaAuthenticatedWhether MFA was usedfalse on a privileged action deserves a look
userIdentity.sessionContext.sourceIdentityOriginal identity when sourceIdentity is enforced on role assumptionMakes shared-role audit possible
requestParametersWhat was asked, such as the group id and ruleMay be null for some services; sensitive values can be redacted
errorCode / errorMessagePresent when the call failed, such as AccessDeniedDenied attempts are early warning signs
readOnlyTrue for read callsFilter on false to see changes only

For IAM Identity Center users the record carries type of IdentityCenterUser with an onBehalfOf block holding the user id and the identity store ARN, rather than a user name. Map user ids to people through the identity store.

AWS Console instructions

Console → CloudTrail → Event history.

  1. Choose the Region where the change happened (event history is per Region).
  2. In Lookup attributes, choose Event name and type AuthorizeSecurityGroupIngress (or choose Resource name and paste the security group id). Only one attribute filter is allowed.
  3. Set the time range, then open the event and use Event record for the raw JSON.
  4. To keep events beyond 90 days: CloudTrail → Trails → Create trail. Choose Management events, Read and Write, enable CloudWatch Logs delivery, SSE-KMS encryption and log file validation.
  5. CloudWatch → Log groups → (trail log group) → Metric filters → Create metric filter with the pattern in the CLI section.
  6. EventBridge → Rules → Create rule, event pattern for AWS API Call via CloudTrail.

AWS CLI

# 1) Who changed any security group rule in the last 24 hours? (event history, management events only)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress \
  --start-time "$(date -u -d '-24 hours' +%Y-%m-%dT%H:%M:%SZ)" \
  --max-results 20 \
  --query 'Events[].{time:EventTime,who:Username,id:EventId}' \
  --profile learn-sandbox --region us-east-1

# 2) Everything that happened to one specific security group (one attribute only).
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0 \
  --profile learn-sandbox --region us-east-1

# 3) Full record of one event, pretty-printed. The CloudTrailEvent field is a JSON string.
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventId,AttributeValue=11111111-2222-3333-4444-555555555555 \
  --profile learn-sandbox --region us-east-1 \
  --query 'Events[0].CloudTrailEvent' --output text | python -m json.tool

# 4) Everything done by one principal (role session names show up as Username).
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=jane.doe \
  --profile learn-sandbox --region us-east-1

# 5) Create a trail (management events, KMS, validation, CloudWatch Logs) -- see the IaC section;
#    the CLI route needs the bucket policy and role first, so IaC is easier.

# 6) Check the status of a trail, including the last delivery error.
aws cloudtrail get-trail-status --name hl-m11-trail --profile learn-sandbox --region us-east-1

# 7) Logs Insights over the trail log group: top callers of changes in the last hour.
aws logs start-query \
  --log-group-name /hl/m11/cloudtrail \
  --start-time $(date -u -d '-1 hour' +%s) --end-time $(date -u +%s) \
  --query-string 'fields @timestamp, eventName, userIdentity.arn, sourceIPAddress
    | filter readOnly = 0
    | stats count() as calls by userIdentity.arn, eventName
    | sort calls desc | limit 20' \
  --profile learn-sandbox --region us-east-1

The lookup attribute keys accepted by lookup-events are, to my knowledge, EventId, EventName, ReadOnly, Username, ResourceType, ResourceName, EventSource and AccessKeyId. ⚠ VERIFY against aws cloudtrail lookup-events help for your CLI version. Event history lookup is rate limited, so do not loop over it in a script. ⚠ VERIFY the current limit.

Infrastructure as Code

The stack creates a KMS key, a trail with file validation and CloudWatch Logs delivery at 7-day retention, a metric filter and alarm for security group changes, and an EventBridge rule that publishes the raw event to SNS.

CloudTrailOpsStack.cs
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.CloudTrail;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.SNS.Subscriptions;
using Constructs;

namespace Harbourline.Monitoring.Infra;

public sealed class CloudTrailOpsStack : Stack
{
    public CloudTrailOpsStack(Construct scope, string id, IStackProps? props = null)
        : base(scope, id, props)
    {
        // Qualified, because Amazon.CDK.Tags.Of(this) would bind to the stack's Tags property (CS0176).
        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l9");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");

        var key = new Key(this, "TrailKey", new KeyProps
        {
            EnableKeyRotation = true,
            Description = "Encrypts the Harbourline lab CloudTrail bucket and log group",
            RemovalPolicy = RemovalPolicy.DESTROY, // learning shortcut
        });

        var bucket = new Bucket(this, "TrailBucket", new BucketProps
        {
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            Encryption = BucketEncryption.KMS,
            EncryptionKey = key,
            EnforceSSL = true,
            Versioned = true,
            RemovalPolicy = RemovalPolicy.DESTROY, // learning shortcut
            AutoDeleteObjects = true,              // learning shortcut
            LifecycleRules = new[]
            {
                new LifecycleRule { Expiration = Duration.Days(30) }, // lab only
            },
        });

        var logGroup = new LogGroup(this, "TrailLogGroup", new Amazon.CDK.AWS.Logs.LogGroupProps
        {
            LogGroupName = "/hl/m11/cloudtrail",
            Retention = RetentionDays.ONE_WEEK,
            EncryptionKey = key,
            RemovalPolicy = RemovalPolicy.DESTROY,
        });

        var trail = new Trail(this, "Trail", new TrailProps
        {
            TrailName = "hl-m11-trail",
            Bucket = bucket,
            EncryptionKey = key,
            EnableFileValidation = true,
            IncludeGlobalServiceEvents = true,
            IsMultiRegionTrail = false, // single Region keeps the lab small
            ManagementEvents = ReadWriteType.ALL,
            SendToCloudWatchLogs = true,
            CloudWatchLogGroup = logGroup,
        });

        var topic = new Topic(this, "OpsTopic", new TopicProps { TopicName = "hl-m11-cloudtrail-alerts" });
        topic.AddSubscription(new EmailSubscription("oncall@example.com")); // replace with your address

        // Metric filter: count security group changes in the trail log group.
        var sgFilter = new MetricFilter(this, "SgChangeFilter", new MetricFilterProps
        {
            LogGroup = logGroup,
            MetricNamespace = "Harbourline/CloudTrail",
            MetricName = "SecurityGroupChanges",
            MetricValue = "1",
            FilterPattern = FilterPattern.Literal(
                "{ ($.eventName = AuthorizeSecurityGroupIngress) || ($.eventName = AuthorizeSecurityGroupEgress) " +
                "|| ($.eventName = RevokeSecurityGroupIngress) || ($.eventName = RevokeSecurityGroupEgress) " +
                "|| ($.eventName = CreateSecurityGroup) || ($.eventName = DeleteSecurityGroup) }"),
        });

        var alarm = new Alarm(this, "SgChangeAlarm", new AlarmProps
        {
            AlarmName = "hl-m11-security-group-changed",
            AlarmDescription = "A security group was created, changed or deleted. Runbook: who changed it, was it approved?",
            Metric = sgFilter.Metric(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(5) }),
            Threshold = 1,
            EvaluationPeriods = 1,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING, // no changes means all is well
        });
        alarm.AddAlarmAction(new SnsAction(topic));

        // EventBridge: near-real-time notification carrying the full CloudTrail event.
        new Rule(this, "SgChangeRule", new RuleProps
        {
            Description = "Security group changes, straight from CloudTrail",
            EventPattern = new EventPattern
            {
                Source = new[] { "aws.ec2" },
                DetailType = new[] { "AWS API Call via CloudTrail" },
                Detail = new Dictionary<string, object>
                {
                    ["eventSource"] = new[] { "ec2.amazonaws.com" },
                    ["eventName"] = new[]
                    {
                        "AuthorizeSecurityGroupIngress", "AuthorizeSecurityGroupEgress",
                        "RevokeSecurityGroupIngress", "RevokeSecurityGroupEgress",
                        "CreateSecurityGroup", "DeleteSecurityGroup",
                    },
                },
            },
            Targets = new IRuleTarget[] { new SnsTopic(topic) },
        });
    }
}

Note: in the CDK version, the CDK-created topic policy is added for you when you use SnsAction and SnsTopic. In Terraform you must write it yourself, as shown.

Complete code

A small command-line tool that answers "who changed this security group?" using the AWS SDK for .NET v4. It reads from event history, so it works with no trail at all. Larger lab code belongs in labs/; this tool is short enough to show in full.

WhoChanged.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <!-- Look up the current 4.x version at https://api.nuget.org/v3-flatcontainer/awssdk.cloudtrail/index.json -->
    <PackageReference Include="AWSSDK.CloudTrail" Version="4.*" />
  </ItemGroup>
</Project>
Program.cs
using System.Text.Json;
using Amazon;
using Amazon.CloudTrail;
using Amazon.CloudTrail.Model;

// Usage: dotnet run -- <security-group-id> [hours]
if (args.Length < 1)
{
    Console.Error.WriteLine("Usage: WhoChanged <security-group-id> [hours-back=24]");
    return 1;
}

var groupId = args[0];
var hoursBack = args.Length > 1 && int.TryParse(args[1], out var h) ? h : 24;

// Credentials come from the default chain (AWS_PROFILE=learn-sandbox), never from code.
using var client = new AmazonCloudTrailClient(RegionEndpoint.USEast1);

var request = new LookupEventsRequest
{
    // Event history allows ONE lookup attribute plus a time range.
    LookupAttributes =
    [
        new LookupAttribute { AttributeKey = LookupAttributeKey.ResourceName, AttributeValue = groupId },
    ],
    StartTime = DateTime.UtcNow.AddHours(-hoursBack),
    EndTime = DateTime.UtcNow,
    MaxResults = 50,
};

var seen = 0;
do
{
    var response = await client.LookupEventsAsync(request);

    foreach (var ev in response.Events ?? [])
    {
        seen++;

        // The Event object has a few top-level fields; the rich detail is a JSON string.
        using var doc = JsonDocument.Parse(ev.CloudTrailEvent);
        var root = doc.RootElement;

        var identity = root.GetProperty("userIdentity");
        var type = identity.GetProperty("type").GetString();
        var arn = identity.TryGetProperty("arn", out var a) ? a.GetString() : "(no arn)";
        var mfa = identity.TryGetProperty("sessionContext", out var sc)
                  && sc.TryGetProperty("attributes", out var attrs)
                  && attrs.TryGetProperty("mfaAuthenticated", out var m)
            ? m.GetString()
            : "n/a";
        var ip = root.TryGetProperty("sourceIPAddress", out var ipEl) ? ipEl.GetString() : "?";
        var agent = root.TryGetProperty("userAgent", out var uaEl) ? uaEl.GetString() : "?";
        var error = root.TryGetProperty("errorCode", out var errEl) ? errEl.GetString() : null;

        Console.WriteLine($"{ev.EventTime:u}  {ev.EventName}");
        Console.WriteLine($"    who   : {type} {arn}  (MFA: {mfa})");
        Console.WriteLine($"    from  : {ip}  using {agent}");
        Console.WriteLine($"    result: {(error is null ? "success" : "FAILED " + error)}");
    }

    request.NextToken = response.NextToken; // null when no more pages
} while (request.NextToken is not null);

Console.WriteLine(seen == 0
    ? "No events found. Check the Region, the group id and the time window."
    : $"{seen} event(s).");
return 0;

Code explained line by line

  1. The package is AWSSDK.CloudTrail; the version range 4.* keeps you on the v4 line, but pin an exact version in a real project.
  2. AmazonCloudTrailClient(RegionEndpoint.USEast1) uses the default credential chain. Locally that is the SSO profile in AWS_PROFILE; on ECS it is the task role. No keys appear in code.
  3. LookupAttributes takes exactly one filter in event history. ResourceName matches the id of the resource named in the event.
  4. StartTime and EndTime bound the 90-day history window. Lookups cover the Region of the client only.
  5. ev.CloudTrailEvent is the full record as a JSON string. Parsing it with System.Text.Json avoids depending on the SDK's partial typed view.
  6. TryGetProperty guards every optional field. arn and sessionContext are not present for every identity type, and errorCode exists only on failures.
  7. The do { ... } while (NextToken is not null) loop follows pagination. Forgetting it is the classic reason a lookup "finds nothing" when results spill onto a second page.
  8. Exit code 1 on bad usage lets you use the tool in scripts.

Security considerations

⚠ Learning shortcut — not for production

The lab trail lives in the same account as the workload, uses RemovalPolicy.DESTROY and AutoDeleteObjects, expires objects after 30 days and covers a single Region. A real bank uses an organisation trail in all Regions delivered to a separate log-archive account with Object Lock, long retention and no delete rights for anyone in the workload account. See CloudTrail for banks.

  • CloudTrail records are evidence. Protect the bucket and log group from change. Enable log file validation so tampering is detectable.
  • Records can contain sensitive values. Request parameters occasionally hold secrets or personal identifiers (for example in some service-specific calls). Treat the log group as sensitive: restrict read access and set retention deliberately, in line with what your data-protection officer agrees under UK GDPR.
  • Alert on the dangerous few, not everything. Root sign-in, ConsoleLogin without MFA, StopLogging, DeleteTrail, PutBucketPolicy changes, IAM policy changes and security group changes. Alerting on every event creates noise that nobody reads.
  • Separation of duties. The people who can change infrastructure should not be the only people who can read or alter the trail.
  • Do not assume attribution. A shared role hides the person unless you enforce session names or sourceIdentity. Plan that before you need it.

Banking use case

Harbourline's platform team keeps a short list of CloudTrail-driven alerts, each with an owner and a runbook:

Trigger (eventName)Why it mattersRoute
ConsoleLogin by RootRoot should never be used day to dayPage security on-call immediately
ConsoleLogin success without MFAWeak authentication on a privileged pathTicket plus email
StopLogging, DeleteTrail, UpdateTrailSomeone is reducing audit visibilityPage security on-call
AuthorizeSecurityGroupIngress with 0.0.0.0/0 on non-web portsPossible exposure of a data tierPage platform on-call
PutBucketPolicy, PutBucketAcl, DeleteBucketPublicAccessBlockData exposure riskTicket, auto-check with Config
AttachRolePolicy with admin-level policiesPrivilege escalationPage security on-call

Operations use the same data differently: when a deployment goes wrong, the first query is "what API calls did the deployment role make in the last hour, and which failed?". A filter on errorCode quickly exposes missing permissions that cause 5xx errors in downstream services.

Audit trail versus application log

CloudTrailApplication log
RecordsAWS API calls (who changed which resource)What your code did (business events, errors, latency)
Written byAWS, automaticallyYour code, through ILogger
Contains customer data?Rarely, but request parameters canOften, so you must mask it
Typical question"Who opened port 1433 to the internet?""Why did payment PAY-… fail?"
Tamper resistanceLog file validation, Object Lock in a separate accountWhatever you build
Retention driverAudit and regulatory expectationsTroubleshooting and cost
Complete?Not for data-plane actions unless data events are onOnly what you chose to log

A bank also needs a business audit trail (the audit_events table in the course's domain model records actor, action, resource, outcome, source IP and correlation ID). CloudTrail does not replace it: it knows an engineer changed a DynamoDB table, not that a customer's standing order was edited.

Common mistakes

  • Searching the wrong Region. Event history is per Region. A global service such as IAM reports in us-east-1.
  • Expecting data events in event history. They are never there; you need a trail with data event selectors.
  • Searching for the person, finding only a role. Use the session name in the arn, or sourceIdentity, to get to the person.
  • Using a trail without validation or encryption. Turn on file validation and SSE-KMS from the start.
  • Alert fatigue. Every Describe* call is a management event. Filter on readOnly = false.
  • Treating CloudTrail as real time. For immediate reaction, use the EventBridge rule; use the log group for counting.
  • Forgetting Lake is closed. Do not design new analytics on CloudTrail Lake.

Troubleshooting

SymptomLikely causeFix
Lookup returns nothingWrong Region, wrong id, or event outside 90 daysSwitch Region, confirm the id, widen the time range
LookupEvents throttledToo many requests per secondBack off and retry; for bulk analysis use the trail in S3 with Athena
Trail shows IsLogging: falseLogging stopped or delivery failedget-trail-status, read LatestDeliveryError, fix bucket policy or KMS policy
No events in the trail log groupRole missing logs:PutLogEvents, or wrong log group ARNVerify the role and that the ARN ends with :*
Metric filter never triggersPattern syntax error, or JSON field mismatchTest with the console Test pattern feature on a real event
Alarm in INSUFFICIENT_DATANo matching events yetExpected for a rare event; use treatMissingData = notBreaching
Alarm fires but no emailTopic subscription unconfirmed, or topic policy missingConfirm the subscription; add the publish policy for cloudwatch.amazonaws.com
EventBridge rule never matchesEvent comes from a different Region, or pattern mismatchCreate the rule in the Region where the API call occurs
AccessDenied on lookup-eventsMissing cloudtrail:LookupEventsAdd the permission to the operator role

Cost and cleanup

ItemLabelEstimate
Event history lookupsfreeNo charge
First copy of management events in one trail to S3freeSee the pricing note above ⚠ VERIFY
S3 storage for trail filescreditsPennies at lab volume
CloudWatch Logs ingestion of trail eventscreditsA few MB in a quiet sandbox; set 7-day retention
Metric filter metric, one alarmcreditsOne custom metric and one alarm for the lab duration
KMS keycreditsPer-key monthly fee, delete after the lab ⚠ VERIFY price
SNS emailfreeNegligible

Free alternative: skip the stack and use only event history, the CLI and the C# tool above. Everything in Lab 11.5 works with no trail.

# Tear down the stack (CDK) ...
cdk destroy CloudTrailOpsStack --profile learn-sandbox
# ... or Terraform
terraform destroy

# KMS keys are scheduled for deletion, not removed immediately.
aws kms list-keys --profile learn-sandbox --region us-east-1
# (if a key remains) aws kms schedule-key-deletion --key-id <key-id> --pending-window-in-days 7 --profile learn-sandbox --region us-east-1

# Confirm nothing tagged Project=cea-lab is left.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'

Exercises

  • 🟢 Guided: In your sandbox create a security group, add an inbound rule for TCP 8080 from your own IP, then delete the group. Find all three events in event history, and note for each the identity type, source IP and user agent.
  • 🟡 Challenge: Extend the C# tool to take an --event-name argument instead of a resource id, and print only events where readOnly is false and errorCode is present. Then explain why that finds permission problems.
  • 🔴 Live problem: Ticket: "Overnight, payments started failing. The platform team says nobody touched anything." You have a trail log group and event history. Describe the five queries you would run in the first ten minutes, in order, and what each outcome would tell you.

Assignment

Write a "CloudTrail alert catalogue" for Harbourline: eight event patterns, each with the metric filter or EventBridge pattern, severity, who is notified, the first action in the runbook, and whether you would alert in real time (EventBridge) or on counts (metric filter and alarm). Commit it with the IaC that creates the first three.

Solutions

Knowledge check

Interview questions

  • A security group was opened to the internet last night. How do you find out who did it? Look up the group id in event history (or the trail's log group), read userIdentity, sourceIPAddress and userAgent, resolve the session to a person, and check whether a pipeline or console made the change.
  • Why does CloudTrail not replace application logging? It records control-plane API calls by default, not what your code did with customer data. They answer different questions and have different retention and masking needs.
  • How would you make sure no one can silently turn CloudTrail off? Alert on StopLogging/DeleteTrail, restrict those permissions with an SCP in a multi-account setup, and deliver to a log-archive account with Object Lock.
  • What are the downsides of data events? They are off by default and priced per event, so busy buckets or tables can generate large bills; enable them selectively on sensitive resources.

Certification practice

Completion checklist

  • I can explain management events, data events and Insights events, and which are free
  • I found my own change in event history and read its userIdentity
  • I can say how CloudTrail differs from an application log and from a business audit trail
  • I built (or designed) a metric filter, alarm and EventBridge rule for a risky action
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • Event history gives you 90 days of management events per Region for free, with no setup.
  • A record answers who, what, when, from where and whether it worked; the userIdentity block is the part to learn.
  • Use EventBridge for immediate single-event alerts and metric filters for counted thresholds.
  • CloudTrail is not an application log and not a business audit trail; a bank needs all three.
  • Protect trail data like evidence: encryption, validation and separate-account storage.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page