AWS CloudTrail for operations: who changed what, and when
Use CloudTrail event history, a trail delivered to CloudWatch Logs and EventBridge rules to answer 'who changed this security group?' in minutes, alert on risky API calls, and keep audit trails separate from application logs.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.CloudTrail) · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what CloudTrail records, and separate management events, data events, Insights events and network activity events.
- Find who made a change, from where and with which credentials, using event history in the console, the CLI and a small C# program.
- Read a CloudTrail record field by field, including the
userIdentityblock for roles, IAM Identity Center and federated users. - Send a trail to CloudWatch Logs and build a metric filter, alarm and EventBridge rule for risky operations such as a security group change.
- Explain how CloudTrail differs from application logging, and why a bank needs both.
Prerequisites
- CloudTrail for banks builds the organisation-wide, immutable audit trail. This lesson is the operations view: using CloudTrail to troubleshoot and alert. Read that one for integrity validation, Object Lock and the log-archive account.
- CloudWatch Logs for .NET and Alarms and composite alarms, because CloudTrail events are routed into log groups and alarmed on.
- Tools: AWS CLI v2 with the
learn-sandboxSSO profile, .NET 10 SDK.
Why this matters in banking
It is 14:10 on a Wednesday. The payments API starts timing out talking to its database. Nothing was deployed. After twenty minutes someone notices that the database security group no longer allows traffic from the application subnets. The obvious next question is not "how do we fix it" (you can fix it in a minute) but "who changed it, with what credentials, from which address, and was it an accident or an attack?".
Without CloudTrail you cannot answer. With it, the answer takes one query, and the same event records are the evidence an auditor or an incident reviewer will ask for. Operational teams use CloudTrail to find the cause of a fault. Security teams use the same data to detect misuse. This lesson is about the first use; Module 5 covers the second in depth.
The idea in plain language
Every call to an AWS API is a request: "create this bucket", "change this security group", "assume this role". CloudTrail writes down each request as a small JSON document: who asked, what they asked for, when, from which IP address and tool, whether it worked, and what came back.
Two ideas matter most:
- It records control-plane calls by default. Actions that manage resources (create, modify, delete, assume a role, sign in) are management events. You get the last 90 days of them for free in event history with no setup.
- It does not record what happens inside your application. Reading an S3 object or invoking a Lambda function is a data event, and those are off by default and cost money when you switch them on. What your own code did with a customer record is not in CloudTrail at all; that is your application log.
Analogy: CloudTrail is the bank's building access log: who badged into which room and at what time. Application logs are the CCTV of what people did once inside. The access log cannot tell you what they did in the vault, but it tells you which door to ask about.
What problem the AWS service solves
CloudTrail answers three questions that otherwise rely on memory and guesswork:
- Who did it? The principal, session, MFA status and source address.
- What exactly changed? The API name and request parameters.
- Did it work? The response elements, and the
errorCodefor failures, which also reveals denied attempts.
Event types, with honest cost notes:
| Event type | What it records | Default | Cost note |
|---|---|---|---|
| Management events | Control-plane API calls (create, modify, delete, sign-in, STS) | On, 90 days in event history | The first copy delivered by a trail to S3 is free; extra copies are priced per 100,000 events ⚠ VERIFY current rate |
| Data events | Data-plane calls: S3 object operations, Lambda invokes, DynamoDB item operations and more | Off | Priced per 100,000 events; can be very large for busy buckets |
| Insights events | Unusual call volume or error rates for management (and optionally data) events | Off | Priced per events analysed |
| Network activity events | Network-level API activity through VPC endpoints for supported services | Off | Priced per events delivered ⚠ VERIFY |
At the time of checking, the pricing page lists management events as free for the first copy per trail and $2.00 per 100,000 for additional copies, data events at $0.10 per 100,000, and Insights at $0.35 per 100,000 management events analysed per insight type. ⚠ VERIFY all prices before you build a budget on them.
Limits that bite:
- Event history shows management events only, for the past 90 days, one Region at a time, with a single attribute filter plus a time range. It cannot show data events.
- A trail is separate from event history. To keep events longer than 90 days or to query them in your own tools, create a trail.
- Delivery is not instant. CloudTrail typically delivers log files to S3 within minutes rather than seconds, so it is a forensic and near-real-time tool, not a transaction log. ⚠ VERIFY current typical delay for your Region.
- CloudTrail Lake is closed to new customers since 2026-05-31. This lesson uses event history, trails with CloudWatch Logs and Athena instead. AWS points to CloudWatch's unified log management as the replacement for analytics.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Activity Log | CloudTrail management events and event history | Event history is 90 days and per Region; Activity Log is per subscription |
| Diagnostic settings for data-plane logs | CloudTrail data events, plus service logs (S3 server access logs, VPC Flow Logs) | Data events are opt-in and billed per event |
| Entra ID sign-in logs and audit logs | CloudTrail ConsoleLogin and IAM events, IAM Identity Center events | Identity events are part of the same trail |
| Activity Log alerts | EventBridge rules on CloudTrail events; metric filters on the trail's log group | Two common patterns; EventBridge is event-driven, metric filters feed alarms |
| Log Analytics query over Activity Log | Athena over trail logs in S3, or Logs Insights over the trail's log group | Pick by retention and cost |
| Microsoft Defender for Cloud recommendations | Security Hub, GuardDuty (read CloudTrail) | Detection services consume the same events |
Architecture
The security-boundary view: the trail's S3 bucket and log group hold evidence, so write access must belong only to the CloudTrail service principal. The operations team gets read access; nobody gets delete access. In a real bank the trail lives in a separate log-archive account with Object Lock, as described in CloudTrail for banks.
Step-by-step implementation
Look at event history first
Before building anything, open event history and find your own last action. This proves the free path works and teaches you the record shape.
Read one record carefully
Check userIdentity, eventSource, eventName, sourceIPAddress, userAgent, requestParameters and errorCode. See the field guide below.
Answer a real question
In your sandbox, change a security group rule on purpose, then find your own change with the console, the CLI and the C# tool. This is Lab 11.5.
Create a trail with CloudWatch Logs delivery
Management events, all Regions where you work, a log group with 7-day retention and a KMS-encrypted bucket.
Add a metric filter, alarm and EventBridge rule
The metric filter and alarm give you a counted, thresholded signal. The EventBridge rule gives you an immediate notification with the full event attached.
Field guide to a CloudTrail record
| Field | What it tells you | Watch for |
|---|---|---|
eventTime | UTC time of the call | Convert from UTC before comparing with local logs |
eventSource / eventName | The service and API, such as ec2.amazonaws.com / AuthorizeSecurityGroupIngress | Console actions map to API names that are not always obvious |
awsRegion | Where it happened | Global services such as IAM report us-east-1 |
sourceIPAddress | Caller IP, or an AWS service name when the call came from another service | A console action through a service can show a service name |
userAgent | Tool: console, aws-cli, Boto3, aws-sdk-dotnet, Terraform, CDK | Quickly separates humans from pipelines |
userIdentity.type | Root, IAMUser, AssumedRole, FederatedUser, IdentityCenterUser, AWSService and others | Any Root event in a bank is an incident until proven otherwise |
userIdentity.arn | Principal that made the call; for roles the ARN includes the session name | Session name is how you find the person behind a shared role |
userIdentity.sessionContext.sessionIssuer | Which role the temporary credentials came from | Use with principalId to follow a session |
userIdentity.sessionContext.attributes.mfaAuthenticated | Whether MFA was used | false on a privileged action deserves a look |
userIdentity.sessionContext.sourceIdentity | Original identity when sourceIdentity is enforced on role assumption | Makes shared-role audit possible |
requestParameters | What was asked, such as the group id and rule | May be null for some services; sensitive values can be redacted |
errorCode / errorMessage | Present when the call failed, such as AccessDenied | Denied attempts are early warning signs |
readOnly | True for read calls | Filter on false to see changes only |
For IAM Identity Center users the record carries type of IdentityCenterUser with an onBehalfOf block holding the user id and the identity store ARN, rather than a user name. Map user ids to people through the identity store.
AWS Console instructions
Console → CloudTrail → Event history.
- Choose the Region where the change happened (event history is per Region).
- In Lookup attributes, choose Event name and type
AuthorizeSecurityGroupIngress(or choose Resource name and paste the security group id). Only one attribute filter is allowed. - Set the time range, then open the event and use Event record for the raw JSON.
- To keep events beyond 90 days: CloudTrail → Trails → Create trail. Choose Management events, Read and Write, enable CloudWatch Logs delivery, SSE-KMS encryption and log file validation.
- CloudWatch → Log groups → (trail log group) → Metric filters → Create metric filter with the pattern in the CLI section.
- EventBridge → Rules → Create rule, event pattern for
AWS API Call via CloudTrail.
AWS CLI
# 1) Who changed any security group rule in the last 24 hours? (event history, management events only)
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress \
--start-time "$(date -u -d '-24 hours' +%Y-%m-%dT%H:%M:%SZ)" \
--max-results 20 \
--query 'Events[].{time:EventTime,who:Username,id:EventId}' \
--profile learn-sandbox --region us-east-1
# 2) Everything that happened to one specific security group (one attribute only).
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0 \
--profile learn-sandbox --region us-east-1
# 3) Full record of one event, pretty-printed. The CloudTrailEvent field is a JSON string.
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventId,AttributeValue=11111111-2222-3333-4444-555555555555 \
--profile learn-sandbox --region us-east-1 \
--query 'Events[0].CloudTrailEvent' --output text | python -m json.tool
# 4) Everything done by one principal (role session names show up as Username).
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=Username,AttributeValue=jane.doe \
--profile learn-sandbox --region us-east-1
# 5) Create a trail (management events, KMS, validation, CloudWatch Logs) -- see the IaC section;
# the CLI route needs the bucket policy and role first, so IaC is easier.
# 6) Check the status of a trail, including the last delivery error.
aws cloudtrail get-trail-status --name hl-m11-trail --profile learn-sandbox --region us-east-1
# 7) Logs Insights over the trail log group: top callers of changes in the last hour.
aws logs start-query \
--log-group-name /hl/m11/cloudtrail \
--start-time $(date -u -d '-1 hour' +%s) --end-time $(date -u +%s) \
--query-string 'fields @timestamp, eventName, userIdentity.arn, sourceIPAddress
| filter readOnly = 0
| stats count() as calls by userIdentity.arn, eventName
| sort calls desc | limit 20' \
--profile learn-sandbox --region us-east-1The lookup attribute keys accepted by lookup-events are, to my knowledge, EventId, EventName, ReadOnly, Username, ResourceType, ResourceName, EventSource and AccessKeyId. ⚠ VERIFY against aws cloudtrail lookup-events help for your CLI version. Event history lookup is rate limited, so do not loop over it in a script. ⚠ VERIFY the current limit.
Infrastructure as Code
The stack creates a KMS key, a trail with file validation and CloudWatch Logs delivery at 7-day retention, a metric filter and alarm for security group changes, and an EventBridge rule that publishes the raw event to SNS.
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.CloudTrail;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.SNS.Subscriptions;
using Constructs;
namespace Harbourline.Monitoring.Infra;
public sealed class CloudTrailOpsStack : Stack
{
public CloudTrailOpsStack(Construct scope, string id, IStackProps? props = null)
: base(scope, id, props)
{
// Qualified, because Amazon.CDK.Tags.Of(this) would bind to the stack's Tags property (CS0176).
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l9");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
var key = new Key(this, "TrailKey", new KeyProps
{
EnableKeyRotation = true,
Description = "Encrypts the Harbourline lab CloudTrail bucket and log group",
RemovalPolicy = RemovalPolicy.DESTROY, // learning shortcut
});
var bucket = new Bucket(this, "TrailBucket", new BucketProps
{
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
Encryption = BucketEncryption.KMS,
EncryptionKey = key,
EnforceSSL = true,
Versioned = true,
RemovalPolicy = RemovalPolicy.DESTROY, // learning shortcut
AutoDeleteObjects = true, // learning shortcut
LifecycleRules = new[]
{
new LifecycleRule { Expiration = Duration.Days(30) }, // lab only
},
});
var logGroup = new LogGroup(this, "TrailLogGroup", new Amazon.CDK.AWS.Logs.LogGroupProps
{
LogGroupName = "/hl/m11/cloudtrail",
Retention = RetentionDays.ONE_WEEK,
EncryptionKey = key,
RemovalPolicy = RemovalPolicy.DESTROY,
});
var trail = new Trail(this, "Trail", new TrailProps
{
TrailName = "hl-m11-trail",
Bucket = bucket,
EncryptionKey = key,
EnableFileValidation = true,
IncludeGlobalServiceEvents = true,
IsMultiRegionTrail = false, // single Region keeps the lab small
ManagementEvents = ReadWriteType.ALL,
SendToCloudWatchLogs = true,
CloudWatchLogGroup = logGroup,
});
var topic = new Topic(this, "OpsTopic", new TopicProps { TopicName = "hl-m11-cloudtrail-alerts" });
topic.AddSubscription(new EmailSubscription("oncall@example.com")); // replace with your address
// Metric filter: count security group changes in the trail log group.
var sgFilter = new MetricFilter(this, "SgChangeFilter", new MetricFilterProps
{
LogGroup = logGroup,
MetricNamespace = "Harbourline/CloudTrail",
MetricName = "SecurityGroupChanges",
MetricValue = "1",
FilterPattern = FilterPattern.Literal(
"{ ($.eventName = AuthorizeSecurityGroupIngress) || ($.eventName = AuthorizeSecurityGroupEgress) " +
"|| ($.eventName = RevokeSecurityGroupIngress) || ($.eventName = RevokeSecurityGroupEgress) " +
"|| ($.eventName = CreateSecurityGroup) || ($.eventName = DeleteSecurityGroup) }"),
});
var alarm = new Alarm(this, "SgChangeAlarm", new AlarmProps
{
AlarmName = "hl-m11-security-group-changed",
AlarmDescription = "A security group was created, changed or deleted. Runbook: who changed it, was it approved?",
Metric = sgFilter.Metric(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(5) }),
Threshold = 1,
EvaluationPeriods = 1,
ComparisonOperator = ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING, // no changes means all is well
});
alarm.AddAlarmAction(new SnsAction(topic));
// EventBridge: near-real-time notification carrying the full CloudTrail event.
new Rule(this, "SgChangeRule", new RuleProps
{
Description = "Security group changes, straight from CloudTrail",
EventPattern = new EventPattern
{
Source = new[] { "aws.ec2" },
DetailType = new[] { "AWS API Call via CloudTrail" },
Detail = new Dictionary<string, object>
{
["eventSource"] = new[] { "ec2.amazonaws.com" },
["eventName"] = new[]
{
"AuthorizeSecurityGroupIngress", "AuthorizeSecurityGroupEgress",
"RevokeSecurityGroupIngress", "RevokeSecurityGroupEgress",
"CreateSecurityGroup", "DeleteSecurityGroup",
},
},
},
Targets = new IRuleTarget[] { new SnsTopic(topic) },
});
}
}Note: in the CDK version, the CDK-created topic policy is added for you when you use SnsAction and SnsTopic. In Terraform you must write it yourself, as shown.
Complete code
A small command-line tool that answers "who changed this security group?" using the AWS SDK for .NET v4. It reads from event history, so it works with no trail at all. Larger lab code belongs in labs/; this tool is short enough to show in full.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<!-- Look up the current 4.x version at https://api.nuget.org/v3-flatcontainer/awssdk.cloudtrail/index.json -->
<PackageReference Include="AWSSDK.CloudTrail" Version="4.*" />
</ItemGroup>
</Project>using System.Text.Json;
using Amazon;
using Amazon.CloudTrail;
using Amazon.CloudTrail.Model;
// Usage: dotnet run -- <security-group-id> [hours]
if (args.Length < 1)
{
Console.Error.WriteLine("Usage: WhoChanged <security-group-id> [hours-back=24]");
return 1;
}
var groupId = args[0];
var hoursBack = args.Length > 1 && int.TryParse(args[1], out var h) ? h : 24;
// Credentials come from the default chain (AWS_PROFILE=learn-sandbox), never from code.
using var client = new AmazonCloudTrailClient(RegionEndpoint.USEast1);
var request = new LookupEventsRequest
{
// Event history allows ONE lookup attribute plus a time range.
LookupAttributes =
[
new LookupAttribute { AttributeKey = LookupAttributeKey.ResourceName, AttributeValue = groupId },
],
StartTime = DateTime.UtcNow.AddHours(-hoursBack),
EndTime = DateTime.UtcNow,
MaxResults = 50,
};
var seen = 0;
do
{
var response = await client.LookupEventsAsync(request);
foreach (var ev in response.Events ?? [])
{
seen++;
// The Event object has a few top-level fields; the rich detail is a JSON string.
using var doc = JsonDocument.Parse(ev.CloudTrailEvent);
var root = doc.RootElement;
var identity = root.GetProperty("userIdentity");
var type = identity.GetProperty("type").GetString();
var arn = identity.TryGetProperty("arn", out var a) ? a.GetString() : "(no arn)";
var mfa = identity.TryGetProperty("sessionContext", out var sc)
&& sc.TryGetProperty("attributes", out var attrs)
&& attrs.TryGetProperty("mfaAuthenticated", out var m)
? m.GetString()
: "n/a";
var ip = root.TryGetProperty("sourceIPAddress", out var ipEl) ? ipEl.GetString() : "?";
var agent = root.TryGetProperty("userAgent", out var uaEl) ? uaEl.GetString() : "?";
var error = root.TryGetProperty("errorCode", out var errEl) ? errEl.GetString() : null;
Console.WriteLine($"{ev.EventTime:u} {ev.EventName}");
Console.WriteLine($" who : {type} {arn} (MFA: {mfa})");
Console.WriteLine($" from : {ip} using {agent}");
Console.WriteLine($" result: {(error is null ? "success" : "FAILED " + error)}");
}
request.NextToken = response.NextToken; // null when no more pages
} while (request.NextToken is not null);
Console.WriteLine(seen == 0
? "No events found. Check the Region, the group id and the time window."
: $"{seen} event(s).");
return 0;Code explained line by line
- The package is
AWSSDK.CloudTrail; the version range4.*keeps you on the v4 line, but pin an exact version in a real project. AmazonCloudTrailClient(RegionEndpoint.USEast1)uses the default credential chain. Locally that is the SSO profile inAWS_PROFILE; on ECS it is the task role. No keys appear in code.LookupAttributestakes exactly one filter in event history.ResourceNamematches the id of the resource named in the event.StartTimeandEndTimebound the 90-day history window. Lookups cover the Region of the client only.ev.CloudTrailEventis the full record as a JSON string. Parsing it withSystem.Text.Jsonavoids depending on the SDK's partial typed view.TryGetPropertyguards every optional field.arnandsessionContextare not present for every identity type, anderrorCodeexists only on failures.- The
do { ... } while (NextToken is not null)loop follows pagination. Forgetting it is the classic reason a lookup "finds nothing" when results spill onto a second page. - Exit code 1 on bad usage lets you use the tool in scripts.
Security considerations
⚠ Learning shortcut — not for production
The lab trail lives in the same account as the workload, uses RemovalPolicy.DESTROY and AutoDeleteObjects, expires objects after 30 days and covers a single Region. A real bank uses an organisation trail in all Regions delivered to a separate log-archive account with Object Lock, long retention and no delete rights for anyone in the workload account. See CloudTrail for banks.
- CloudTrail records are evidence. Protect the bucket and log group from change. Enable log file validation so tampering is detectable.
- Records can contain sensitive values. Request parameters occasionally hold secrets or personal identifiers (for example in some service-specific calls). Treat the log group as sensitive: restrict read access and set retention deliberately, in line with what your data-protection officer agrees under UK GDPR.
- Alert on the dangerous few, not everything. Root sign-in,
ConsoleLoginwithout MFA,StopLogging,DeleteTrail,PutBucketPolicychanges, IAM policy changes and security group changes. Alerting on every event creates noise that nobody reads. - Separation of duties. The people who can change infrastructure should not be the only people who can read or alter the trail.
- Do not assume attribution. A shared role hides the person unless you enforce session names or
sourceIdentity. Plan that before you need it.
Banking use case
Harbourline's platform team keeps a short list of CloudTrail-driven alerts, each with an owner and a runbook:
| Trigger (eventName) | Why it matters | Route |
|---|---|---|
ConsoleLogin by Root | Root should never be used day to day | Page security on-call immediately |
ConsoleLogin success without MFA | Weak authentication on a privileged path | Ticket plus email |
StopLogging, DeleteTrail, UpdateTrail | Someone is reducing audit visibility | Page security on-call |
AuthorizeSecurityGroupIngress with 0.0.0.0/0 on non-web ports | Possible exposure of a data tier | Page platform on-call |
PutBucketPolicy, PutBucketAcl, DeleteBucketPublicAccessBlock | Data exposure risk | Ticket, auto-check with Config |
AttachRolePolicy with admin-level policies | Privilege escalation | Page security on-call |
Operations use the same data differently: when a deployment goes wrong, the first query is "what API calls did the deployment role make in the last hour, and which failed?". A filter on errorCode quickly exposes missing permissions that cause 5xx errors in downstream services.
Audit trail versus application log
| CloudTrail | Application log | |
|---|---|---|
| Records | AWS API calls (who changed which resource) | What your code did (business events, errors, latency) |
| Written by | AWS, automatically | Your code, through ILogger |
| Contains customer data? | Rarely, but request parameters can | Often, so you must mask it |
| Typical question | "Who opened port 1433 to the internet?" | "Why did payment PAY-… fail?" |
| Tamper resistance | Log file validation, Object Lock in a separate account | Whatever you build |
| Retention driver | Audit and regulatory expectations | Troubleshooting and cost |
| Complete? | Not for data-plane actions unless data events are on | Only what you chose to log |
A bank also needs a business audit trail (the audit_events table in the course's domain model records actor, action, resource, outcome, source IP and correlation ID). CloudTrail does not replace it: it knows an engineer changed a DynamoDB table, not that a customer's standing order was edited.
Common mistakes
- Searching the wrong Region. Event history is per Region. A global service such as IAM reports in
us-east-1. - Expecting data events in event history. They are never there; you need a trail with data event selectors.
- Searching for the person, finding only a role. Use the session name in the
arn, orsourceIdentity, to get to the person. - Using a trail without validation or encryption. Turn on file validation and SSE-KMS from the start.
- Alert fatigue. Every
Describe*call is a management event. Filter onreadOnly = false. - Treating CloudTrail as real time. For immediate reaction, use the EventBridge rule; use the log group for counting.
- Forgetting Lake is closed. Do not design new analytics on CloudTrail Lake.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Lookup returns nothing | Wrong Region, wrong id, or event outside 90 days | Switch Region, confirm the id, widen the time range |
LookupEvents throttled | Too many requests per second | Back off and retry; for bulk analysis use the trail in S3 with Athena |
Trail shows IsLogging: false | Logging stopped or delivery failed | get-trail-status, read LatestDeliveryError, fix bucket policy or KMS policy |
| No events in the trail log group | Role missing logs:PutLogEvents, or wrong log group ARN | Verify the role and that the ARN ends with :* |
| Metric filter never triggers | Pattern syntax error, or JSON field mismatch | Test with the console Test pattern feature on a real event |
Alarm in INSUFFICIENT_DATA | No matching events yet | Expected for a rare event; use treatMissingData = notBreaching |
| Alarm fires but no email | Topic subscription unconfirmed, or topic policy missing | Confirm the subscription; add the publish policy for cloudwatch.amazonaws.com |
| EventBridge rule never matches | Event comes from a different Region, or pattern mismatch | Create the rule in the Region where the API call occurs |
AccessDenied on lookup-events | Missing cloudtrail:LookupEvents | Add the permission to the operator role |
Cost and cleanup
| Item | Label | Estimate |
|---|---|---|
| Event history lookups | free | No charge |
| First copy of management events in one trail to S3 | free | See the pricing note above ⚠ VERIFY |
| S3 storage for trail files | credits | Pennies at lab volume |
| CloudWatch Logs ingestion of trail events | credits | A few MB in a quiet sandbox; set 7-day retention |
| Metric filter metric, one alarm | credits | One custom metric and one alarm for the lab duration |
| KMS key | credits | Per-key monthly fee, delete after the lab ⚠ VERIFY price |
| SNS email | free | Negligible |
Free alternative: skip the stack and use only event history, the CLI and the C# tool above. Everything in Lab 11.5 works with no trail.
# Tear down the stack (CDK) ...
cdk destroy CloudTrailOpsStack --profile learn-sandbox
# ... or Terraform
terraform destroy
# KMS keys are scheduled for deletion, not removed immediately.
aws kms list-keys --profile learn-sandbox --region us-east-1
# (if a key remains) aws kms schedule-key-deletion --key-id <key-id> --pending-window-in-days 7 --profile learn-sandbox --region us-east-1
# Confirm nothing tagged Project=cea-lab is left.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'Exercises
- 🟢 Guided: In your sandbox create a security group, add an inbound rule for TCP 8080 from your own IP, then delete the group. Find all three events in event history, and note for each the identity type, source IP and user agent.
- 🟡 Challenge: Extend the C# tool to take an
--event-nameargument instead of a resource id, and print only events wherereadOnlyis false anderrorCodeis present. Then explain why that finds permission problems. - 🔴 Live problem: Ticket: "Overnight, payments started failing. The platform team says nobody touched anything." You have a trail log group and event history. Describe the five queries you would run in the first ten minutes, in order, and what each outcome would tell you.
Assignment
Write a "CloudTrail alert catalogue" for Harbourline: eight event patterns, each with the metric filter or EventBridge pattern, severity, who is notified, the first action in the runbook, and whether you would alert in real time (EventBridge) or on counts (metric filter and alarm). Commit it with the IaC that creates the first three.
Solutions
Use aws ec2 create-security-group, authorize-security-group-ingress and delete-security-group. In event history, events named CreateSecurityGroup, AuthorizeSecurityGroupIngress and DeleteSecurityGroup appear after a short delay. With the SSO profile, userIdentity.type is typically AssumedRole (or IdentityCenterUser), the userAgent begins with aws-cli, and the sourceIPAddress is your public IP. Making the same change in the console shows a console user agent instead. The lesson: the same API action leaves different fingerprints depending on the tool, which helps you separate humans from pipelines.
Replace the lookup attribute with LookupAttributeKey.EventName, filter after parsing: root.GetProperty("readOnly").GetBoolean() == false && root.TryGetProperty("errorCode", out _). Failed write calls reveal missing permissions or invalid requests, for example a deployment role getting AccessDenied on ecs:UpdateService, which in turn explains a rollout that silently never happened.
Knowledge check
Answer: 90 days, per account and per Region. A trail is needed for anything longer or for events across Regions in one place.
Answer: No. Reading objects is a data event, and event history shows management events only. You would need a trail with S3 data events enabled before the read happened.
Answer: Use the session name at the end of userIdentity.arn, or sourceIdentity where enforced, and for IAM Identity Center use onBehalfOf.userId with the identity store. Then correlate with your identity provider's records.
Answer: When you need an immediate notification that carries the full event for a single rare action (root login, StopLogging). Metric filters suit counts and thresholds ("more than 10 denied calls in 5 minutes").
Answer: It has been closed to new customers since 2026-05-31; existing customers continue. For new designs use trails with S3 and Athena, or CloudWatch's unified log management. Trails themselves are fully supported.
Interview questions
- A security group was opened to the internet last night. How do you find out who did it? Look up the group id in event history (or the trail's log group), read
userIdentity,sourceIPAddressanduserAgent, resolve the session to a person, and check whether a pipeline or console made the change. - Why does CloudTrail not replace application logging? It records control-plane API calls by default, not what your code did with customer data. They answer different questions and have different retention and masking needs.
- How would you make sure no one can silently turn CloudTrail off? Alert on
StopLogging/DeleteTrail, restrict those permissions with an SCP in a multi-account setup, and deliver to a log-archive account with Object Lock. - What are the downsides of data events? They are off by default and priced per event, so busy buckets or tables can generate large bills; enable them selectively on sensitive resources.
Certification practice
An operations engineer must be notified within a minute whenever any user deletes a CloudTrail trail in an account. Which solution is the most direct?
A. A daily Athena query over trail files in S3.
B. An EventBridge rule matching AWS API Call via CloudTrail with eventName of DeleteTrail, targeting an SNS topic.
C. A CloudWatch alarm on the S3 bucket's BucketSizeBytes metric.
D. AWS Config rule s3-bucket-versioning-enabled.
Answer: B. EventBridge receives management events near real time and can notify immediately. A is not timely. C does not relate to trail deletion. D checks versioning, not trail changes.
A team needs to alert when more than five console sign-in failures happen within ten minutes. Their trail already delivers to CloudWatch Logs. What is the best approach?
A. Create a CloudWatch Logs metric filter on ConsoleLogin failures and an alarm on its sum over 10 minutes.
B. Poll LookupEvents every second from a Lambda function.
C. Turn on S3 data events.
D. Enable VPC Flow Logs.
Answer: A. A metric filter turns matching events into a metric that an alarm can threshold. B is throttled and wasteful. C and D record different things.
An auditor asks for proof that trail log files were not altered. Which feature provides this?
A. S3 Transfer Acceleration. B. CloudTrail log file integrity validation. C. CloudWatch Logs retention. D. VPC endpoints for S3.
Answer: B. Digest files with hashes and signatures let you validate log integrity. The others do not address tampering.
Completion checklist
- I can explain management events, data events and Insights events, and which are free
- I found my own change in event history and read its
userIdentity - I can say how CloudTrail differs from an application log and from a business audit trail
- I built (or designed) a metric filter, alarm and EventBridge rule for a risky action
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Event history gives you 90 days of management events per Region for free, with no setup.
- A record answers who, what, when, from where and whether it worked; the
userIdentityblock is the part to learn. - Use EventBridge for immediate single-event alerts and metric filters for counted thresholds.
- CloudTrail is not an application log and not a business audit trail; a bank needs all three.
- Protect trail data like evidence: encryption, validation and separate-account storage.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
CloudWatch Application Signals: service maps, SLOs and error budgets
Turn OpenTelemetry telemetry from an ASP.NET Core service into an automatic service map, standard Latency/Fault/Error metrics and service level objectives with burn-rate alarms, and know what it adds over raw X-Ray traces.
Log aggregation and error monitoring across accounts
Centralise logs from many AWS accounts without copying everything twice, group thousands of exception lines into a handful of distinct errors, and route the right alerts to the right people with severity-based paths.