Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Logs Insights queries: five saved queries that answer real incident questions

Write CloudWatch Logs Insights queries for the questions an on-call engineer asks during an incident (what happened to this request, are customers failing, where is it slow), save them, run them from the CLI and from C#, and keep their cost under control.

intermediate⏱ 1 h 30 min🟡 Low costSOA-C03DOP-C02DVA-C03

Assumes: .NET 10 LTS · AWSSDK.CloudWatchLogs 4.1.0.2 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what Logs Insights is, which fields it discovers automatically, and how it differs from filter-log-events.
  • Write queries with fields, filter, stats, sort, limit, parse, bin, pctile, count_distinct and ispresent.
  • Turn five incident questions into saved queries and keep them in source control next to the service.
  • Run a query from the AWS CLI and from a .NET 10 console app using the AWS SDK v4, polling until it completes.
  • Estimate and limit the cost of a query, and know when a metric or an alarm is a better tool than a log query.

Prerequisites

  • CloudWatch Logs for .NET: the flat JSON log shape (level, message, correlationId, path, statusCode, elapsedMs, traceId) used by every query here.
  • Observability fundamentals: why each query exists.
  • Tools: AWS CLI v2 with the learn-sandbox profile; optional .NET 10 SDK or Docker.

Why this matters in banking

At 09:14 on salary day a customer reports that a transfer "just spun and failed". Support gives you one thing: the X-Correlation-ID header from the error response. Three minutes later the incident manager asks: "Is this one customer or many? Since when? Which step is slow?"

If your only tool is scrolling a log stream, you lose twenty minutes. With a handful of rehearsed queries you answer all three in under three. Logs Insights is the tool for interactive questions over recent logs. It is not a reporting database and not a replacement for metrics, but for "what is happening right now" it is the fastest path from a symptom to a cause, and, being priced per gigabyte scanned, it rewards queries that are narrow in time and in log groups.

The idea in plain language

CloudWatch Logs Insights lets you query one or more log groups with a pipe-based language. Each command takes the rows produced by the previous one:

fields @timestamp, level, path
| filter level = "Error"
| sort @timestamp desc
| limit 20

Read it like a PowerShell pipeline or a KQL query: pick columns, keep matching rows, sort, cut off.

What Insights knows about your logs without any setup:

  • System fields, always present: @timestamp (the event time), @message (the raw line), @logStream, @log (the log group, useful when you query several), @ingestionTime.
  • Discovered fields: for JSON log lines, each top-level key becomes a field (level, statusCode, correlationId). This is why the previous lesson insisted on a flat JSON shape. For nested JSON, Insights flattens with dots (for example request.path) and for arrays with an index, which is clumsy.
  • Unstructured lines need parse to extract fields at query time.

Analogy: The log group is a filing cabinet of bank statements. Insights is a clerk who can read every page in a date range, but is paid per page read. A good query says "only the May drawer, only current accounts, only rows over £1,000" before the clerk starts reading. A lazy query says "everything, ever" and arrives with an invoice.

What problem the AWS service solves

Logs Insights answers three kinds of question:

KindExampleCommand pattern
Find specific eventsAll lines for correlation ID Xfilter correlationId = "..."
AggregateRequests and errors per five minutesstats count(*) by bin(5m)
Compare / summarisep50 and p99 latency by pathstats pctile(elapsedMs, 99) by path

The commands and functions this lesson uses (checked against the AWS documentation on 2026-11-18, with the points that need re-verifying flagged):

ItemWhat it does
fieldsChoose or compute columns; fields floor(statusCode / 100) as statusClass
filterKeep matching rows; operators =, !=, <, >, like, in, boolean and/or/not
statsAggregate: count, count_distinct, sum, avg, min, max, pctile, optionally by one or more fields and bin(period)
sort, limitOrder and cut the result
parseExtract fields from @message with a glob or a regular expression
displayChoose output columns at the end
dedup, unnest, pattern, diff, filterIndexNewer commands: duplicate removal, array expansion, automatic pattern grouping, comparison with an earlier period, and filtering on indexed fields ⚠ VERIFY exact syntax and availability for your account
Functionsbin, ispresent, coalesce, floor, strlen, toupper, substr, concat and more

The AWS documentation also lists other query languages (OpenSearch PPL and SQL) for Logs Insights ⚠ VERIFY current support; this module uses the default Logs Insights language.

Limits and honest caveats ⚠ VERIFY current values before relying on them:

  • The StartQuery API is limited to 10 requests per second per account per Region (verified against the quotas page), so a dashboard with many log widgets can throttle.
  • A query runs against a time range you choose and up to a documented maximum number of log groups ⚠ VERIFY; queries have a time limit, and the console and API return a capped number of rows (10,000 by default for the console, and the API's Limit defaults to 1,000 ⚠ VERIFY).
  • Query results are eventually consistent with ingestion: lines from the last few seconds may not be searchable yet.
  • You pay per GB scanned (see the CloudWatch pricing page) ⚠ VERIFY the current price. Scan cost depends on the time range and the number of log groups, not on how many rows the query returns.
  • Insights is for investigation. For anything you need continuously (error rate for an alarm), emit a metric (lesson 4) or use a metric filter. Insights can also back log alarms that run a query on a schedule and alarm on the result ⚠ VERIFY availability and pricing, which is convenient but bills for scanned data on every run.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
KQL: AppRequests | where ResultCode >= 500 | summarize count() by bin(TimeGenerated, 5m)filter statusCode >= 500 | stats count(*) by bin(5m)Fewer commands; no joins across tables; bin(5m) has no field argument
projectfields / displayfields can compute new columns
summarize ... bystats ... byResults sorted by time bucket automatically only if you sort
extendfields expr as nameSame idea
parse / extractparse @message "..." as a, b (glob) or a regexGlob syntax uses * as a wildcard
percentile(x, 99)pctile(x, 99)Approximate percentile on large result sets
Saved queries / query packsSaved query definitions (put-query-definition)Stored per account and Region; export to Git
Log Analytics cost per GB ingested and retainedPer GB scanned per queryA wide, long-range query is the expensive thing
join across workspacesNot supported the same wayQuery several log groups at once and use @log to tell them apart

Architecture

Security-boundary view. Query permissions are separate from read permissions on raw events. An operator needs logs:StartQuery, logs:GetQueryResults (and logs:StopQuery) on the log groups they query. A dashboard's log widget runs the query as the viewer, so viewers need those permissions too. Because queries can reveal personal data that slipped into logs, restrict production log groups to a named operations role and review who holds it.

The five incident queries

These are exactly the queries in labs/m11-observe-banking-api/src/Harbourline.Observability/Queries/IncidentQueries.cs, and a unit test fails if a log field they depend on is renamed.

1. Follow one request

"What happened to this one request, across every log line?"

fields @timestamp, level, category, message, path, statusCode
| filter correlationId = "01a1edab-ab5a-7c55-bda9-d95e2a2212e4"
| sort @timestamp asc
| limit 200

correlationId is a discovered field because the log line is flat JSON. Sorting ascending reads the request like a story. Expected output for a failed transfer in the lab:

@timestamplevelcategorymessagepathstatusCode
08:00:13.177InformationProgramTransfer requested, reference rent, call me on [phone]
08:00:13.203ErrorProgramTransfer failed: sanctions partner unavailable
08:00:13.237ErrorCorrelationMiddlewareRequest completed POST /v1/transfers 502 89.1/v1/transfers502

2. Are customers failing right now?

"What share of requests fail, and since when?"

filter ispresent(statusCode)
| fields floor(statusCode / 100) as statusClass
| stats count(*) as requests by bin(5m), statusClass
| sort bin(5m) desc

ispresent skips lines that are not request logs. floor(statusCode / 100) turns 200, 201, 502 into classes 2 and 5, so you see 2xx, 4xx and 5xx per five-minute bucket. A sudden column of 5 rows is the incident. (Computing a ratio directly in one query is possible but easier to get wrong; a metric-based alarm is the right tool for ratios.)

3. Which endpoints are slow?

"Which paths are slow at the median and in the tail?"

filter ispresent(elapsedMs)
| stats count(*) as requests,
        pctile(elapsedMs, 50) as p50,
        pctile(elapsedMs, 95) as p95,
        pctile(elapsedMs, 99) as p99
  by path
| sort p99 desc
| limit 20

A p50 of 20 ms with a p99 of 1,900 ms says "most requests are fine but a tail is terrible", which points to a slow dependency or lock contention rather than load.

4. What are the most common errors?

"Which errors happen most, and on which paths?"

filter level = "Error"
| stats count(*) as occurrences, count_distinct(correlationId) as affectedRequests by message, path
| sort occurrences desc
| limit 20

count_distinct separates "one request retried 500 times" from "500 customers affected". The EMF metric lines in the same log group have level = "Metric", so they never pollute this query.

5. Which requests were slowest?

"Give me individual slow requests so I can open their traces."

filter ispresent(elapsedMs) and elapsedMs > 1000
| fields @timestamp, correlationId, traceId, path, statusCode, elapsedMs
| sort elapsedMs desc
| limit 25

Copy a traceId into the trace viewer (Distributed tracing). This is the bridge from logs to traces.

Unstructured logs: parse

Third-party components and older code write plain text. parse extracts fields at query time:

fields @message
| parse @message "partner=* status=* latencyMs=*" as partner, status, latencyMs
| filter status != "200"
| stats count(*) as failures, avg(latencyMs) as avgMs by partner

The * wildcards capture text between the literal parts. Every value arrives as a string; Insights converts where you use it in arithmetic. Parsing is slower and more fragile than JSON fields, so the better fix is to log JSON.

Step-by-step implementation

Write the question first

Before writing a query, write the sentence it answers ("Which endpoints are slow?"). The sentence becomes the saved-query name and the runbook line.

Narrow the scope

Choose the smallest time range and the fewest log groups that can contain the answer. Scan cost and wait time both follow scope.

Start with filter, then aggregate

Put selective filter commands early. Check the row count and the bytes scanned (the console shows it) before running a heavy stats.

Save it as a query definition

Store the text in Git (IncidentQueries.cs in the lab), and deploy it with put-query-definition so the whole team sees the same names, for example hl/incident/errors-by-path.

Rehearse in a game day

Inject a fault (the lab's X-Fault: partner-down) and run the queries against real output. A query you have never run under pressure is a guess.

AWS Console instructions

Console → CloudWatch → Logs → Logs Insights. Choose the query language (Logs Insights QL), select one or more log groups, set the time range, paste the query and choose Run query. Results appear as a table; the Visualization tab charts stats ... by bin() results as a time series or bar chart. Choose Save to store the query with a name; folders can be created with a / in the name. Use History for recent queries. A query that runs long can be cancelled. Menu names change from time to time ⚠ VERIFY.

AWS CLI

# Start a query. Times are epoch SECONDS. This asks for the last hour.
QID=$(aws logs start-query --profile learn-sandbox \
  --log-group-name /hl-m11/transfers-api \
  --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
  --query-string 'filter level = "Error" | stats count(*) as occurrences by message | sort occurrences desc | limit 10' \
  --query queryId --output text)

# Poll until the status is Complete, then read the rows.
aws logs get-query-results --query-id "$QID" --profile learn-sandbox \
  --query '{status:status,scannedBytes:statistics.bytesScanned,rows:results}'

# Save a query definition so the whole team sees the same name and text.
aws logs put-query-definition --profile learn-sandbox \
  --name "hl/incident/follow-one-request" \
  --log-group-names /hl-m11/transfers-api \
  --query-string 'fields @timestamp, level, message, path, statusCode | filter correlationId = "REPLACE-ME" | sort @timestamp asc | limit 200'

# List saved definitions
aws logs describe-query-definitions --query-definition-name-prefix hl/ --profile learn-sandbox --query 'queryDefinitions[].name'

# Cancel a runaway query
aws logs stop-query --query-id "$QID" --profile learn-sandbox

On Windows PowerShell, replace date -d '-1 hour' +%s with [DateTimeOffset]::UtcNow.AddHours(-1).ToUnixTimeSeconds().

Infrastructure as Code

Saved queries can be managed as code so they survive console clean-ups.

QueryDefinitions.cs
using Amazon.CDK;
using Amazon.CDK.AWS.Logs;
using Constructs;

public sealed class QueriesStack : Stack
{
    public QueriesStack(Construct scope, string id) : base(scope, id)
    {
        _ = new QueryDefinition(this, "TopErrors", new QueryDefinitionProps
        {
            QueryDefinitionName = "hl/incident/top-errors",
            QueryString = new QueryString(new QueryStringProps
            {
                FilterStatements = new[] { "level = \"Error\"" },
                StatsStatements = new[] { "count(*) as occurrences, count_distinct(correlationId) as affectedRequests by message, path" },
                Sort = "occurrences desc",
                Limit = 20,
            }),
            LogGroups = [LogGroup.FromLogGroupName(this, "Api", "/hl-m11/transfers-api")],
        });
    }
}

Complete code

A small .NET 10 console tool that runs any query and prints the rows. It uses the AWS SDK for .NET v4 and is the shape of tool a support team might wrap around the five queries.

Program.cs
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;

// Runs a Logs Insights query and prints the rows.
// Usage: dotnet run -- /hl-m11/transfers-api 60 "stats count(*) as lines by level"
var logGroup = args.Length > 0 ? args[0] : "/hl-m11/transfers-api";
var minutes = args.Length > 1 ? int.Parse(args[1]) : 60;
var query = args.Length > 2 ? args[2] : "stats count(*) as lines by level";

using var logs = new AmazonCloudWatchLogsClient();

var end = DateTimeOffset.UtcNow;
var start = await logs.StartQueryAsync(new StartQueryRequest
{
    LogGroupNames = [logGroup],
    StartTime = end.AddMinutes(-minutes).ToUnixTimeSeconds(),
    EndTime = end.ToUnixTimeSeconds(),
    QueryString = query,
    Limit = 100,
});

GetQueryResultsResponse result;
var delay = TimeSpan.FromMilliseconds(500);
do
{
    await Task.Delay(delay);
    delay = TimeSpan.FromMilliseconds(Math.Min(delay.TotalMilliseconds * 2, 5000));
    result = await logs.GetQueryResultsAsync(new GetQueryResultsRequest { QueryId = start.QueryId });
}
while (result.Status == QueryStatus.Running || result.Status == QueryStatus.Scheduled);

if (result.Status != QueryStatus.Complete)
{
    Console.Error.WriteLine($"Query ended with status {result.Status}.");
    return 1;
}

foreach (var row in result.Results)
{
    Console.WriteLine(string.Join("  ", row.Where(f => f.Field != "@ptr").Select(f => $"{f.Field}={f.Value}")));
}
Console.WriteLine($"Scanned {result.Statistics.BytesScanned:N0} bytes, matched {result.Statistics.RecordsMatched:N0} records.");
return 0;

Project file: net10.0, Nullable and ImplicitUsings enabled, and <PackageReference Include="AWSSDK.CloudWatchLogs" Version="4.1.0.2" />. The code compiles against that package version (checked in Docker on 2026-11-18).

Code explained line by line

  1. AmazonCloudWatchLogsClient() uses the default credential chain: your SSO profile locally (AWS_PROFILE=learn-sandbox), a role in AWS. No access keys appear in code.
  2. StartQueryAsync starts the query and returns immediately with a QueryId. Insights is asynchronous: a query may take seconds to minutes.
  3. StartTime and EndTime are Unix epoch seconds, not milliseconds. Mixing this up is the most common bug (the query then looks at 1970 and returns nothing).
  4. The loop polls GetQueryResults with exponential back-off capped at five seconds; the GetQueryResults API is limited to 10 requests per second per account, so tight polling can throttle.
  5. QueryStatus is a constant class in SDK v4, compared with ==, not an enum. Scheduled and Running mean "keep waiting".
  6. Each result row is a list of ResultField (name and value pairs). The @ptr field is an internal pointer used to fetch the full log record; the tool hides it.
  7. Statistics.BytesScanned shows what the query cost you in scan volume. Print it every time, so engineers learn to feel the cost.

Security considerations

⚠ Learning shortcut — not for production

The lab lets one sandbox identity run queries on the log group. In production, give an ApiOperators role logs:StartQuery, logs:GetQueryResults and logs:StopQuery on named log-group ARNs, require MFA on that role, and log who ran queries (CloudTrail records StartQuery calls).

  • Queries reveal data. If personal data leaked into logs, anyone who can query can read it. Fix the source (lesson 2) and restrict the role.
  • Injection into saved queries. Do not splice raw user input into a query string. The lab's IncidentQuery.Render rejects quotes, backslashes, pipes and newlines in placeholder values, because a pasted x" or 1=1 | fields @message would change the query.
  • Cost as an attack surface. A stolen operator session can run expensive queries repeatedly. Keep the scope of the role small and set a billing alarm (see Free tier and billing).
  • Dashboards run queries as the viewer. A shared dashboard does not bypass log-group permissions.

Banking use case

Harbourline's payments on-call runbook begins with the same three queries every time: error rate in five-minute buckets, top errors by message and path, and slowest requests. They are saved under hl/incident/*, deployed from the service repository, and reviewed in the quarterly game day. When a customer supplies a correlation ID, support runs query 1 and attaches the result to the ticket. A weekly job runs "top errors" and files tickets for any message that exceeds a threshold, turning an ad hoc query into a trend signal. All data is fictional.

Common mistakes

  • Querying a 30-day range "to be safe". It multiplies scan cost and wait time. Start with 15 to 60 minutes.
  • Filtering late. Put selective filter first; aggregate after.
  • Using Insights for continuous alerting. Use metrics and alarms; reserve Insights for investigation.
  • Forgetting ispresent. A stats over mixed log lines (request logs and EMF lines) counts the wrong rows.
  • Percentiles of averages. pctile over raw elapsedMs is right; averaging per-minute p99 values is not.
  • Epoch milliseconds in the API. The API wants seconds.
  • Unbounded limit. Large result sets slow the console and the SDK.

Troubleshooting

SymptomLikely causeFix
Query returns 0 rowsWrong time range, wrong log group, or field name differs (correlationId vs CorrelationId)Run fields @message | limit 5 first; check discovered fields in the console sidebar
Field not found style behaviour with nested JSONField is nestedUse dotted paths, or flatten the log shape
ThrottlingException on StartQueryMore than 10 queries per secondBack off; reduce dashboard log widgets
Query is slowWide time range or many log groupsNarrow both; add an early filter
Times look wrongConsole shows local time zone; API uses UTCSet the console to UTC for incident work
AccessDeniedExceptionMissing logs:StartQuery on the group, or KMS decrypt for an encrypted groupAdd permissions; for KMS add kms:Decrypt

Cost and cleanup

Insights bills per GB scanned (see the CloudWatch pricing page ⚠ VERIFY the current rate), and the free tier includes scans of up to 5 GB per month. The lab's log group holds kilobytes, so queries cost effectively nothing; the habit matters more than the amount. Saved queries are free. To clean up:

# Delete the saved query definitions you created
for id in $(aws logs describe-query-definitions --query-definition-name-prefix hl/ --query 'queryDefinitions[].queryDefinitionId' --output text --profile learn-sandbox); do
  aws logs delete-query-definition --query-definition-id "$id" --profile learn-sandbox
done

Delete the log group and stack as described in CloudWatch Logs for .NET.

Exercises

  • 🟢 Guided: In the lab, start the API, send ten transfers with scripts/load.sh, then run stats count(*) as lines by level on the log group (after deploying, or against a local copy of the output using jq). Save the five queries as definitions.
  • 🟡 Challenge: Write a query that shows, per five-minute bucket, the number of requests with elapsedMs above 1,000. Then write a second that lists the correlation IDs with the most Error lines, with the number of lines for each.
  • 🔴 Live problem: "Some customers say transfers are slow on Monday mornings but your p99 chart looks fine." Describe how you would use Insights to investigate: which fields, which comparisons (for example diff with last week ⚠ VERIFY), and what you would change in the logging if you cannot answer.

Assignment

Create queries/README.md in a service of your choice with: five saved queries (the question, the query text, a sample output), the IAM policy that allows an operations role to run them on named log groups only, and a CDK or Terraform snippet that deploys them as query definitions.

Solutions

Knowledge check

Interview questions

  • How would you find everything that happened to one failed payment? Take the correlation ID, run filter correlationId = "..." | sort @timestamp asc over the relevant log groups, and use the traceId to open the trace.
  • How do you keep Insights costs under control? Narrow time range and log groups, filter early, avoid dashboards that refresh heavy queries, and use metrics for anything continuous.
  • What is the difference between filter-log-events and Logs Insights? filter-log-events matches a pattern and returns raw events; Insights supports aggregation, computed fields and statistics.
  • Why prefer JSON logs for Insights? Top-level keys become fields automatically, so you avoid fragile parse patterns.
  • How would you share queries across a team? Keep them in Git and deploy as query definitions with IaC under a naming convention such as hl/incident/....

Certification practice

Completion checklist

  • I can write filter, stats ... by bin() and pctile queries from memory
  • I ran the five incident queries (or their local equivalents) and read the results
  • I ran a query from the CLI and understand the asynchronous start/poll pattern
  • I saved the queries as definitions and deleted them afterwards
  • I can explain what a query costs and how to reduce it

Key takeaways

  • Flat JSON logs make Insights queries short and fast; correlation and trace IDs make them useful.
  • Cost follows data scanned: narrow the time range and the log groups, and filter early.
  • Keep five rehearsed queries per service and store them in source control.
  • Insights is for investigation; use metrics and alarms for anything continuous.
  • Queries can expose personal data, so restrict who can run them.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page