Skip to Content
Cloud Engineering Academy
7. Data Engineering & ETL Pipelines

Module 7 cheat sheet

One-page revision summary for Data Engineering & ETL Pipelines: services, Azure mapping, CLI commands, limits, decision rules, exam traps and cost traps.

DEA-C01SAA-C03

Assumes: AWS CLI v2 · AWS SDK for .NET v4 · Glue (Spark) · Region us-east-1 · facts as stated in the Module 7 lessons; items marked VERIFY there are not repeated as certain here. Verified against official sources on 2026-11-18.

A printable summary of Module 7. Each row links back to the lesson with the detail.

Key services

ServiceWhat it is for
S3 (raw / curated / gold zones)The data lake: one bucket and one KMS key per zone, Hive-style partitions
Glue Data Catalog + crawlersTable and partition metadata shared by Athena, Glue, Redshift Spectrum
Glue Spark jobsLarge-scale validate, mask and transform; job bookmarks; Data Quality (DQDL)
LambdaSmall, quick, per-file transforms and stream consumers
AthenaServerless SQL on S3, billed by bytes scanned
Redshift / ServerlessColumnar warehouse for heavy, repeated, concurrent BI
Kinesis Data Streams / Firehose / MSKStreaming ingestion; managed delivery to S3 (with Parquet conversion); Kafka compatibility
Step Functions, EventBridge, Scheduler, MWAAOrchestration, event triggers, time triggers, managed Airflow
DMSFull load plus CDC from SQL Server
Lake FormationRow, column, cell and tag-based permissions on the catalogue

.NET / Azure → AWS

Microsoft / AzureAWS
SSIS / Azure Data FactoryGlue jobs + Step Functions
ADLS Gen2S3 with zones
Synapse serverless SQLAthena
Synapse dedicated poolRedshift / Redshift Serverless
Purview + access policiesGlue Data Catalog + Lake Formation
Event Hubs / Event Hubs CaptureKinesis Data Streams / Data Firehose
SQL Server CDCDMS full load + CDC
LINQ over collectionsSpark DataFrames

Most useful CLI commands

TaskCommand
Run an Athena queryaws athena start-query-execution --query-string "..." --work-group WG
See bytes scannedaws athena get-query-execution --query-execution-id ID
Start / inspect a Glue jobaws glue start-job-run --job-name NAME · aws glue get-job-runs --job-name NAME
Check partitionsaws glue get-partitions --database-name DB --table-name T
Start a workflowaws stepfunctions start-execution --state-machine-arn ARN --input file://in.json
Put a stream recordaws kinesis put-record --stream-name NAME --partition-key KEY --data ...
Stream capacityaws kinesis describe-stream-summary --stream-name NAME
Grant on the catalogueaws lakeformation grant-permissions ... · aws lakeformation list-permissions
Leftover lab resourcesaws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab

Key limits and defaults

ItemValueLesson
Kinesis shard1 MB/s or 1,000 records/s write; 2 MB/s read; 24 h default retentionStreaming ingestion
PutRecordsAt most 500 records and 5 MB per call; can fail partiallyStreaming ingestion
Lambda15 min, 128-10,240 MB memory, /tmp 512-10,240 MB, 6 MB sync / 1 MB async payloadLambda transforms
AthenaUSD 5 per TB scanned, 10 MB minimum per query; per-query cutoff minimum 10 MBAmazon Athena
Athena managed resultsKept 24 hours, then deletedAmazon Athena
Glue SparkUSD 0.44 per DPU-hour, per second, 1-minute minimum; Flex USD 0.29; minimum 2 workersPySpark in Glue
Parquet file sizeAbout 128 MB to 1 GB for analytical scansS3 data lake design
Step Functions Standard25,000 history events; 256 KB state payload; history kept 90 daysOrchestration
Redshift ServerlessBase capacity from 4 RPU; set a daily RPU-hour usage limitRedshift

Decision rules

If…Then…
Seconds matter (fraud, velocity limits)Streaming; otherwise default to batch
Validation and masking before anyone queriesETL in Glue or Lambda; then ELT (SQL) for aggregates
Small file, a few seconds of workLambda
Large, joined or multi-file workGlue Spark
Ad hoc queries on the lakeAthena on partitioned Parquet
Many concurrent dashboards on joined martsRedshift
Row-level changes, MERGE, time travelIceberg tables
Analysts must not see raw personal dataLake Formation filters, tokenised curated data, no direct S3 access
AWS-native multi-service pipelineStep Functions Standard with .sync Glue; MWAA only for Airflow shops
Table schema must be stableDefine curated and gold schemas as code; crawlers for discovery

Common exam traps

  • Crawlers are not a schema contract: silent drift needs LOG behaviour and table-version diffs.
  • Partition projection is Athena-only; Glue and Redshift Spectrum do not use it.
  • Lake Formation filters are bypassed if IAMAllowedPrincipals or direct S3 access remains.
  • Time travel is not retention, and a DELETE on an Iceberg table is not erasure.
  • PutRecords is not all-or-nothing: retry only the failed entries.
  • Lambda has a 15-minute limit; large files go to Glue.
  • Gzip on large CSV blocks parallelism; Parquet compression does not.
  • Order CDC records by commit time, not by arrival.
  • Do not design masking on S3 Object Lambda (closed to new customers); mask in the Glue job.
  • Step Functions Local is unsupported; test with TestState.

Cost traps

TrapAvoid it by
SELECT * on CSVParquet, partition filters, workgroup scan limit
Kinesis shards left runningDeploy, test and destroy within an hour
Redshift Serverless with no usage limitSet a daily RPU-hour limit; storage still bills while idle
MWAA and MSK environmentsDesign-only, or run Airflow locally in Docker
Glue crawlers run too oftenRegister partitions from the writer, or use projection
Many tiny files from streamingBuffer in Firehose; compact
Quarantine and results buckets with no expiryLifecycle rules; retention from the records policy
Customer managed KMS keys left behindDelete after the waiting period; use S3 Bucket Keys

Secure pipeline and reporting (lessons 15 and 16)

TopicRule of thumbLesson
EncryptionSSE-KMS with a customer managed key and S3 Bucket Keys; a Glue security configuration covers S3 output, CloudWatch Logs and job bookmarks; Athena results encrypted and enforced by the workgroupSecure pipeline design
Key policyAdministrators manage the key; only pipeline roles use it; CloudWatch Logs needs its own statementSecure pipeline design
Job rolesOne role per function; raw is read-only for the transform job; ingest is write-only to its prefixSecure pipeline design
Glue and VPCS3-to-S3 jobs need no VPC; a network connection needs an S3 gateway endpoint, a KMS route (endpoint or NAT) and a self-referencing security groupSecure pipeline design
Endpoint costS3 gateway endpoint: no hourly charge; interface endpoints: about USD 0.01 per AZ-hour plus USD 0.01 per GB ⚠ VERIFYSecure pipeline design
PIIDrop first; mask for display; keyed HMAC token to join; never a plain hash of a low-entropy value; rotate the key by re-tokenisingSecure pipeline design
Quick Sight namingQuick Sight is a feature of AWS Quick; CLI, SDK and IAM keep quicksightQuick Sight reporting
SPICE or direct querySmall, daily gold data: SPICE with a scheduled refresh; live data: direct query on a capped Athena workgroupQuick Sight reporting
Row-level securityEnterprise edition; a rules dataset of user or group to allowed values; text columns only; no rule means no rowsQuick Sight reporting
Two layersQuick Sight queries Athena with its service role, so Lake Formation filters do not restrict dashboard viewersQuick Sight reporting

Exam traps: encryption is not authorisation; a Glue job does not need a VPC for S3-to-S3 work; Quick Sight RLS is not available in Standard edition; a dashboard shared with a dataset owner shows every row.

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page