Module 7 cheat sheet
One-page revision summary for Data Engineering & ETL Pipelines: services, Azure mapping, CLI commands, limits, decision rules, exam traps and cost traps.
Assumes: AWS CLI v2 · AWS SDK for .NET v4 · Glue (Spark) · Region us-east-1 · facts as stated in the Module 7 lessons; items marked VERIFY there are not repeated as certain here. Verified against official sources on 2026-11-18.
A printable summary of Module 7. Each row links back to the lesson with the detail.
Key services
| Service | What it is for |
|---|---|
| S3 (raw / curated / gold zones) | The data lake: one bucket and one KMS key per zone, Hive-style partitions |
| Glue Data Catalog + crawlers | Table and partition metadata shared by Athena, Glue, Redshift Spectrum |
| Glue Spark jobs | Large-scale validate, mask and transform; job bookmarks; Data Quality (DQDL) |
| Lambda | Small, quick, per-file transforms and stream consumers |
| Athena | Serverless SQL on S3, billed by bytes scanned |
| Redshift / Serverless | Columnar warehouse for heavy, repeated, concurrent BI |
| Kinesis Data Streams / Firehose / MSK | Streaming ingestion; managed delivery to S3 (with Parquet conversion); Kafka compatibility |
| Step Functions, EventBridge, Scheduler, MWAA | Orchestration, event triggers, time triggers, managed Airflow |
| DMS | Full load plus CDC from SQL Server |
| Lake Formation | Row, column, cell and tag-based permissions on the catalogue |
.NET / Azure → AWS
| Microsoft / Azure | AWS |
|---|---|
| SSIS / Azure Data Factory | Glue jobs + Step Functions |
| ADLS Gen2 | S3 with zones |
| Synapse serverless SQL | Athena |
| Synapse dedicated pool | Redshift / Redshift Serverless |
| Purview + access policies | Glue Data Catalog + Lake Formation |
| Event Hubs / Event Hubs Capture | Kinesis Data Streams / Data Firehose |
| SQL Server CDC | DMS full load + CDC |
| LINQ over collections | Spark DataFrames |
Most useful CLI commands
| Task | Command |
|---|---|
| Run an Athena query | aws athena start-query-execution --query-string "..." --work-group WG |
| See bytes scanned | aws athena get-query-execution --query-execution-id ID |
| Start / inspect a Glue job | aws glue start-job-run --job-name NAME · aws glue get-job-runs --job-name NAME |
| Check partitions | aws glue get-partitions --database-name DB --table-name T |
| Start a workflow | aws stepfunctions start-execution --state-machine-arn ARN --input file://in.json |
| Put a stream record | aws kinesis put-record --stream-name NAME --partition-key KEY --data ... |
| Stream capacity | aws kinesis describe-stream-summary --stream-name NAME |
| Grant on the catalogue | aws lakeformation grant-permissions ... · aws lakeformation list-permissions |
| Leftover lab resources | aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab |
Key limits and defaults
| Item | Value | Lesson |
|---|---|---|
| Kinesis shard | 1 MB/s or 1,000 records/s write; 2 MB/s read; 24 h default retention | Streaming ingestion |
PutRecords | At most 500 records and 5 MB per call; can fail partially | Streaming ingestion |
| Lambda | 15 min, 128-10,240 MB memory, /tmp 512-10,240 MB, 6 MB sync / 1 MB async payload | Lambda transforms |
| Athena | USD 5 per TB scanned, 10 MB minimum per query; per-query cutoff minimum 10 MB | Amazon Athena |
| Athena managed results | Kept 24 hours, then deleted | Amazon Athena |
| Glue Spark | USD 0.44 per DPU-hour, per second, 1-minute minimum; Flex USD 0.29; minimum 2 workers | PySpark in Glue |
| Parquet file size | About 128 MB to 1 GB for analytical scans | S3 data lake design |
| Step Functions Standard | 25,000 history events; 256 KB state payload; history kept 90 days | Orchestration |
| Redshift Serverless | Base capacity from 4 RPU; set a daily RPU-hour usage limit | Redshift |
Decision rules
| If… | Then… |
|---|---|
| Seconds matter (fraud, velocity limits) | Streaming; otherwise default to batch |
| Validation and masking before anyone queries | ETL in Glue or Lambda; then ELT (SQL) for aggregates |
| Small file, a few seconds of work | Lambda |
| Large, joined or multi-file work | Glue Spark |
| Ad hoc queries on the lake | Athena on partitioned Parquet |
| Many concurrent dashboards on joined marts | Redshift |
Row-level changes, MERGE, time travel | Iceberg tables |
| Analysts must not see raw personal data | Lake Formation filters, tokenised curated data, no direct S3 access |
| AWS-native multi-service pipeline | Step Functions Standard with .sync Glue; MWAA only for Airflow shops |
| Table schema must be stable | Define curated and gold schemas as code; crawlers for discovery |
Common exam traps
- Crawlers are not a schema contract: silent drift needs
LOGbehaviour and table-version diffs. - Partition projection is Athena-only; Glue and Redshift Spectrum do not use it.
- Lake Formation filters are bypassed if
IAMAllowedPrincipalsor direct S3 access remains. - Time travel is not retention, and a
DELETEon an Iceberg table is not erasure. PutRecordsis not all-or-nothing: retry only the failed entries.- Lambda has a 15-minute limit; large files go to Glue.
- Gzip on large CSV blocks parallelism; Parquet compression does not.
- Order CDC records by commit time, not by arrival.
- Do not design masking on S3 Object Lambda (closed to new customers); mask in the Glue job.
- Step Functions Local is unsupported; test with TestState.
Cost traps
| Trap | Avoid it by |
|---|---|
SELECT * on CSV | Parquet, partition filters, workgroup scan limit |
| Kinesis shards left running | Deploy, test and destroy within an hour |
| Redshift Serverless with no usage limit | Set a daily RPU-hour limit; storage still bills while idle |
| MWAA and MSK environments | Design-only, or run Airflow locally in Docker |
| Glue crawlers run too often | Register partitions from the writer, or use projection |
| Many tiny files from streaming | Buffer in Firehose; compact |
| Quarantine and results buckets with no expiry | Lifecycle rules; retention from the records policy |
| Customer managed KMS keys left behind | Delete after the waiting period; use S3 Bucket Keys |
Secure pipeline and reporting (lessons 15 and 16)
| Topic | Rule of thumb | Lesson |
|---|---|---|
| Encryption | SSE-KMS with a customer managed key and S3 Bucket Keys; a Glue security configuration covers S3 output, CloudWatch Logs and job bookmarks; Athena results encrypted and enforced by the workgroup | Secure pipeline design |
| Key policy | Administrators manage the key; only pipeline roles use it; CloudWatch Logs needs its own statement | Secure pipeline design |
| Job roles | One role per function; raw is read-only for the transform job; ingest is write-only to its prefix | Secure pipeline design |
| Glue and VPC | S3-to-S3 jobs need no VPC; a network connection needs an S3 gateway endpoint, a KMS route (endpoint or NAT) and a self-referencing security group | Secure pipeline design |
| Endpoint cost | S3 gateway endpoint: no hourly charge; interface endpoints: about USD 0.01 per AZ-hour plus USD 0.01 per GB ⚠ VERIFY | Secure pipeline design |
| PII | Drop first; mask for display; keyed HMAC token to join; never a plain hash of a low-entropy value; rotate the key by re-tokenising | Secure pipeline design |
| Quick Sight naming | Quick Sight is a feature of AWS Quick; CLI, SDK and IAM keep quicksight | Quick Sight reporting |
| SPICE or direct query | Small, daily gold data: SPICE with a scheduled refresh; live data: direct query on a capped Athena workgroup | Quick Sight reporting |
| Row-level security | Enterprise edition; a rules dataset of user or group to allowed values; text columns only; no rule means no rows | Quick Sight reporting |
| Two layers | Quick Sight queries Athena with its service role, so Lake Formation filters do not restrict dashboard viewers | Quick Sight reporting |
Exam traps: encryption is not authorisation; a Glue job does not need a VPC for S3-to-S3 work; Quick Sight RLS is not available in Standard edition; a dashboard shared with a dataset owner shows every row.
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Module project: banking transaction processing pipeline
Deploy, run, break and audit a complete batch pipeline: raw files in an encrypted S3 zone, an event-driven Step Functions and Glue workflow that validates, tokenises and reconciles, quarantine with reason codes, partitioned Parquet, Athena reports and one audit record per run, with 27 .NET and 18 Python tests that need no AWS account.
Module 8: Messaging, Events & Microservices
Learn how AWS services pass messages and events between systems reliably: SQS, SNS, EventBridge, Step Functions, Kinesis, MSK and Amazon MQ. You apply retries, dead-letter queues, idempotency, sagas and the outbox pattern in .NET, and build a banking notification system that survives duplicates and failures.