Integrating with core banking and mainframes: hybrid networks, private APIs and MQ messaging
Connect cloud services to an on-premises core banking system safely: choose between Site-to-Site VPN and Direct Connect, expose and consume private APIs, use request/reply messaging with Amazon MQ (or RabbitMQ locally), and translate COBOL copybook records with EBCDIC and packed decimals into a clean C# domain model behind an anti-corruption layer.
Assumes: .NET 10 LTS · RabbitMQ.Client 7.2.2 · Amazon MQ (RabbitMQ 3.13/4.x, ActiveMQ 5.18/5.19) · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Service availability — verified 2026-11-17
AWS Mainframe Modernization managed runtime environments closed to new customers on 2025-11-07, and the self-managed experience entered maintenance. This lesson does not build on that service. It integrates with a mainframe that stays where it is, through private networking, APIs and messaging.
Cost: Amazon MQ and hybrid networking bill per hour
An Amazon MQ broker bills per instance-hour from creation until deletion (for example about USD 0.27 per hour for a RabbitMQ mq.m7g.large instance in us-east-1, verified 2026-11-18), plus storage. Site-to-Site VPN connections and Direct Connect ports also bill hourly. The free path runs RabbitMQ in Docker with exactly the same C# client code; VPN and Direct Connect are design-only exercises.
Learning objectives
By the end of this lesson you can:
- Explain the shape of a core banking system and why integration, not replacement, is the normal starting point.
- Choose between Site-to-Site VPN, Direct Connect and combinations (with Transit Gateway) for hybrid connectivity, including encryption and resilience.
- Design both directions of private API access: AWS services calling on-premises APIs (NLB with IP targets, VPC Link) and on-premises systems calling AWS (private REST APIs through an interface VPC endpoint).
- Use MQ-style messaging with Amazon MQ (ActiveMQ or RabbitMQ) for request/reply with correlation IDs, timeouts and idempotent consumers, and explain how IBM MQ estates fit.
- Parse COBOL copybook records in C#: EBCDIC text, zoned decimals and COMP-3 packed decimals, and map them through an anti-corruption layer into Harbourline's domain model.
Prerequisites
- Container-backed APIs and VPC Link
- Calling partner APIs safely — timeouts, retries and circuit breakers apply to the core
- Idempotency keys
- ISO 20022 message handling and SWIFT and file-based integration — the other two big legacy channels
- Module 2: Networking — VPCs, routing, private subnets
- Tools: .NET 10 SDK, Docker Desktop. AWS CLI v2 with
learn-sandboxonly for the optional Amazon MQ deploy
Why this matters in banking
Harbourline Bank (fictional) runs its accounts, balances and postings on a 30-year-old core banking system on a mainframe in its data centre. The new mobile app, built on AWS, needs balances in under 300 ms. The first prototype calls the core directly from a Lambda function over the internet through a reverse proxy, opens a new connection per request, and retries every timeout three times.
On payday morning the core's transaction monitor hits its connection limit. Retries triple the load; balance enquiries for branch staff start timing out too; a duplicated debit posting appears because a "timed-out" request had actually succeeded. The incident report has three findings: the network path was neither private nor resilient, the integration had no back-pressure, and the cloud code leaked mainframe concepts (field names like LEDGER-BAL, packed decimals) everywhere.
Most banks will run their core for years alongside the cloud. Integrating with it safely, privately and politely (without overloading it) is one of the most valuable skills a cloud engineer at a bank can have.
The idea in plain language
A core banking system is the system of record for accounts, balances and postings. Mainframe cores typically expose functionality as transactions (programs in CICS or IMS), batch jobs that run at night, and data in Db2 or VSAM files. Data formats come from COBOL: fixed-width records described by copybooks, text in EBCDIC (not ASCII), and numbers in zoned or packed decimal (COMP-3).
Integration has three parts:
- A private, reliable network path between AWS and the data centre: an encrypted VPN over the internet, a dedicated Direct Connect link, or both.
- An integration style:
- Synchronous APIs (the mainframe team publishes REST APIs through an API layer such as z/OS Connect ⚠ VERIFY product names your bank uses) for quick queries like balance enquiry.
- Messaging (IBM MQ on the mainframe side; Amazon MQ or a bridge on AWS) for postings and anything that must not be lost, with request/reply when an answer is needed.
- Files and change data capture for bulk and analytics (nightly extracts, CDC streams).
- An anti-corruption layer (ACL): one well-tested component that translates between the core's world (copybooks, codes like
'O'/'F', EBCDIC) and your clean domain (Account,AccountStatus.Open,decimal). Nothing outside it knows the mainframe exists.
Analogy: The core banking system is a senior colleague in another building who speaks a different language, works carefully and cannot be rushed. You build a private corridor to their building (VPN/Direct Connect), agree to send requests through their in-tray rather than barging in (message queue), hire one interpreter who knows both languages (anti-corruption layer), and never send the same request twice without a reference number (idempotency).
What problem the AWS services solve
| Need | AWS building block | Honest limitations |
|---|---|---|
| Encrypted connectivity over the internet | AWS Site-to-Site VPN (two IPsec tunnels per connection, to a virtual private gateway or Transit Gateway) | Internet path: variable latency; per-tunnel bandwidth limits (about 1.25 Gbps ⚠ VERIFY); billed per connection-hour plus data transfer |
| Dedicated private connectivity | AWS Direct Connect (dedicated or hosted connections at a Direct Connect location) | Weeks to provision; not encrypted by default (use MACsec on supported dedicated connections, or VPN over Direct Connect); a single connection is a single point of failure |
| Hub for many VPCs and on-prem links | AWS Transit Gateway | Per-attachment-hour and per-GB processing charges |
| AWS → on-prem HTTP APIs | Network Load Balancer with on-premises IP targets, reached from API Gateway through a VPC Link or directly from services | IP targets outside the VPC must be in supported private ranges and reachable via VPN/DX ⚠ VERIFY current rules |
| On-prem → AWS private APIs | API Gateway private REST API with an interface VPC endpoint (execute-api) and a resource policy | Private APIs are REST only; on-prem DNS must resolve the endpoint (Route 53 Resolver inbound endpoint) |
| Managed message broker for JMS/AMQP/MQTT | Amazon MQ for ActiveMQ (5.18, 5.19) or RabbitMQ (3.13 on t3/m5/m7g; 4.2 and 4.3 on m7g only) | Hourly per broker instance; not IBM MQ; RabbitMQ streams are not supported |
| Cloud-native queues and events | SQS, SNS, EventBridge | Not protocol-compatible with JMS/AMQP clients on the mainframe side |
Amazon MQ exists for protocol compatibility: when the other side speaks JMS, AMQP, STOMP, MQTT or OpenWire and you cannot change it. For new cloud-to-cloud messaging, SQS/SNS/EventBridge are cheaper and simpler.
IBM MQ. Many mainframes use IBM MQ, and Amazon MQ does not run IBM MQ or its native protocol. Common patterns: keep IBM MQ as the mainframe's broker and run an IBM MQ queue manager (licensed) on EC2 or containers in AWS connected by MQ channels; or bridge IBM MQ to ActiveMQ/RabbitMQ with a JMS bridge application; or have the mainframe team expose the transaction through an API instead ⚠ VERIFY licensing and supported bridge options with your vendor.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure VPN Gateway | Site-to-Site VPN (virtual private gateway or Transit Gateway) | Two tunnels per connection by default; you configure both on your device |
| ExpressRoute | Direct Connect | Similar model (locations, partners); encryption via MACsec or VPN overlay |
| Virtual WAN hub | Transit Gateway | Route tables per attachment; inter-Region peering |
| APIM with internal VNet | API Gateway private REST API + interface VPC endpoint | Resource policy restricts by VPC endpoint |
| Azure Service Bus (AMQP 1.0) | Amazon MQ for ActiveMQ (AMQP 1.0, JMS) or RabbitMQ (AMQP 0-9-1) | Amazon MQ runs open-source brokers; Service Bus is proprietary |
| Host Integration Server (HIS) for CICS/IMS and EBCDIC | No direct equivalent; your .NET code (ACL) plus mainframe-side API layers | You parse copybooks yourself or with a library |
System.Text.Encoding.CodePages | Same in .NET 10 | Register CodePagesEncodingProvider to get EBCDIC code pages such as 037 |
Architecture
Security-boundary view.
- Network: no internet path to the core. Traffic flows over Direct Connect (MACsec or IPsec overlay) with VPN as encrypted backup; security groups and on-prem firewalls allow only the specific broker and API ports.
- Identity: service-to-service credentials (broker users, mTLS client certificates) are in Secrets Manager and rotated; the mainframe side maps them to RACF/ACF2/Top Secret identities with least privilege ⚠ VERIFY your security product.
- Data: TLS on every hop (AMQPS, HTTPS). Copybook records contain customer data; they never leave the ACL in raw form and are never logged.
- Load: the ACL enforces concurrency limits and timeouts so cloud traffic cannot overwhelm the core.
Step-by-step implementation
Agree the network design with the network and mainframe teams
Typical choice for a bank: two Direct Connect connections at two locations (resilience) into a Direct Connect gateway and Transit Gateway, with Site-to-Site VPN as an encrypted backup path. Decide encryption (MACsec on dedicated connections, or IPsec VPN over Direct Connect), BGP routing preferences, and IP ranges (no overlaps between VPC CIDRs and data centre ranges). Use VPN alone for early phases and labs.
Pick the integration style per use case
| Use case | Style | Why |
|---|---|---|
| Balance enquiry for the app | Synchronous API (or request/reply over MQ) with a cache | Read-only, latency-sensitive |
| Posting a payment | Asynchronous messaging, idempotent, with confirmation message | Must not be lost or duplicated |
| Customer data for analytics | CDC or nightly extract to S3 | Bulk; no load on online transactions |
| Statement generation | Files | Batch by nature |
Build the anti-corruption layer
One .NET component owns: copybook layouts, EBCDIC encoding, packed decimal conversion, code mappings ('O' → Open), error code translation (core return codes → domain errors), and the transport (MQ or HTTP). It exposes a clean interface such as ICoreAccounts.GetBalanceAsync(accountId). Contract tests run against recorded core responses.
Messaging: request/reply with correlation and timeouts
For each request: generate a correlation ID, publish to the request queue with reply_to set to the adapter's reply queue and a message expiration equal to your timeout (so stale requests are discarded instead of processed late), and wait on a TaskCompletionSource keyed by correlation ID. Replies with unknown correlation IDs (late arrivals) are logged and dropped.
Make writes idempotent end to end
Posting requests carry a business idempotency key (for example payment_id). The core-side program (or a table in front of it) records processed keys, and the adapter treats "timeout" as "unknown outcome": it queries the status by key before retrying. Never blindly retry a posting.
Protect the core
Limit concurrency per adapter instance (for example a SemaphoreSlim of 20), use a circuit breaker that opens when the core is slow, serve balance reads from a short-lived cache, and coordinate capacity with the mainframe team: they know the transaction limits and batch windows (when online processing is restricted).
Plan for data movement
For analytics, use change data capture or nightly extracts rather than querying the core. AWS DMS lists IBM Db2 for z/OS as a supported source with limitations ⚠ VERIFY current DMS support and CDC capabilities; specialist third-party CDC tools are also common. For modernising code later, look at AWS Transform for mainframe ⚠ VERIFY current availability, not the closed Mainframe Modernization runtime.
AWS Console instructions
- Amazon MQ broker: Console → Amazon MQ → Brokers → Create brokers → engine RabbitMQ → deployment Single-instance broker (lab) or Cluster deployment (production) → broker instance type (for a lab, the smallest offered for RabbitMQ 3.13) → Access type: Private access, choose a VPC, private subnet(s) and a security group allowing TCP 5671 from your application security group only → set an admin user name and a strong password (store it in Secrets Manager) → Maintenance: enable automatic minor version upgrades → add tags → Create broker. Creation takes several minutes.
- Endpoints: broker → Connections shows the AMQPS endpoint (
amqps://b-....mq.us-east-1.on.aws:5671) and the web console URL. - Site-to-Site VPN (design only): VPC → Customer gateways (your data centre's public IP and BGP ASN) → Transit gateways or Virtual private gateways → Site-to-Site VPN connections → Create → download the configuration for your device vendor.
- Private API: API Gateway → Create API → REST API → endpoint type Private → attach a resource policy allowing
execute-api:Invokeonly whenaws:SourceVpceequals your interface endpoint; create theexecute-apiinterface endpoint in VPC → Endpoints.
AWS CLI
PROFILE=learn-sandbox
REGION=us-east-1
# See which engine versions and instance types are offered right now
aws mq describe-broker-instance-options --engine-type RABBITMQ \
--query 'BrokerInstanceOptions[].{type:HostInstanceType,versions:SupportedEngineVersions[].Name}' \
--profile $PROFILE --region $REGION
# Read the broker password from Secrets Manager (created beforehand)
MQ_PASS=$(aws secretsmanager get-secret-value --secret-id hl-m04-mq-admin \
--query SecretString --output text --profile $PROFILE --region $REGION)
# Create a private single-instance RabbitMQ broker (billing starts at creation)
aws mq create-broker --broker-name hl-m04-core-mq \
--engine-type RABBITMQ --engine-version 3.13 \
--host-instance-type mq.t3.micro \
--deployment-mode SINGLE_INSTANCE \
--no-publicly-accessible \
--subnet-ids subnet-0aaa1111bbbb2222c \
--security-groups sg-0ccc3333dddd4444e \
--auto-minor-version-upgrade \
--users "Username=hl-admin,Password=$MQ_PASS" \
--tags Project=cea-lab,Lab=m04-l22,Environment=learn,Owner=you \
--profile $PROFILE --region $REGION
# Wait until RUNNING, then get the AMQPS endpoint
aws mq describe-broker --broker-id <broker-id> \
--query '{state:BrokerState,endpoints:BrokerInstances[0].Endpoints}' \
--profile $PROFILE --region $REGION
# Design-only: what a VPN connection would look like (do NOT run unless you have a real customer gateway)
# aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id cgw-... --transit-gateway-id tgw-...mq.t3.micro is offered for RabbitMQ 3.13 but not for 4.x, which runs only on mq.m7g instances; check the describe-broker-instance-options output before creating. Single-instance and micro brokers are for development only.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.AmazonMQ;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.SecretsManager;
using Constructs;
namespace Harbourline.CoreIntegration.Infra;
public sealed class CoreMessagingStackProps : StackProps
{
public required IVpc Vpc { get; init; }
public required ISecurityGroup AppSecurityGroup { get; init; }
}
public sealed class CoreMessagingStack : Stack
{
public CoreMessagingStack(Construct scope, string id, CoreMessagingStackProps props) : base(scope, id, props)
{
var brokerSg = new SecurityGroup(this, "BrokerSg", new SecurityGroupProps
{
Vpc = props.Vpc,
AllowAllOutbound = false,
Description = "Amazon MQ: AMQPS from the core adapter only"
});
brokerSg.AddIngressRule(props.AppSecurityGroup, Port.Tcp(5671), "AMQPS from core adapter");
var admin = new Secret(this, "MqAdmin", new SecretProps
{
SecretName = "hl-m04-mq-admin",
GenerateSecretString = new SecretStringGenerator
{
SecretStringTemplate = "{\"username\":\"hl-admin\"}",
GenerateStringKey = "password",
ExcludeCharacters = ",:=", // characters RabbitMQ broker passwords cannot contain ⚠ VERIFY
PasswordLength = 32
}
});
var subnet = props.Vpc.SelectSubnets(new SubnetSelection { SubnetType = SubnetType.PRIVATE_WITH_EGRESS }).SubnetIds[0];
new CfnBroker(this, "CoreMq", new CfnBrokerProps
{
BrokerName = "hl-m04-core-mq",
EngineType = "RABBITMQ",
EngineVersion = "3.13",
HostInstanceType = "mq.t3.micro",
DeploymentMode = "SINGLE_INSTANCE", // production: CLUSTER_MULTI_AZ on mq.m7g
PubliclyAccessible = false,
AutoMinorVersionUpgrade = true,
SubnetIds = [subnet],
SecurityGroups = [brokerSg.SecurityGroupId],
Users = new object[]
{
new CfnBroker.UserProperty
{
Username = admin.SecretValueFromJson("username").UnsafeUnwrap(),
Password = admin.SecretValueFromJson("password").UnsafeUnwrap()
}
},
Logs = new CfnBroker.LogListProperty { General = true }
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m04-l22");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "you");
}
}UnsafeUnwrap() passes a dynamic reference to the secret into the template (resolved by CloudFormation at deploy time), so the password is not written into the synthesized template; the method name is a reminder to use it only for properties that accept secrets.
Complete code
A .NET 10 console app that runs entirely against RabbitMQ in Docker. It starts a fake core banking responder (which answers balance enquiries with a 62-byte EBCDIC copybook record containing packed decimals) and a core adapter that sends request/reply messages with correlation IDs and timeouts, parses the copybook and maps it through the anti-corruption layer to Harbourline's Account.
# Free local broker: management UI on http://localhost:15672 (guest/guest works only from localhost)
docker run -d --name hl-rabbit -p 5672:5672 -p 15672:15672 rabbitmq:4-management<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="RabbitMQ.Client" Version="7.2.2" />
</ItemGroup>
</Project>using System.Collections.Concurrent;
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); // enables EBCDIC code page 037
var factory = new ConnectionFactory
{
HostName = Environment.GetEnvironmentVariable("MQ_HOST") ?? "localhost",
Port = int.Parse(Environment.GetEnvironmentVariable("MQ_PORT") ?? "5672"),
UserName = Environment.GetEnvironmentVariable("MQ_USER") ?? "guest",
Password = Environment.GetEnvironmentVariable("MQ_PASSWORD") ?? "guest",
// Amazon MQ: Port = 5671 and Ssl = new SslOption { Enabled = true, ServerName = host }
ClientProvidedName = "hl-core-adapter"
};
await using var connection = await factory.CreateConnectionAsync();
// ---- Fake core banking system (in reality: CICS behind IBM MQ, reached through a bridge) ----
await using var coreChannel = await connection.CreateChannelAsync();
await FakeCore.StartAsync(coreChannel);
// ---- Core adapter (anti-corruption layer + request/reply client) ----
await using var adapter = await CoreAdapter.CreateAsync(connection, timeout: TimeSpan.FromSeconds(2), maxConcurrency: 20);
foreach (var id in new[] { "ACC-0000123456", "ACC-0000123457", "ACC-0000999999" })
{
var result = await adapter.GetAccountAsync(id);
Console.WriteLine(result switch
{
CoreResult<Account>.Ok ok => $"{ok.Value.AccountId} {ok.Value.Status,-6} {ok.Value.Product,-8} {ok.Value.Balance,12:N2} {ok.Value.Currency} (available {ok.Value.AvailableBalance:N2})",
CoreResult<Account>.NotFound => $"{id} not found in core",
CoreResult<Account>.Unavailable u => $"{id} core unavailable: {u.Reason}",
_ => "unexpected"
});
}
// =================== Domain (Harbourline) ===================
public enum AccountStatus { Open, Frozen, Closed }
public enum Product { Current, Savings, Business, CreditCard }
public sealed record Account(string AccountId, string CustomerId, string Currency, Product Product,
AccountStatus Status, decimal Balance, decimal AvailableBalance, DateOnly OpenedOn);
public abstract record CoreResult<T>
{
public sealed record Ok(T Value) : CoreResult<T>;
public sealed record NotFound : CoreResult<T>;
public sealed record Unavailable(string Reason) : CoreResult<T>;
}
// =================== Copybook layout ===================
// 01 ACCT-BALANCE-REC. offset length
// 05 ACCT-NO PIC X(14). 0 14
// 05 CUST-NO PIC X(12). 14 12
// 05 CCY PIC X(3). 26 3
// 05 PRODUCT-CD PIC X(2). 29 2 CA, SA, BU, CC
// 05 STATUS-CD PIC X(1). 31 1 O, F, C
// 05 LEDGER-BAL PIC S9(13)V99 COMP-3. 32 8
// 05 AVAIL-BAL PIC S9(13)V99 COMP-3. 40 8
// 05 OPEN-DATE PIC 9(8). 48 8 YYYYMMDD, zoned display
// 05 FILLER PIC X(6). 56 6
public static class AcctBalanceRec
{
public const int Length = 62;
public static readonly Encoding Ebcdic = Encoding.GetEncoding(37); // IBM037, US/Canada EBCDIC ⚠ check your core's CCSID
public sealed record Raw(string AcctNo, string CustNo, string Ccy, string ProductCd, string StatusCd,
decimal LedgerBal, decimal AvailBal, string OpenDate);
public static Raw Parse(ReadOnlySpan<byte> rec)
{
if (rec.Length != Length) throw new FormatException($"Expected {Length} bytes, got {rec.Length}");
return new Raw(
AcctNo: Text(rec, 0, 14),
CustNo: Text(rec, 14, 12),
Ccy: Text(rec, 26, 3),
ProductCd: Text(rec, 29, 2),
StatusCd: Text(rec, 31, 1),
LedgerBal: PackedDecimal.Decode(rec.Slice(32, 8), scale: 2),
AvailBal: PackedDecimal.Decode(rec.Slice(40, 8), scale: 2),
OpenDate: Text(rec, 48, 8));
static string Text(ReadOnlySpan<byte> r, int offset, int length) => Ebcdic.GetString(r.Slice(offset, length)).TrimEnd();
}
public static byte[] Build(Raw r) // used by the fake core and by tests
{
var buf = new byte[Length];
void Put(int offset, int length, string s) => Ebcdic.GetBytes(s.PadRight(length).AsSpan(0, length), buf.AsSpan(offset, length));
Put(0, 14, r.AcctNo); Put(14, 12, r.CustNo); Put(26, 3, r.Ccy); Put(29, 2, r.ProductCd); Put(31, 1, r.StatusCd);
PackedDecimal.Encode(r.LedgerBal, 2, buf.AsSpan(32, 8));
PackedDecimal.Encode(r.AvailBal, 2, buf.AsSpan(40, 8));
Put(48, 8, r.OpenDate); Put(56, 6, "");
return buf;
}
}
// COMP-3: two decimal digits per byte; the last nibble is the sign (C or F = positive, D = negative).
public static class PackedDecimal
{
public static decimal Decode(ReadOnlySpan<byte> bytes, int scale)
{
decimal value = 0;
for (var i = 0; i < bytes.Length; i++)
{
int hi = bytes[i] >> 4, lo = bytes[i] & 0x0F;
value = value * 10 + Digit(hi);
if (i < bytes.Length - 1) value = value * 10 + Digit(lo);
else
{
return lo switch
{
0xC or 0xF => value / Pow10(scale),
0xD => -value / Pow10(scale),
_ => throw new FormatException($"Invalid COMP-3 sign nibble 0x{lo:X}")
};
}
}
throw new FormatException("Empty COMP-3 field");
static int Digit(int n) => n <= 9 ? n : throw new FormatException($"Invalid COMP-3 digit nibble 0x{n:X}");
}
public static void Encode(decimal value, int scale, Span<byte> dest)
{
var digits = decimal.Round(Math.Abs(value) * Pow10(scale), 0).ToString("0", System.Globalization.CultureInfo.InvariantCulture);
var nibbles = dest.Length * 2 - 1;
if (digits.Length > nibbles) throw new OverflowException($"{value} does not fit in {dest.Length} bytes");
digits = digits.PadLeft(nibbles, '0');
var sign = value < 0 ? 0xD : 0xC;
for (var i = 0; i < dest.Length; i++)
{
var hi = digits[i * 2] - '0';
var lo = i < dest.Length - 1 ? digits[i * 2 + 1] - '0' : sign;
dest[i] = (byte)((hi << 4) | lo);
}
}
private static decimal Pow10(int n) { decimal p = 1; for (var i = 0; i < n; i++) p *= 10; return p; }
}
// =================== Anti-corruption layer mapping ===================
public static class CoreAccountMapper
{
public static Account ToDomain(AcctBalanceRec.Raw r) => new(
AccountId: r.AcctNo,
CustomerId: r.CustNo,
Currency: r.Ccy is "GBP" or "EUR" ? r.Ccy : throw new FormatException($"Unsupported currency {r.Ccy}"),
Product: r.ProductCd switch
{
"CA" => Product.Current, "SA" => Product.Savings, "BU" => Product.Business, "CC" => Product.CreditCard,
_ => throw new FormatException($"Unknown product code '{r.ProductCd}'")
},
Status: r.StatusCd switch
{
"O" => AccountStatus.Open, "F" => AccountStatus.Frozen, "C" => AccountStatus.Closed,
_ => throw new FormatException($"Unknown status code '{r.StatusCd}'")
},
Balance: r.LedgerBal,
AvailableBalance: r.AvailBal,
OpenedOn: DateOnly.ParseExact(r.OpenDate, "yyyyMMdd", System.Globalization.CultureInfo.InvariantCulture));
}
// =================== Request/reply adapter ===================
public sealed class CoreAdapter : IAsyncDisposable
{
private const string RequestQueue = "core.balance.requests";
private readonly IChannel _channel;
private readonly string _replyQueue;
private readonly TimeSpan _timeout;
private readonly SemaphoreSlim _concurrency;
private readonly ConcurrentDictionary<string, TaskCompletionSource<byte[]>> _pending = new();
private CoreAdapter(IChannel channel, string replyQueue, TimeSpan timeout, int maxConcurrency)
=> (_channel, _replyQueue, _timeout, _concurrency) = (channel, replyQueue, timeout, new SemaphoreSlim(maxConcurrency));
public static async Task<CoreAdapter> CreateAsync(IConnection connection, TimeSpan timeout, int maxConcurrency)
{
var channel = await connection.CreateChannelAsync();
// Exclusive, auto-delete reply queue for this adapter instance.
var reply = await channel.QueueDeclareAsync(queue: "", durable: false, exclusive: true, autoDelete: true);
var adapter = new CoreAdapter(channel, reply.QueueName, timeout, maxConcurrency);
var consumer = new AsyncEventingBasicConsumer(channel);
consumer.ReceivedAsync += (_, ea) =>
{
var id = ea.BasicProperties.CorrelationId;
if (id is not null && adapter._pending.TryRemove(id, out var tcs))
tcs.TrySetResult(ea.Body.ToArray()); // copy: the body buffer is reused after the handler returns
else
Console.WriteLine($"Dropped late or unknown reply {id}");
return Task.CompletedTask;
};
await channel.BasicConsumeAsync(reply.QueueName, autoAck: true, consumer);
return adapter;
}
public async Task<CoreResult<Account>> GetAccountAsync(string accountId, CancellationToken ct = default)
{
if (!await _concurrency.WaitAsync(_timeout, ct))
return new CoreResult<Account>.Unavailable("adapter at concurrency limit"); // back-pressure, protects the core
var correlationId = Guid.CreateVersion7().ToString("N");
var tcs = new TaskCompletionSource<byte[]>(TaskCreationOptions.RunContinuationsAsynchronously);
_pending[correlationId] = tcs;
try
{
var props = new BasicProperties
{
CorrelationId = correlationId,
ReplyTo = _replyQueue,
ContentType = "application/x-ebcdic-fixed",
Expiration = ((int)_timeout.TotalMilliseconds).ToString(System.Globalization.CultureInfo.InvariantCulture) // stale requests expire
};
var request = AcctBalanceRec.Ebcdic.GetBytes($"BALINQ {accountId,-14}");
await _channel.BasicPublishAsync(exchange: "", routingKey: RequestQueue, mandatory: false, basicProperties: props, body: request, cancellationToken: ct);
using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
timeoutCts.CancelAfter(_timeout);
byte[] reply;
try { reply = await tcs.Task.WaitAsync(timeoutCts.Token); }
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
{
return new CoreResult<Account>.Unavailable($"no reply within {_timeout.TotalSeconds}s");
}
// Reply: 2-byte return code ("00" ok, "04" not found) + record
var rc = AcctBalanceRec.Ebcdic.GetString(reply, 0, 2);
return rc switch
{
"00" => new CoreResult<Account>.Ok(CoreAccountMapper.ToDomain(AcctBalanceRec.Parse(reply.AsSpan(2)))),
"04" => new CoreResult<Account>.NotFound(),
_ => new CoreResult<Account>.Unavailable($"core return code {rc}")
};
}
finally
{
_pending.TryRemove(correlationId, out _);
_concurrency.Release();
}
}
public async ValueTask DisposeAsync()
{
await _channel.CloseAsync();
_channel.Dispose();
_concurrency.Dispose();
}
}
// =================== Fake core banking responder ===================
public static class FakeCore
{
private static readonly Dictionary<string, AcctBalanceRec.Raw> Accounts = new()
{
["ACC-0000123456"] = new("ACC-0000123456", "CUS-00012345", "GBP", "CA", "O", 1520.40m, 1420.40m, "20190314"),
["ACC-0000123457"] = new("ACC-0000123457", "CUS-00012345", "GBP", "CC", "F", -245.99m, 0m, "20210602")
};
public static async Task StartAsync(IChannel channel)
{
await channel.QueueDeclareAsync("core.balance.requests", durable: true, exclusive: false, autoDelete: false);
await channel.BasicQosAsync(prefetchSize: 0, prefetchCount: 10, global: false);
var consumer = new AsyncEventingBasicConsumer(channel);
consumer.ReceivedAsync += async (_, ea) =>
{
var text = AcctBalanceRec.Ebcdic.GetString(ea.Body.Span);
var accountId = text.Length >= 22 ? text[8..22].Trim() : "";
var reply = Accounts.TryGetValue(accountId, out var rec)
? [.. AcctBalanceRec.Ebcdic.GetBytes("00"), .. AcctBalanceRec.Build(rec)]
: AcctBalanceRec.Ebcdic.GetBytes("04");
var props = new BasicProperties { CorrelationId = ea.BasicProperties.CorrelationId };
await channel.BasicPublishAsync(exchange: "", routingKey: ea.BasicProperties.ReplyTo!, mandatory: false, basicProperties: props, body: reply);
await channel.BasicAckAsync(ea.DeliveryTag, multiple: false);
};
await channel.BasicConsumeAsync("core.balance.requests", autoAck: false, consumer);
}
}Run it:
cd src/Harbourline.CoreIntegration
dotnet run
# ACC-0000123456 Open Current 1,520.40 GBP (available 1,420.40)
# ACC-0000123457 Frozen CreditCard -245.99 GBP (available 0.00)
# ACC-0000999999 not found in core
# Simulate a core outage: stop the broker, run again, and observe "core unavailable" / connection errors
docker stop hl-rabbitThe rabbitmq:4-management image tag tracks the latest 4.x release; pin an exact tag in real projects ⚠ VERIFY the current tag on Docker Hub. To run against Amazon MQ, set MQ_HOST, MQ_PORT=5671, MQ_USER, MQ_PASSWORD and enable TLS in ConnectionFactory as noted in the code comment.
Code explained line by line
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance)makes legacy code pages available in .NET; without it,Encoding.GetEncoding(37)throws. Code page 037 is common for US/UK mainframes, but your core may use another CCSID (for example 285 for UK, 1140 with the euro sign); confirm with the mainframe team.ConnectionFactoryreads host and credentials from environment variables; in AWS these come from Secrets Manager. Amazon MQ requires TLS (AMQPS on 5671).- One connection, multiple channels: connections are expensive TCP + TLS sessions; channels are lightweight. Creating a connection per request was one of the incident's root causes.
- The copybook comment block documents offsets and lengths next to the code. Keep the original copybook in the repository and generate or check offsets in tests.
AcctBalanceRec.Parseslices the fixed-width record. Text fields are EBCDIC and right-padded with spaces, henceTrimEnd().PackedDecimal.Decodereads two digits per byte; the last byte holds one digit and the sign nibble (C/Fpositive,Dnegative).PIC S9(13)V99 COMP-3is 15 digits + sign = 16 nibbles = 8 bytes, with an implied decimal point two digits from the right (scale: 2). Invalid nibbles throw: never silently accept corrupted money.PackedDecimal.Encodeis the inverse, used by the fake core and in round-trip tests.CoreAccountMapperis the anti-corruption layer: two-letter product codes and one-letter status codes become enums, unknown codes throw, and only GBP/EUR are accepted. The rest of the platform sees onlyAccount.CoreAdapter.CreateAsyncdeclares an exclusive, auto-delete reply queue (server-named) and one consumer that completes pending requests by correlation ID.ea.Body.ToArray()copies the message body; in RabbitMQ.Client 7 the body memory is only valid during the handler._concurrency(aSemaphoreSlim) caps in-flight requests per adapter instance. When the core is slow, callers get a fast "unavailable" instead of piling up: back-pressure protects the core.Expirationon the request message makes the broker discard it if it waits in the queue longer than the timeout, so the core never processes a request whose caller has already given up.tcs.Task.WaitAsync(timeoutCts.Token)waits for the reply with a timeout. A timeout is reported asUnavailable; for reads the caller may retry or show cached data; for writes it means "outcome unknown" and must be followed by a status query, never a blind retry.- Late replies (after timeout) have no pending entry and are dropped with a log line.
- The fake core uses
BasicQosAsync(prefetchCount: 10)and manual acks, like a well-behaved consumer: it takes at most 10 messages at a time and acknowledges only after replying.
Security considerations
⚠ Learning shortcut — not for production
The local demo uses RabbitMQ's default guest account without TLS, and the fake core runs in the same process. In production: TLS everywhere (Amazon MQ requires it), one broker user per application with permissions limited to its own queues (RabbitMQ vhost/queue permissions or ActiveMQ authorization entries), credentials in Secrets Manager with rotation, and brokers in private subnets with security groups allowing only the adapter.
⚠ Learning shortcut — not for production
The lab broker is SINGLE_INSTANCE on a micro instance. Production core integrations use a cluster (RabbitMQ) or active/standby (ActiveMQ) deployment across Availability Zones, with alarms on queue depth, consumer count and broker health.
- No internet exposure. Do not create publicly accessible brokers for core integration. Use private access and reach on-premises over VPN/Direct Connect.
- Encrypt the network path. Direct Connect is not encrypted by default; use MACsec where supported or IPsec VPN over Direct Connect if your policy requires encryption in transit on private links.
- Least privilege on both sides. The mainframe security team maps each cloud integration identity to a minimal set of transactions.
- Data minimisation. Request only the fields needed; never log raw copybook records (they contain account and customer identifiers).
- Poison messages. A malformed reply must not crash the adapter in a loop; use dead-letter exchanges/queues on the broker and alarms on their depth.
- Change control. Copybook changes on the mainframe side are breaking changes for you. Agree versioning and run contract tests on every change.
Banking use case
Harbourline's core integration (fictional) after the incident:
- Connectivity: two Direct Connect connections at different locations into a Direct Connect gateway and Transit Gateway; Site-to-Site VPN as encrypted backup; BGP prefers Direct Connect.
- Reads: balance and account details via request/reply over Amazon MQ (RabbitMQ cluster) bridged to the mainframe's IBM MQ, with a 2-second timeout and a 10-second cache in the Accounts API. When the core is in its nightly batch window, the API serves the last known balance with an "as of" timestamp.
- Writes: payment postings as persistent messages carrying
payment_idas the idempotency key; the core replies with a posting reference; a reconciliation job compares postings withpaymentsevery hour. - Analytics: nightly extracts and CDC into the data platform, never direct queries against Db2 from the cloud.
- Inbound from the core: batch jobs call a private REST API through an
execute-apiinterface endpoint to publish end-of-day events into AWS. - Operations: dashboards for queue depth, reply latency (p99), timeouts, circuit breaker state and Direct Connect/VPN tunnel status; runbooks agreed with the mainframe operations team.
Common mistakes
- Calling the core over the internet "temporarily".
- New connection per request to the broker or the core API.
- Blind retries on postings. Timeouts mean unknown outcome; query status by idempotency key.
- No message expiration. The core processes requests long after callers gave up.
- Leaking mainframe concepts (copybook field names, codes, EBCDIC) beyond the adapter.
- Wrong code page. Characters like
£,#or[differ between EBCDIC code pages; test with real samples. - Treating COMP-3 like text. Packed bytes converted with an encoding produce garbage; slice them out first.
- Single Direct Connect connection with no backup path.
- Assuming Amazon MQ speaks IBM MQ. It does not; plan a bridge or a different interface.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
No data is available for encoding 37 | Code pages provider not registered | Encoding.RegisterProvider(CodePagesEncodingProvider.Instance) at startup |
Names show à or odd symbols | Wrong EBCDIC code page | Confirm the CCSID with the mainframe team (037, 285, 1140…) |
Invalid COMP-3 sign nibble | Offsets wrong, or field is zoned/binary (COMP) not packed | Recheck the copybook; compare a hex dump with expected values |
| Connection refused to Amazon MQ | Security group, wrong port (5671), or broker not RUNNING | Allow 5671 from the app SG; check describe-broker |
| TLS handshake fails to Amazon MQ | TLS not enabled in ConnectionFactory or wrong ServerName | Ssl = new SslOption { Enabled = true, ServerName = host } |
| Replies arrive after timeouts | Core slow, or queue backlog | Check queue depth; lower concurrency; agree capacity; add caching |
ACCESS_REFUSED on publish | Broker user lacks permission on the queue/vhost | Grant configure/write/read permissions on the specific resources |
| VPN tunnel up but no traffic | Route tables/propagation or on-prem firewall | Check TGW/VGW route propagation and on-prem routes for the VPC CIDR |
Cost and cleanup
- Free path: RabbitMQ in Docker and the C# app cost nothing. VPN and Direct Connect are design exercises.
- Amazon MQ (paid): billed per broker instance-hour from creation to deletion plus storage (for example about USD 0.27/hour per
mq.m7g.largeRabbitMQ instance and USD 0.10 per GB-month of EBS storage for RabbitMQ; a three-node cluster triples the instance cost). A micro single-instance broker for one hour costs cents ⚠ VERIFY themq.t3.microrate on the pricing page. New accounts may cover this from free plan credits. - Site-to-Site VPN: per connection-hour plus data transfer out. Direct Connect: port-hours plus data transfer. Neither is needed for this lesson.
# Delete the broker (billing stops after deletion completes)
aws mq delete-broker --broker-id <broker-id> --profile learn-sandbox --region us-east-1
aws secretsmanager delete-secret --secret-id hl-m04-mq-admin --force-delete-without-recovery \
--profile learn-sandbox --region us-east-1
docker rm -f hl-rabbit
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandbox --region us-east-1With CDK or Terraform, cdk destroy / terraform destroy.
Exercises
- 🟢 Guided: Run the demo against Docker RabbitMQ. Then change the fake core's ledger balance for
ACC-0000123456to-0.01mand confirm the adapter shows-0.01. Look at the 8 packed bytes in a hex dump (Convert.ToHexString) and identify the sign nibble. - 🟡 Challenge: Make the fake core slow: add
await Task.Delay(3000)before replying for one account. Show that the adapter returns "unavailable" after 2 seconds, that the late reply is dropped, and that the request message expires if the core is stopped. Then add a simple circuit breaker (open after 5 consecutive timeouts for 30 seconds). - 🔴 Live problem: The mainframe team announces: "Next month ACCT-BALANCE-REC gains a 3-byte OVERDRAFT-LIMIT field (PIC S9(5) COMP-3) after AVAIL-BAL; FILLER shrinks to 3 bytes; total length unchanged." Your adapter is deployed in 4 services. Plan the change so nothing breaks on the day the mainframe deploys, and explain how you test it in advance.
Assignment
Write an integration design for "post a payment to the core": sequence diagram, message formats (request, reply, error), idempotency key handling, timeout and "unknown outcome" procedure, reconciliation, capacity limits agreed with the core team, and the network path. Add the copybook, the C# layout class and round-trip tests to your portfolio repository.
Solutions
-0.01 with scale 2 is the digit string 000000000000001 and sign D, so the last byte is 0x1D and the field is 000000000000001D in hex. The adapter prints -0.01. Why it matters: the sign lives in the last nibble, not in a separate character; that is exactly what a naive EBCDIC-to-text conversion destroys.
With the delay, GetAccountAsync returns Unavailable("no reply within 2s"); about 1 second later the console prints "Dropped late or unknown reply …" because the pending entry was already removed. If you stop the fake core's consumer, request messages disappear from core.balance.requests after 2 seconds thanks to Expiration (visible in the management UI). For the circuit breaker, keep a counter of consecutive timeouts and an "open until" timestamp in the adapter; while open, return Unavailable("circuit open") immediately without publishing. In production you would use Microsoft.Extensions.Resilience pipelines (as in Calling partner APIs safely) around the request method. Why: fail fast while the core is struggling, so you do not add load to a system that is already overloaded.
"Total length unchanged" hides a breaking change: in this copybook OPEN-DATE sits right after AVAIL-BAL, so inserting a 3-byte field there moves OPEN-DATE from offset 48 to 51, and today's parser would read the wrong bytes as the date. Plan: (1) get the new copybook and real sample records now; (2) add a second layout (AcctBalanceRecV2: OVERDRAFT-LIMIT at 48 for 3 bytes, OPEN-DATE at 51, FILLER at 59); (3) decide how the adapter knows which layout it receives: ideally the mainframe adds a record version or the reply header carries one; otherwise use a feature flag switched in the same change window as the mainframe deployment; (4) add round-trip and contract tests for both layouts with the real samples; (5) deploy the adapter that understands both versions to all 4 services before the mainframe change; (6) remove V1 after the switch. Why: fixed-width formats have no field names, so any offset change is breaking; versioned layouts plus early deployment avoid a big-bang release.
Knowledge check
Answer: When the other side needs a standard broker protocol (JMS/OpenWire, AMQP, STOMP, MQTT) and you cannot change it, as with many on-premises and mainframe integrations. For new cloud-native messaging, SQS/SNS/EventBridge are simpler and cheaper.
Answer: No. Use MACsec on supported dedicated connections, or run Site-to-Site VPN over Direct Connect, if you need encryption in transit on that path; application-level TLS should be used regardless.
Answer: 8 bytes: 15 digits plus a sign nibble make 16 nibbles. The sign is the last (low) nibble of the last byte: C or F positive, D negative.
Answer: Not blindly. A timeout means the outcome is unknown; query the status using the idempotency key first, and retry only if the core confirms it never processed the request (or if the core-side operation is idempotent on that key).
Answer: Use a private REST API reachable through an execute-api interface VPC endpoint, with on-premises DNS resolving it (for example via a Route 53 Resolver inbound endpoint) over VPN or Direct Connect, and a resource policy restricting access to that endpoint.
Interview questions
- "How would you integrate a new cloud app with a mainframe core?" — Private network path (DX + VPN backup), choose API vs messaging per use case, an anti-corruption layer for copybooks and codes, idempotent writes with status queries on timeout, back-pressure and caching to protect the core, contract tests, and joint runbooks with the mainframe team.
- "What is COMP-3?" — Packed decimal: two digits per byte with a sign nibble at the end; implied decimal places come from the copybook's
V. Decode it from bytes, not via text encoding. - "Does Amazon MQ support IBM MQ?" — No. It runs ActiveMQ and RabbitMQ. Bridge from IBM MQ with a JMS bridge, run IBM MQ yourself on AWS, or use an API interface.
- "VPN or Direct Connect?" — VPN is quick, encrypted, internet-based and good for starting or as backup; Direct Connect gives consistent latency and bandwidth but takes time, is not encrypted by default, and needs redundancy.
- "What is an anti-corruption layer?" — A translation boundary that keeps a legacy model's concepts out of your domain, so the legacy system can change or be replaced without rewriting everything.
Certification practice
A bank needs consistent, low-latency private connectivity from AWS to its data centre for core banking traffic, with an encrypted backup path if the primary fails. Which design meets the requirements MOST cost-effectively?
A. Two Site-to-Site VPN connections over the internet. B. One AWS Direct Connect connection as primary and an AWS Site-to-Site VPN connection as backup, both terminating on a Transit Gateway. C. VPC peering between the VPC and the data centre. D. Amazon CloudFront with an origin in the data centre.
Answer: B. Direct Connect provides consistent performance; VPN gives an encrypted, low-cost backup. A lacks consistent latency. C: VPC peering connects VPCs, not data centres. D is a CDN for content delivery, not private connectivity. (For maximum resilience, banks often add a second Direct Connect location.)
An on-premises application uses JMS over OpenWire to exchange messages with a message broker. The company wants to move the broker to AWS with minimal application changes. Which service should they use?
A. Amazon SQS B. Amazon MQ for ActiveMQ C. Amazon Kinesis Data Streams D. Amazon SNS
Answer: B. Amazon MQ for ActiveMQ supports JMS and OpenWire, so clients keep working. SQS, Kinesis and SNS use AWS APIs and would require code changes.
A service sends request messages to a broker and waits up to 2 seconds for replies. During core outages, requests pile up in the queue and the core later processes thousands of stale requests. What should the developer do?
A. Increase the wait time to 30 seconds. B. Set a message expiration (TTL) equal to the client timeout on each request. C. Use a single shared reply queue for all instances. D. Disable acknowledgements on the core side.
Answer: B. Expired requests are discarded by the broker instead of being processed after the caller has given up. A makes callers wait longer and still processes stale work. C does not address staleness. D risks losing messages.
Completion checklist
- I can compare Site-to-Site VPN and Direct Connect, including encryption and resilience
- I can design private API access in both directions
- I ran the RabbitMQ request/reply demo with correlation IDs and timeouts
- I can decode EBCDIC text and COMP-3 packed decimals in C#
- I can explain the anti-corruption layer and why postings need idempotency and status queries
- I deleted any Amazon MQ broker and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Integrate with the core over private, redundant, encrypted paths; never over the open internet.
- Choose APIs for quick reads, messaging for reliable writes, files/CDC for bulk; protect the core with timeouts, concurrency limits and caching.
- Amazon MQ (ActiveMQ/RabbitMQ) exists for protocol compatibility; it is not IBM MQ.
- Keep copybooks, EBCDIC and COMP-3 inside one anti-corruption layer with round-trip and contract tests.
- A timed-out posting is an unknown outcome: query by idempotency key, never blindly retry.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
SWIFT and file-based integration: SFTP with AWS Transfer Family, PGP and checksums
Exchange batch payment and statement files with partners safely: understand where Swift and scheme file channels fit, run SFTP into S3 with AWS Transfer Family (or a free local SFTP server), encrypt and decrypt with OpenPGP in C#, verify SHA-256 manifests and S3 checksums, and process each file exactly once.
Module project: the Account Information API (Cognito, ownership checks, idempotency, throttling and WAF)
Run, attack and optionally deploy a .NET 10 Account Information API where each customer sees only their own accounts: Cognito access tokens and scopes, server-side customer mapping, validation, idempotent statement requests, ProblemDetails, correlation IDs, throttling and AWS WAF.