Skip to Content
Cloud Engineering Academy
4. API Development & Integration

Integrating with core banking and mainframes: hybrid networks, private APIs and MQ messaging

Connect cloud services to an on-premises core banking system safely: choose between Site-to-Site VPN and Direct Connect, expose and consume private APIs, use request/reply messaging with Amazon MQ (or RabbitMQ locally), and translate COBOL copybook records with EBCDIC and packed decimals into a clean C# domain model behind an anti-corruption layer.

advanced⏱ 2 h 30 min🔴 Paid — optionalSAA-C03DVA-C03SCS-C03

Assumes: .NET 10 LTS · RabbitMQ.Client 7.2.2 · Amazon MQ (RabbitMQ 3.13/4.x, ActiveMQ 5.18/5.19) · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Service availability — verified 2026-11-17

AWS Mainframe Modernization managed runtime environments closed to new customers on 2025-11-07, and the self-managed experience entered maintenance. This lesson does not build on that service. It integrates with a mainframe that stays where it is, through private networking, APIs and messaging.

Cost: Amazon MQ and hybrid networking bill per hour

An Amazon MQ broker bills per instance-hour from creation until deletion (for example about USD 0.27 per hour for a RabbitMQ mq.m7g.large instance in us-east-1, verified 2026-11-18), plus storage. Site-to-Site VPN connections and Direct Connect ports also bill hourly. The free path runs RabbitMQ in Docker with exactly the same C# client code; VPN and Direct Connect are design-only exercises.

Learning objectives

By the end of this lesson you can:

  • Explain the shape of a core banking system and why integration, not replacement, is the normal starting point.
  • Choose between Site-to-Site VPN, Direct Connect and combinations (with Transit Gateway) for hybrid connectivity, including encryption and resilience.
  • Design both directions of private API access: AWS services calling on-premises APIs (NLB with IP targets, VPC Link) and on-premises systems calling AWS (private REST APIs through an interface VPC endpoint).
  • Use MQ-style messaging with Amazon MQ (ActiveMQ or RabbitMQ) for request/reply with correlation IDs, timeouts and idempotent consumers, and explain how IBM MQ estates fit.
  • Parse COBOL copybook records in C#: EBCDIC text, zoned decimals and COMP-3 packed decimals, and map them through an anti-corruption layer into Harbourline's domain model.

Prerequisites

Why this matters in banking

Harbourline Bank (fictional) runs its accounts, balances and postings on a 30-year-old core banking system on a mainframe in its data centre. The new mobile app, built on AWS, needs balances in under 300 ms. The first prototype calls the core directly from a Lambda function over the internet through a reverse proxy, opens a new connection per request, and retries every timeout three times.

On payday morning the core's transaction monitor hits its connection limit. Retries triple the load; balance enquiries for branch staff start timing out too; a duplicated debit posting appears because a "timed-out" request had actually succeeded. The incident report has three findings: the network path was neither private nor resilient, the integration had no back-pressure, and the cloud code leaked mainframe concepts (field names like LEDGER-BAL, packed decimals) everywhere.

Most banks will run their core for years alongside the cloud. Integrating with it safely, privately and politely (without overloading it) is one of the most valuable skills a cloud engineer at a bank can have.

The idea in plain language

A core banking system is the system of record for accounts, balances and postings. Mainframe cores typically expose functionality as transactions (programs in CICS or IMS), batch jobs that run at night, and data in Db2 or VSAM files. Data formats come from COBOL: fixed-width records described by copybooks, text in EBCDIC (not ASCII), and numbers in zoned or packed decimal (COMP-3).

Integration has three parts:

  1. A private, reliable network path between AWS and the data centre: an encrypted VPN over the internet, a dedicated Direct Connect link, or both.
  2. An integration style:
    • Synchronous APIs (the mainframe team publishes REST APIs through an API layer such as z/OS Connect ⚠ VERIFY product names your bank uses) for quick queries like balance enquiry.
    • Messaging (IBM MQ on the mainframe side; Amazon MQ or a bridge on AWS) for postings and anything that must not be lost, with request/reply when an answer is needed.
    • Files and change data capture for bulk and analytics (nightly extracts, CDC streams).
  3. An anti-corruption layer (ACL): one well-tested component that translates between the core's world (copybooks, codes like 'O'/'F', EBCDIC) and your clean domain (Account, AccountStatus.Open, decimal). Nothing outside it knows the mainframe exists.

Analogy: The core banking system is a senior colleague in another building who speaks a different language, works carefully and cannot be rushed. You build a private corridor to their building (VPN/Direct Connect), agree to send requests through their in-tray rather than barging in (message queue), hire one interpreter who knows both languages (anti-corruption layer), and never send the same request twice without a reference number (idempotency).

What problem the AWS services solve

NeedAWS building blockHonest limitations
Encrypted connectivity over the internetAWS Site-to-Site VPN (two IPsec tunnels per connection, to a virtual private gateway or Transit Gateway)Internet path: variable latency; per-tunnel bandwidth limits (about 1.25 Gbps ⚠ VERIFY); billed per connection-hour plus data transfer
Dedicated private connectivityAWS Direct Connect (dedicated or hosted connections at a Direct Connect location)Weeks to provision; not encrypted by default (use MACsec on supported dedicated connections, or VPN over Direct Connect); a single connection is a single point of failure
Hub for many VPCs and on-prem linksAWS Transit GatewayPer-attachment-hour and per-GB processing charges
AWS → on-prem HTTP APIsNetwork Load Balancer with on-premises IP targets, reached from API Gateway through a VPC Link or directly from servicesIP targets outside the VPC must be in supported private ranges and reachable via VPN/DX ⚠ VERIFY current rules
On-prem → AWS private APIsAPI Gateway private REST API with an interface VPC endpoint (execute-api) and a resource policyPrivate APIs are REST only; on-prem DNS must resolve the endpoint (Route 53 Resolver inbound endpoint)
Managed message broker for JMS/AMQP/MQTTAmazon MQ for ActiveMQ (5.18, 5.19) or RabbitMQ (3.13 on t3/m5/m7g; 4.2 and 4.3 on m7g only)Hourly per broker instance; not IBM MQ; RabbitMQ streams are not supported
Cloud-native queues and eventsSQS, SNS, EventBridgeNot protocol-compatible with JMS/AMQP clients on the mainframe side

Amazon MQ exists for protocol compatibility: when the other side speaks JMS, AMQP, STOMP, MQTT or OpenWire and you cannot change it. For new cloud-to-cloud messaging, SQS/SNS/EventBridge are cheaper and simpler.

IBM MQ. Many mainframes use IBM MQ, and Amazon MQ does not run IBM MQ or its native protocol. Common patterns: keep IBM MQ as the mainframe's broker and run an IBM MQ queue manager (licensed) on EC2 or containers in AWS connected by MQ channels; or bridge IBM MQ to ActiveMQ/RabbitMQ with a JMS bridge application; or have the mainframe team expose the transaction through an API instead ⚠ VERIFY licensing and supported bridge options with your vendor.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure VPN GatewaySite-to-Site VPN (virtual private gateway or Transit Gateway)Two tunnels per connection by default; you configure both on your device
ExpressRouteDirect ConnectSimilar model (locations, partners); encryption via MACsec or VPN overlay
Virtual WAN hubTransit GatewayRoute tables per attachment; inter-Region peering
APIM with internal VNetAPI Gateway private REST API + interface VPC endpointResource policy restricts by VPC endpoint
Azure Service Bus (AMQP 1.0)Amazon MQ for ActiveMQ (AMQP 1.0, JMS) or RabbitMQ (AMQP 0-9-1)Amazon MQ runs open-source brokers; Service Bus is proprietary
Host Integration Server (HIS) for CICS/IMS and EBCDICNo direct equivalent; your .NET code (ACL) plus mainframe-side API layersYou parse copybooks yourself or with a library
System.Text.Encoding.CodePagesSame in .NET 10Register CodePagesEncodingProvider to get EBCDIC code pages such as 037

Architecture

Security-boundary view.

  • Network: no internet path to the core. Traffic flows over Direct Connect (MACsec or IPsec overlay) with VPN as encrypted backup; security groups and on-prem firewalls allow only the specific broker and API ports.
  • Identity: service-to-service credentials (broker users, mTLS client certificates) are in Secrets Manager and rotated; the mainframe side maps them to RACF/ACF2/Top Secret identities with least privilege ⚠ VERIFY your security product.
  • Data: TLS on every hop (AMQPS, HTTPS). Copybook records contain customer data; they never leave the ACL in raw form and are never logged.
  • Load: the ACL enforces concurrency limits and timeouts so cloud traffic cannot overwhelm the core.

Step-by-step implementation

Agree the network design with the network and mainframe teams

Typical choice for a bank: two Direct Connect connections at two locations (resilience) into a Direct Connect gateway and Transit Gateway, with Site-to-Site VPN as an encrypted backup path. Decide encryption (MACsec on dedicated connections, or IPsec VPN over Direct Connect), BGP routing preferences, and IP ranges (no overlaps between VPC CIDRs and data centre ranges). Use VPN alone for early phases and labs.

Pick the integration style per use case

Use caseStyleWhy
Balance enquiry for the appSynchronous API (or request/reply over MQ) with a cacheRead-only, latency-sensitive
Posting a paymentAsynchronous messaging, idempotent, with confirmation messageMust not be lost or duplicated
Customer data for analyticsCDC or nightly extract to S3Bulk; no load on online transactions
Statement generationFilesBatch by nature

Build the anti-corruption layer

One .NET component owns: copybook layouts, EBCDIC encoding, packed decimal conversion, code mappings ('O' → Open), error code translation (core return codes → domain errors), and the transport (MQ or HTTP). It exposes a clean interface such as ICoreAccounts.GetBalanceAsync(accountId). Contract tests run against recorded core responses.

Messaging: request/reply with correlation and timeouts

For each request: generate a correlation ID, publish to the request queue with reply_to set to the adapter's reply queue and a message expiration equal to your timeout (so stale requests are discarded instead of processed late), and wait on a TaskCompletionSource keyed by correlation ID. Replies with unknown correlation IDs (late arrivals) are logged and dropped.

Make writes idempotent end to end

Posting requests carry a business idempotency key (for example payment_id). The core-side program (or a table in front of it) records processed keys, and the adapter treats "timeout" as "unknown outcome": it queries the status by key before retrying. Never blindly retry a posting.

Protect the core

Limit concurrency per adapter instance (for example a SemaphoreSlim of 20), use a circuit breaker that opens when the core is slow, serve balance reads from a short-lived cache, and coordinate capacity with the mainframe team: they know the transaction limits and batch windows (when online processing is restricted).

Plan for data movement

For analytics, use change data capture or nightly extracts rather than querying the core. AWS DMS lists IBM Db2 for z/OS as a supported source with limitations ⚠ VERIFY current DMS support and CDC capabilities; specialist third-party CDC tools are also common. For modernising code later, look at AWS Transform for mainframe ⚠ VERIFY current availability, not the closed Mainframe Modernization runtime.

AWS Console instructions

  • Amazon MQ broker: Console → Amazon MQ → Brokers → Create brokers → engine RabbitMQ → deployment Single-instance broker (lab) or Cluster deployment (production) → broker instance type (for a lab, the smallest offered for RabbitMQ 3.13) → Access type: Private access, choose a VPC, private subnet(s) and a security group allowing TCP 5671 from your application security group only → set an admin user name and a strong password (store it in Secrets Manager) → Maintenance: enable automatic minor version upgrades → add tags → Create broker. Creation takes several minutes.
  • Endpoints: broker → Connections shows the AMQPS endpoint (amqps://b-....mq.us-east-1.on.aws:5671) and the web console URL.
  • Site-to-Site VPN (design only): VPC → Customer gateways (your data centre's public IP and BGP ASN) → Transit gateways or Virtual private gateways → Site-to-Site VPN connections → Create → download the configuration for your device vendor.
  • Private API: API Gateway → Create API → REST API → endpoint type Private → attach a resource policy allowing execute-api:Invoke only when aws:SourceVpce equals your interface endpoint; create the execute-api interface endpoint in VPC → Endpoints.

AWS CLI

PROFILE=learn-sandbox
REGION=us-east-1

# See which engine versions and instance types are offered right now
aws mq describe-broker-instance-options --engine-type RABBITMQ \
  --query 'BrokerInstanceOptions[].{type:HostInstanceType,versions:SupportedEngineVersions[].Name}' \
  --profile $PROFILE --region $REGION

# Read the broker password from Secrets Manager (created beforehand)
MQ_PASS=$(aws secretsmanager get-secret-value --secret-id hl-m04-mq-admin \
  --query SecretString --output text --profile $PROFILE --region $REGION)

# Create a private single-instance RabbitMQ broker (billing starts at creation)
aws mq create-broker --broker-name hl-m04-core-mq \
  --engine-type RABBITMQ --engine-version 3.13 \
  --host-instance-type mq.t3.micro \
  --deployment-mode SINGLE_INSTANCE \
  --no-publicly-accessible \
  --subnet-ids subnet-0aaa1111bbbb2222c \
  --security-groups sg-0ccc3333dddd4444e \
  --auto-minor-version-upgrade \
  --users "Username=hl-admin,Password=$MQ_PASS" \
  --tags Project=cea-lab,Lab=m04-l22,Environment=learn,Owner=you \
  --profile $PROFILE --region $REGION

# Wait until RUNNING, then get the AMQPS endpoint
aws mq describe-broker --broker-id <broker-id> \
  --query '{state:BrokerState,endpoints:BrokerInstances[0].Endpoints}' \
  --profile $PROFILE --region $REGION

# Design-only: what a VPN connection would look like (do NOT run unless you have a real customer gateway)
# aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id cgw-... --transit-gateway-id tgw-...

mq.t3.micro is offered for RabbitMQ 3.13 but not for 4.x, which runs only on mq.m7g instances; check the describe-broker-instance-options output before creating. Single-instance and micro brokers are for development only.

Infrastructure as Code

infra/cdk/CoreMessagingStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.AmazonMQ;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.SecretsManager;
using Constructs;

namespace Harbourline.CoreIntegration.Infra;

public sealed class CoreMessagingStackProps : StackProps
{
    public required IVpc Vpc { get; init; }
    public required ISecurityGroup AppSecurityGroup { get; init; }
}

public sealed class CoreMessagingStack : Stack
{
    public CoreMessagingStack(Construct scope, string id, CoreMessagingStackProps props) : base(scope, id, props)
    {
        var brokerSg = new SecurityGroup(this, "BrokerSg", new SecurityGroupProps
        {
            Vpc = props.Vpc,
            AllowAllOutbound = false,
            Description = "Amazon MQ: AMQPS from the core adapter only"
        });
        brokerSg.AddIngressRule(props.AppSecurityGroup, Port.Tcp(5671), "AMQPS from core adapter");

        var admin = new Secret(this, "MqAdmin", new SecretProps
        {
            SecretName = "hl-m04-mq-admin",
            GenerateSecretString = new SecretStringGenerator
            {
                SecretStringTemplate = "{\"username\":\"hl-admin\"}",
                GenerateStringKey = "password",
                ExcludeCharacters = ",:=", // characters RabbitMQ broker passwords cannot contain ⚠ VERIFY
                PasswordLength = 32
            }
        });

        var subnet = props.Vpc.SelectSubnets(new SubnetSelection { SubnetType = SubnetType.PRIVATE_WITH_EGRESS }).SubnetIds[0];

        new CfnBroker(this, "CoreMq", new CfnBrokerProps
        {
            BrokerName = "hl-m04-core-mq",
            EngineType = "RABBITMQ",
            EngineVersion = "3.13",
            HostInstanceType = "mq.t3.micro",
            DeploymentMode = "SINGLE_INSTANCE",       // production: CLUSTER_MULTI_AZ on mq.m7g
            PubliclyAccessible = false,
            AutoMinorVersionUpgrade = true,
            SubnetIds = [subnet],
            SecurityGroups = [brokerSg.SecurityGroupId],
            Users = new object[]
            {
                new CfnBroker.UserProperty
                {
                    Username = admin.SecretValueFromJson("username").UnsafeUnwrap(),
                    Password = admin.SecretValueFromJson("password").UnsafeUnwrap()
                }
            },
            Logs = new CfnBroker.LogListProperty { General = true }
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m04-l22");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "you");
    }
}

UnsafeUnwrap() passes a dynamic reference to the secret into the template (resolved by CloudFormation at deploy time), so the password is not written into the synthesized template; the method name is a reminder to use it only for properties that accept secrets.

Complete code

A .NET 10 console app that runs entirely against RabbitMQ in Docker. It starts a fake core banking responder (which answers balance enquiries with a 62-byte EBCDIC copybook record containing packed decimals) and a core adapter that sends request/reply messages with correlation IDs and timeouts, parses the copybook and maps it through the anti-corruption layer to Harbourline's Account.

# Free local broker: management UI on http://localhost:15672 (guest/guest works only from localhost)
docker run -d --name hl-rabbit -p 5672:5672 -p 15672:15672 rabbitmq:4-management
src/Harbourline.CoreIntegration/Harbourline.CoreIntegration.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="RabbitMQ.Client" Version="7.2.2" />
  </ItemGroup>
</Project>
src/Harbourline.CoreIntegration/Program.cs
using System.Collections.Concurrent;
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;

Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); // enables EBCDIC code page 037

var factory = new ConnectionFactory
{
    HostName = Environment.GetEnvironmentVariable("MQ_HOST") ?? "localhost",
    Port = int.Parse(Environment.GetEnvironmentVariable("MQ_PORT") ?? "5672"),
    UserName = Environment.GetEnvironmentVariable("MQ_USER") ?? "guest",
    Password = Environment.GetEnvironmentVariable("MQ_PASSWORD") ?? "guest",
    // Amazon MQ: Port = 5671 and Ssl = new SslOption { Enabled = true, ServerName = host }
    ClientProvidedName = "hl-core-adapter"
};

await using var connection = await factory.CreateConnectionAsync();

// ---- Fake core banking system (in reality: CICS behind IBM MQ, reached through a bridge) ----
await using var coreChannel = await connection.CreateChannelAsync();
await FakeCore.StartAsync(coreChannel);

// ---- Core adapter (anti-corruption layer + request/reply client) ----
await using var adapter = await CoreAdapter.CreateAsync(connection, timeout: TimeSpan.FromSeconds(2), maxConcurrency: 20);

foreach (var id in new[] { "ACC-0000123456", "ACC-0000123457", "ACC-0000999999" })
{
    var result = await adapter.GetAccountAsync(id);
    Console.WriteLine(result switch
    {
        CoreResult<Account>.Ok ok => $"{ok.Value.AccountId} {ok.Value.Status,-6} {ok.Value.Product,-8} {ok.Value.Balance,12:N2} {ok.Value.Currency} (available {ok.Value.AvailableBalance:N2})",
        CoreResult<Account>.NotFound => $"{id} not found in core",
        CoreResult<Account>.Unavailable u => $"{id} core unavailable: {u.Reason}",
        _ => "unexpected"
    });
}

// =================== Domain (Harbourline) ===================
public enum AccountStatus { Open, Frozen, Closed }
public enum Product { Current, Savings, Business, CreditCard }

public sealed record Account(string AccountId, string CustomerId, string Currency, Product Product,
    AccountStatus Status, decimal Balance, decimal AvailableBalance, DateOnly OpenedOn);

public abstract record CoreResult<T>
{
    public sealed record Ok(T Value) : CoreResult<T>;
    public sealed record NotFound : CoreResult<T>;
    public sealed record Unavailable(string Reason) : CoreResult<T>;
}

// =================== Copybook layout ===================
//   01 ACCT-BALANCE-REC.                       offset length
//      05 ACCT-NO       PIC X(14).                 0     14
//      05 CUST-NO       PIC X(12).                14     12
//      05 CCY           PIC X(3).                 26      3
//      05 PRODUCT-CD    PIC X(2).                 29      2   CA, SA, BU, CC
//      05 STATUS-CD     PIC X(1).                 31      1   O, F, C
//      05 LEDGER-BAL    PIC S9(13)V99 COMP-3.     32      8
//      05 AVAIL-BAL     PIC S9(13)V99 COMP-3.     40      8
//      05 OPEN-DATE     PIC 9(8).                 48      8   YYYYMMDD, zoned display
//      05 FILLER        PIC X(6).                 56      6
public static class AcctBalanceRec
{
    public const int Length = 62;
    public static readonly Encoding Ebcdic = Encoding.GetEncoding(37); // IBM037, US/Canada EBCDIC ⚠ check your core's CCSID

    public sealed record Raw(string AcctNo, string CustNo, string Ccy, string ProductCd, string StatusCd,
        decimal LedgerBal, decimal AvailBal, string OpenDate);

    public static Raw Parse(ReadOnlySpan<byte> rec)
    {
        if (rec.Length != Length) throw new FormatException($"Expected {Length} bytes, got {rec.Length}");
        return new Raw(
            AcctNo: Text(rec, 0, 14),
            CustNo: Text(rec, 14, 12),
            Ccy: Text(rec, 26, 3),
            ProductCd: Text(rec, 29, 2),
            StatusCd: Text(rec, 31, 1),
            LedgerBal: PackedDecimal.Decode(rec.Slice(32, 8), scale: 2),
            AvailBal: PackedDecimal.Decode(rec.Slice(40, 8), scale: 2),
            OpenDate: Text(rec, 48, 8));

        static string Text(ReadOnlySpan<byte> r, int offset, int length) => Ebcdic.GetString(r.Slice(offset, length)).TrimEnd();
    }

    public static byte[] Build(Raw r) // used by the fake core and by tests
    {
        var buf = new byte[Length];
        void Put(int offset, int length, string s) => Ebcdic.GetBytes(s.PadRight(length).AsSpan(0, length), buf.AsSpan(offset, length));
        Put(0, 14, r.AcctNo); Put(14, 12, r.CustNo); Put(26, 3, r.Ccy); Put(29, 2, r.ProductCd); Put(31, 1, r.StatusCd);
        PackedDecimal.Encode(r.LedgerBal, 2, buf.AsSpan(32, 8));
        PackedDecimal.Encode(r.AvailBal, 2, buf.AsSpan(40, 8));
        Put(48, 8, r.OpenDate); Put(56, 6, "");
        return buf;
    }
}

// COMP-3: two decimal digits per byte; the last nibble is the sign (C or F = positive, D = negative).
public static class PackedDecimal
{
    public static decimal Decode(ReadOnlySpan<byte> bytes, int scale)
    {
        decimal value = 0;
        for (var i = 0; i < bytes.Length; i++)
        {
            int hi = bytes[i] >> 4, lo = bytes[i] & 0x0F;
            value = value * 10 + Digit(hi);
            if (i < bytes.Length - 1) value = value * 10 + Digit(lo);
            else
            {
                return lo switch
                {
                    0xC or 0xF => value / Pow10(scale),
                    0xD => -value / Pow10(scale),
                    _ => throw new FormatException($"Invalid COMP-3 sign nibble 0x{lo:X}")
                };
            }
        }
        throw new FormatException("Empty COMP-3 field");

        static int Digit(int n) => n <= 9 ? n : throw new FormatException($"Invalid COMP-3 digit nibble 0x{n:X}");
    }

    public static void Encode(decimal value, int scale, Span<byte> dest)
    {
        var digits = decimal.Round(Math.Abs(value) * Pow10(scale), 0).ToString("0", System.Globalization.CultureInfo.InvariantCulture);
        var nibbles = dest.Length * 2 - 1;
        if (digits.Length > nibbles) throw new OverflowException($"{value} does not fit in {dest.Length} bytes");
        digits = digits.PadLeft(nibbles, '0');
        var sign = value < 0 ? 0xD : 0xC;
        for (var i = 0; i < dest.Length; i++)
        {
            var hi = digits[i * 2] - '0';
            var lo = i < dest.Length - 1 ? digits[i * 2 + 1] - '0' : sign;
            dest[i] = (byte)((hi << 4) | lo);
        }
    }

    private static decimal Pow10(int n) { decimal p = 1; for (var i = 0; i < n; i++) p *= 10; return p; }
}

// =================== Anti-corruption layer mapping ===================
public static class CoreAccountMapper
{
    public static Account ToDomain(AcctBalanceRec.Raw r) => new(
        AccountId: r.AcctNo,
        CustomerId: r.CustNo,
        Currency: r.Ccy is "GBP" or "EUR" ? r.Ccy : throw new FormatException($"Unsupported currency {r.Ccy}"),
        Product: r.ProductCd switch
        {
            "CA" => Product.Current, "SA" => Product.Savings, "BU" => Product.Business, "CC" => Product.CreditCard,
            _ => throw new FormatException($"Unknown product code '{r.ProductCd}'")
        },
        Status: r.StatusCd switch
        {
            "O" => AccountStatus.Open, "F" => AccountStatus.Frozen, "C" => AccountStatus.Closed,
            _ => throw new FormatException($"Unknown status code '{r.StatusCd}'")
        },
        Balance: r.LedgerBal,
        AvailableBalance: r.AvailBal,
        OpenedOn: DateOnly.ParseExact(r.OpenDate, "yyyyMMdd", System.Globalization.CultureInfo.InvariantCulture));
}

// =================== Request/reply adapter ===================
public sealed class CoreAdapter : IAsyncDisposable
{
    private const string RequestQueue = "core.balance.requests";
    private readonly IChannel _channel;
    private readonly string _replyQueue;
    private readonly TimeSpan _timeout;
    private readonly SemaphoreSlim _concurrency;
    private readonly ConcurrentDictionary<string, TaskCompletionSource<byte[]>> _pending = new();

    private CoreAdapter(IChannel channel, string replyQueue, TimeSpan timeout, int maxConcurrency)
        => (_channel, _replyQueue, _timeout, _concurrency) = (channel, replyQueue, timeout, new SemaphoreSlim(maxConcurrency));

    public static async Task<CoreAdapter> CreateAsync(IConnection connection, TimeSpan timeout, int maxConcurrency)
    {
        var channel = await connection.CreateChannelAsync();
        // Exclusive, auto-delete reply queue for this adapter instance.
        var reply = await channel.QueueDeclareAsync(queue: "", durable: false, exclusive: true, autoDelete: true);
        var adapter = new CoreAdapter(channel, reply.QueueName, timeout, maxConcurrency);

        var consumer = new AsyncEventingBasicConsumer(channel);
        consumer.ReceivedAsync += (_, ea) =>
        {
            var id = ea.BasicProperties.CorrelationId;
            if (id is not null && adapter._pending.TryRemove(id, out var tcs))
                tcs.TrySetResult(ea.Body.ToArray()); // copy: the body buffer is reused after the handler returns
            else
                Console.WriteLine($"Dropped late or unknown reply {id}");
            return Task.CompletedTask;
        };
        await channel.BasicConsumeAsync(reply.QueueName, autoAck: true, consumer);
        return adapter;
    }

    public async Task<CoreResult<Account>> GetAccountAsync(string accountId, CancellationToken ct = default)
    {
        if (!await _concurrency.WaitAsync(_timeout, ct))
            return new CoreResult<Account>.Unavailable("adapter at concurrency limit"); // back-pressure, protects the core

        var correlationId = Guid.CreateVersion7().ToString("N");
        var tcs = new TaskCompletionSource<byte[]>(TaskCreationOptions.RunContinuationsAsynchronously);
        _pending[correlationId] = tcs;
        try
        {
            var props = new BasicProperties
            {
                CorrelationId = correlationId,
                ReplyTo = _replyQueue,
                ContentType = "application/x-ebcdic-fixed",
                Expiration = ((int)_timeout.TotalMilliseconds).ToString(System.Globalization.CultureInfo.InvariantCulture) // stale requests expire
            };
            var request = AcctBalanceRec.Ebcdic.GetBytes($"BALINQ  {accountId,-14}");
            await _channel.BasicPublishAsync(exchange: "", routingKey: RequestQueue, mandatory: false, basicProperties: props, body: request, cancellationToken: ct);

            using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
            timeoutCts.CancelAfter(_timeout);
            byte[] reply;
            try { reply = await tcs.Task.WaitAsync(timeoutCts.Token); }
            catch (OperationCanceledException) when (!ct.IsCancellationRequested)
            {
                return new CoreResult<Account>.Unavailable($"no reply within {_timeout.TotalSeconds}s");
            }

            // Reply: 2-byte return code ("00" ok, "04" not found) + record
            var rc = AcctBalanceRec.Ebcdic.GetString(reply, 0, 2);
            return rc switch
            {
                "00" => new CoreResult<Account>.Ok(CoreAccountMapper.ToDomain(AcctBalanceRec.Parse(reply.AsSpan(2)))),
                "04" => new CoreResult<Account>.NotFound(),
                _ => new CoreResult<Account>.Unavailable($"core return code {rc}")
            };
        }
        finally
        {
            _pending.TryRemove(correlationId, out _);
            _concurrency.Release();
        }
    }

    public async ValueTask DisposeAsync()
    {
        await _channel.CloseAsync();
        _channel.Dispose();
        _concurrency.Dispose();
    }
}

// =================== Fake core banking responder ===================
public static class FakeCore
{
    private static readonly Dictionary<string, AcctBalanceRec.Raw> Accounts = new()
    {
        ["ACC-0000123456"] = new("ACC-0000123456", "CUS-00012345", "GBP", "CA", "O", 1520.40m, 1420.40m, "20190314"),
        ["ACC-0000123457"] = new("ACC-0000123457", "CUS-00012345", "GBP", "CC", "F", -245.99m, 0m, "20210602")
    };

    public static async Task StartAsync(IChannel channel)
    {
        await channel.QueueDeclareAsync("core.balance.requests", durable: true, exclusive: false, autoDelete: false);
        await channel.BasicQosAsync(prefetchSize: 0, prefetchCount: 10, global: false);
        var consumer = new AsyncEventingBasicConsumer(channel);
        consumer.ReceivedAsync += async (_, ea) =>
        {
            var text = AcctBalanceRec.Ebcdic.GetString(ea.Body.Span);
            var accountId = text.Length >= 22 ? text[8..22].Trim() : "";
            var reply = Accounts.TryGetValue(accountId, out var rec)
                ? [.. AcctBalanceRec.Ebcdic.GetBytes("00"), .. AcctBalanceRec.Build(rec)]
                : AcctBalanceRec.Ebcdic.GetBytes("04");

            var props = new BasicProperties { CorrelationId = ea.BasicProperties.CorrelationId };
            await channel.BasicPublishAsync(exchange: "", routingKey: ea.BasicProperties.ReplyTo!, mandatory: false, basicProperties: props, body: reply);
            await channel.BasicAckAsync(ea.DeliveryTag, multiple: false);
        };
        await channel.BasicConsumeAsync("core.balance.requests", autoAck: false, consumer);
    }
}

Run it:

cd src/Harbourline.CoreIntegration
dotnet run
# ACC-0000123456 Open   Current      1,520.40 GBP (available 1,420.40)
# ACC-0000123457 Frozen CreditCard    -245.99 GBP (available 0.00)
# ACC-0000999999 not found in core

# Simulate a core outage: stop the broker, run again, and observe "core unavailable" / connection errors
docker stop hl-rabbit

The rabbitmq:4-management image tag tracks the latest 4.x release; pin an exact tag in real projects ⚠ VERIFY the current tag on Docker Hub. To run against Amazon MQ, set MQ_HOST, MQ_PORT=5671, MQ_USER, MQ_PASSWORD and enable TLS in ConnectionFactory as noted in the code comment.

Code explained line by line

  1. Encoding.RegisterProvider(CodePagesEncodingProvider.Instance) makes legacy code pages available in .NET; without it, Encoding.GetEncoding(37) throws. Code page 037 is common for US/UK mainframes, but your core may use another CCSID (for example 285 for UK, 1140 with the euro sign); confirm with the mainframe team.
  2. ConnectionFactory reads host and credentials from environment variables; in AWS these come from Secrets Manager. Amazon MQ requires TLS (AMQPS on 5671).
  3. One connection, multiple channels: connections are expensive TCP + TLS sessions; channels are lightweight. Creating a connection per request was one of the incident's root causes.
  4. The copybook comment block documents offsets and lengths next to the code. Keep the original copybook in the repository and generate or check offsets in tests.
  5. AcctBalanceRec.Parse slices the fixed-width record. Text fields are EBCDIC and right-padded with spaces, hence TrimEnd().
  6. PackedDecimal.Decode reads two digits per byte; the last byte holds one digit and the sign nibble (C/F positive, D negative). PIC S9(13)V99 COMP-3 is 15 digits + sign = 16 nibbles = 8 bytes, with an implied decimal point two digits from the right (scale: 2). Invalid nibbles throw: never silently accept corrupted money.
  7. PackedDecimal.Encode is the inverse, used by the fake core and in round-trip tests.
  8. CoreAccountMapper is the anti-corruption layer: two-letter product codes and one-letter status codes become enums, unknown codes throw, and only GBP/EUR are accepted. The rest of the platform sees only Account.
  9. CoreAdapter.CreateAsync declares an exclusive, auto-delete reply queue (server-named) and one consumer that completes pending requests by correlation ID.
  10. ea.Body.ToArray() copies the message body; in RabbitMQ.Client 7 the body memory is only valid during the handler.
  11. _concurrency (a SemaphoreSlim) caps in-flight requests per adapter instance. When the core is slow, callers get a fast "unavailable" instead of piling up: back-pressure protects the core.
  12. Expiration on the request message makes the broker discard it if it waits in the queue longer than the timeout, so the core never processes a request whose caller has already given up.
  13. tcs.Task.WaitAsync(timeoutCts.Token) waits for the reply with a timeout. A timeout is reported as Unavailable; for reads the caller may retry or show cached data; for writes it means "outcome unknown" and must be followed by a status query, never a blind retry.
  14. Late replies (after timeout) have no pending entry and are dropped with a log line.
  15. The fake core uses BasicQosAsync(prefetchCount: 10) and manual acks, like a well-behaved consumer: it takes at most 10 messages at a time and acknowledges only after replying.

Security considerations

⚠ Learning shortcut — not for production

The local demo uses RabbitMQ's default guest account without TLS, and the fake core runs in the same process. In production: TLS everywhere (Amazon MQ requires it), one broker user per application with permissions limited to its own queues (RabbitMQ vhost/queue permissions or ActiveMQ authorization entries), credentials in Secrets Manager with rotation, and brokers in private subnets with security groups allowing only the adapter.

⚠ Learning shortcut — not for production

The lab broker is SINGLE_INSTANCE on a micro instance. Production core integrations use a cluster (RabbitMQ) or active/standby (ActiveMQ) deployment across Availability Zones, with alarms on queue depth, consumer count and broker health.

  • No internet exposure. Do not create publicly accessible brokers for core integration. Use private access and reach on-premises over VPN/Direct Connect.
  • Encrypt the network path. Direct Connect is not encrypted by default; use MACsec where supported or IPsec VPN over Direct Connect if your policy requires encryption in transit on private links.
  • Least privilege on both sides. The mainframe security team maps each cloud integration identity to a minimal set of transactions.
  • Data minimisation. Request only the fields needed; never log raw copybook records (they contain account and customer identifiers).
  • Poison messages. A malformed reply must not crash the adapter in a loop; use dead-letter exchanges/queues on the broker and alarms on their depth.
  • Change control. Copybook changes on the mainframe side are breaking changes for you. Agree versioning and run contract tests on every change.

Banking use case

Harbourline's core integration (fictional) after the incident:

  • Connectivity: two Direct Connect connections at different locations into a Direct Connect gateway and Transit Gateway; Site-to-Site VPN as encrypted backup; BGP prefers Direct Connect.
  • Reads: balance and account details via request/reply over Amazon MQ (RabbitMQ cluster) bridged to the mainframe's IBM MQ, with a 2-second timeout and a 10-second cache in the Accounts API. When the core is in its nightly batch window, the API serves the last known balance with an "as of" timestamp.
  • Writes: payment postings as persistent messages carrying payment_id as the idempotency key; the core replies with a posting reference; a reconciliation job compares postings with payments every hour.
  • Analytics: nightly extracts and CDC into the data platform, never direct queries against Db2 from the cloud.
  • Inbound from the core: batch jobs call a private REST API through an execute-api interface endpoint to publish end-of-day events into AWS.
  • Operations: dashboards for queue depth, reply latency (p99), timeouts, circuit breaker state and Direct Connect/VPN tunnel status; runbooks agreed with the mainframe operations team.

Common mistakes

  • Calling the core over the internet "temporarily".
  • New connection per request to the broker or the core API.
  • Blind retries on postings. Timeouts mean unknown outcome; query status by idempotency key.
  • No message expiration. The core processes requests long after callers gave up.
  • Leaking mainframe concepts (copybook field names, codes, EBCDIC) beyond the adapter.
  • Wrong code page. Characters like £, # or [ differ between EBCDIC code pages; test with real samples.
  • Treating COMP-3 like text. Packed bytes converted with an encoding produce garbage; slice them out first.
  • Single Direct Connect connection with no backup path.
  • Assuming Amazon MQ speaks IBM MQ. It does not; plan a bridge or a different interface.

Troubleshooting

SymptomLikely causeFix
No data is available for encoding 37Code pages provider not registeredEncoding.RegisterProvider(CodePagesEncodingProvider.Instance) at startup
Names show à or odd symbolsWrong EBCDIC code pageConfirm the CCSID with the mainframe team (037, 285, 1140…)
Invalid COMP-3 sign nibbleOffsets wrong, or field is zoned/binary (COMP) not packedRecheck the copybook; compare a hex dump with expected values
Connection refused to Amazon MQSecurity group, wrong port (5671), or broker not RUNNINGAllow 5671 from the app SG; check describe-broker
TLS handshake fails to Amazon MQTLS not enabled in ConnectionFactory or wrong ServerNameSsl = new SslOption { Enabled = true, ServerName = host }
Replies arrive after timeoutsCore slow, or queue backlogCheck queue depth; lower concurrency; agree capacity; add caching
ACCESS_REFUSED on publishBroker user lacks permission on the queue/vhostGrant configure/write/read permissions on the specific resources
VPN tunnel up but no trafficRoute tables/propagation or on-prem firewallCheck TGW/VGW route propagation and on-prem routes for the VPC CIDR

Cost and cleanup

  • Free path: RabbitMQ in Docker and the C# app cost nothing. VPN and Direct Connect are design exercises.
  • Amazon MQ (paid): billed per broker instance-hour from creation to deletion plus storage (for example about USD 0.27/hour per mq.m7g.large RabbitMQ instance and USD 0.10 per GB-month of EBS storage for RabbitMQ; a three-node cluster triples the instance cost). A micro single-instance broker for one hour costs cents ⚠ VERIFY the mq.t3.micro rate on the pricing page. New accounts may cover this from free plan credits.
  • Site-to-Site VPN: per connection-hour plus data transfer out. Direct Connect: port-hours plus data transfer. Neither is needed for this lesson.
# Delete the broker (billing stops after deletion completes)
aws mq delete-broker --broker-id <broker-id> --profile learn-sandbox --region us-east-1
aws secretsmanager delete-secret --secret-id hl-m04-mq-admin --force-delete-without-recovery \
  --profile learn-sandbox --region us-east-1
docker rm -f hl-rabbit
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandbox --region us-east-1

With CDK or Terraform, cdk destroy / terraform destroy.

Exercises

  • 🟢 Guided: Run the demo against Docker RabbitMQ. Then change the fake core's ledger balance for ACC-0000123456 to -0.01m and confirm the adapter shows -0.01. Look at the 8 packed bytes in a hex dump (Convert.ToHexString) and identify the sign nibble.
  • 🟡 Challenge: Make the fake core slow: add await Task.Delay(3000) before replying for one account. Show that the adapter returns "unavailable" after 2 seconds, that the late reply is dropped, and that the request message expires if the core is stopped. Then add a simple circuit breaker (open after 5 consecutive timeouts for 30 seconds).
  • 🔴 Live problem: The mainframe team announces: "Next month ACCT-BALANCE-REC gains a 3-byte OVERDRAFT-LIMIT field (PIC S9(5) COMP-3) after AVAIL-BAL; FILLER shrinks to 3 bytes; total length unchanged." Your adapter is deployed in 4 services. Plan the change so nothing breaks on the day the mainframe deploys, and explain how you test it in advance.

Assignment

Write an integration design for "post a payment to the core": sequence diagram, message formats (request, reply, error), idempotency key handling, timeout and "unknown outcome" procedure, reconciliation, capacity limits agreed with the core team, and the network path. Add the copybook, the C# layout class and round-trip tests to your portfolio repository.

Solutions

Knowledge check

Interview questions

  • "How would you integrate a new cloud app with a mainframe core?" — Private network path (DX + VPN backup), choose API vs messaging per use case, an anti-corruption layer for copybooks and codes, idempotent writes with status queries on timeout, back-pressure and caching to protect the core, contract tests, and joint runbooks with the mainframe team.
  • "What is COMP-3?" — Packed decimal: two digits per byte with a sign nibble at the end; implied decimal places come from the copybook's V. Decode it from bytes, not via text encoding.
  • "Does Amazon MQ support IBM MQ?" — No. It runs ActiveMQ and RabbitMQ. Bridge from IBM MQ with a JMS bridge, run IBM MQ yourself on AWS, or use an API interface.
  • "VPN or Direct Connect?" — VPN is quick, encrypted, internet-based and good for starting or as backup; Direct Connect gives consistent latency and bandwidth but takes time, is not encrypted by default, and needs redundancy.
  • "What is an anti-corruption layer?" — A translation boundary that keeps a legacy model's concepts out of your domain, so the legacy system can change or be replaced without rewriting everything.

Certification practice

Completion checklist

  • I can compare Site-to-Site VPN and Direct Connect, including encryption and resilience
  • I can design private API access in both directions
  • I ran the RabbitMQ request/reply demo with correlation IDs and timeouts
  • I can decode EBCDIC text and COMP-3 packed decimals in C#
  • I can explain the anti-corruption layer and why postings need idempotency and status queries
  • I deleted any Amazon MQ broker and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • Integrate with the core over private, redundant, encrypted paths; never over the open internet.
  • Choose APIs for quick reads, messaging for reliable writes, files/CDC for bulk; protect the core with timeouts, concurrency limits and caching.
  • Amazon MQ (ActiveMQ/RabbitMQ) exists for protocol compatibility; it is not IBM MQ.
  • Keep copybooks, EBCDIC and COMP-3 inside one anti-corruption layer with round-trip and contract tests.
  • A timed-out posting is an unknown outcome: query by idempotency key, never blindly retry.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page