Interview weight: 25–40% (integration-heavy roles; async Apex ~15–20% of the loop) · Estimated time: 6–8 sessions (~90 min each) Target: By the end, you can design an integration (auth, sync/async choice, error handling, monitoring, idempotency) in 2 minutes closed notes — and explain why a given async failure happened and which tool you'd pick. Integration + Async is the second-biggest block after core Apex, and the async decision (future vs queueable vs batch) is the classic "senior filter" at 3–4 YOE.

M0 — THE MAP (read this first, 5–10 min)

The one idea everything hangs on: THE CALL IS A PROMISE

Every concept in this module — sync vs async, Named Credentials, callout limits, Platform Events, Bulk API, retries, monitoring — is a consequence of one design decision Salesforce made:

Your code never talks to the outside world directly. Every conversation with an external system happens through a policed doorway with three rules: (1) you can't call out from inside a database transaction, (2) the call is charged to a strict budget (count, time, size), and (3) the other side may fail — so the design must assume failure, not hope.

Think of it as a physical trading post:

• The door = the callout: only one direction (outbound via HttpRequest, inbound via @RestResource), never from inside an open transaction (triggers can't call out — that's rule #1).
• The gatekeeper = limits: 100 callouts per transaction, 10 s default timeout (120 s cumulative), 6 MB sync / 12 MB async payloads. Push too much through → CalloutException.
• The pass = Named Credentials: instead of carrying a password around in your pocket (hardcoded secrets), the gatekeeper holds your credentials in a vault and issues you a pass (callout:My_NC/...) — rotatable, admin-managed, audited.
• The ferry = async Apex: when the conversation can't fit in the synchronous boat (callouts from triggers, big payloads, long processing), the platform gives you new boats with new budgets: @future (a kayak — small, one-shot), Queueable (a proper boat — takes cargo, can chain), Batch (a barge — millions of records, chunked), Scheduled (the ferry timetable).
• The post office = Platform Events / CDC: when you don't need an answer at all — you drop a letter (publish) and the other side subscribes. Decoupled, high-volume, no waiting. (Change Data Capture = the post office's change journal for record changes.)
• The harbor master = monitoring and failure design: retries with backoff, idempotency keys, delivery-status checks, alerts. The senior answer to "how do you make it resilient?" is always: timeouts + retry + idempotency + monitoring.

Why this map matters (the bridge): Every "hard" integration interview question — the trigger callout, the leaked credential, the 6 MB wall, the vanished event, the 5M-record migration, the 3 AM outage — is a specific incident where someone forgot one of these facts:

1. No callouts from inside a transaction.
2. Callouts are budgeted (count, time, size) and the budget is shared.
3. Async = new budgets, but different rules (no direct result, no future-from-future).
4. External systems fail — the design must retry, dedupe, and monitor.
5. Secrets live in Named Credentials, never in code.

By the end of this module, "knowing it" looks like this: given any one of the 9 problems below, you can (a) name the mechanism that failed, (b) explain the fix on a whiteboard, (c) write the corrected class/design from memory, and (d) say which interview question it maps to.

The incidents (choose your own adventure — recommended order)

Protocol reminder (from file 00): attempt in writing FIRST (≥2 hypotheses + 2 solution attempts), hard 45-min cap, hint ladder, then reveal, then REDO, then retrieval drill. The sealed answer sheet lives in 03b_Topic03_Answer_Sheet.md. You are expected to fail. The failure is the task.

INCIDENT 1 — THE TRIGGER THAT TRIED TO CALL THE INTERNET

STAKES

Monday, 10:00 AM. A "realtime credit check" feature ships: an Account after-update trigger calls a payment gateway's REST API to re-score the credit limit. Deploy succeeds. The first save in production throws: System.CalloutException: You have uncommitted work pending. Please commit or rollback before calling out. Then the dev "fixes" it by adding @future(callout=true) inside the trigger — and now a DIFFERENT error appears, and the payment gateway starts seeing requests with the wrong Account data.

THE INCIDENT

// V1 — the trigger:
trigger AccountAfterUpdate on Account (after update) {
    for (Account a : Trigger.new) {
        HttpResponse res = scoreCredit(a.Id);   // ← sync callout from a trigger
    }
}

// V2 — the "fix":
trigger AccountAfterUpdate on Account (after update) {
    for (Account a : Trigger.new) {
        scoreCreditAsync(a.Id);                  // ← @future from a trigger loop
    }
}

The second error: System.LimitException: Too many future calls: 51.

THE PROBLEM

Three distinct failures, three mechanisms: (1) why does the sync callout throw exactly that message, (2) what is V2's real problem beyond the 51-limit error, (3) what is the correct design for "score a credit limit when an Account changes, once, reliably, with fresh data"?

Write: (1) ≥2 hypotheses, (2) the correct architecture (trigger → what?), (3) the queueable class with the guards you'd add (change detection, one job per batch).

HINT LADDER

• Hint 1 (the avenue): (1) Triggers run inside an open transaction — the DB has uncommitted work; a callout would freeze DB locks while waiting on the internet. The error is literal. (2) What limit do 200 future calls hit, and what does "no direct result" mean for the gateway's payload? (3) What does a Queueable give you that @future can't (params, JobId, AllowsCallouts)?
• Hint 2 (the mechanism): (1) Sync callouts are blocked from triggers by design (DB lock + rollback-consistency: the gateway would see records that later get rolled back). (2) 200 records → 200 @future calls → blows the 50 future-calls-per-transaction limit. Even below 50: future methods can't be chained, can't pass sObjects, have no JobId — and they run later, in a new transaction, re-querying whatever state exists then — the wrong-data garbage. (3) Queueable with Database.AllowsCallouts gets complex params, a JobId, fresh async limits (200 SOQL, 12 MB heap, 60 s CPU) — and the correct design is ONE job holding a Set of IDs, not one job per record.
• Hint 3 (the skeleton): Trigger collects changed IDs (change detection via oldMap) → single System.enqueueJob(new CreditScoreJob(ids)) per transaction; the job implements Database.AllowsCallouts, uses a Named Credential endpoint (Incident 2), per-call timeouts, per-record try/catch, retry via queueable chaining with an exit condition (Incident 8), and failure logging.

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — StackExchange "Callout from trigger", Salesforce docs "Callout Limits", 2026 interview banks):

Failure 1 — the literal error: A trigger runs inside the same database transaction as the save. The platform forbids HTTP callouts while there is uncommitted work — a callout would hold the transaction's DB locks open for seconds while the internet round-trips, and the gateway would receive data that could still be rolled back. The message is not a bug — it's the platform refusing to let you send data that doesn't exist yet. (The same rule blocks callouts from synchronous flow invocations.)

Failure 2 — the "fix" that made it worse: @future calls are capped at 50 per transaction — 200 records = 200 calls = Too many future calls: 51. But even under 50 records the design is wrong: future methods can't pass sObjects, can't be chained, can't be monitored (no JobId), and they execute later, in a new transaction — whatever the Account's state is then (after the trigger's own DML and any other automation) is what the gateway sees. The "wrong data" is the delayed snapshot.

Failure 3 — the correct design:

// Trigger: collect + enqueue ONCE per transaction (never per record)
trigger AccountAfterUpdate on Account (after update) {
    Set<Id> changedIds = new Set<Id>();
    for (Account a : Trigger.new) {
        if (Trigger.oldMap.get(a.Id).Credit_Score_Date__c == a.Credit_Score_Date__c) continue;
        changedIds.add(a.Id);
    }
    if (!changedIds.isEmpty()) {
        System.enqueueJob(new CreditScoreJob(changedIds));   // one job, not 200
    }
}

public class CreditScoreJob implements Queueable, Database.AllowsCallouts {
    Set<Id> ids;
    public CreditScoreJob(Set<Id> ids) { this.ids = ids; }

    public void execute(QueueableContext qc) {
        for (Id id : ids) {
            try {
                HttpRequest req = new HttpRequest();
                req.setEndpoint('callout:PaymentGateway/score');  // Named Credential
                req.setTimeout(5000);                             // explicit, < 120s cumulative
                // ... send, parse response, DML the score ...
            } catch (CalloutException ex) {
                // log + retry via queueable chain with exit condition (Incident 8)
            }
        }
    }
}

Rules extracted: (1) never call out from a trigger — async only; (2) one async job per batch of records, not one per record (the 50-future lesson); (3) Database.AllowsCallouts is mandatory on Queueable/Batch/Schedulable classes that call out; (4) change detection so you don't re-score unchanged records.

KNOWLEDGE EXTRACTION (interview-ready answers you just earned)

• "Can you make HTTP callouts from a trigger?" → No. Triggers run inside an open transaction; callouts are blocked (uncommitted work + DB locks). Use @future(callout=true) (legacy) or Queueable with Database.AllowsCallouts (modern default).
• "What are the callout limits?" → 100 callouts per transaction (sync and async); default timeout 10 s, cumulative 120 s per transaction; request/response 6 MB sync / 12 MB async.
• "@future vs Queueable for callouts?" → Future: primitive params only, no chaining, no JobId, 50/transaction. Queueable: sObjects/collections, chaining (1 enqueue per execute), JobId monitoring, AllowsCallouts. Queueable is the default for new async work.
• "Can a future call a future? A batch a future?" → No to both (AsyncException). Batch → queueable in finish().
• "What is Database.AllowsCallouts?" → The interface that explicitly permits HTTP callouts in Queueable/Batch/Schedulable execute() (and batch start()/finish()). Required since API 39; without it callouts throw.
• "How do you know the async job actually ran?" → AsyncApexJob (Status, JobItemsProcessed, TotalJobItems), the JobId from System.enqueueJob, BatchApexErrorEvent, debug logs.

THE REDO (compressed, from memory — 15 min)

Write: the trigger (change detection + ONE enqueue), the queueable (AllowsCallouts + per-record try/catch + Named Credential endpoint). Then say which limit each line protects.

RETRIEVAL DRILL (closed-book, written)

1. Why are callouts blocked from triggers? (Two reasons in the error's own words.)
2. Callout limits: count, default timeout, cumulative timeout, payload sync/async.
3. Three things @future can't do that Queueable can.
4. What interface must a Queueable implement to call out, and what error appears without it?
5. One async job per record vs per batch — which limit does per-record design hit?

INTERVIEW MAPPING

"Call an external shipping API when an Opportunity closes — architecture?" (Accenture L2) and "Can you make callouts from a trigger?" are guaranteed questions at your band. The 51-future-calls detail + "one job per batch" + AllowsCallouts is the level-above answer — most candidates stop at "@future(callout=true)".

INCIDENT 2 — THE PASSWORD IN THE GIT REPO

STAKES

A security review of an integration with a logistics provider: the Apex class contains String pass = 'P@ssw0rd!2024'; String user = 'admin@shipping'; committed 14 months ago in a repo with 20 people. The review also finds: the endpoint URL is hardcoded, the provider's API key sits in a Custom Setting readable by every integration user, and when the key rotated last month the integration was down for 9 hours because "only the one dev knew where to change it."

THE INCIDENT

public class ShipmentSync {
    // TODO: move to custom setting (never did)
    private static final String ENDPOINT = 'https://shipping.example.com/v2/track';
    private static final String USER = 'admin@shipping';
    private static final String PASS = 'P@ssw0rd!2024';
    private static final String API_KEY = 'x7k2...';

    public static void sync(String trackingNumber) {
        HttpRequest req = new HttpRequest();
        req.setEndpoint(ENDPOINT);
        String auth = EncodingUtil.base64Encode(Blob.valueOf(USER + ':' + PASS));
        req.setHeader('Authorization', 'Basic ' + auth);
        req.setHeader('X-Api-Key', API_KEY);
        // ...
    }
}

THE PROBLEM

Four distinct security defects. Name all four, explain the real-world blast radius of each, and rewrite the class using Named Credentials — including the interviewer's follow-up: "the provider doesn't support OAuth, only an API key — what now?"

Write: (1) the 4 defects, (2) the Named Credential rewrite, (3) the API-key-only answer.

HINT LADDER

• Hint 1 (the avenue): (1) What's wrong with secrets in code — and in a repo with history? (2) What's wrong with Basic auth (base64 ≠ encryption)? (3) Why is an API key in a Custom Setting — readable by any user with Read on it, and shipped in change sets/backups — still a leak? (4) What does the 9-hour outage reveal about operational design?
• Hint 2 (the mechanism): Named Credentials store endpoint + auth in Setup (admin-controlled, encrypted at rest, never in source control). Apex references them via callout:'Name/path' and never touches the secret. Auth types: OAuth 2.0 (Authorization Code, Client Credentials, JWT Bearer), Username-Password (legacy), and custom auth for API-key-style providers.
• Hint 3 (the skeleton): Setup: Named Credential ShipmentNC → endpoint + auth → req.setEndpoint('callout:ShipmentNC/track'); no headers, no secrets in code. For API-key-only providers: a custom auth provider attached to the NC (the key lives in Setup, injected at request time). Rotation = a 2-minute Setup change, no release.

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — sfdcpoint.com NC guides, Salesforce docs, 2025–26 security-review red-flag lists):

The four defects:

1. Secrets in source code — visible to everyone with repo access (and clones, forks, backups, CI logs). Blast radius: full account takeover of the provider API; the password is permanently compromised regardless of rotation (git history keeps it). Reviewer's verdict: block release.
2. Basic auth — username:password base64 (reversible, not encryption) sent on every request; sniffable without forced TLS; long-lived and reusable from any stolen client. Modern answer: OAuth 2.0 with scoped, short-lived, revocable tokens (JWT Bearer or Client Credentials for server-to-server).
3. API key in a Custom Setting — Custom Settings are user-readable metadata (anyone with Read on the setting; also ships in change sets/backups). "Not in the repo" is not "in a vault." Keys belong in Named/External Credentials (encrypted at rest, referenced by name, never fetched into code).
4. The 9-hour outage — the operational defect: rotation required a code change + release. The senior fix: credentials live in Setup, so rotation = a 2-minute admin action with no release and no single point of knowledge.

Why the "obvious fixes" failed (the contrast):

• "Move it to a Custom Setting" → better than hardcode, but Custom Settings are readable config, not secrets; every sandbox refresh and backup carries the key.
• "Use a Protected Custom Setting" → protected only blocks package access, not org users.
• "Encrypt it in code and decrypt at runtime" → the decryption key would be... in the code. Security theater.
• "Just use OAuth" → the fix isn't just the protocol — it's where the secret lives (NC vault) + how it rotates (Setup, no release) + who can read it (no one — it never appears in code).

The Named Credential rewrite:

public class ShipmentSync {
    public static void sync(String trackingNumber) {
        HttpRequest req = new HttpRequest();
        req.setEndpoint('callout:ShipmentNC/track');   // endpoint + auth live in Setup
        req.setMethod('GET');
        req.setTimeout(5000);
        // NO headers with secrets — the Named Credential injects auth
        HttpResponse res = new Http().send(req);
    }
}

The API-key-only answer: Named Credentials support a custom auth type — a custom auth provider holds the key in Setup and injects it at request time; the platform adds the header/value, and the code never sees the secret. Rotation = update the credential in Setup.

KNOWLEDGE EXTRACTION (interview-ready)

• "What are Named Credentials?" → A Setup-defined bundle of endpoint URL + authentication (OAuth 2.0 flows, username-password, JWT, custom) referenced in Apex via callout:'Name/path'. Benefits: no secrets in code, admin-managed rotation, outbound-callout security, central audit.
• "Auth flows for server-to-server?" → OAuth 2.0 Client Credentials (org-level, machine-to-machine) or JWT Bearer (certificate-signed assertion — the 2026 default for provider-supported setups); Authorization Code = user-driven; Username-Password = legacy/deprecated.
• "Certificate-based auth setup?" → Generate cert in Setup (Key & Certificates) → share public key with the provider → JWT header + claims (iss = client id, sub = the provider's connected-app user, aud = token endpoint) signed with the private key → exchange for an access token.
• "What about inbound (someone calling US)?" → Connected App + OAuth scopes for APIs; verify signatures/API keys server-side for webhooks; never trust the caller's identity claim.
• "Where should an API key live?" → In the Named/External Credential vault (Setup, encrypted), injected by the platform — not in code, not in Custom Settings, not in Custom Metadata.

THE REDO

Rewrite the class from memory using callout:ShipmentNC/.... Then the rotation story: "the provider rotates the key — what happens to your integration?" (Expected: nothing — Setup change, no release.)

RETRIEVAL DRILL

1. Four reasons hardcoded credentials are fatal (name the blast radius of each).
2. What do Named Credentials actually store, and where?
3. Name the two server-to-server OAuth flows and when you'd pick each.
4. Why is a Custom Setting still a bad place for an API key?
5. What does "rotation" mean in the Named Credential world?

INTERVIEW MAPPING

Named Credentials are the expected 2026 answer for any auth question. Security-review-style questions ("found a password in code — now what?") appear in Deloitte/Accenture security blocks. The API-key-only follow-up is the level-above probe.

INCIDENT 3 — THE "QUICK" FUTURE THAT ATE THE NIGHT

STAKES

An integration exports every changed Order to the warehouse: "it's just a sync, I'll use @future — it's quick." Friday 11:00 PM, a Data Loader job pushes 12,000 Orders (60 chunks × 200). By 11:05 the error log is a wall of System.LimitException: Too many future calls: 51. The dev "fixes" it by making the future method iterate inside itself — and now the warehouse gets 60,000 duplicate sync messages and the org burns its entire 250,000 async-executions-per-day budget by 2:00 AM, starving every other scheduled job in the org.

THE INCIDENT

public class OrderSync {
    // V1: called from trigger, per record
    @future(callout=true)
    public static void syncOrder(Id orderId) { /* callout + DML */ }

    // V2: the "fix" — one future iterates internally
    @future(callout=true)
    public static void syncAll(Set<Id> orderIds) { /* loop, callout per order inside */ }
}

V2's single future loops 12,000 orders, doing a callout + a DML per order, inside one future transaction.

THE PROBLEM

Three mechanisms: (1) why does V1 blow the 51-limit exactly at 12,000 records (do the math), (2) why is V2 worse — which limits does one "mega-future" hit, and what does the warehouse see, (3) what is the correct async design for "12,000 changed Orders → warehouse, every night, monitored"?

Write: (1) the math, (2) V2's failure modes, (3) the corrected design (tool choice + chunking + monitoring + idempotency).

HINT LADDER

• Hint 1 (the avenue): (1) Future calls are capped per transaction — how many per 200-record chunk? (2) Inside ONE future: 100 callouts/transaction, 150 DML, 10,000 DML rows, 60 s CPU — 12,000 callouts is 120× the callout cap. (3) The async-executions/day budget is global per org (250,000 or licenses × 200) — what does per-record async do to it?
• Hint 2 (the mechanism): (1) 200 records per chunk → 200 future calls → the 50-future limit trips on the first chunk ("Too many future calls: 51" = the 51st call). (2) V2: one future, 12,000 iterations → dies at the 100th callout AND the 10,000 DML rows AND the 60 s CPU wallet AND the 12 MB heap; the DML that ran before the exception is committed — a partial, non-atomic sync = duplicates/reordering at the warehouse; a failed future is also retried by the platform → more duplicate messages. (3) The right tool is Batch Apex (or Bulk API 2.0 — Incident 7): chunked execute() with fresh limits per chunk, Database.AllowsCallouts, small stateful counters, finish() reconciliation.
• Hint 3 (the skeleton): Batch OrderSyncBatch implements Database.Batchable<sObject>, Database.AllowsCallouts, Database.Stateful: start() → QueryLocator of changed Orders (LIMIT-able); execute() → per-chunk: one query, callouts with per-call try/catch, DML, counters; finish() → compare counts, alert, optionally chain. Schedule via System.scheduleBatch. Idempotency at the warehouse via the Order's external ID.

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — Salesforce Dictionary async guide, mytutorialrack queueable limits, StackExchange async dumps 2025–26):

The V1 math: Future calls are capped per transaction. Each 200-record chunk is one trigger transaction → 200 future calls → the 50-call limit trips on the 51st call of the very first chunk. 60 chunks × the same death. The limit is per-transaction, so no batch-size tweak below 50 records per chunk would have "fixed" it either — the trigger would just fire more transactions, each still capped.

Why V2 is worse: One future, 12,000 iterations, one transaction:

• Dies at the 100th callout (Callout limit exceeded) — 12,000 needed.
• Before dying it committed whatever DML it had done: partial, non-atomic sync. The warehouse received 60K messages (duplicates from platform retries of the failed future + the trigger re-firing on the DML) with no ordering or dedupe.
• The 250,000 async executions/day budget is global per org — per-record async design burns it in hours, starving every other scheduled/batch job org-wide (jobs fail with AsyncApexExecutions exceeded).

The correct design:

public class OrderSyncBatch implements Database.Batchable<sObject>,
        Database.AllowsCallouts, Database.Stateful {
    private Integer processed = 0;
    private List<String> errors = new List<String>();

    public Database.QueryLocator start(Database.BatchableContext bc) {
        return Database.getQueryLocator(
            [SELECT Id, Name, External_Id__c FROM Order
             WHERE LastModifiedDate > :cutoff ORDER BY CreatedDate]);
    }

    public void execute(Database.BatchableContext bc, List<sObject> scope) {
        for (sObject o : scope) {
            try {
                // one callout per record, but chunk = 200 → 200 callouts ≤ 100 cap?
                // NO — 200 > 100! Set batch size to 100 (or 50 for safety margin).
                // req.setTimeout(5000); send; parse; DML with external-id upsert.
                processed++;
            } catch (CalloutException ex) { errors.add(String.valueOf(o.Id)); }
        }
    }

    public void finish(Database.BatchableContext bc) {
        // reconcile processed vs AsyncApexJob.TotalJobItems; alert on mismatch.
    }
}

Key insight: callouts cap at 100 per chunk → batch size = 100 ÷ callouts-per-record (with a safety margin; here batch size 100). Scheduling: System.scheduleBatch(batch, 'Nightly Order Sync', cron, 100). Idempotency: the warehouse keys on External_Id__c (upsert), so retries never duplicate.

Why the "obvious fixes" failed (the contrast):

• "Reduce the trigger batch to 50" → future calls are capped per transaction, not per record-count; the cap is the same per transaction regardless.
• "One mega-future" → the single-transaction budget (100 callouts / 10K DML rows / 60 s CPU) is the whole problem, not the solution.
• "Just retry failures" → retrying a partial, non-atomic run re-sends committed work → duplicates. Idempotency keys, not retries, fix this.

KNOWLEDGE EXTRACTION (interview-ready)

• "Future vs Queueable vs Batch — the decision?" → Future: fire-and-forget, primitives only, 50/transaction. Queueable: complex params, chaining (1 enqueue per execute, depth 5 in prod), JobId, AllowsCallouts — the default for new async work. Batch: >~50K records, chunked, fresh limits per chunk, stateful counters, finish() — the only choice for volume + monitoring.
• "Batch size vs callout cap?" → 100 callouts per chunk → scope must be ≤ 100 when each record needs a callout (use 50 for headroom).
• "Async Apex executions per day?" → 250,000 per org per 24 h (or user licenses × 200, whichever is greater). Per-record async designs burn it.
• "Stateful vs stateless batch?" → Default stateless (instance vars reset per chunk); Database.Stateful keeps counters/errors across chunks — never data (Module 1 Incident 4).
• "How do you test async?" → Test.startTest()/Test.stopTest() (forces async to run synchronously) + HttpCalloutMock.
• "@future from a batch?" → Not allowed. Use Queueable from finish().

THE REDO

From memory: the batch class (start/execute/finish), the batch-size math for one-callout-per-record, the scheduling line, and the idempotency mechanism.

RETRIEVAL DRILL

1. Do the V1 math: why "too many future calls: 51" at 12,000 records?
2. Which 4 limits does the mega-future hit?
3. What does a failed future do that creates duplicates?
4. Batch scope for 1 callout/record — and why not 200?
5. What is the global async budget, and what design burns it?

INTERVIEW MAPPING

Async tool-choice questions are the senior filter — "you need to sync 500K records nightly, walk me through it" is a top-5 scenario question at your band (golinuxcloud, salesforcedictionary 2026). The batch-size-÷-callouts math and the global async budget are the level-above details.

INCIDENT 4 — THE WEBHOOK THAT CREATED 400 DUPLICATES

STAKES

An external CRM pushes Account updates to Salesforce via webhook. The integration "worked" for months. Then the provider's delivery service had a 45-minute glitch — and retried every webhook 4–6 times. Your org now has 400 duplicate Accounts (same Source_Id__c), duplicate contacts, and a data-cleanup ticket that's been open for two weeks. The provider claims your side returned errors — but your debug logs show 200 OK.

THE INCIDENT

@RestResource(urlMapping='/accounts/webhook')
global with sharing class AccountWebhookResource {
    @HttpPost
    global static void handle() {
        RestRequest req = RestContext.request;
        Map<String, Object> body = (Map<String, Object>) JSON.deserializeUntyped(
            req.requestBody.toString());
        upsert new Account(
            Name = (String) body.get('name'),
            Source_Id__c = (String) body.get('sourceId')   // ← the dedupe key
        );
    }
}

The provider retries on ANY non-2xx OR on a timeout. Your endpoint always returns 200 because "the method is void."

THE PROBLEM

Two defects, two mechanisms: (1) why does the upsert on Source_Id__c NOT dedupe — what's missing from the design (hint: it's a field, not a query), and (2) what's the correct contract: idempotency, the response status, and the async processing pattern? Rewrite the resource.

Write: (1) the 2 mechanisms, (2) the rewritten resource (idempotent + correct status + async handoff), (3) the signature-verification answer the interviewer will probe.

HINT LADDER

• Hint 1 (the avenue): (1) upsert dedupes on the external ID field being marked as external ID — upsert without the External_Id__c argument uses Id only! (2) Returning 200 while the work happens synchronously (and can still fail silently) breaks the provider's retry contract — the retry logic needs a signal.
• Hint 2 (the mechanism): (1) upsert record (no field name) = upsert on Id — new payloads always get new Ids → duplicates. Fix: upsert account Source_Id__c (requires the field to be marked External ID in the schema), or check-then-insert inside a queueable with a unique index. (2) The webhook contract: process fast, return 2xx only when accepted; if you accept-then-process synchronously, a timeout mid-processing → provider retries → duplicates. The senior pattern: return 202 Accepted immediately, hand the payload to a Queueable (async), and make the processing idempotent. (3) Signature/secret verification server-side — never trust an unauthenticated endpoint.
• Hint 3 (the skeleton):

@HttpPost
global static void handle() {
    // 1. verify signature (HMAC over body with the shared secret from a Named Credential/Custom Metadata)
    // 2. deserialize; 3. enqueue Queueable with the payload; 4. return 202.
    // Processing queueable: upsert with External_Id__c + unique-check; log failures; retry with backoff.
}

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — inbound REST patterns, StackExchange webhook-idempotency threads, Deloitte 2026 interview scenario):

Defect 1 — the upsert that doesn't dedupe: upsert record; without a field argument upserts on Id only. New payloads (no Id) → always insert → duplicate Accounts with the same Source_Id__c. The field was intended as the key but was never honored: upsert record Source_Id__c; requires the field to be marked as an External ID in the schema (unique index enforced by the platform). That single line was the whole feature — and it was silently not doing it.

Defect 2 — the broken retry contract: The provider retries on non-2xx and on timeout. A void REST method that processes synchronously and returns 200 "success" gives the provider no signal that anything failed — and if a timeout happens during synchronous processing, the provider retries while the first processing is still running → the second insert lands as a duplicate (the 400 copies). The correct contract:

1. Verify the caller (signature over the raw body using a shared secret — HMAC — stored in a Named Credential or protected Custom Metadata; never trust an unauthenticated endpoint).
2. Accept fast: deserialize → hand to a Queueable (System.enqueueJob) → return 202 Accepted immediately. The provider sees a definitive accept; no ambiguity, no retry storm.
3. Process idempotently: the queueable upserts on Source_Id__c (external-ID-marked), so even a retry that does arrive is a no-op update, not an insert.
4. Fail loudly: log to a custom object + alert; the provider only retries what you didn't accept.

The rewritten resource:

@RestResource(urlMapping='/accounts/webhook')
global with sharing class AccountWebhookResource {
    @HttpPost
    global static String handle() {
        verifySignature();                                   // HMAC check — reject non-2xx
        Map<String, Object> body = (Map<String, Object>)
            JSON.deserializeUntyped(RestContext.request.requestBody.toString());
        System.enqueueJob(new AccountWebhookProcessor(body)); // accept fast
        RestContext.response.statusCode = 202;                // Accepted — definitive signal
        return 'accepted';
    }
}

public class AccountWebhookProcessor implements Queueable {
    // upsert account Source_Id__c;   ← external-ID-marked field
    // catch DmlException per payload; log + alert; retry with backoff via chaining (exit condition!)
}

Why the "obvious fixes" failed (the contrast):

• "Add a SELECT ... WHERE Source_Id__c = :x check before insert" → works until two concurrent webhooks pass the check simultaneously (race) — the external-ID index is the only real guard; the query is a mitigation, not a guarantee.
• "Return 500 on any failure" → the provider retries everything unambiguously — but synchronous processing means retries can double-insert before your failure is even logged. Fast-accept + async + idempotent processing is the contract.
• "It worked for months" → of course — the provider never retried before. The failure mode was latent: the design was wrong from day one, exposed by a retry.

KNOWLEDGE EXTRACTION (interview-ready)

• "Inbound REST resource?" → @RestResource(urlMapping=...) + @HttpGet/@HttpPost/..., RestContext.request/response, global class, parse via JSON.deserializeUntyped or typed DTOs.
• "How do you make a webhook safe?" → (1) authenticate the caller (signature/HMAC over the raw body, secret in Setup); (2) accept fast (202) + process async; (3) idempotency via external-ID upsert; (4) failure logging + alerts; (5) rate-limit considerations.
• "Idempotency?" → The property that retrying the same call produces the same result. Mechanisms: external-ID upsert, idempotency keys echoed in responses, or check-then-act inside a single transaction with a unique index.
• "External ID requirements?" → Marked External ID in the schema (platform-enforced unique index); upsert records Field__c; also used for Database.upsert(records, Field__c, false) partial success.
• "202 Accepted vs 200 OK?" → 202 = "accepted for processing, not necessarily done" — the correct response for accepted-but-async webhooks; 200 = "done" — implies the work completed.

THE REDO

Rewrite the resource from memory (verify → enqueue → 202) + the processing queueable (external-ID upsert + failure log). Then the probe answer: "what if the same payload arrives twice in the same second?"

RETRIEVAL DRILL

1. Why did upsert record; not dedupe on Source_Id__c?
2. What's required on the field for upsert records Source_Id__c to dedupe?
3. Why is synchronous processing + 200 a retry trap?
4. Name the 3 parts of the webhook contract (accept / process / signal).
5. What does the queueable do that the REST method must not?

INTERVIEW MAPPING

Inbound webhooks + idempotency are standard scenario questions (Deloitte "duplicate payloads — how do you dedupe?" is nearly verbatim). The "upsert needs a schema-marked external ID" detail filters most candidates — they know upsert exists but not what activates it.

INCIDENT 5 — THE 6 MB WALL

STAKES

A file-sync integration: the org must push 8 MB JSON exports to an external document store. Works in the sandbox with test payloads (2 MB). In production, real exports: System.CalloutException: Length of request body is 8388608 bytes, which is larger than the 6291456 maximum — the literal 6 MB wall. The dev's "fix": System.JSON.serialize the payload a different way. Same error. Then he moves the callout to a Queueable — and it dies at 12,590,336 instead.

THE INCIDENT

public class DocExport {
    public static void push(String jsonPayload) {        // 8 MB string
        HttpRequest req = new HttpRequest();
        req.setEndpoint('callout:DocStore/upload');
        req.setMethod('POST');
        req.setBody(jsonPayload);                        // ← 6 MB sync wall
        Http h = new Http();
        h.send(req);
    }
}

THE PROBLEM

The payload doesn't fit the door. Three questions: (1) why does "serialize differently" not help, and which two limits does the wall actually consist of (sync AND async), (2) what are the correct patterns for moving >6 MB (name at least two, with the right Salesforce objects/APIs), (3) what is the wrong pattern interviewers watch for (hint: it involves ContentVersion misuse or parsing the whole thing into heap)?

Write: (1) the limit math, (2) two correct patterns, (3) the anti-pattern.

HINT LADDER

• Hint 1 (the avenue): The wall is size: 6,291,456 bytes sync (6 MB) / 12,582,912 async (12 MB) — and it's both request and response. Re-serialization doesn't change bytes. (2) What Salesforce object stores files up to 2 GB and supports chunked uploads? (3) What does JSON.createParser stream instead of materializing?
• Hint 2 (the mechanism): Patterns: (a) files via ContentDocument/ContentVersion + REST chunked upload (2 GB limit, Content-Version with chunked transfer) — or Salesforce Files Connect/External Storage for big binary; (b) stream, don't materialize: build the payload with a streaming writer or JSON.createGenerator and parse responses with JSON.createParser — heap is 6 MB sync too, so a giant string already breaks heap before the callout; (c) chunk the data — split the export into pages of records and POST per page (idempotent upsert at the destination). The anti-pattern: JSON.serialize of a huge object graph into a single string, or base64-ing binaries into the payload (1.33× inflation).
• Hint 3 (the skeleton): Real-world: exports >6 MB → write each page to ContentVersion? No — better: callout per page with an idempotency key; binaries → ContentVersion + the standard file upload API; never one mega-payload. Heap rule: never hold the full payload + its serialized copy simultaneously.

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — Salesforce callout limits docs, SFDC dictionary, StackExchange 6-MB threads):

The limit math: The wall is request/response size: 6,291,456 bytes (6 MB) synchronous, 12,582,912 (12 MB) asynchronous. "Serialize differently" changes nothing — the byte count is the limit. And a second wall hides underneath: heap is 6 MB sync / 12 MB async — an 8 MB string plus the objects it came from can exceed heap before the callout ever runs. The Queueable version died at 12,590,336 — the async ceiling, exactly as math predicts.

The correct patterns:

1. Files → ContentDocument/ContentVersion (or Files Connect/External Storage). Files support up to 2 GB with chunked uploads through the standard APIs. For document stores: upload the file once, then send the destination a reference/URL in a small callout — the 6 MB wall never sees the bytes.
2. Chunk the data, not the payload. Split the export into pages (e.g., 2,000 records each → ~1 MB JSON), POST per page with an idempotency key; the destination assembles/upserts by external ID. Each callout is small; retries are per-page, not per-export.
3. Stream instead of materialize. JSON.createGenerator to build incrementally and JSON.createParser to consume responses — heap stays bounded; never hold the full string plus a second serialized copy.

The anti-pattern (what interviewers watch for): JSON.serialize(bigObjectGraph) into one string (heap ×2, then the 6 MB wall), or EncodingUtil.base64Encode of binaries into a JSON payload (33% inflation, double heap). Also: trying to push binary via req.setBody — binaries belong to the file API, not Apex callouts.

Why the "obvious fixes" failed (the contrast):

• "Move to Queueable" → correct instinct (12 MB ceiling) but the payload is 8 MB with a 2× serialization footprint — still over; the real fix is not shipping the bytes in Apex at all.
• "Compress it" → gzip helps only if the provider accepts compressed bodies (many don't); it's a mitigation, not a design.
• "Split the string in two and concatenate on the other side" → the wall is per-request, and the destination would need to reassemble idempotently — per-page semantic chunks are cleaner than blind string halves.

KNOWLEDGE EXTRACTION (interview-ready)

• "Callout size limits?" → Request and response: 6 MB sync / 12 MB async; callout timeout 10 s default / 120 s cumulative; 100 callouts/transaction.
• "Large file transfer in Salesforce?" → ContentVersion (2 GB, chunked upload API); Files Connect / External Storage for remote binaries; never Apex payloads for binaries.
• "JSON streaming?" → JSON.createParser / JSON.createGenerator — constant memory instead of full materialization.
• "How do you know a payload is too big before sending?" → payload.length() / Blob.valueOf(payload).size() check against the ceiling; fail fast with a clear message instead of a CalloutException.
• "Chunked exports pattern?" → Page by external ID or CreatedDate, idempotency key per page, destination upserts by key — retries are per-page and safe.

THE REDO

Design (no full code needed): "8 MB export nightly → external store." Write the architecture in 6 lines: what moves, how it's chunked, what the idempotency key is, where the heap/6-MB checks live.

RETRIEVAL DRILL

1. The two size ceilings (sync/async) — bytes.
2. Why does "serialize differently" never help?
3. Two correct patterns for >6 MB.
4. What is the base64 trap?
5. Which object + limit handles real files?

INTERVIEW MAPPING

"Large file upload / 50 MB export" is a classic scenario (real interviews: "how do you send 50 MB to an external system?"). The answer "ContentVersion for files, chunked calls for data, streaming for JSON" is the complete senior response — most candidates only know the 6 MB number exists.

INCIDENT 6 — THE EVENT THAT VANISHED AT 2:00 AM

STAKES

A realtime integration: record changes → Platform Event → an external subscriber (via Streaming API) does follow-up work. It worked in the pilot. Three months in: the ops dashboard shows event volume spiked 4× overnight and the subscriber processed nothing for 90 minutes. No errors in Apex. No failed jobs. The events were published — EventBus.publish returned true — and yet the other side saw nothing. Then they just... arrived, 90 minutes late, in a flood.

THE INCIDENT

// Publisher (trigger):
public static void publishChange(Id recordId) {
    Change_Event__e evt = new Change_Event__e(Record_Id__c = recordId);
    Database.SaveResult sr = EventBus.publish(evt);
    System.debug('published: ' + sr.isSuccess());   // true, every time
}

Subscriber: an external service on the Streaming API (CometD), which "always worked in the pilot."

THE PROBLEM

Two mechanisms: (1) what does a true from EventBus.publish actually mean, and what does it NOT mean, (2) what are the real delivery-failure modes of Platform Events at volume, and what monitoring surfaces them? Bonus: Platform Events vs Change Data Capture — when would you switch?

Write: (1) publish-return semantics, (2) the delivery failure modes + the three monitoring tools, (3) the PE-vs-CDC decision.

HINT LADDER

• Hint 1 (the avenue): (1) EventBus.publish returns success = accepted into the event bus, not delivered to a subscriber. (2) Delivery failures: subscriber offline/backpressure (Streaming API replay policy), event delivery lag under volume, EventBusSubscriber status, PlatformEventDeliveryStatus records. (3) CDC vs PE: which one is record-change journaling (automatic, DML-tied) vs custom payload (manual publish)?
• Hint 2 (the mechanism): (1) publish returns true when the event is enqueued — delivery is asynchronous and unguaranteed to any specific subscriber. (2) Volume 4× → subscriber's CometD channel backpressure → events buffered/lagged; if the subscriber's replay-policy position is wrong (or it reconnects without replay), events are skipped; PlatformEventDeliveryStatus (per subscriber, per event) shows Delivered/Failed/Unknown; EventBusSubscriber shows position + lag. (3) CDC: enabled per object, automatic change events with before/after values, same Streaming API delivery; PE: manual, custom fields, high-volume custom payloads, cross-org via EventBridge.
• Hint 3 (the skeleton): Monitor: (a) EventBusSubscriber (subscriber name, position vs latest, isActive), (b) PlatformEventDeliveryStatus for failed deliveries, (c) event-monitoring/usage dashboards for lag. Fix: subscriber must replay from its last position on reconnect (CometD replay extension), backpressure → throttle publishes server-side (batch to a lower rate), and add a dead-letter/retry queue (re-publish undelivered events, idempotent consumer). CDC when you need record changes with field deltas automatically; PE when you need custom payloads, high volume, or cross-org.

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — Platform Events dev guide, salesforceben PE-vs-CDC, event-delivery monitoring threads 2025–26):

What true really means: EventBus.publish returns success when the event is accepted into the event bus — that is all. It is NOT a delivery receipt to any subscriber. Delivery depends on: subscriber connectivity, channel capacity, replay position, and platform backpressure. The debug log's true every time was never evidence of delivery — a classic monitoring blind spot.

The failure modes at volume:

1. Backpressure/lag: 4× volume flooded the subscriber's CometD channel; the bus kept accepting (publish = true) while delivery lagged 90 minutes. Nothing errored — it was slow, not failed.
2. Replay-position loss: when the subscriber's connection dropped and reconnected, if it resumed without replay (or the replay buffer for the event type was exhausted), the lagged events were silently skipped. This is why the flood arrived late — and why some events never arrived at all.
3. Delivery-status visibility: PlatformEventDeliveryStatus records (one per subscriber-event pair) and EventBusSubscriber (per subscriber: Position, IsActive, LastError) are the two surfaces — neither was being read.

The monitoring fix (name all three in an interview):

• EventBusSubscriber — subscriber health, position vs bus head, reconnect state.
• PlatformEventDeliveryStatus — per-event delivery verdicts (Delivered / Failed / Unknown).
• Usage/event-monitoring dashboards — lag and volume trends (the 4× spike was visible for weeks).

The resilience fix: subscriber replays from its last acknowledged position on reconnect (Streaming API replay extension); publisher throttles under load (batch publishes at a sustainable rate); a dead-letter queue re-publishes undelivered events; the consumer is idempotent (external-ID upsert — Incident 4 discipline).

PE vs CDC (the decision): Change Data Capture = automatic change journaling per object (create/update/delete/undelete + field-level deltas) delivered via the same Streaming API — choose it when you replicate record changes with before/after values to an external system, zero custom publish code. Platform Events = manual publish of custom payloads (any data, not just record deltas), higher volume, and EventBridge for cross-org propagation — choose it for custom signals, decoupling, and multi-org. Both share: async delivery, subscriber replay, PlatformEventDeliveryStatus monitoring.

Why the "obvious fixes" failed (the contrast):

• "Check the Apex logs" → the publisher did its job; the failure was in the delivery plane, invisible to Apex logs.
• "Just call the subscriber's API directly instead" → reintroduces the sync-callout coupling (Incident 1) and loses the decoupling the event bus exists for; the fix is monitoring + replay, not architecture reversal.
• "It worked in the pilot" → pilots run at 1× volume with an attentive subscriber; production found the unmonitored failure mode.

KNOWLEDGE EXTRACTION (interview-ready)

• "Platform Event vs CDC?" → PE: manual publish, custom payload, high volume, decoupling, EventBridge cross-org. CDC: automatic per-object change journal with field deltas, same delivery channel. Both async, both replayable, both monitored via PlatformEventDeliveryStatus/EventBusSubscriber.
• "What does EventBus.publish returning true mean?" → Accepted into the bus — not delivered. Delivery is subscriber-dependent.
• "How do you monitor event delivery?" → EventBusSubscriber (position/lag/health), PlatformEventDeliveryStatus (per-event verdicts), event-monitoring dashboards (volume/lag trends).
• "Replay?" → Subscribers reconnect with a replay position (CometD replay extension / ReplayFrom in empApi); missed events redeliver if within the replay buffer.
• "When is the event bus the wrong tool?" → When you need a synchronous answer (request/response), or guaranteed per-subscriber delivery with transactional semantics — that's Queueable/callout territory.

THE REDO

Design "external system must know every Account change in realtime": PE or CDC? Which monitoring? What happens when the subscriber is down for 2 hours? (4-line answer.)

RETRIEVAL DRILL

1. EventBus.publish true = what exactly?
2. Two subscriber-side failure modes that need no Apex error.
3. Name the three monitoring surfaces for event delivery.
4. PE vs CDC — decision in one sentence each.
5. What does the dead-letter queue do?

INTERVIEW MAPPING

"Platform Events vs Change Data Capture — when do you use which?" is a top-5 integration question at your band (salesforceben, real dumps). The "publish=true ≠ delivered" fact plus the two monitoring tables is the level-above answer.

INCIDENT 7 — THE MIGRATION THAT DIED AT 5 MILLION RECORDS

STAKES

A data migration: 5 million Opportunity records from a legacy system into production, over a weekend. The dev "architected" it as: a scheduled Apex job that calls the REST API per record (/services/data/v62.0/sobjects/Opportunity/) in a loop. Friday 8:00 PM: job starts. Friday 8:15 PM: the org starts throwing AsyncApexExecutions exceeded — every scheduled job in the org fails. The migration is at record 1,847. The dev's defense: "It's just REST, it's the standard."

THE INCIDENT

// Scheduled job: per-record REST calls into the same org (self-integration)
public class MigrationJob implements Schedulable {
    public void execute(SchedulableContext sc) {
        for (Integer i = 0; i < 5000; i++) {
            // HttpRequest POST /services/data/v62.0/sobjects/Opportunity/
            // one request per record — 5,000,000 total
        }
    }
}

THE PROBLEM

The design is wrong at three levels. (1) Why does per-record REST kill the org — name the exact budgets it burns (async executions/day, callouts, time), (2) what is the correct tool for this exact job (REST vs Bulk API 2.0 vs Batch Apex — decide and justify), (3) if you must keep it in Salesforce, what is the migration-safe architecture (throttling, monitoring, restartability)?

Write: (1) the budget math, (2) the tool decision, (3) the restartable design.

HINT LADDER

• Hint 1 (the avenue): (1) REST-per-record from inside the org = async executions (if @future/batch) or sync callouts (if scheduled sync — but then what happens at 100 callouts?). (2) The platform's own tool for volume: Bulk API 2.0 — designed for exactly this. (3) "Restartable" = idempotent + resumable from a checkpoint.
• Hint 2 (the mechanism): (1) Inside-org per-record REST: synchronous = dies at the 100-callout wall; async = each record = one async execution → 5M records ≈ 20 days of the entire org's 250K/day async budget — and that budget is shared, so every other job in the org fails (the 8:15 PM symptom). (2) Bulk API 2.0: one job, CSV/JSON ingestion, server-side batching — 5M records in hours; the migration outbound alternative is Batch Apex + Database.getQueryLocator (50M records) — but self-ingest belongs to Bulk API. REST is for interactive/small payloads; SOAP is legacy enterprise; BULK is volume. (3) Restartability: checkpoint via a custom object (last processed external ID), idempotent upsert on the external ID, dry-run first, monitor job status + AsyncApexJob/Bulk job states.
• Hint 3 (the skeleton): Decision: Bulk API 2.0 for the bulk load (create job → upload → poll → results), with the source system writing files directly; Batch Apex only if the transformation must happen in Salesforce. Budget reality: 250K async/day shared → never per-record async for volume; sync per-record REST caps at 100 callouts/transaction → 2 hours just for 720K records of callout time alone (plus timeout risk).

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — Bulk API 2.0 docs, StackExchange migration threads, module-1 batch knowledge):

The budget math (three walls):

1. Async executions/day (250,000 per org): per-record async design = 5M executions ≈ 20 days of the org's entire daily async budget, and the budget is org-global — which is exactly why every scheduled job died at 8:15 PM (AsyncApexExecutions exceeded). A migration should use a fraction of 1% of a day's budget.
2. Callouts: 100 per transaction (sync) — a per-record loop dies on the 101st; a scheduled job that calls the API 5,000 times sync = 50 transactions' worth — plus each request pays serialization + response parsing.
3. Time: 5M sequential REST round-trips at ~300 ms each ≈ 2+ weeks of pure callout time, with 10 s timeouts multiplying on any hiccup.

The tool decision (memorize this table logic):

The migration-safe architecture (restartable):

1. Bulk API 2.0 job lifecycle: create job → upload data (CSV/JSON) → poll status (Open → UploadComplete → InProgress → JobComplete) → retrieve results (successfulResults/failedResults).
2. Checkpoint: a custom object tracking the last successfully processed external ID per batch — on failure, resume from the checkpoint instead of restarting.
3. Idempotency: upsert on the external ID field — a re-run of a completed batch is a no-op, not a duplicate (Incident 4 discipline).
4. Dry run first (small sample, validate field mapping) — then the full run in a maintenance window.
5. Monitor: job state + failed-results file; alert on partial failure; never declare success on JobComplete alone if failedResults is non-empty (Module 1 Incident 4's "Completed ≠ done" lesson, now on the Bulk API side).
6. Budget hygiene: any remaining per-record async work must respect the shared 250K/day budget — design with batch/queueable chunking, not per-record.

Why the "obvious fixes" failed (the contrast):

• "It's just REST, it's the standard" → REST is standard for interactive integration; volume is BULK's reason to exist. Confusing the two is the exact filter question.
• "Use @future for each record" → the 250K/day shared budget burns in minutes — and every other job in the org pays.
• "Batch Apex with QueryLocator" → correct for outbound processing with logic (50M records); but pure ingest into the same org = Bulk API 2.0, which is faster and doesn't consume the Apex async budget the same way.

KNOWLEDGE EXTRACTION (interview-ready)

• "REST vs SOAP vs BULK?" → REST: JSON, synchronous, interactive, modest volume. SOAP: legacy enterprise WSDL contracts. BULK 2.0: jobs + batches for >2,000 records — migrations, backfills, ETL. The one-liner: "REST for interaction, BULK for volume, SOAP for legacy."
• "Bulk API 2.0 lifecycle?" → Create job → upload CSV/JSON → poll (Open → UploadComplete → InProgress → JobComplete) → retrieve results (successful/failed files). Async by design; no per-record Apex.
• "Composite API?" → Up to 25 subrequests in one round-trip (/composite), for related-record workflows and reduced round-trips — not a volume tool.
• "How many records does the QueryLocator support?" → 50M in batch start(); getQueryLocator in sync Apex: 10,000.
• "250K async executions/day — what is it?" → The per-org daily ceiling on async Apex executions (or licenses × 200, whichever greater) — shared by futures, queueables, batches, and scheduled jobs. Budget it; never per-record.

THE REDO

From memory: the tool decision for "5M records migration", the Bulk API 2.0 lifecycle (5 steps), and the 3 restartability mechanisms.

RETRIEVAL DRILL

1. Which 3 budgets does per-record REST burn?
2. Bulk API 2.0 job lifecycle — write the 5 states.
3. REST vs SOAP vs BULK — one-line decision each.
4. What is JobComplete NOT proof of?
5. QueryLocator: sync vs batch record ceilings.

INTERVIEW MAPPING

"Migrations, backfills, >2,000 records" → Bulk API 2.0 is the expected 2026 answer; candidates who say "REST" fail the filter. The 250K/day shared-budget insight is the senior differentiator — it connects Module 1's limits to integration design.

INCIDENT 8 — THE 3:00 AM BLACKOUT

STAKES

The billing provider has a 40-minute outage at 3:00 AM. Your nightly invoice-sync job: every callout times out at the default 10 s, every job in the chain fails once and is never retried, and nobody notices until the morning report shows 12,000 invoices unsynced. The provider's status page was green the whole time ("regional routing issue — we're fine"). Postmortem asks you for the failure design that should have been there from day one.

THE INCIDENT

// The nightly sync — as shipped:
public class InvoiceSync {
    public static void run() {
        for (Invoice__c inv : [SELECT Id FROM Invoice__c WHERE Synced__c = false]) {
            HttpRequest req = new HttpRequest();
            req.setEndpoint('callout:Billing/invoices');
            req.setMethod('POST');
            // NO setTimeout — default 10 s
            Http h = new Http();
            HttpResponse res = h.send(req);       // ← throws on timeout; job dies; never retried
            if (res.getStatusCode() == 200) { /* mark synced */ }
        }
    }
}

THE PROBLEM

Design the failure handling that's missing: (1) timeouts — what's wrong with the default, and what should be set, (2) retry — the pattern with backoff, where it lives (which async construct), and the exit condition, (3) monitoring — how the org should have known at 3:15, not 7:00 AM. Bonus: the idempotency answer.

Write: (1) timeout policy, (2) the retry-with-backoff design (queueable chain, exit condition), (3) the monitoring stack (3 surfaces).

HINT LADDER

• Hint 1 (the avenue): (1) Default timeout 10 s; cumulative 120 s/transaction. Timeouts throw CalloutException — unhandled = the whole transaction dies (and in a batch, the chunk). (2) Retries need their own transaction + state + a stop condition — that's Queueable chaining with a retry counter and backoff (2 s → 4 s → 8 s...). (3) Monitoring: AsyncApexJob/Queueable failures, BatchApexErrorEvent, custom error-log object + an alerting path.
• Hint 2 (the mechanism): (1) Set req.setTimeout(15000) explicitly (or lower for fast APIs); cumulative 120 s caps the total — a per-record loop of 12 s timeouts dies at 10 calls. (2) Retry design: on CalloutException or non-2xx → enqueue a retry queueable with attempt = attempt + 1; backoff via System.enqueueJob chaining + Time.now().addSeconds(...)? No — queueables can't sleep; backoff = schedule the next attempt via System.schedule or chain with a count check; exit condition (max 3 attempts) or you get an infinite chain (queueable depth 5 in prod). (3) Monitoring: error-log records (custom object) per failed invoice, BatchApexErrorEvent subscriber, alerting via email/platform event at thresholds.
• Hint 3 (the skeleton): Per-invoice: timeout set; on failure → increment attempts on the Invoice (a field) → if < 3, enqueue a retry queueable (or re-schedule); if = 3, mark Failed + log + alert. Backoff: chain with System.schedule at now + 2^attempt minutes. Idempotency: the billing side keys on invoice number — retries can't double-post. Monitoring: BatchApexErrorEvent, error-log dashboard, nightly reconciliation job (count unsynced vs zero).

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — every production integration postmortem; StackExchange retry-with-backoff patterns; Salesforce docs):

Timeout policy: The default timeout is 10 s — and it counts against the 120 s cumulative per transaction. A loop of per-invoice calls that each hang 10 s dies at the 11th call (110 s + overhead) even with no network failure — and during the provider's 40-minute blip, every call hit the wall. Correct policy: explicit setTimeout per call (e.g., 5–15 s depending on the API's SLA), never the default assumption; and a total-budget check (Limits.getCallouts() + elapsed time) when looping.

Retry with backoff (the pattern):

public class InvoiceSyncJob implements Queueable, Database.AllowsCallouts {
    private Id invoiceId;
    private Integer attempt;

    public void execute(QueueableContext qc) {
        try {
            // callout with explicit timeout; on 2xx → mark Synced; done.
        } catch (CalloutException ex) {
            if (attempt < 3) {
                // backoff: retry in 2^attempt minutes via schedule (queueables can't sleep)
                System.schedule('InvoiceRetry-' + invoiceId,
                    '0 ' + (now.minute() + Math.pow(2, attempt)) + ' * * * ?',
                    new InvoiceRetryScheduler(invoiceId, attempt + 1));
            } else {
                markFailedAndAlert(invoiceId);   // dead-letter: log + alert + dashboard
            }
        }
    }
}

Design rules: (1) exit condition — max attempts (3) or queueable chaining becomes an infinite chain (prod depth 5); (2) backoff — exponential (2 s, 4 s, 8 s...) or time-based via scheduled retry; (3) dead-letter — after max attempts, mark Failed + log + alert instead of silent loss; (4) idempotency — the destination keys on the invoice number, so a retry after a successful-but-lost-response cannot double-post (the classic retry trap from Incident 4).

Monitoring (the 3:15 AM answer, not 7:00 AM):

1. BatchApexErrorEvent (or queueable-failure monitoring) — fires on async job errors; subscriber logs + alerts.
2. Error-log object — every failed callout writes Error_Log__c (endpoint, record, error, attempt) — the dashboard that shows 12,000 unsynced as it happens.
3. Reconciliation job — a nightly (or hourly) count of Synced__c = false vs expected zero — the "Completed ≠ done" discipline (Module 1 Incident 4) applied to integrations.
4. Threshold alerts — platform event/email when failures exceed N in 15 minutes — the human is paged at 3:15.

Why the "obvious fixes" failed (the contrast):

• "Add a try/catch and continue" → better, but a swallowed failure without retry + alert is how 12,000 invoices silently don't sync. Fail-loudly is the discipline.
• "Just raise the timeout" → raises the wall from 10 s to 30 s — the cumulative 120 s cap and the outage still break the run; the fix is retry + backoff + monitoring, not a bigger timeout.
• "The provider's status page was green" → your status page is your own telemetry; never depend on theirs.

KNOWLEDGE EXTRACTION (interview-ready)

• "Retry pattern in Apex?" → Queueable chaining with a retry counter + exponential backoff (via scheduling) + exit condition (max attempts) + dead-letter handling. Never infinite chains.
• "Timeout guidance?" → Explicit setTimeout; default 10 s; cumulative 120 s/transaction; budget-check loops.
• "Idempotency in retries?" → The destination must tolerate duplicate delivery (external-ID upsert / idempotency key) — otherwise "retry after a lost response" double-posts.
• "How do you monitor async failures?" → AsyncApexJob/Queueable status, BatchApexErrorEvent, custom error logs, reconciliation jobs, threshold alerts.
• "Queueable depth limit?" → 5 chained jobs in production; 50 enqueues per sync transaction; 1 enqueue per queueable execution.

THE REDO

From memory: the retry-with-backoff queueable skeleton (timeout, exit condition, dead-letter) and the monitoring list (3 surfaces).

RETRIEVAL DRILL

1. Default vs cumulative timeout numbers.
2. Why can't a queueable "sleep" for backoff — and what's the alternative?
3. The exit condition for retry chains (and the depth limit).
4. Two reasons a retry can double-post, and the fix.
5. Name the 4 monitoring surfaces for an integration.

INTERVIEW MAPPING

"How do you handle timeouts/retries in callouts?" is the standard follow-up after any sync/async question (agent 03: "what if the external system is down?" is the expected probe). Exponential backoff + dead-letter + monitoring is the complete senior answer.

🏆 CAPSTONE — THE INTEGRATION THAT ATE THE ORG

STAKES

Friday, 11:34 PM. You're on-call. A single external system's 40-minute blip has triggered a cascade: three different error classes are flooding the log, duplicate records are landing, and the org's async budget is bleeding. You have 45 minutes to present a triage to leadership. You have all the knowledge from Incidents 1–8. The clues are real.

THE INCIDENT (the evidence file)

• Clue A: System.CalloutException: You have uncommitted work pending... — from a trigger that "was reviewed" two weeks ago.
• Clue B: 400+ duplicate Account records with the same Source_Id__c — and the integration's own log shows the webhook "returned 200 every time."
• Clue C: System.LimitException: Too many future calls: 51 — from the nightly sync that was "fixed" last month to use @future.
• Clue D: Callout limit exceeded in a scheduled job — every record now times out at 10 s because the provider is degraded.
• Clue E: AsyncApexExecutions exceeded — every scheduled job in the org is failing; the ops dashboard shows the org's async budget burned to 0 by 1:00 AM.
• Clue F: The provider's retries during the blip are re-arriving — and the webhook handler is processing them synchronously (2 s each), queueing up 90 minutes of lag.

THE PROBLEM (the transfer test — the real interview scenario round)

Produce, in writing, a complete incident report:

1. For each clue: name the mechanism (1 line), the root cause (2–3 lines), and the fix (pointer to the pattern — no full code).
2. Prioritize: what do you fix tonight vs Monday?
3. Identify the shared root cause that connects at least 4 clues (there is one — find it).
4. Write the 3 regression tests you'd add before Monday's release.
5. Role-play the interview: leadership asks "why did this happen and how do you guarantee it won't again?" — answer in 2 minutes, closed notes.

This is deliberately hard. Produce your best report even if incomplete — the comparison with the model answer below is where the learning lives. (40–45 min cap.)

THE MODEL REPORT (reveal after your attempt)

1. Clue A → Incident 1: Sync callout inside a trigger — blocked by design (uncommitted work). Fix: Queueable with Database.AllowsCallouts, one job per batch with a Set of IDs, change detection, Named Credentials.

2. Clue B → Incident 4: upsert record; upserts on Id only — Source_Id__c was never marked External ID, so every retry inserted. The 200s were "accepted" but processing was sync (Clue F) so timeouts caused duplicate retries. Fix: mark the field External ID, upsert ... Source_Id__c, accept-fast (202) + async processing + idempotent consumer + signature verification.

3. Clue C → Incident 3: @future per record — 50-call per-transaction cap trips on the first 200-record chunk. Fix: Batch Apex (or Bulk API 2.0 for pure ingest) with AllowsCallouts, chunk size ≤ 100 for one-callout-per-record, stateful counters, finish() reconciliation.

4. Clue D → Incident 8: No explicit timeout policy — default 10 s; the provider's degradation turned every call into a 10 s hang; cumulative 120 s killed the transaction. Fix: explicit setTimeout, retry-with-backoff via queueable chaining + exit condition (max 3), dead-letter + alert.

5. Clue E → Incidents 3 + 7: Per-record async design burns the org-global 250K async executions/day budget — and the failures cascade: every other scheduled job dies (AsyncApexExecutions exceeded). Fix: bulk tools for volume (Batch/Bulk API 2.0), budget-conscious design, monitoring.

6. Clue F → Incident 4 (sync processing): Synchronous webhook processing + 200 = retry trap; during the blip, 2 s-per-payload processing lagged 90 minutes. Fix: verify → enqueue → 202; process async; replay-safe consumer.

7. Priorities: Tonight — stop the bleeding: (a) disable/deactivate the trigger callout (Clue A) and the @future sync (Clue C); (b) pause the webhook handler or make it async + 202 (Clue F); (c) dedupe the 400 Accounts (merge by Source_Id__c, archive extras); (d) raise the provider incident + hold retries at the provider side if possible. Monday — permanent fixes: NC + queueable architecture for A, external-ID upsert + signature for B, batch for C, timeout/retry/backoff for D, budget monitoring + alerts for E, async-accept for F; add the 3 regression tests; add the monitoring stack (error logs, BatchApexErrorEvent, reconciliation).

8. The shared root cause: "Every integration was designed as fire-and-forget synchronous code: no async boundary, no idempotency, no retry, no budget awareness, no telemetry." The provider's 40-minute blip was just the trigger that exposed it. That single sentence is also the interview answer to "why did this fail in production?" — and it's the one thing to say first, before the clue-by-clue details.

9. The 3 regression tests: (a) Callout-from-trigger guard test — save an Account through the trigger chain, assert zero sync callouts and exactly one enqueued job (Limits.getQueueableJobs()), with HttpCalloutMock stubbing the queueable's calls; (b) Webhook idempotency test — POST the same payload twice (signature included), assert exactly one Account per Source_Id__c; plus a test that the endpoint returns 202 and processes async; (c) Retry/budget test — simulate a failing provider (HttpCalloutMock throwing CalloutException), assert the retry chain stops at max attempts, the invoice is marked Failed + logged, and the run's async executions stay within a bounded budget.

KNOWLEDGE EXTRACTION (the meta-lesson)

The capstone is a model of the integration scenario round — interviewers give you a multi-symptom integration story and watch how you prioritize, connect, and communicate. The 3-sentence summary you should now be able to produce:

"Every integration failure I've seen — the trigger callout, the duplicate webhooks, the burned async budget, the silent timeout — is the same disease: synchronous fire-and-forget thinking, designed without an async boundary, without idempotency, without retries, and without telemetry. The fix is always the same discipline: async boundary at the transaction edge, external-ID idempotency, retry-with-backoff and a dead-letter, Named Credentials for secrets, and monitoring surfaces that page a human before the users do."

THE KNOWLEDGE SPINE (the allowed 5% — memorize after the incidents)

The async decision matrix (reproduce cold)

Limits to know cold (integration-relevant)

The 5 integration patterns that fix 90% of incidents

1. Async boundary: no callouts from triggers/flows — Queueable with Database.AllowsCallouts, one job per batch of IDs.
2. Secrets in Setup: Named Credentials (callout:NC/path) — OAuth JWT/Client Credentials; never code, Custom Settings, or metadata.
3. Idempotency everywhere: external-ID upsert (field marked External ID), webhook = verify → 202 → async, retries can't double-post.
4. Failure design: explicit timeouts, retry-with-backoff + exit condition, dead-letter + alert, reconciliation jobs.
5. Telemetry: AsyncApexJob/EventBusSubscriber/PlatformEventDeliveryStatus, error-log objects, BatchApexErrorEvent, threshold paging.

Quick API/tool one-liners

• REST = JSON + Bearer token; BULK 2.0 = jobs + polling for >2K records; SOAP = legacy WSDL enterprise.
• Composite API = ≤25 subrequests per round-trip.
• Platform Events = manual publish, custom payload, cross-org via EventBridge. CDC = automatic per-object change journal (before/after field deltas).
• EventBus.publish true = accepted into the bus, NOT delivered.
• Streaming API = push (CometD) — subscribe in connectedCallback, unsubscribe in disconnectedCallback (LWC) / replay on reconnect.
• ContentVersion = files up to 2 GB (chunked upload); never binaries in Apex payloads.
• Bulk API 2.0 lifecycle: create job → upload → poll (Open→UploadComplete→InProgress→JobComplete) → retrieve results.
• Bulk 2.0 vs REST decision: interactive → REST; volume → BULK; legacy → SOAP.

Rapid-fire trick questions (module 3 scope)

INTERLEAVED PRACTICE SET (do 1–2 per session, closed-book)

Pick the technique before solving — the choice is the training. Mixes Modules 1, 2, and 3.

1. Limit hunt (which wallet, which module): (a) a trigger that calls @future per record on a 200-record chunk; (b) a queueable that builds a 13 MB JSON string; (c) a batch with scope 200 doing one callout per record; (d) a webhook handler that upserts without an external ID on a retry storm; (e) a nightly sync whose provider times out at default 10 s × 20 invoices.
2. Design (2 minutes, closed notes): "external billing system calls Salesforce to create invoices; our org must also notify them when invoices change." Which direction is webhook (inbound) and which is callout (outbound)? Auth for each? Async or sync for each? Idempotency for each?
3. Module-1 bridge: an Account trigger updates a field, and a Queueable re-queries the Account 5 minutes later for a callout. What Module-1 facts make this dangerous (order of execution? triggers re-firing? mixed DML?) — name three.
4. Module-2 bridge: an LWC subscribes to Platform Events for realtime updates. Where does it subscribe, where does it unsubscribe, and what does the subscription need on reconnect (empApi replay)?
5. The one-card answer: write the complete "how do you make an integration resilient?" answer in 5 bullet lines — then say which Incident each bullet maps to.

SPACED REPETITION SCHEDULE (log it in the canvas)

• Today: after each incident — retrieval drill + redo.
• Tomorrow: re-answer the 5-question drills from Incidents 1–4 (closed-book).
• +1 week: the Interleaved Practice Set + rapid-fire bank (all three modules).
• +1 month: the Capstone (re-do from memory) + Modules 1–2 capstones back to back.

Incident sources (real, for your curiosity): Salesforce docs — Callout Limits, Asynchronous Apex, Platform Events Developer Guide, Change Data Capture Developer Guide, Bulk API 2.0 Developer Guide, Named Credentials, REST/SOAP API Guides; sfdcpoint.com (Named Credentials, callout limits); salesforcedictionary.com (async Apex, callouts); salesforceben.com (Platform Events vs CDC, org-to-org); mytutorialrack.com (queueable chaining limits); technoeric.in (async Apex Q&A); flutterant.com (real-time async scenarios); golinuxcloud.com (45+ SFDC interview questions 2026); MuleSoft blog (API-led connectivity); LinkedIn real dumps (Deloitte/Capgemini/Accenture integration-heavy roles); StackExchange (callout-from-trigger, retry-with-backoff, webhook idempotency threads). Full URL list in _research/round1_master_report/agent_03_integrations_async/sources.md + links_master.md.