Companion to 03_Topic03_Integrations_Async.md — open ONLY after you have written your own attempt.
Protocol (file 00, M1→M3): write ≥2 hypotheses + 2 solution attempts before reading. Then compare your attempt against this sheet like an investigator compares a suspect's story to the evidence. The contrast IS the learning. Then REDO from memory, then do the drill.
The trigger "worked in dev" and died in prod. Why does the platform block callouts from triggers, what exactly does the error say, and what is the correct architecture for "send to ERP whenever an Account changes"?
Why blocked: triggers run inside the database transaction; a callout is a second, external transaction that cannot be coordinated with it. Salesforce forbids it by design — the exact error: System.CalloutException: You have uncommitted work pending. Please commit or rollback before calling out.
The budget fact: even in async, callouts are capped at 100 per transaction (sync AND async), 10 s default timeout, 120 s cumulative per transaction.
The fix (async boundary):
public class AccountERPSync implements Queueable, Database.AllowsCallouts { private Set<Id> accountIds; public AccountERPSync(Set<Id> ids) { this.accountIds = ids; } public void execute(QueueableContext qc) { // read fresh, then one callout per record (or batched payload) // callout:ERP endpoint via Named Credential }}// in trigger (after insert/after update):// 1. change detection: compare old/new maps — only sync Accounts with changes// 2. System.enqueueJob(new AccountERPSync(changedIds));
Two refinements: (1) Change detection — never call out for unchanged records (compare Trigger.oldMap/Trigger.newMap on the fields the ERP cares about); (2) one job per transaction — enqueue a single Queueable with a Set<Id>, not one job per record (queueable enqueue budget: 50 sync, and the 100-callout cap applies inside).
trigger AccountTrigger on Account (after insert, after update) { Set<Id> changed = new Set<Id>(); if (Trigger.isInsert) { changed.addAll(Trigger.newMap.keySet()); } else if (Trigger.isUpdate) { for (Account a : Trigger.new) { Account o = Trigger.oldMap.get(a.Id); if (a.ERP_Status__c != o.ERP_Status__c || a.BillingCountry != o.BillingCountry) { changed.add(a.Id); } } } if (!changed.isEmpty()) System.enqueueJob(new AccountERPSync(changed));}
No callout in the trigger; one job; only changed records; callouts happen in the queueable with AllowsCallouts.
Can a trigger call out? → No. CalloutException: You have uncommitted work pending. Please commit or rollback before calling out. Callouts are blocked from inside any open transaction (triggers, also flows).
The async pattern? → Queueable with Database.AllowsCallouts (or @future(callout=true) — legacy), one job per transaction carrying the IDs; re-query inside; call out there.
Callout limits? → 100 per transaction; 10 s default per callout (120 s cumulative); 6 MB sync / 12 MB async payloads.
Why is @future the legacy choice? → Primitives-only params, no chaining, no monitoring job ID, fire-and-forget. Queueable gives JobId, chaining, and serializable params.
What does Limits.getCallouts() tell you? → Calls used vs remaining in the current transaction — budget-check loops before calling.
The credential leaked through a GitHub push. What is the correct vaulting mechanism (and its auth modes), why can't you patch this with a Constant, and what makes NCs the required answer?
The rule: secrets never live in code — not in Constants, not in Custom Labels, not in Custom Settings (visible to every admin with Metadata API access). They live in Named Credentials (Setup → Named Credentials) — the platform's credential vault: the endpoint, auth, and secrets are stored in Setup and exposed to code as callout:My_NC/path.
Auth modes (know all three): (1) OAuth 2.0 (client credentials / authorization code) — delegate username/password flow to the NC; (2) JWT bearer — certificate-based server-to-server (org-to-org, service accounts); (3) Custom — for API-key style auth: the NC stores the key; your code adds it via req.setHeader('X-Api-Key', 'callout:My_NC')? No — for custom auth the NC holds the header value and injects it per the NC config; code just references callout:My_NC/....
Why NCs win: (1) credentials are rotatable without code deploys (the leaked key is killed and replaced in Setup); (2) centralized audit (who/what uses which credential); (3) the Principal stays in Setup, not in a repo; (4) External Credential + Named Credential split allows credential reuse across endpoints.
Interview note: "How do you store API keys?" → "Named Credential with custom auth; JWT bearer for org-to-org; rotation in Setup, never in code."
HttpRequest req = new HttpRequest();req.setEndpoint('callout:Billing_ERP/invoices'); // NC does endpoint + authreq.setMethod('POST');req.setHeader('Content-Type', 'application/json');// No token in code. NC supplies client credentials / JWT / API key.
Where do integration secrets live? → Named Credentials (Setup). Code references callout:Name/path; the vault holds endpoint, auth protocol, and credentials.
Why JWT over client credentials for org-to-org? → Certificate-based, no shared secret in either system, short-lived signed assertions, standard for MuleSoft/SF-to-SF integrations.
Rotating a leaked key? → Update the NC in Setup → zero code changes, zero deploy; audit trail shows which integrations use it.
Principal identity? → The identity the external system sees (user or service account) — configured in the NC; your code doesn't handle tokens.
Limits violated: (1) 50 @future calls per transaction — a 200-record chunk trips System.LimitException: Too many future calls: 51; (2) no future-from-future (can't re-enqueue); (3) the org-global 250,000 async executions/day budget — per-record futures on volume eat the whole org's budget, starving every other scheduled job (AsyncApexExecutions exceeded).
The fix — Batch Apex:
public class SyncBatch implements Database.Batchable<SObject>, Database.AllowsCallouts { public Database.QueryLocator start(Database.BatchableContext bc) { return Database.getQueryLocator([SELECT Id, ERP_Status__c FROM Account WHERE Needs_Sync__c = true]); } public void execute(Database.BatchableContext bc, List<Account> scope) { for (Account a : scope) { /* 1 callout per record */ } // scope ≤ 100 } public void finish(Database.BatchableContext bc) { // reconciliation: count leftover Needs_Sync__c records, log, alert } }
The sizing rule: batch scope default 200, max 2,000 — but with one callout per record you must cap scope at 100 (the callout-per-transaction limit). With batched payloads (one callout for N records), you can use larger scopes.
Design refinements:Iterable + Stateful for counters/cursors; finish() runs the reconciliation ("Completed ≠ done" from Module 1 Incident 4).
@future calls per transaction? → 50; 0 from batch/future (AsyncException).
Async executions per 24 h? → 250,000 (or licenses × 200, whichever is greater) — org-global, shared by all async constructs.
Why can't you future-from-future? → Async context rule — System.AsyncException: Async callout from future/batch is not allowed; chaining belongs to Queueable/Batch.
Correct bulk tool for "update 500K records via callout"? → Batch Apex (chunked, querylocator, finish()), scope ≤ 100 for 1-callout-per-record; for pure external ingest into the org → Bulk API 2.0 (Incident 7).
Stateful batch for what? → Aggregating counters across execute() invocations (e.g., total synced/failed), cursors, or post-processing state.
The schema bug:upsert record; with no field argument upserts on Id only — Source_Id__c was never marked External ID in the schema, so retried payloads inserted instead of matching. Fix: mark the field External ID in Setup → upsert accounts Source_Id__c;.
The contract bug: the handler processed synchronously and returned 200 after processing — during timeouts the provider retried, and each retry either inserted (schema bug) or double-processed. Correct webhook contract: verify → accept → process async:
Verify the signature (HMAC with a shared secret — never trust the caller's identity on URL alone).
Accept fast: enqueue a Queueable with the payload (or persist to an object) and return 202 Accepted immediately.
Process async & idempotently: upsert on the External ID — a retry of an already-processed payload is a no-op.
The retry trap: a 200/202 with a lost response, followed by a retry, is normal — idempotency is what makes it harmless. "The other side may fail" (M0 rule #4).
Bonus: inbound auth options — signature (HMAC), IP allowlist, JWT; never process on trust.
upsert record; upserts on what? → Id only. Deduping on a business key requires a field marked External ID in the schema, passed as the second arg.
The webhook contract? → Verify signature → accept (202) → process asynchronously → idempotent upsert on the external key.
Why 200-after-processing is a trap? → Timeouts/lost responses trigger retries that re-process — duplicates or double side-effects; accept-fast + async + idempotency makes retries safe.
What is an External ID? → A field marked as unique + external in schema (type External ID), usable in upsert, Upsert.External_Id__c matching, and external lookups.
How do you verify a webhook sender? → HMAC signature with a shared secret (or IP allowlist / JWT) — never trust the URL alone.
The wall: request+response together must fit 6 MB sync (6,291,456 B) / 12 MB async (12,582,912 B) — System.CalloutException: Request size exceeded limit.
Workarounds in order of preference:
Compress — setHeader('Content-Encoding','gzip') with a compressed body (JSON compresses 5–10×).
Chunk — split the payload into batches of records, one callout per chunk (respecting the 100-callout cap and cumulative 120 s).
Stream via JSON generator/parser — JSON.createGenerator/JSON.createParser write/read incrementally instead of building one giant string in memory.
Beyond the wall (large files): binaries and huge documents go through ContentVersion (files up to 2 GB, chunked upload), never through callout payloads — attach a ContentDocumentLink + expose a REST resource/files endpoint for the external system.
Design rule: the 6 MB/12 MB limits are on the transaction payload, so batch the data, not the megabytes.
Callout payload limits? → 6,291,456 B (sync) / 12,582,912 B (async), request+response combined.
Three ways to fit under the wall? → gzip; chunk records across callouts; streaming generator/parser.
Files that genuinely exceed the wall? → ContentVersion (2 GB, chunked), not callout payloads.
Does the 120 s cumulative cap affect big payloads? → Yes — time + size both budgeted; long serialization + slow API can hit the cumulative wall before size does.
JSON.createParser benefit? → Streams tokens without materializing the whole string — memory-safe for large responses.
EventBus.publish returned true but nothing arrived. What does true actually mean, where do you inspect delivery, and how do you architect replay + the PE-vs-CDC decision?
The lie of true:EventBus.publish returning true means accepted into the bus, not delivered. Delivery is asynchronous and best-effort; failures land in PlatformEventDeliveryStatus records (the inbox of undeliverable events — monitor DeliveryStatus: Failed).
Where to inspect:
PlatformEventDeliveryStatus — per-event delivery outcome (the "did it arrive?" surface).
EventBusSubscriber — who's subscribed and is Position (replay) advancing; a stuck position = a dead subscriber.
Replay architecture: events are retained 24 h; subscribers replay from a position (on reconnect) — the consumer must be idempotent (Incident 4 discipline) because replay can redeliver.
PE vs CDC (know the one-liner): Platform Events = manually published custom payloads for your own signals (cross-org, decoupled workflows, audit); CDC = automatic change journal for standard/custom record changes with before/after deltas (replicate to data warehouse, cache invalidation). CDC for "I need to know what changed"; PE for "I decide what to announce".
Cross-org (2026 answer):EventBridge (manageable event catalog, first-class governance) or PE via EventBridge connector; the older org-to-org PE config is the legacy path.
Consumer must handle: out-of-order, duplicates (replay), and missed events (reconciliation job — Module 1 Incident 4 discipline).
// publish (fire-and-forget by design):List<Invoice_Event__e> evts = new List<Invoice_Event__e>();evts.add(new Invoice_Event__e(Invoice_Number__c = 'INV-123', Amount__c = 100));List<Database.SaveResult> results = EventBus.publish(evts);// results[i].isSuccess() = accepted; then monitor PlatformEventDeliveryStatus
Monitoring: scheduled job counts PlatformEventDeliveryStatus rows with DeliveryStatus = 'Failed' in the last hour → alert; dashboard watches EventBusSubscriber.Position advancing.
EventBus.publish returns true means? → Accepted into the bus. NOT delivered. Delivery status lives in PlatformEventDeliveryStatus.
Where do failed deliveries go? → PlatformEventDeliveryStatus (DeliveryStatus Failed) — poll/alert on it.
How do you know a subscriber is alive? → EventBusSubscriber — advancing Position = healthy; stuck = dead.
Replay? → Events retained 24 h; subscriber replays from a position on reconnect — so consumers must be idempotent and tolerate duplicates/out-of-order.
PE vs CDC? → PE: manual publish, custom payload, your signals, cross-org via EventBridge. CDC: automatic per-record change journal (before/after deltas) for replication/cache invalidation. Events "manually published by you" vs "automatically recorded for you".
The three budgets burned: (1) async executions/day — 250,000 org-global (per-record async on 5M records ≈ 20 days of the entire org's budget — every other scheduled job dies, the 8:15 PM symptom); (2) callouts — 100/transaction sync dies on the 101st; (3) time — 5M sequential round-trips ≈ weeks of callout time (10 s timeouts multiply).
The tool decision:Bulk API 2.0 — jobs + polling (create job → upload CSV/JSON → poll Open→UploadComplete→InProgress→JobComplete → retrieve successfulResults/failedResults), server-side batching, built for millions. REST = interactive/small; SOAP = legacy WSDL; BULK = volume (>2,000 records). Batch Apex only if transformation must happen in Salesforce (and then scope ≤ 100 for 1-callout-per-record).
Restartable design (3 mechanisms): (1) idempotent upsert on the External ID — re-running a completed batch is a no-op; (2) checkpoint — custom object storing the last processed external ID per batch; resume from it, never restart; (3) verify, don't trust — JobComplete is NOT proof of success: check failedResults and reconcile counts (Module 1 Incident 4's "Completed ≠ done").
Budget hygiene: the shared 250K/day budget is the constraint all async design must respect.
Timeouts: default is 10 s (and 120 s cumulative per transaction) — a per-invoice loop that hangs 10 s each dies around the 11th call even with no network fault. Set explicit setTimeout per call (5–15 s per the API's SLA) and budget-check loops (Limits.getCallouts()).
Retry-with-backoff (the pattern):
public class InvoiceSyncJob implements Queueable, Database.AllowsCallouts { private Id invoiceId; private Integer attempt; public void execute(QueueableContext qc) { try { // callout with explicit timeout; on 2xx → mark Synced } catch (CalloutException ex) { if (attempt < 3) { // backoff: schedule the next attempt at now + 2^attempt minutes // (queueables can't sleep — backoff via System.schedule) } else { markFailedAndAlert(invoiceId); // dead-letter: log + alert + dashboard } } }}
Rules: exit condition (max attempts — never infinite chains; prod chain depth 5), exponential backoff (2 s → 4 s → 8 s, or schedule-based minutes), dead-letter after max attempts (log + alert, never silent), idempotency so a retry after a lost-success can't double-post (Incident 4).
Monitoring (know at 3:15, not 7:00): (1) BatchApexErrorEvent / async-failure events → subscriber logs + alerts; (2) error-log object — every failed callout writes endpoint, record, error, attempt → the dashboard; (3) reconciliation job — count Synced__c = false vs expected zero, hourly; (4) threshold alerts — platform event/email when failures exceed N in 15 min.
The one-liner: resilience = timeouts + retry-with-backoff + idempotency + monitoring; never depend on the provider's status page — your telemetry is your status page.
Default vs cumulative timeout? → 10 s per callout; 120 s cumulative per transaction.
Why can't a queueable sleep for backoff? → Queueables execute immediately when enqueued; no delay primitive — backoff = schedule the next attempt via System.schedule (or chain with a time-based scheduler).
Exit condition for retry chains? → Max attempts (e.g., 3), then dead-letter; prod queueable chain depth is 5 — infinite chains hit it and error.
Two ways a retry double-posts, and the fix? → (a) retry after a lost success response; (b) retry of a failed attempt that actually committed. Fix: idempotent consumer (external-ID upsert / idempotency key on the destination).
A → Incident 1: callout in trigger (uncommitted work). Fix: Queueable + AllowsCallouts, change detection.
B → Incident 4: upsert on Id only — Source_Id__c not marked External ID; retries inserted. Fix: External ID in schema, upsert ... Source_Id__c, verify → 202 → async.
C → Incident 3: @future-per-record, 50-call cap trips. Fix: Batch Apex with AllowsCallouts, scope ≤ 100, finish() reconciliation.
D → Incident 8: default 10 s timeouts + provider degradation. Fix: explicit timeouts, retry-with-backoff + exit condition, dead-letter.
E → Incidents 3 + 7: per-record async burned the org-global 250K/day budget → every scheduled job dies. Fix: bulk tools for volume; budget-aware design; monitor AsyncApexJob.
Priorities: Tonight — stop the bleeding (deactivate the trigger callout and @future sync; pause/make-async the webhook handler; dedupe the 400 Accounts by Source_Id__c; hold provider retries). Monday — permanent architecture (NC + queueable, External-ID upsert + signature, batch, timeout/retry/backoff, budget monitoring, async-accept), regression tests, monitoring stack.
The shared root cause (find it):"Every integration was designed fire-and-forget synchronous: no async boundary, no idempotency, no retry, no budget awareness, no telemetry." The provider's blip only exposed it. Say this first in the interview.
The 3 regression tests:
(a) Trigger-callout guard — save an Account through the trigger chain, assert 0 sync callouts + exactly 1 enqueued job (Limits.getQueueableJobs()), HttpCalloutMock stubs the queueable.
(b) Webhook idempotency — POST the same payload twice (with signature), assert exactly 1 Account per Source_Id__c; assert 202 + async processing.
The 2-minute answer (say out loud): "The failure wasn't the provider's outage — it was design: callouts from triggers, no async boundary, no idempotency, per-record async burning a shared budget, no timeouts, no retries, no telemetry. The fix is the same discipline everywhere: async boundary at the transaction edge, external-ID idempotency, retry-with-backoff with a dead-letter, Named Credentials for secrets, and monitoring that pages a human — before the users do. And I'd prove it with three regression tests: a trigger-callout guard, a webhook-idempotency test, and a retry-exit-condition test."
Limit hunt: (a) Module 1/3 — 50 @future per transaction on a 200-record chunk; (b) Module 3 — 12 MB async payload wall (6 MB sync); (c) Module 3 — 100 callouts/transaction → scope must be ≤ 100; (d) Module 3 — upsert on Id only → External ID missing (duplicates); (e) Module 3 — 10 s default × 20 sequential = 200 s > 120 s cumulative (timeout policy + retry).
Design (2 min): Outbound (our org → billing): callout, Queueable + AllowsCallouts, Named Credential auth (JWT bearer), idempotent by invoice number on their side. Inbound (billing → our org): webhook, verify signature → 202 → async queueable → idempotent upsert on External ID. Sync only for interactive; everything else async.
Module-1 bridge (three dangers): (1) order of execution — trigger re-fire on the DML from the queueable (recursion guard needed); (2) mixed DML — if the queueable touches User/Setup objects with DML on SObjects in the same transaction → Mixed DML Operation; (3) limits — the queueable re-query + trigger re-fire both consume SOQL/DML in one transaction context; plus the 50-enqueue cap if the trigger fires in bulk. (Also: Limits.getQueueableJobs() in tests.)
Module-2 bridge: Subscribe in connectedCallback (empApi), unsubscribe in disconnectedCallback, replay on reconnect (empApi stores replayId; re-subscribe with last position) — same lifecycle discipline as LWC lifecycle (Incident 5, Module 2).
One-card answer (5 bullets + incident map): (1) Async boundary — no callouts in triggers (I1); (2) Secrets in Named Credentials (I2); (3) Idempotency — external-ID upsert, verify→202→async (I4); (4) Failure design — explicit timeouts, retry-with-backoff + exit condition, dead-letter (I8); (5) Telemetry — error logs, BatchApexErrorEvent, reconciliation, threshold alerts (I8/I6/I3). Bulk tools for volume, never per-record async (I7/I3).