Interview weight: 8–12% (a fixed scripted block in most loops — "walk me through your deployment process" is asked even of pure developer candidates) · Estimated time: 4–5 sessions (~90 min each) Target: By the end, you can explain source-driven development end to end (Git → SFDX → CI pipeline → org), diagnose real Change-Set/scratch-org/sandbox/packaging failures, name the exact numeric limits, and — critically — never answer "how do you deploy?" with Change Sets alone. Interviewer consensus: naming only Change Sets in 2026 dates the candidate badly.

M0 — THE MAP (read this first, 5–10 min)

The one idea everything hangs on: METADATA WITHOUT VERSION CONTROL IS A RUMOR, NOT A RECORD

Every concept in this module — Change Sets, SFDX, scratch orgs, packaging, CI/CD, sandbox refresh, DevOps tooling — is a consequence of one realization:

An org's metadata is a live, mutable, multi-editor document. Without Git as the single source of truth, "what's actually in prod" is whatever the last person clicked — unrecorded, unreviewable, un-rollback-able. Every tool in this module exists to answer one question with evidence instead of memory: "what changed, who approved it, and can we undo it?"

Think of it as a supply chain for metadata:

• The warehouse ledger = Git — the one place "current state" is provably true. Everything else (Change Sets, sandboxes, scratch orgs) is a shipment, not the ledger.
• The packing list format = SFDX / sf CLI — sfdx-project.json, force-app/main/default, .forceignore — metadata as text files a diff tool can read, not a black-box XML export.
• The disposable test kitchen = scratch orgs — spun up from a definition file, thrown away in 30 days, never a place to store irreplaceable work.
• The staging warehouses = sandboxes (Developer, Developer Pro, Partial Copy, Full) — each with a different capacity and refresh cadence, and every refresh is a factory reset of anything not in source control.
• The shipping manifest = packages (Unmanaged / Unlocked / Managed) — how you bundle metadata for repeatable, versioned delivery instead of ad hoc copying.
• The loading dock = CI/CD pipeline — branch strategy, validation-only deploys, test levels — the automated, reviewable gate between a laptop and production.
• The forklift you outgrow = Change Sets — fine for a single admin move, structurally incapable of being the answer to "how does your team ship."

Why this map matters (the bridge): The research is blunt: "naming only Change Sets in 2026 dates the candidate badly." Every incident below is a real team that trusted a shipment (a Change Set, a scratch org, a sandbox) as if it were the ledger (Git) — and paid for it. The fix is always the same five disciplines:

1. Git is the single source of truth — every environment is disposable except the repo.
2. Know the exact capacity/expiry numbers for every environment type (Change Set 10,000 files/400MB/30 days; scratch org 30 days/200MB; sandbox refresh cadences) — and design around them, not around hope.
3. Validate before you deploy — test levels, validation-only runs, and Quick Deploy's 10-day cache window are not optional ceremony.
4. Deletion is a distinct, deliberate act (destructiveChanges.xml) — never assumed, never silent.
5. Pick the packaging/CI tool that matches team maturity (Unlocked packages + native CI vs Copado vs Gearset) — and be able to say why.

By the end of this module, "knowing it" looks like this: given any one of the 9 problems below, you can (a) name the mechanism that failed, (b) explain the fix on a whiteboard, (c) describe the corrected pipeline/config from memory, and (d) say which interview question it maps to.

The incidents (choose your own adventure — recommended order)

Protocol reminder (from file 00): attempt in writing FIRST (≥2 hypotheses + 2 solution attempts), hard 45-min cap, hint ladder, then reveal, then REDO, then retrieval drill. The sealed answer sheet lives in 07b_Topic07_DevOps_SFDX_Answer_Sheet.md. You are expected to fail. The failure is the task.

INCIDENT 1 — THE FIELD THAT WOULDN'T DIE

STAKES

A security review flags a deprecated field, Legacy_SSN__c, still exposed on a page layout in production — six months after the admin "removed" it via Change Set from sandbox to prod. The admin insists: "I deleted it and moved the Change Set." Compliance wants an explanation by end of day for the auditor.

THE INCIDENT

Sandbox: Legacy_SSN__c field deleted from the Account object, page layout updated.
Outbound Change Set "Q2 Cleanup" created → components added: Account page layout (updated),
  a validation rule (updated) → uploaded → deployed to Production.
Result: page layout in prod STILL shows Legacy_SSN__c. The field still exists in prod.
Change Set "Deploy" showed 100% success.

THE PROBLEM

The Change Set reported 100% success. Why is the field still in production? Name the exact capability Change Sets lack, the one artifact that can fix it, and the corrected release process for this cleanup.

Write: (1) what Change Sets structurally cannot do, (2) the mechanism that actually deletes metadata, (3) the corrected process.

HINT LADDER

• Hint 1 (the avenue): (1) Change Sets are additive/metadata-only — they can add or update components, but they cannot delete a component in the target org, ever. (2) The only programmatic deletion path is a destructive changes manifest. (3) The fix isn't "try harder with Change Sets" — it's a different tool entirely for the deletion half of the change.
• Hint 2 (the mechanism): Change Sets are a whitelist of components to add/update; there is no "delete" checkbox because Salesforce deliberately keeps Change Sets non-destructive (a safety design, not a bug). The field deletion happened only in sandbox's local state — nothing in the Change Set told production to delete anything, so production kept the field (and the layout reference the admin thought was gone came along as an "updated" layout that still, in the org's actual state, has the field available to reference). The only way to delete metadata via any repeatable/programmatic mechanism is destructiveChangesPre.xml (pre-deploy deletions) or destructiveChanges.xml (post-deploy deletions), packaged with a package.xml, deployed via Metadata API/SFDX (sf project deploy start --pre-destructive-changes / --post-destructive-changes).
• Hint 3 (the skeleton): Corrected process: move off Change Sets for this class of change entirely → SFDX project → destructiveChanges.xml listing CustomField: Account.Legacy_SSN__c → deploy with sf project deploy start --manifest package.xml --post-destructive-changes destructiveChanges.xml → verify via sf project retrieve start diff that the field is gone from the retrieved metadata. Use destructiveChangesPre.xml when the deletion must happen BEFORE the rest of the deploy (e.g., deleting a field before deploying a picklist that reuses the API name).

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — the classic "Change Set says success but nothing was deleted" story; every org with an admin who assumes Change Sets are symmetric):

Change Sets are fundamentally a whitelist of metadata to add or update in the target org — Salesforce designed them to be non-destructive by omission: there is no "delete this component" action available in the Change Set UI at all. The sandbox correctly deleted the field locally; the Change Set simply never carried a delete instruction, because it structurally cannot. The "100% success" was true and irrelevant — every component that WAS in the Change Set deployed fine; the deletion was never in it.

The fix: the only programmatic path to delete metadata is a destructive-changes manifest:

• destructiveChangesPre.xml — deletions applied BEFORE the rest of the package deploys (use when a rename/replace needs the old component gone first).
• destructiveChanges.xml — deletions applied AFTER the rest of the package deploys (the common case: pure cleanup).
• Both are paired with a package.xml and deployed via Metadata API, Ant migration tool, or (2026 standard) sf project deploy start with --pre-destructive-changes / --post-destructive-changes flags, or committed into a source-driven CI pipeline.

Corrected process for this exact case: author destructiveChanges.xml with <types><members>Legacy_SSN__c</members><name>CustomField</name></types>, deploy it explicitly, then verify by retrieving metadata from prod and confirming the field is absent — never trust the deploy log alone; verify the retrieved state.

Why the "obvious fixes" failed (the contrast):

• "Re-upload the Change Set" → re-uploads the same non-destructive whitelist; nothing changes.
• "Manually delete the field in prod via Setup" → works once, but isn't repeatable/auditable and doesn't fix the process — the next cleanup hits the same wall.
• "Change Sets have a 'destructive changes' related list — use it" → that UI feature (destructive change Change Sets) exists in some orgs but is limited, easy to miss, and still not how a source-driven team should be deleting metadata — the durable fix is the manifest-based approach in a CI-managed pipeline.

KNOWLEDGE EXTRACTION (interview-ready)

• "Can a Change Set delete a component?" → No. Change Sets are additive/metadata-only; they can add or update, never delete. This is the Change Set limitation interviewers probe for.
• "How do you delete metadata programmatically?" → destructiveChangesPre.xml (pre-deploy) or destructiveChanges.xml (post-deploy) with a package.xml, deployed via Metadata API / SFDX / CI pipeline.
• "Other Change Set limits?" → 10,000 files / 400MB per Change Set; one-directional (needs a deployment connection between the two specific orgs); 30-day expiry after upload; no rollback of a successful deploy; no version control, no code review, no CI.
• "Validate vs Deploy vs Quick Deploy?" → Validate = test-compile without saving to org (used for RunAllTestsInOrg / release-gate checks); Deploy = validate + save; Quick Deploy = reuses a successful validation's test results within a 10-day cache window, skipping re-running tests — huge time saver for large orgs with long test suites.

THE REDO

From memory: what Change Sets cannot do, the two destructive-changes manifests and when each applies, and the verification step that would have caught this in month one.

RETRIEVAL DRILL

1. Can a Change Set delete a component in the target org?
2. Name the two destructive-changes manifests and the difference between them.
3. Change Set size/file limits?
4. Change Set expiry after upload?
5. What does Quick Deploy reuse, and what's its cache window?

INTERVIEW MAPPING

"Walk me through how you'd remove a deprecated field from production" is a scripted follow-up after any deployment question — the answer that stops at "Change Set" fails; naming destructiveChanges.xml is the senior signal.

INCIDENT 2 — THE SCRATCH ORG THAT VANISHED AT MIDNIGHT

STAKES

A developer has been building a complex LWC + Apex feature in a scratch org for three weeks — "it's basically done, I'll push to Git this weekend." Friday morning: sf org list shows the scratch org expired and deleted. No commits since day 4. The sprint demo is Monday. The developer's Slack message to the team: "is there any way to recover it?"

THE INCIDENT

$ sf org list
ALIAS       USERNAME                    STATUS
my-scratch  test-abc123@example.com     Expired

$ sf org open --target-org my-scratch
ERROR: This scratch org has expired and is no longer accessible.

Last commit in the feature branch: 17 days ago.
Scratch org created: 30 days ago (definition file default: durationDays: 30).

THE PROBLEM

Is the work recoverable — and regardless of the answer, name the exact scratch-org lifecycle rule that made this inevitable, the Dev Hub limits that shape scratch-org strategy, and the workflow discipline that prevents this from ever happening again.

Write: (1) the recovery verdict + reasoning, (2) the lifecycle numbers, (3) the corrected workflow.

HINT LADDER

• Hint 1 (the avenue): (1) Scratch orgs have a hard maximum life of 30 days; on expiry they are permanently deleted — there is no recycle bin, no export, no support ticket that brings it back. (2) Dev Hub itself has active/daily creation caps that shape how teams use scratch orgs. (3) The discipline: scratch orgs are disposable compute, never storage — the only durable copy of work is Git, committed continuously.
• Hint 2 (the mechanism): durationDays in the scratch org definition file caps at 30 days maximum (default is often 7; teams raise it up to 30). At expiry the org and everything in it — unpushed metadata, test data, debug logs — is gone permanently; Salesforce does not offer recovery. A Dev Hub has a limit of roughly 40 active scratch orgs at once and 80 created per day (edition-dependent) and each scratch org is capped around 200MB of data storage — small by design, because it's meant to be rebuilt from source, not preserved. The core discipline failure here is identical to "my laptop died and I lost my code" — except the platform itself enforces the deletion on a clock.
• Hint 3 (the skeleton): Verdict: not recoverable — the last 13+ days of uncommitted work are gone. Numbers to state cold: 30-day max scratch-org life, permanent deletion on expiry, ~40 active / 80 daily per Dev Hub, ~200MB storage, scratch ≠ sandbox (sandboxes don't expire this way). Workflow fix: commit to a feature branch daily (small, incremental commits — not "done" milestones), treat the scratch org as recreate-able at any time (sf org create scratch -f config/project-scratch-def.json -a my-scratch --duration-days 7, then sf project deploy start from source to rebuild state in minutes), and add a team norm/CI check that flags scratch orgs nearing expiry with no recent commits.

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — "I'll push it later" meets a 30-day platform-enforced deletion; the scratch-org equivalent of an unsaved document):

The verdict: the work is gone. Scratch orgs have a hard-coded maximum lifespan of 30 days; when the clock runs out, the org and every unpushed metadata change, custom object, test record, and debug artifact inside it is permanently deleted — not archived, not recoverable via support. There is no override.

The Dev Hub economics that shape scratch-org strategy: a Dev Hub allows roughly 40 active scratch orgs concurrently and up to 80 created per day (varies by edition/allotment), each capped around 200MB. These numbers exist because the platform's mental model of a scratch org is "a disposable, reproducible-from-source sandbox for a feature branch" — not a place anyone should treat as durable storage. Scratch org ≠ sandbox: a Developer sandbox degrades gracefully and can be refreshed without total data loss of its metadata baseline; a scratch org simply ceases to exist.

The corrected workflow (the discipline, not the tool):

1. Commit daily, in small increments — the feature branch is the source of truth from hour one, not "when it's done."
2. Treat the scratch org as ephemeral compute: sf org create scratch from a definition file, then sf project deploy start to push source-controlled metadata into it — rebuilding a scratch org from Git should take minutes, so losing one is an inconvenience, not a catastrophe.
3. Track expiry proactively: sf org list shows days remaining; teams script a Slack reminder or CI check when a scratch org is within 3-5 days of expiry with uncommitted local changes (sf project retrieve start --dry-run style diff against the branch).
4. Never store test data uniquely in a scratch org — data setup should be scripted (sf data import tree or Apex @TestSetup-style seed scripts checked into the repo) so it's reproducible too.

Why the "obvious fixes" failed (the contrast):

• "Ask Salesforce support to restore it" → scratch orgs are explicitly excluded from any restore/backup process; this is by design, not a support gap.
• "Extend the scratch org's duration next time" → 30 days is the platform maximum, not a configurable ceiling you can raise further; it delays the same mistake, doesn't fix it.
• "Use a sandbox instead so it doesn't expire" → sandboxes solve the expiry problem but reintroduce the "which sandbox has the real state" problem (Incident 5) — the actual fix is Git discipline, independent of environment type.

KNOWLEDGE EXTRACTION (interview-ready)

• "Scratch org max life?" → 30 days maximum, permanently deleted on expiry, no recovery.
• "Scratch org vs sandbox?" → Scratch org: ephemeral, source-driven, created from a definition file against a Dev Hub, ~200MB, for feature-branch development. Sandbox: a copy of production's org shell (metadata and optionally data), refreshed on a schedule, used for QA/UAT/staging — not disposable in the same sense.
• "Dev Hub limits?" → Roughly 40 active scratch orgs / 80 created per day per Dev Hub (edition-dependent) — a reason CI pipelines clean up scratch orgs after each PR (sf org delete scratch).
• "What's the actual source of truth in an SFDX workflow?" → Git, always — every org (scratch, sandbox, or prod) is a deployable/disposable target, never the record of truth.

THE REDO

From memory: the 30-day hard expiry + permanent deletion, the Dev Hub capacity numbers, scratch org vs sandbox distinction, and the four-point commit-discipline workflow.

RETRIEVAL DRILL

1. Maximum scratch org lifespan?
2. What happens to data/metadata in a scratch org on expiry?
3. Approximate Dev Hub active/daily scratch-org limits?
4. Scratch org storage cap?
5. What is the actual single source of truth in an SFDX-based team?

INTERVIEW MAPPING

"What's your Git branching / scratch org workflow?" is a standard scripted DevOps question; this incident is its failure-mode form — it tests whether the candidate understands scratch orgs are compute, not storage.

INCIDENT 3 — THE PIPELINE THAT PASSED AND STILL BROKE PROD

STAKES

CI is green. Every PR check passed, the release branch merged clean, the deploy to production succeeded with "All tests passed." Ninety minutes later, a support ticket: the nightly integration batch job is throwing System.NullPointerException in a class nobody touched this release. The release manager's first words: "But CI said it passed."

THE INCIDENT

# CI pipeline step (GitHub Actions-style)
- name: Deploy to Production
  run: |
    sf project deploy start \
      --target-org prod \
      --test-level RunLocalTests \
      --wait 30

# "RunLocalTests" = all Apex tests belonging to namespaces/classes local to this org's
# unmanaged code — excludes managed package test classes.
# The failing class, NightlyIntegrationBatch, calls a method in an installed
# managed package that changed behavior in a recent package upgrade — but no local
# test exercises that integration path, and the managed package's own tests
# aren't run by RunLocalTests (they're not "local").

THE PROBLEM

"All tests passed" and yet production broke on a code path CI never exercised. Name the four Salesforce test levels and what each actually covers, explain precisely why RunLocalTests let this through, and design the release process fix.

Write: (1) the four test levels, (2) why RunLocalTests missed this, (3) the fix.

HINT LADDER

• Hint 1 (the avenue): (1) The four levels: NoTestRun, RunSpecifiedTests, RunLocalTests, RunAllTestsInOrg. (2) RunLocalTests deliberately excludes managed package test classes — "local" means code that isn't part of an installed managed package's namespace. (3) The gap is coverage-of-integration-surface, not coverage-percentage — 75%+ org-wide coverage says nothing about whether the interaction with the managed package was tested.
• Hint 2 (the mechanism): RunLocalTests runs every Apex test in the org except tests belonging to managed packages — this is intentional (you don't own or want to re-run a vendor's package tests on every deploy), but it means a regression introduced by your code's new dependency on a package method is only caught if your own local test class explicitly exercises that call path. Here, nobody wrote a test for NightlyIntegrationBatch calling into the package method at all (a test-coverage gap that still satisfied the org's aggregate coverage threshold because other classes had plenty of coverage) — the deploy's test run was 100% green because the one test that would have caught it didn't exist, not because the level ran the wrong tests. RunAllTestsInOrg would have re-run everything including any of the package's own tests, but would still not create a test that doesn't exist in your codebase — the real gap is a missing test, exposed by the wrong test-level choice for a change of this risk profile.
• Hint 3 (the skeleton): Fix: (a) reserve RunLocalTests for low-risk releases; use RunSpecifiedTests deliberately naming the tests that cover every changed/dependent class (forces someone to think about blast radius) or RunAllTestsInOrg for releases touching integration points, on a schedule where the longer runtime is acceptable; (b) mandate that any class calling into a managed package's public API gets an explicit unit test with a mock/stub for that boundary; (c) add a validation-only deploy on every PR (sf project deploy start --dry-run / checkonly) against a full-copy or partial sandbox that actually has the managed package installed, so integration surface is exercised pre-merge, not just pre-prod.

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — the "green CI, broken batch job" story every team with managed-package dependencies eventually hits):

The four test levels (memorize with what each excludes):

1. NoTestRun — no tests execute at all; only legal for deploys to sandboxes/scratch orgs (not production), and used for validation-only fast checks of metadata-only changes.
2. RunSpecifiedTests — only the explicitly named test classes run; fastest meaningful option, but requires the deployer to correctly identify every affected test — a manual, error-prone step unless automated by dependency analysis.
3. RunLocalTests — every Apex test class in the org's own namespace runs, excluding any test classes that belong to an installed managed package. This is the default many pipelines use because it's "everything I own," but "everything I own" is not the same as "everything my code touches."
4. RunAllTestsInOrg — every test class in the org runs, including managed package tests; required for production deploys when the org has none of the other levels satisfying coverage, and the safest (slowest) option for high-risk releases.

Why RunLocalTests let this through: the level correctly ran every local test — the gap was that no local test existed exercising NightlyIntegrationBatch's call into the managed package's changed method. The org still cleared the required ≥75% coverage threshold in aggregate because unrelated classes were well-tested. Aggregate coverage percentage is not the same as coverage of the specific code paths at risk in this release — the same lie a green CI badge tells when it measures the wrong thing.

The release-process fix:

1. Risk-based test-level selection: low-risk/metadata-only changes → RunLocalTests is fine; any release touching a class that calls a managed package, or introducing a new integration surface → escalate to RunSpecifiedTests (explicitly including the dependency's boundary tests) or RunAllTestsInOrg.
2. Mandatory boundary tests: any Apex method calling into a package's public API gets a dedicated test with the package call mocked/stubbed (or run against a sandbox with the actual package installed) — this is the same "test the seam" discipline as Module 6's mocking lesson, applied to package boundaries.
3. Validation-only pre-merge gate: every PR runs a checkonly/--dry-run deploy against a persistent integration sandbox that mirrors production's installed packages — catching this class of break before merge, not after prod deploy.

Why the "obvious fixes" failed (the contrast):

• "Just always run RunAllTestsInOrg" → catches vendor test regressions in the package itself, but does nothing for a missing test in your code — the actual gap here. It also makes every deploy far slower without fixing the root cause.
• "Increase code coverage requirement to 90%" → coverage percentage is not coverage of the right paths; you can hit 90% while never exercising the one risky integration call.
• "Blame the managed package vendor's upgrade" → the package changed behavior (their right, if documented), but the local codebase had zero test defense at that boundary — that's the fixable half.

KNOWLEDGE EXTRACTION (interview-ready)

• "The four test levels — name and describe?" → NoTestRun (sandbox/scratch/validation-only only), RunSpecifiedTests (named classes), RunLocalTests (everything local, excludes managed-package tests), RunAllTestsInOrg (everything, including package tests) — production deploys require ≥75% aggregate coverage under RunLocalTests/RunAllTestsInOrg.
• "Why would CI pass and prod still break?" → Aggregate coverage ≠ coverage of the actually-changed/at-risk code paths; RunLocalTests specifically excludes managed package test classes, so a regression in the seam between local code and a package is invisible unless a local test targets it.
• "How do you test a managed-package integration seam?" → A dedicated unit test at that boundary (mock/stub the package call, or run in a sandbox with the package installed) — treat it like any external dependency (Module 6's mocking discipline).
• "Validate vs Deploy in CI?" → Validation-only (checkonly/--dry-run) deploys on every PR give you the test-run and metadata-compile signal without committing to prod — the loading-dock inspection before the truck leaves.

THE REDO

From memory: the four test levels and what each excludes, why aggregate coverage lied here, and the three-part release-process fix.

RETRIEVAL DRILL

1. Name the four Apex test levels.
2. What exactly does RunLocalTests exclude?
3. What's the minimum aggregate coverage requirement for a production deploy?
4. Why did 100% test-pass + adequate coverage still miss this bug?
5. What's the fix for testing a managed-package integration seam?

INTERVIEW MAPPING

"How do you decide which test level to run in your pipeline?" is a standard CI/CD scripted question; the follow-up ("what if CI passes but prod still breaks?") is exactly this incident — most candidates only know the test levels exist, not why RunLocalTests is a trap with managed packages.

INCIDENT 4 — THE MANAGED PACKAGE THAT BROKE EVERY SUBSCRIBER

STAKES

An ISV ships version 4.2 of its managed package to every subscriber org via a scheduled push upgrade. Within an hour, support tickets flood in from a dozen customers: a Flow that calls the package's @InvocableMethod now throws INVALID_TYPE_ON_FIELD_IN_RECORD — the method's input parameter changed shape. Every subscriber's automation that depended on the old signature is now broken, in production, simultaneously, with no warning.

THE INCIDENT

// v4.1 (what every subscriber's Flow currently calls):
@InvocableMethod(label='Calculate Discount')
public static List<DiscountResult> calculate(List<DiscountRequest> requests) { ... }

public class DiscountRequest {
    @InvocableVariable public Decimal amount;
    @InvocableVariable public String tier;
}

// v4.2 (the "improvement" the ISV shipped):
public class DiscountRequest {
    @InvocableVariable public Decimal orderAmount;   // renamed from "amount"
    @InvocableVariable public String customerTier;   // renamed from "tier"
}
// Every subscriber Flow's action call still maps to the OLD field names → breaks.

THE PROBLEM

Name the packaging model that should have prevented this (1GP vs 2GP, and the versioning discipline within it), explain exactly why a field rename in a managed package is catastrophic in a way it wouldn't be in unmanaged code, and design the ISV's correct release process going forward.

Write: (1) the packaging/versioning concept that was violated, (2) why renames are uniquely dangerous in managed packages, (3) the corrected release discipline.

HINT LADDER

• Hint 1 (the avenue): (1) Managed packages (1GP legacy or 2GP modern) exist specifically to let an ISV ship versioned, backward-compatible upgrades to subscribers who never see the source — that promise was broken. (2) A public API element (an @InvocableVariable/@InvocableMethod signature) in a managed package is a contract; subscribers' Flows/Apex reference it by name, and the package's own code obfuscation means subscribers can't "just fix it" on their end. (3) The fix is API versioning discipline: never rename/remove a public element; add new ones and deprecate the old ones over multiple major versions.
• Hint 2 (the mechanism): In a managed package, once a field, method, or class is marked global/public and shipped, Salesforce's backward-compatibility rules (and ISV best practice) treat it as permanent — you can add to it, you cannot rename or remove it without breaking every subscriber referencing it, because subscribers integrate via declarative tools (Flow, Process Builder-successors) that store the name as a string reference, invisible to any compiler check on the ISV's side. A push upgrade (the ISV scheduling a mandatory version bump for all subscribers) amplifies the blast radius to every customer simultaneously with zero opt-out window. 2GP (second-generation packaging) doesn't prevent bad API design — it improves modularity, dependency management, and CI-friendliness, but the backward-compatibility discipline is still the developer's responsibility, enforced by "Salesforce checks you're not breaking global-scoped members" at package-version creation, which is a lint check, not a design guarantee — a rename that keeps the type compatible but changes the name can still slip through global-member deletion protection only if the linter deems it a removal; a straightforward rename via delete-old/add-new is exactly the removal the protections are meant to catch, meaning the ISV likely bypassed or didn't hit that check (e.g., by keeping the class name identical while altering internal variable names insufficiently protected as "global").
• Hint 3 (the skeleton): Corrected process: (a) treat every global/public member as permanent once shipped — new requirements become NEW fields/methods (orderAmountV2) with the old ones marked deprecated but functional; (b) push upgrades only after a beta/pilot cohort and a deprecation notice window — never a silent mandatory bump to 100% of subscribers; (c) version-gate breaking changes into a new major package version that subscribers opt into deliberately, never inherit via automatic push; (d) maintain a compatibility test suite that runs the previous major version's public contract against the new package build before every release.

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — the AppExchange ISV "silent breaking change" story; the reason managed-package API discipline is its own specialty):

The packaging concept violated: managed packages (whether legacy 1GP or the modern 2GP) exist to let an ISV ship code that subscribers install as a black box and receive versioned upgrades for — the entire value proposition rests on backward compatibility as a promise, not a courtesy. Salesforce enforces some of this mechanically (you cannot delete a global class/method/field from a package once it's been released in a version — that's a build-time error), but a rename achieves the same subscriber-facing break while technically being "add a new member, stop using an old one" if the old one wasn't itself marked global in a way the platform's protections tracked as removal-worthy, or if the ISV simply didn't run the compatibility check. The practical result is identical to a deletion: every subscriber Flow referencing amount/tier by name now fails.

Why this is uniquely dangerous in managed packages (vs. unmanaged/internal code): in an internal org, a field rename is a one-repo, one-team problem — you grep for references and fix them in the same change. In a managed package, the subscribers — dozens or hundreds of independent orgs, each with their own Flows, reports, and Apex referencing the package's public surface by name — have no visibility into the ISV's code and no ability to pre-empt the break. The obfuscation that protects the ISV's IP also means subscribers can't patch around a breaking change; they can only wait for a fix or roll back (which for managed packages is itself limited — package versions generally can't be downgraded without an uninstall/reinstall, which can cascade data loss for dependent custom objects).

The corrected release discipline:

1. Additive-only public API evolution: never rename or remove a global-scoped member; add orderAmountV2 alongside the deprecated amount, map internally, deprecate over 2+ major versions with clear release notes.
2. Staged rollout, not silent push: beta/pilot subscriber cohort first, deprecation-notice window, opt-in major-version upgrades for breaking changes — mandatory push upgrades reserved for security patches and additive changes only.
3. Automated compatibility testing: a test suite that instantiates and calls the previous major version's documented public contract against the new build before every package version is created — CI-enforced, not tribal knowledge.
4. Semantic versioning communicated to subscribers, and 2GP's dependency-management features used to let subscribers pin a package version rather than being force-upgraded.

Why the "obvious fixes" failed (the contrast):

• "Roll back the push upgrade" → managed package version downgrades are heavily restricted; you often cannot simply revert without uninstall, which itself risks deleting dependent custom data.
• "Tell subscribers to update their Flows" → shifts the ISV's bug onto every customer simultaneously, at the worst possible time (mid-incident), with no advance notice.
• "It's just a rename, subscribers should adapt" → subscribers integrate declaratively by name; there is no compiler warning them in advance, and no code review on their side catches an ISV-side rename.

KNOWLEDGE EXTRACTION (interview-ready)

• "Unmanaged vs Unlocked vs Managed packages?" → Unmanaged: one-time transfer, fully editable after install, no upgrade path (starter kits/templates). Unlocked: modern CI/CD standard for internal multi-team orgs — upgradeable, source-trackable, not IP-protected, ideal for org-internal modularization. Managed (1GP legacy / 2GP modern): IP-protected (obfuscated Apex), AppExchange-distributable, versioned upgrade path, subject to backward-compatibility rules and (for AppExchange listing) a security review.
• "1GP vs 2GP?" → 1GP: package tied to a single Dev Hub/namespace org, harder CI integration, being phased toward 2GP. 2GP: source-driven, better dependency management, ancestry/branching for versions, CI/CD-friendly, still enforces the same backward-compatibility rule for global members.
• "Why can't you delete a global member from a managed package?" → Once released in a version, Salesforce blocks deletion of global-scoped Apex members at package-version creation — the backward-compatibility guarantee subscribers depend on; renames can achieve the same practical break if not caught.
• "Push upgrade vs subscriber-initiated upgrade?" → Push: ISV forces the new version onto installed orgs (used for critical/security fixes, or by policy) — reserve for additive/safe changes; a breaking change should be a subscriber-initiated major-version upgrade with notice.

THE REDO

From memory: the packaging model (1GP/2GP + managed backward-compatibility rule), why renames break subscribers uniquely, and the four-point release discipline.

RETRIEVAL DRILL

1. Difference between Unlocked and Managed packages — when do you use each?
2. Can you delete a global Apex member from an already-released package version?
3. Why is a field rename in a managed package as dangerous as a deletion?
4. What's a push upgrade, and when should it be avoided?
5. What compatibility safeguard should run before every new package version ships?

INTERVIEW MAPPING

"Unmanaged vs Unlocked vs Managed — when would you use each?" is a standard packaging question; this incident is the scenario form testing whether the candidate understands the consequence of the managed-package promise, not just the definitions.

INCIDENT 5 — THE REFRESH THAT ERASED THE INTEGRATION

STAKES

QA requests a Full sandbox refresh to get realistic production data before UAT. The Basis admin runs it Saturday night. Monday morning: the middleware integration that syncs Orders to the ERP system is throwing INVALID_SESSION_ID on every call, and worse, users report they've all been logged out and their passwords no longer work in the sandbox. The integration team says "nothing changed on our end." The admin says "I just refreshed a sandbox — that shouldn't touch anything."

THE INCIDENT

Sandbox: UAT (Full copy) refreshed from Production, Saturday 11:00 PM.

Monday: 
- Named Credential / OAuth connected app tokens for the ERP middleware: INVALID_SESSION_ID
- All sandbox user passwords: reset, login requires password reset flow
- Outbound email: routed to a "System" test address instead of real users (as designed)
- A custom "PostRefreshSetup" Apex class that used to auto-reconnect integrations: never ran

THE PROBLEM

"A refresh shouldn't touch anything" — explain precisely what a sandbox refresh always resets regardless of sandbox type, name the interface that exists specifically to automate post-refresh fixes, and design the corrected refresh runbook.

Write: (1) what always resets on refresh, (2) the interface/mechanism to automate recovery, (3) the runbook.

HINT LADDER

• Hint 1 (the avenue): (1) A sandbox refresh is a full reset of the sandbox's org shell from production, not an incremental sync — it always invalidates OAuth tokens/sessions, resets user passwords, and routes system email to sandbox-only addresses, by design (security/isolation), regardless of Developer/Partial/Full type. (2) Salesforce provides SandboxPostCopy — an Apex interface a class can implement to run custom logic automatically right after a refresh completes. (3) The runbook needs the OAuth/Named Credential reconnection and password reset communicated and, ideally, automated via that interface, BEFORE the refresh, not discovered after.
• Hint 2 (the mechanism): Every sandbox refresh, no matter the type (Developer/Developer Pro/Partial Copy/Full), performs a full metadata (and for Partial/Full, data) copy from production, and as part of that copy Salesforce always: invalidates all existing session/OAuth tokens (so any Named Credential or Connected App relying on a previously-authorized token breaks), resets all user passwords (forcing a reset flow), and switches outbound email deliverability to "System email only" (sandbox emails go only to the sandbox's designated admin, not real recipients) — these are hard platform behaviors, not configuration the admin controls. The org DID have a PostRefreshSetup idea in mind (the ticket mentions it) but it was never actually implemented as the platform's SandboxPostCopy Apex interface — a class implementing SandboxPostCopy.runApexClass(SandboxContext context) is automatically invoked by Salesforce right after the refresh completes, and is the documented place to re-authenticate Named Credentials programmatically (or at least flag/notify), reset feature flags, and re-point integration endpoints to sandbox-safe URLs.
• Hint 3 (the skeleton): Runbook: (a) BEFORE refresh — notify integration owners of the exact date/time, since OAuth/session invalidation is guaranteed; (b) implement a class implementing SandboxPostCopy that runs automatically post-refresh to re-establish Named Credential auth (or at minimum send a checklist email to the integration team with exact reconnection steps), reset any sandbox-only Custom Metadata/Custom Settings that should point away from prod endpoints, and disable/redirect outbound integrations that shouldn't fire against stale data; (c) AFTER refresh — a documented checklist: re-authorize Named Credentials, communicate password-reset requirement to test users, verify email deliverability setting, re-run any org-specific setup (feature flags, sandbox-only integration users).

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — the "the refresh broke everything and nobody warned us" story; a rite of passage for every team that treats sandbox refresh as a data-only operation):

What a refresh ALWAYS resets, regardless of sandbox type: a sandbox refresh replaces the sandbox with a fresh full copy of production's metadata (and, for Partial Copy/Full, a data sample or full copy per the sandbox type's rules) — and as an unavoidable consequence: (1) all OAuth tokens and active sessions are invalidated — anything connected via a Named Credential, Connected App, or a previously-issued session ID has to reauthenticate; (2) every user's password is reset and login requires the password-reset flow; (3) outbound email deliverability defaults to "System email only", routing test emails away from real recipients — a safety feature so sandbox testing never spams real customers, but a surprise if the team assumed email would flow normally. None of this is a bug; it's the platform's isolation guarantee for a refreshed environment, and it fires every single refresh, whether Developer, Developer Pro, Partial Copy, or Full.

The interface built for exactly this: SandboxPostCopy is an Apex interface — implement runApexClass(SandboxContext context) in a class that implements it, and Salesforce automatically invokes that class immediately after a refresh completes (per sandbox, since you can have a different post-copy class per sandbox or a shared one keyed by sandbox name via context.sandboxName()). This is the documented, correct place to: kick off Named Credential re-authentication flows or notification emails, reset feature-flag Custom Metadata to sandbox-safe values, redirect integration endpoints to sandbox mocks/test endpoints, and log a "refresh completed, here's what needs manual follow-up" summary.

The corrected runbook:

1. Pre-refresh notice to every integration owner with the exact scheduled time — OAuth invalidation is guaranteed, not a maybe.
2. SandboxPostCopy class implemented and deployed in advance, so the moment the refresh completes, automated remediation (or at minimum, notification) fires without anyone having to remember.
3. Post-refresh checklist (documented, in the same repo as the SFDX project): re-authorize every Named Credential, communicate the password-reset requirement, verify the email deliverability setting matches the intended test posture, re-point sandbox-only Custom Metadata/Custom Settings.
4. Never refresh silently — treat it with the same change-management rigor as a production deploy, because its blast radius (every integration, every login) is comparable.

Why the "obvious fixes" failed (the contrast):

• "Just re-authenticate the Named Credential" → fixes the symptom this time; doesn't prevent the next refresh from doing the same thing to a different team who wasn't warned.
• "Ask Salesforce support why passwords reset" → this is documented, guaranteed platform behavior, not a support case — the fix is process, not a ticket.
• "Avoid refreshing Full sandboxes" → the reset behaviors (tokens, passwords, email) happen on every refresh type, not just Full — avoiding Full doesn't dodge this bullet, it just changes the data footprint.

KNOWLEDGE EXTRACTION (interview-ready)

• "What does a sandbox refresh always reset?" → OAuth tokens/active sessions (invalidated), user passwords (reset, forces reset flow), outbound email deliverability (System email only) — regardless of sandbox type.
• "How do you automate post-refresh recovery?" → Implement the SandboxPostCopy Apex interface; Salesforce automatically invokes it right after the refresh completes, per sandbox.
• "Sandbox types and refresh cadence?" → Developer (200MB, metadata only, 1-day refresh interval), Developer Pro (1GB, metadata only, 1-day), Partial Copy (5GB data + 5GB files, sampled via template, max ~10k records/object, 5-day refresh), Full (complete copy of prod data+metadata, 29-day refresh).
• "Sandbox vs scratch org, again?" → Sandbox: refresh-based, long-lived, tied to production's org shell. Scratch org: definition-file-based, source-driven, max 30 days, no "refresh" concept — you just delete and recreate it.

THE REDO

From memory: the three guaranteed refresh resets, the SandboxPostCopy interface and what it's for, and the four-part runbook.

RETRIEVAL DRILL

1. Name the three things every sandbox refresh resets, no matter the type.
2. What interface automates post-refresh remediation, and what method does it require?
3. Refresh intervals for Developer, Partial Copy, and Full sandboxes?
4. Data/storage caps for Partial Copy vs Full?
5. Does a Developer Pro sandbox refresh behave differently from a Full sandbox regarding OAuth/password resets?

INTERVIEW MAPPING

"Walk me through sandbox types and when you'd use each" is a scripted DevOps question; the deeper senior probe is "what breaks after a refresh, and how do you automate around it" — exactly this incident, and SandboxPostCopy is the differentiator most candidates miss.

INCIDENT 6 — THE PROMOTION THAT JUMPED THE QUEUE

STAKES

Two feature branches — a pricing-engine fix (User Story #4021) and a page-layout update (User Story #4033) — are both in the release pipeline, promoted through environments via a Copado-style tool. A release manager, under pressure to hotfix a production bug, promotes #4033 directly from UAT to Production out of pipeline order, skipping the integration environment where #4021 is still sitting. Production deploy succeeds — but now Production has metadata from #4033 built on top of a different version of a shared Apex class than what #4021 (still pending) expects, and the next regular release deploy for #4021 fails with a merge conflict Copado's own conflict detection didn't catch until deploy time.

THE INCIDENT

Pipeline: Feature → Dev/Scratch → Integration → UAT → Production
User Story #4021 (pricing fix): Dev → Integration → UAT (currently here, awaiting next release window)
User Story #4033 (layout hotfix): Dev → Integration → UAT → PROMOTED DIRECTLY TO PROD (skip order)

Shared file touched by both: PricingCalculator.cls
  #4021's UAT version: PricingCalculator.cls (v2, awaiting promotion)
  #4033's promotion to Prod: carried an OLDER snapshot of PricingCalculator.cls (v1)
    because the tool's promotion is "everything in this org's current UAT state" —
    and UAT still had v1 checked in for #4033's branch lineage.

Result: Production PricingCalculator.cls silently REVERTED to v1 behavior.
Next release (with #4021) fails deploy: "conflicting changes detected in PricingCalculator.cls"

THE PROBLOM

Name the specific promotion-tooling mechanism that caused a silent regression (not a deploy failure — a successful deploy that reverted code), the conflict-detection gap, and the release-governance fix that allows legitimate hotfixes without this risk.

Write: (1) the mechanism that reverted the class, (2) why conflict detection didn't catch it, (3) the governance fix (hotfix lane design).

HINT LADDER

• Hint 1 (the avenue): (1) Promotion-based tools (Copado's Promotion vs Deployment distinction, Gearset's org-to-org diff) move a snapshot of an environment's current state for the selected user story/branch — if that environment's snapshot of a shared file is behind another pending story's version, promoting the hotfix silently carries the OLDER file along with it. (2) Conflict detection tools typically compare the incoming change against the target org's current state, not against other pending, not-yet-promoted work sitting in earlier pipeline stages — so a conflict between two branches that haven't both reached the same environment yet is invisible until they collide at deploy time. (3) The fix: a dedicated hotfix lane that promotes directly from a hotfix branch cut from current production (not from an environment that's behind), plus back-merging the hotfix into every pending release branch immediately.
• Hint 2 (the mechanism): Copado/Gearset-style promotion moves metadata associated with a user story by diffing/snapshotting the environment the story currently lives in — when #4033 was promoted "out of order" straight to Production, the tool pulled the full metadata footprint associated with that promotion path, which included whatever version of PricingCalculator.cls existed in the environment it was promoted from (UAT, at a point where #4021's newer version hadn't yet landed there in a way tied to #4033's story). Conflict detection in these tools generally works by comparing the components a specific promotion will change against the target org's current metadata — it does not simulate "what if User Story A and User Story B, still sitting in different environments, both eventually reach Production" — that's a lineage/ordering problem, not a same-target-org diff problem, so it's structurally invisible to point-in-time conflict checks.
• Hint 3 (the skeleton): Governance fix: (a) a dedicated hotfix lane — cut the hotfix branch from Production's actual current state (or the latest tag deployed to Production), not from an upstream environment that may be behind; (b) promote the hotfix through an expedited but still-ordered path (hotfix branch → fast-track validation → Production), never by reaching into a mid-pipeline environment and cherry-picking; (c) immediately back-merge the hotfix into every active release branch (including #4021's) so the next regular promotion carries both changes forward consistently; (d) require conflict detection to run not just against the target org but against all other open/pending promotions touching the same files — or, pragmatically, enforce a file-ownership/communication norm (Slack alert: "PricingCalculator.cls is touched by #4021 and #4033, coordinate before promoting either").

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — the "hotfix reverted a pending release" story; the classic pitfall of promotion-based release tooling used without lineage discipline):

The mechanism that silently reverted the class: Copado (and similarly Gearset) model releases around User Stories moving through Pipelines of connected Org Connections, with Promotion meaning "commit the metadata associated with this story from its current environment forward" and Deployment meaning "push a specific package/version to a target." When the release manager promoted #4033 directly to Production, the tool correctly moved everything associated with #4033's story — but the version of the shared PricingCalculator.cls bundled in that promotion was whatever existed in the environment #4033 was promoted from, which was behind #4021's still-pending, newer edit to the same file. The result wasn't a deploy failure — it was a successful deploy that silently regressed shared code, because the tool has no concept of "another story, elsewhere in the pipeline, has a newer version of a file you're about to overwrite."

Why conflict detection missed it: these tools' conflict detection is typically a point-in-time diff between the incoming promotion and the target org's current state — it correctly would have flagged a conflict if two promotions targeting the SAME org at the SAME time touched the same file with divergent changes. It does not (by default) simulate cross-environment lineage — "story A is sitting in UAT with an old version of file X, story B has a newer version of file X sitting one stage further along, and B hasn't reached Production yet" — because that requires reasoning about pipeline order and future promotions, not just current-state diffing.

The governance fix (hotfix lane design):

1. Dedicated hotfix branch/lane cut from Production's true current state (or the last-deployed tag), never reached into from a mid-pipeline environment that might be behind on shared files.
2. Fast-tracked but still-ordered promotion for the hotfix — expedited review/approval gates, not "skip the pipeline entirely."
3. Mandatory immediate back-merge of the hotfix into every open release branch (here, #4021's) so the next regular promotion inherits both changes consistently instead of colliding.
4. File-level ownership/communication norm — when two active stories touch the same Apex class, the tool (or a lightweight process — a Slack bot watching the repo) flags it so promotions are coordinated, not surprises.

Why the "obvious fixes" failed (the contrast):

• "Just don't allow out-of-order promotions, ever" → real production incidents need real hotfixes faster than the standard pipeline; the fix is a safe hotfix lane, not a ban that gets bypassed under pressure anyway (as it clearly was here).
• "Trust the tool's conflict detection" → it did exactly what it's designed to do (diff against target state); the gap is architectural (cross-environment lineage), not a tool bug to file a ticket about.
• "Revert Production back to v2 manually" → fixes the immediate symptom but doesn't address why the hotfix lane doesn't exist, so the next emergency repeats this exact failure.

KNOWLEDGE EXTRACTION (interview-ready)

• "Copado vocabulary — Promotion vs Deployment?" → Promotion: commit a User Story's associated metadata forward from its current environment into the pipeline (source-control-like). Deployment: push a specific package/version to a target org (the actual metadata deploy). Org Connections link pipeline stages; Pipelines model the branch/environment sequence.
• "What's a hotfix lane, and why does it matter?" → A pre-approved fast path for emergency production fixes that's cut from Production's actual current state (never from a possibly-behind mid-pipeline environment) and is immediately back-merged into all pending release branches — prevents exactly this "successful deploy, silent regression" class of bug.
• "Why can conflict detection miss a real conflict?" → Most tools diff the incoming promotion against the target org's current state only — cross-environment/cross-story lineage conflicts (two branches with divergent changes to the same file, neither yet merged) are invisible until they actually collide at a shared target.
• "Gearset's model, for contrast?" → Metadata comparison/diff-driven deploys with automated pipelines, scheduled backups, and one-click rollback — its conflict detection is similarly a diff against target state; the lineage-blind-spot lesson applies to any promotion/pipeline tool, not just Copado.

THE REDO

From memory: what Promotion vs Deployment mean in Copado's vocabulary, why the hotfix reverted a shared class without a deploy error, and the four-part hotfix-lane governance fix.

RETRIEVAL DRILL

1. Copado: difference between Promotion and Deployment?
2. Why did the hotfix's successful promotion silently revert shared code?
3. What kind of conflict does standard conflict detection catch, and what kind does it miss?
4. Name the four parts of a safe hotfix lane.
5. What's Gearset's headline differentiator versus a pure Git+CI pipeline?

INTERVIEW MAPPING

"Have you used Copado/Gearset/DevOps Center — how do you handle hotfixes in a multi-team pipeline?" is asked in any role touching release management; naming the promotion-vs-deployment vocabulary and the lineage-conflict blind spot is the senior differentiator.

INCIDENT 7 — THE JWT THAT WORKED ON TUESDAY

STAKES

The nightly CI pipeline has deployed to the integration sandbox flawlessly for three months using JWT-based auth (no interactive login, no stored password). Wednesday morning: every pipeline run fails at the auth step with invalid_grant: user hasn't approved this consumer or invalid_grant: authentication failure. Nobody touched the CI config. The release is blocked and the team's first instinct is "Salesforce broke something."

THE INCIDENT

# CI auth step
- name: Authenticate to Salesforce
  run: |
    sf org login jwt \
      --client-id $CONSUMER_KEY \
      --jwt-key-file server.key \
      --username ci-bot@company.com.integration \
      --instance-url https://test.salesforce.com

# Error:
# Error authenticating with JWT: invalid_grant: user hasn't approved this consumer

THE PROBLEM

List the JWT Bearer flow's moving parts (the certificate, the Connected App, the consumer key, the pre-authorization, the IP-relaxation/login-IP-range setting) and, for each, name the specific way it silently breaks over time. Then design the CI auth setup that's resistant to this class of failure.

Write: (1) the JWT flow's moving parts and their individual failure modes, (2) the most likely culprit given the symptom, (3) the resilient setup.

HINT LADDER

• Hint 1 (the avenue): (1) JWT Bearer flow needs: a self-signed certificate uploaded to a Connected App, the Connected App configured with that certificate and pre-authorized for the integration user (or "Admin approved users are pre-authorized"), the private key held by CI matching the uploaded cert, and (often) IP relaxation / a trusted IP range so the non-interactive flow isn't blocked. (2) invalid_grant: user hasn't approved this consumer specifically means the pre-authorization was lost — e.g., a permission-set/profile change removed the user's assignment to the Connected App, or someone edited the Connected App's OAuth policy from "Admin approved users are pre-authorized" back to a mode requiring interactive consent. (3) The resilient setup: dedicated integration user with a permission set (not profile-only) granting Connected App access, certificate rotation calendar tracked outside anyone's memory, and a CI health-check job that fails loudly on auth before the real deploy runs.
• Hint 2 (the mechanism): The JWT Bearer flow's failure modes, one by one: certificate expiry (self-signed certs have an expiration date; if nobody tracks it, the cert silently expires and every JWT signed with the now-invalid key is rejected — but the error there is usually invalid_grant: invalid assertion, not this one); Connected App consumer key/secret rotation (if someone regenerates the consumer key without updating the CI secret, auth fails immediately); pre-authorization removal (the specific error here) — the Connected App must be set to "Admin approved users are pre-authorized" with the integration user's profile/permission set explicitly assigned; if a permission-set change (e.g., a cleanup of "unused" permission sets, exactly the kind Module 5 warns about) accidentally un-assigns the integration user from that permission set, the platform now requires the (impossible, headless) interactive OAuth consent screen — hence "user hasn't approved this consumer"; IP relaxation (a Connected App or profile's login IP ranges tightened, and the CI runner's IP isn't in the allowed range) throws a different error (invalid_grant: ip restricted) but is worth ruling out. Given the exact error text, the most likely culprit is the pre-authorization/permission-set assignment, not the certificate.
• Hint 3 (the skeleton): Resilient setup: dedicated ci-bot integration user with a permission set (never profile-only, so it survives profile refactors) explicitly assigned "Connected App access" via the Connected App's policy set to "Admin approved users are pre-authorized," certificate rotation tracked on a calendar/expiry-monitoring script (openssl x509 -enddate) well before the ~1-2 year expiry, consumer key stored in CI secrets with a documented rotation runbook, and IP relaxation configured explicitly for the CI runner's egress IP range (or "Relax IP restrictions: Apply Login IP restrictions to API only, if browser is used" configured deliberately) — plus a lightweight auth-only smoke-test job that runs before the real deploy step and pages the team the moment auth breaks, isolating "CI auth is broken" from "the deploy itself failed."

THE REVEAL — POSTMORTEM

What actually happened (real class of incidents — the "CI auth silently rotted" story; every team running headless JWT auth for months without an expiry/permission audit):

The JWT Bearer flow's moving parts and how each breaks:

1. Self-signed certificate — uploaded to the Connected App; the CI holds the matching private key to sign the JWT assertion. Breaks via expiry (certs typically issued for 1-2 years; if untracked, silent time-bomb) — error is usually invalid_grant: invalid assertion or similar, not the one seen here.
2. Connected App consumer key — the client ID CI presents. Breaks if regenerated without updating CI's stored secret — immediate, obvious auth failure.
3. Pre-authorization ("Admin approved users are pre-authorized") — the Connected App's OAuth policy must have this set, AND the integration user must be assigned to a profile or permission set with access to the Connected App. Breaks if that assignment is removed — e.g., a permission-set cleanup, a profile re-org, or someone switching the Connected App's policy to require interactive user consent. This is the exact error text observed: invalid_grant: user hasn't approved this consumer — the platform is saying, correctly, that headless JWT auth can't get consent because the pre-authorization path is gone.
4. IP relaxation / login IP ranges — if a profile or Connected App's IP restrictions tighten and the CI runner's IP falls outside them, auth fails with a distinct ip restricted message — worth checking but not the likely culprit given the exact error text here.

The resilient CI auth setup:

1. Dedicated integration user, access granted via a permission set (not baked into a profile that gets refactored) — permission sets are additive and less likely to be silently altered during unrelated profile cleanup.
2. Certificate expiry monitored by an automated check (openssl x509 -in server.crt -noout -enddate in a scheduled job) well ahead of expiry, with a documented rotation runbook.
3. A pre-flight, auth-only smoke test in CI that runs before the real deploy and fails fast with a clear "auth broke" signal — distinguishing "CI infra problem" from "the deploy itself is bad" saves hours of misdiagnosis.
4. Change-management awareness: any permission-set/profile cleanup review explicitly checks "does this remove Connected App / integration-user access" before merging — the same governance discipline Module 5 teaches for security reviews, applied to CI credentials.

Why the "obvious fixes" failed (the contrast):

• "Salesforce broke something — file a case" → the platform is behaving correctly; a local permission-set change removed the pre-authorization. No incident on Salesforce's side.
• "Regenerate the consumer key and certificate" → treats the symptom of a different failure mode (key/cert rotation) and doesn't address the actual cause (missing pre-authorization); would not fix this specific error.
• "Re-run the pipeline, it's probably a fluke" → an invalid_grant config error doesn't self-heal on retry; wastes CI minutes and delays diagnosis.

KNOWLEDGE EXTRACTION (interview-ready)

• "How does JWT Bearer flow work for CI, and why?" → No interactive login, no stored password — a self-signed certificate authenticates a Connected App on behalf of a pre-authorized integration user via a signed JWT assertion. Ideal for headless CI/CD.
• "What are the moving parts and their failure modes?" → Certificate (expiry), consumer key (rotation without secret update), pre-authorization/permission-set assignment (removed by an unrelated cleanup — the classic silent break), IP relaxation (tightened ranges blocking the CI runner).
• "What does invalid_grant: user hasn't approved this consumer specifically mean?" → The Connected App's pre-authorization for that user is gone — either the policy changed away from "Admin approved users are pre-authorized," or the user's profile/permission set no longer grants access to the Connected App.
• "Why use a permission set instead of profile access for the integration user?" → Permission sets are additive and reviewed independently; a profile refactor/cleanup is far more likely to accidentally strip Connected App access than a dedicated, clearly-named permission set would be.

THE REDO

From memory: the four JWT moving parts and each one's specific failure signature, why THIS error points to pre-authorization not the certificate, and the four-part resilient CI auth setup.

RETRIEVAL DRILL

1. What are the four moving parts of a JWT Bearer CI auth setup?
2. What error text specifically indicates a lost pre-authorization vs. an expired certificate?
3. Why use a permission set rather than a profile for the integration user's Connected App access?
4. What's the value of a pre-flight auth-only smoke test in CI?
5. Besides certs and permissions, what other setting can silently block headless auth?

INTERVIEW MAPPING

"How do you set up CI/CD auth without storing a password?" is a standard scripted DevOps/security question; the failure-mode breakdown here (which error means what) is the senior-level follow-up most junior candidates can't answer.

INCIDENT 8 — THE CANDIDATE WHO ONLY KNEW CHANGE SETS

STAKES

A final-round interview, 3-4 YOE band. The interviewer asks: "Walk me through how your team deploys changes from sandbox to production." The candidate answers fluently and confidently — entirely in terms of Change Sets: outbound Change Sets, deployment connections, "you upload it and then deploy it in prod." The interviewer's next three follow-up questions all fall flat. This incident dramatizes exactly what goes wrong and why it's a hard red flag in 2026.

THE INCIDENT

Interviewer: "How do you handle version control — what's in your Git repo?"
Candidate:   "We don't really use Git for Salesforce, the Change Set is like our record
             of what moved."

Interviewer: "What if you need to roll back after a bad Change Set deploy?"
Candidate:   "You'd create a new Change Set to undo it, I think? We haven't had to."

Interviewer: "How do you handle 10+ developers working in parallel branches?"
Candidate:   "Each person makes their changes and we Change-Set them up when ready,
             usually one person merges everyone's sandbox changes manually."

Interviewer: [silence] "...Okay. Let's talk about scratch orgs — have you set one up?"
Candidate:   "I've heard of them but haven't used one personally."

THE PROBLEM

Diagnose exactly what each of the candidate's three answers reveals is missing from their process (not just "they don't know SFDX" — name the specific capability gap behind each answer), and then write the answer a 3-4 YOE candidate SHOULD give to the original question, in under 2 minutes, that would NOT trigger any of these follow-ups.

Write: (1) the three specific gaps exposed, (2) why "we don't really use Git" is the worst of the three, (3) the 2-minute model answer to the original question.

HINT LADDER

• Hint 1 (the avenue): (1) Gap 1 (no Git): no diff history, no code review, no CI, no single source of truth — everything downstream (rollback, parallel work, testing gates) becomes manual and fragile because of this ONE root gap. (2) Gap 2 (no rollback plan): Change Sets have no rollback of a successful deploy — the "make a new Change Set to undo it" answer reveals the candidate doesn't know this limitation exists, let alone that Gearset's one-click rollback or a Git revert + redeploy are the real answers. (3) Gap 3 (manual merge of parallel sandbox work): this is the "everyone works in shared/individual sandboxes and merges by hand" antipattern Git branching + scratch orgs exist to eliminate.
• Hint 2 (the mechanism): All three gaps are downstream symptoms of the same root cause: metadata was never treated as text under version control. Without Git: there's no diff to review before merging (so "who changed what" is tribal memory), there's no automated CI test gate (so quality control is "did it work in sandbox"), there's no branch-per-feature model (so parallel work collides in a shared sandbox and someone manually reconciles conflicting XML by hand — exactly the fragile, error-prone process that source-driven development replaced). "We don't really use Git" is the worst answer of the three specifically because it's not a missing feature, it's a missing foundation — every other 2026-relevant DevOps capability (CI/CD, code review, automated testing gates, packaging, scratch-org-per-feature) is built ON TOP of Git being the source of truth; without it, none of the rest is even possible to bolt on later without a foundational migration.
• Hint 3 (the skeleton): The 2-minute model answer needs, in order: (1) Git as the source of truth, SFDX project structure (sfdx-project.json, force-app), feature branches per story; (2) scratch orgs (or sandboxes) per feature, PRs with validation-only deploys + RunSpecifiedTests/RunLocalTests as a CI gate; (3) merge to a develop/integration branch → deploy to an integration sandbox; (4) release branch → UAT; (5) main → Production via CI with RunAllTestsInOrg or equivalent coverage gate, using unlocked packages if the org is modular; (6) rollback story: Git revert + redeploy the prior commit, or a tool like Gearset's one-click rollback — NOT "make a new Change Set." Mentioning Change Sets only as "useful for a single quick admin-only fix, not our primary pipeline" is the correct, calibrated way to bring them up at all.

THE REVEAL — POSTMORTEM

What actually happened (this is literally the research's headline trap, dramatized: "naming only Change Sets in 2026 dates the candidate badly" — and here's the mechanism, not just the verdict):

The three gaps, precisely:

1. "We don't really use Git" — the foundational gap. Every other DevOps capability — code review, CI gates, automated testing, rollback, packaging, parallel branch development — is architecturally built on top of "metadata is text, tracked in a repo, diffable and mergeable." Without it, the candidate's team isn't doing DevOps at all; they're doing manual configuration management with an upload button. This is why it's the worst of the three: it's not a missing feature, it's a missing foundation that makes every other 2026 practice structurally impossible to add without first fixing this.
2. "Make a new Change Set to undo it" reveals the candidate doesn't know Change Sets cannot roll back a successful deploy — there's no "undo" primitive; you'd have to manually reconstruct the prior state as a fresh Change Set (error-prone, and impossible for deletions, since Change Sets can't delete either — compounding Incident 1's lesson). The correct answer, once Git exists, is trivial: revert the commit, redeploy the prior state — or use a tool with built-in rollback (Gearset's one-click rollback, which works because it tracks metadata state over time the way Change Sets never do).
3. "One person merges everyone's sandbox changes manually" reveals no branch-per-feature model and no scratch-org-per-developer isolation — parallel work collides in a single shared sandbox, and reconciliation is manual XML surgery instead of a Git merge with conflict markers a diff tool can show you.

The model 2-minute answer (say this instead): "Git is our source of truth — an SFDX project with force-app/main/default, feature branches per user story. Developers work in scratch orgs spun up per feature from a definition file, or a shared dev sandbox for smaller teams. Every PR triggers a validation-only deploy with RunSpecifiedTests or RunLocalTests as a CI gate before merge. Feature branches merge to develop, which deploys to an integration sandbox; a release branch promotes to UAT; main deploys to production via CI with full test coverage. If something needs to unwind, we revert the commit and redeploy the prior state — Change Sets don't support rollback of a successful deploy, so we don't rely on them as the primary pipeline; they're fine for a one-off admin-only tweak, but the real pipeline is Git-driven."

Why "the obvious fixes" failed (the contrast, in interview terms):

• "I'll just say I know SFDX exists" → naming a tool without explaining the pipeline it enables (Git → branch → CI gate → promote) doesn't answer "walk me through your process"; it invites exactly these follow-ups.
• "Focus on the technical Change Set limits instead of the bigger picture" → interviewers aren't testing trivia recall here, they're testing whether the candidate has actually worked in — or understands — a real team pipeline; reciting the 10,000-file limit without a Git-based answer doesn't fix the red flag.
• "Avoid mentioning Change Sets at all" → also wrong; a calibrated senior answer correctly places Change Sets as a legitimate minor tool for narrow admin-only use, not by pretending they don't exist.

KNOWLEDGE EXTRACTION (interview-ready)

• "Why is 'we use Change Sets' alone a red flag in 2026?" → It signals no Git-based source of truth, which cascades into no rollback story, no CI gate, no code review, no scalable parallel-development model — Change Sets are a legitimate minor tool, never the whole answer.
• "What's the correct branching strategy to describe?" → Feature branch → scratch org/dev sandbox → PR with validation-only CI gate → develop/integration branch → integration sandbox → release branch → UAT → main → Production, each step matching a Git branch to an environment.
• "What's the correct rollback story?" → Git revert + redeploy the prior commit's state, or a tool with built-in rollback (Gearset) — never "make a new Change Set to undo it," since Change Sets have no rollback of successful deploys and can't delete components either.
• "How do you bring up Change Sets without it being a red flag?" → Position them precisely: fine for a single quick admin-driven metadata move between two connected orgs with no CI need; never the primary pipeline for a team shipping regularly.

THE REDO

From memory: the three gaps behind the candidate's three answers, why "no Git" is the worst of the three, and the full 2-minute model pipeline answer.

RETRIEVAL DRILL

1. What's structurally missing when a team's entire deployment story is "Change Sets"?
2. Why can't you "make a new Change Set to undo" a bad deploy?
3. What replaces "one person manually merges everyone's sandbox changes"?
4. Name the branch → environment mapping for a standard SFDX pipeline.
5. What is the ONE correct, calibrated way to mention Change Sets in an interview answer without it being a red flag?

INTERVIEW MAPPING

This IS the scripted question — "walk me through your deployment process" — presented as the failure mode itself. Recognizing which of your own instincts would trigger these follow-ups is the entire point of this incident.

🏆 CAPSTONE — THE RELEASE THAT ATE ITSELF

STAKES

You've just joined as the senior developer on a struggling Salesforce team. The VP of Engineering hands you the postmortem doc from last week's release — three separate, independently-reported symptoms, all from the same release window, all still unresolved. You have a 2:00 PM leadership sync in two hours and need to walk in with a root-cause diagnosis, a triage plan, and a verification checklist. Below is the actual incident report.

THE INCIDENT (multi-part incident report)

INCIDENT REPORT — Release 2026.08.21

SYMPTOM A (Support, 9:12 AM):
  "Customers report the discount calculator managed-package integration stopped
   working after this morning's package auto-update. Error: unexpected parameter."

SYMPTOM B (DevOps, 10:40 AM):
  "The nightly CI deploy to the Integration sandbox has been failing since Tuesday
   with 'invalid_grant: user hasn't approved this consumer.' We didn't touch the
   Connected App. We DID do a permission-set consolidation project last week
   (removed 14 'unused' permission sets org-wide)."

SYMPTOM C (QA Lead, 11:15 AM):
  "The Full sandbox refresh from Saturday broke our automated Selenium login
   tests — every test fails at login. Also separately: the field
   'Discount_Override__c' that was supposed to be removed from the Account page
   layout via last month's Change Set cleanup is STILL showing up for reps in
   production. Compliance flagged it again."

BACKGROUND (from the release branch history):
  - This release included a hotfix (US-5190, a page-layout text fix) that was
    promoted directly from UAT to Production, skipping the Integration stage,
    because "it was just a label change and urgent."
  - The team has 30+ Apex triggers/flows and no documented ownership map.
  - Nobody has looked at whether the org uses Unlocked or Managed packages
    internally — "we just install what the AppExchange vendor gives us."

THE PROBLEM

For each of the three symptoms, name the exact root-cause mechanism (which of the module's incidents it maps to) and the fix. Then write: (1) a 2-minute leadership-sync script that ties all three to one systemic root cause, and (2) a regression/verification checklist your team will run before every future release to catch this entire class of problem again.

HINT LADDER

• Hint 1 (the avenue): Map each symptom to its mechanism: (A) a managed-package breaking API change hitting subscribers (Incident 4's class); (B) JWT pre-authorization broken by an unrelated permission-set consolidation (Incident 7's class, plus the exact trigger event: "removed 14 unused permission sets" is precisely the change-management gap the incident predicted); (C) has TWO separate root causes bundled together — sandbox refresh guaranteed resets (Incident 5's class) breaking Selenium logins, AND a Change Set that never actually deleted the field because Change Sets can't delete components (Incident 1's class, resurfacing a month later because nobody implemented the destructive-changes fix the first time).
• Hint 2 (the mechanism): The systemic root cause tying all of this together: the org has no unified change-management discipline — a vendor's managed package auto-updates with no compatibility testing on the subscriber side (A); a permission-set cleanup ran with no "does this touch Connected App access" check (B); a sandbox refresh ran with no pre-refresh notice or SandboxPostCopy automation and, separately, a "field deletion" a month ago was never actually completed because the team used the wrong tool for deletion and never verified the retrieved metadata state (C, two failures compounding). The hotfix background detail (US-5190 skipping Integration) is a live example of Incident 6's promotion-order risk sitting in the same release, even though it's not the direct cause of A/B/C here — flag it as a latent risk for the leadership sync, not a fourth root cause to over-attribute.
• Hint 3 (the skeleton): 2-minute script structure: one sentence naming the systemic root cause (no unified change-management/verification discipline across packages, permissions, sandboxes, and destructive changes) → one sentence per symptom mapping to its specific mechanism and fix → one sentence on the immediate triage priority (which fire to put out first and why) → one sentence on the structural fix (the verification checklist + governance change) that prevents a repeat. Checklist: pre-release compatibility check for any managed-package boundary code; permission-set/profile change review that explicitly checks Connected App and integration-user access; sandbox refresh runbook with pre-notice + SandboxPostCopy; every destructive change verified by retrieving metadata post-deploy, not trusting deploy-success alone; hotfix lane discipline with mandatory back-merge.

THE REVEAL — expected structure of a strong answer (self-graded against the answer sheet)

A complete answer names, for EACH symptom, the specific mechanism (not just "something broke"), states the fix, and then — critically — steps back to name the ONE thing that would have prevented all three: a change-management discipline that treats every "small, unrelated" change (a permission-set cleanup, a sandbox refresh, a vendor package update, a Change Set cleanup) as a release-worthy event requiring its own verification step, not a background task nobody reviews. The full model answer, 2-minute script, and verification checklist are in the sealed answer sheet — write your own version first.

RETRIEVAL DRILL (for the capstone)

1. Symptom A — name the mechanism and the fix in one sentence each.
2. Symptom B — name the mechanism and the fix in one sentence each.
3. Symptom C has two separate root causes — name both.
4. What's the one systemic root cause tying A, B, and C together?
5. Name three items that belong on a pre-release verification checklist as a direct result of this incident.

INTERVIEW MAPPING

"Tell me about a time you had to diagnose a complex production incident" / "how do you think about release risk across a whole org, not just one feature" — this capstone is the rehearsal for that exact senior-level, multi-thread diagnostic question.

End of Module 7 workbook. Proceed to the sealed answer sheet only after writing your own attempt for every incident, including the capstone's 2-minute script and verification checklist.