Add SDKConfigBenchmark: legacy vs config endpoint benchmark suite

[facumenzella]

Motivation

Visual companion (SDK wiring): https://claude.ai/code/artifact/ba8a51b9-d976-487a-b833-c5313a690347
Visual companion (app-launch tier): https://claude.ai/code/artifact/7ceb8f6d-fb74-4b53-a6b0-cd0d430535ac

We need reproducible, SDK-side numbers to compare the current offerings system against the config endpoint system (and its kill switch) before releasing config + workflows, and a way to keep monitoring both systems afterwards.

Description

Adds SDKConfigBenchmark, a macOS command-line benchmark (own Tuist project, compiles Sources/ directly) that measures time-to-offerings-delivered through the real manager-level SDK code: OfferingsManager.updateOfferingsCache + RemoteConfigManager refresh, called exactly like Purchases.updateAllCaches does. Three modes (legacy, config, config-killswitch), two scenarios (cold, warm with real 304/204 revalidation enforced per iteration), two transports:

• simulated (default): deterministic in-process fixtures with network profiles (ideal/wifi/lte) and a seeded packet-loss model. Same-seed loss-free rows are byte-identical across processes.
• live: real requests against the pinned stress-test project (5f07e7e3), through a recording passthrough so live rows carry the same per-request metrics. PROJECT_ID=<id> swaps the target project (key resolved via mafdet, rows labeled so cross-project results never compare as equivalents).

Output is JSONL (percentiles, phase timings, request/byte counts, error accounting); compare.py diffs two runs and fails on invalid, duplicate, missing, or mislabeled rows, so it works directly as a regression gate. run-matrix.sh runs the curated matrix. Methodology, determinism contract, sample numbers, and known limitations are in Tests/Benchmarks/SDKConfigBenchmark/CONFIG_ENDPOINT_BENCHMARKS.md.

Headline numbers so far: config cold on LTE is ~10x legacy purely because all fixture workflows are prefetch-gated (prefetch scope is the lever to sweep next); config warm delivers at parity with legacy (the manifest 204 revalidates after delivery, matching production); the kill switch costs ~1 API round trip per launch. Live runs against the stress project confirm the full config→blob-store→warm-proof chain, with blobs currently delivered inline by khepri.

SDK sources are untouched except four benchmark-gated hooks (#if SDK_CONFIG_BENCHMARK): the transport injection in HTTPClient, a disk-root override in DirectoryHelper (container dirs ignore $HOME, so runs sandbox all caches under a temp root), and two guards that disable the legacy Documents migrations while sandboxed. No API keys live in source: live runs resolve the target project's key at run time (run-matrix.sh via mafdet, or --api-key/SDK_CONFIG_BENCHMARK_API_KEY for direct binary runs).

App-launch tier. On top of the CLI, SDKConfigBenchmarkApp (own Tuist project under Tests/TestingApps/) measures what the CLI can't: a real iOS app linking the stock RevenueCat/RevenueCatUI products (built Release) that calls Purchases.configure and times configure → first customer info → offerings → paywall content on screen. The headline percentile is configure + getOfferings, with or without workflows compiled in (the team's release-gating metric, matching the CLI tier's total); the paywall marks (wrapper mount, and the SDK's content-appeared event) stay in the row as secondary phases. An XCUITest relaunches the app once per iteration (each a true process cold start; cold wipes SDK state per launch, warm retains it), and run-app-launch.sh builds the two SDK variants by rewriting SWIFT_ACTIVE_COMPILATION_CONDITIONS in Local.xcconfig (which Package.swift feeds into the SPM-built SDK). Variant labels are proven twice: the build log must show the flag reached the SDK compile, and every measured launch must report at runtime whether the config path actually ran (mismatch fails the run, cached builds included).

Every launch also registers blobs: counts split inline vs CDN-downloaded, byte totals, and size extremes (max_inline_blob_bytes / min_downloaded_blob_bytes) that empirically bracket the backend's inline-size budget (blobs under ~3MB should arrive inline; a small blob arriving via CDN is a regression the rows catch). The app observes this through the stock SDK's log stream with a parser pinned to RemoteConfigStrings by unit tests, plus config_persisted / last_blob_stored phase marks; the CLI registers the same fields via its blob store and transport events. The registration caught a real signal on its first Release run: alongside the 2 inline workflow blobs (max 70.8KB), 4 blobs were CDN-fetched, the smallest just 65 bytes — far under the inline budget, worth raising with the config team.

The goal of this tier: run the real app N times through any automation that can drive xcodebuild test (CI, the baguette CLI, a cron box) and collect the same comparable, gateable JSONL the CLI matrix produces, so both systems get measured end to end exactly as a customer app experiences them. Rows are compare.py-compatible (app-launch-legacy / app-launch-config, profile = simulator/device). First Release numbers (simulator, live network, p50 over configure + getOfferings): config cold 1354ms vs legacy 1221ms (+11%), warm 1094ms vs 888ms (+23%).

How to run

One-time setup:

tuist install && tuist generate SDKConfigBenchmark

Full simulated matrix (all modes x scenarios x profiles + loss sweep, ~10 min) and a results table:

bash Tests/Benchmarks/SDKConfigBenchmark/run-matrix.sh > results.jsonl
python3 Tests/Benchmarks/SDKConfigBenchmark/compare.py results.jsonl

Against the real backend (stress-test project by default; key resolved via mafdet):

TRANSPORT=live bash Tests/Benchmarks/SDKConfigBenchmark/run-matrix.sh > live.jsonl
PROJECT_ID=<other-project> TRANSPORT=live bash Tests/Benchmarks/SDKConfigBenchmark/run-matrix.sh > other.jsonl

Regression comparison between two branches (nonzero exit on invalid/duplicate/missing rows):

python3 Tests/Benchmarks/SDKConfigBenchmark/compare.py baseline.jsonl candidate.jsonl

Single configuration:

.build/sdk-config-benchmark-derived-data/Build/Products/Release/SDKConfigBenchmark \
  --mode config --scenario cold --profile lte --loss-percent 20 \
  --iterations 25 --warmup-iterations 3 --paywalls 50 --workflows 100

App-launch tier (real app, live network; builds both SDK variants, ~15-25 min):

bash Tests/TestingApps/SDKConfigBenchmarkApp/run-app-launch.sh > app-launch.jsonl
python3 Tests/Benchmarks/SDKConfigBenchmark/compare.py app-launch.jsonl

Pass DESTINATION="platform=iOS,id=<udid>" for a physical device, ITERATIONS=25 for real numbers. To poke at the app from Xcode: TUIST_BENCH_API_KEY=<key> tuist generate SDKConfigBenchmarkApp injects the key into the (gitignored) scheme.

Unit tests (86):

xcodebuild -workspace RevenueCat-Tuist.xcworkspace -scheme SDKConfigBenchmarkTests \
  -destination platform=macOS test

Every run sandboxes the SDK's disk caches under a temp root; simulated runs cannot touch the network. Full docs: Tests/Benchmarks/SDKConfigBenchmark/README.md.

AI session context

AI Context

Metadata

• PR: this PR
• Branch: facu/sdk-config-benchmark-v2
• Author / human owner: facumenzella
• Agent(s): Claude Code (Fable 5), Codex (review loop via codex-companion)
• Session source: current conversation
• Generated: 2026-07-09 (v3: blob registration, config-path phases, Release + impression + runtime-variant-proof review fixes)
• Context document version: 3

Goal

A repeatable benchmark suite that measures the legacy offerings system vs the config endpoint system through real SDK code, producing numbers trustworthy enough for release decisions and ongoing regression monitoring.

Initial Prompt

Review an earlier codex-authored benchmark harness ("what do you think?"), then: "Let's plan for it. Once you're ready, implement it. Then run it through /codex-review-fix-loop." The plan replaced the earlier harness's hand-rolled /v2/config calls (an endpoint that doesn't exist on main) with real manager-level SDK flows.

Important Follow-up Prompts

• "We should have it hardcoded to https://app.revenuecat.com/projects/5f07e7e3" then "I want this to hit the backend" — added the live transport against production instead of captured fixtures.
• "Once you're done and you both agree, let's make sure this is the best regression test suite, and that it outputs real useful numbers for comparing and monitoring both systems" — framed the review loop's acceptance bar.
• "but POST /v1/config/app does not get a workflow? we should test the whole path" — led to verifying live inline blob delivery end to end after workflows were published in the project.
• "it should be cool to be able to swap the target project" — added PROJECT_ID support.
• "Do you think there's a way to measure this with a real app?" then "I think we should build it around PaywallsTester, and run it only on CI / locally" and "we should mimic a real world scenario like configuring the SDK first" — added the app-launch tier (built as a sibling of PaywallsTester rather than inside it, to keep that project's side effects out).
• "make the cli and the app register the #of blobs, if they are inline or not" and "the goal should be to also measure the blob size limit. Right now if it's smaller than 3MB it should be inlined" — added blob registration with size extremes to both tiers.
• "we're only measuring the offerings path. Why can't we do both?" — added config-path phases (config_persisted, last_blob_stored) to the app tier via the SDK's log stream.
• "the goal is to be able to run the app N times (maybe using baguette cli or any other automation) so we can measure it like we do with the CLI" — stated in the description, docs, and artifacts.

Agent Contribution

• All code in Tests/Benchmarks/SDKConfigBenchmark/, Projects/SDKConfigBenchmark/, and the four gated SDK hooks; 65 unit tests; matrix runner; compare script; methodology doc.
• Ran the full simulated matrix (18 rows) and live matrices against production; verified cross-process determinism empirically.
• Drove a 6-pass adversarial review loop (Codex reviewer + adversarial reviewer + a 4-agent simplify pass), applying 16 findings and rejecting 3 with reasons.
• App-launch tier: all code in Tests/TestingApps/SDKConfigBenchmarkApp/ and Projects/SDKConfigBenchmarkApp/, 6 additional unit tests, the variant runner script, and doc sections; verified end to end on simulator against the live project (4 rows, zero errors).

Human Decisions

• Decision: hit the real backend in live mode rather than capture fixtures (agent had recommended fixture capture; human overrode).
• Decision: initially hardcode the stress-test project's public keys (human's explicit instruction, flagged by agent and reviewer); later reversed by the human — keys removed from source, branch history rewritten and force-pushed to scrub the literals, and resolution moved to run time (mafdet / env var).
• Decision: run the review loop to convergence ("until you both agree").

Key Implementation Decisions

• Decision: compile Sources/** into the benchmark target and drive OfferingsManager.updateOfferingsCache + RemoteConfigManager.refreshRemoteConfig directly, in production enqueue order.
  • Rationale: measure shipping code, not an imitation; offerings(appUserID:) adds a main-actor hop production's launch path doesn't have.
  • Rejected: hand-rolled API calls (the prior harness's approach).
• Decision: derive each simulated request's network plan from a stable key (seed, iteration, URL, attempt) instead of a shared RNG.
  • Rationale: request arrival order is scheduler-dependent; order-dependent sampling made same-seed runs disagree across processes.
• Decision: strict per-iteration warm validation (offerings all-304, config all-204 with full prefetched_blobs proof, kill-switch all-4xx, zero blob re-downloads) with nonzero exits on violation.
  • Rationale: a warm row silently measuring cold behavior is worse than no row.
• Decision: per-process disk sandbox via a gated DirectoryHelper override, with legacy Documents migrations disabled while sandboxed.
  • Rationale: container directories ignore $HOME; without this, concurrent runs corrupted each other (observed) and migrations could move/delete real ~/Documents/RevenueCat files.
• Decision (app tier): link the stock RevenueCat/RevenueCatUI products and switch variants via the ENABLE_REMOTE_CONFIG compile condition in Local.xcconfig, with a build-log check (-DENABLE_REMOTE_CONFIG present/absent) refusing mislabeled rows.
  • Rationale: measure the shipping binary; the SPM manifest caches evaluations and the xcconfig is not part of the cache key, so an unverified switch could silently build the wrong variant. Linking the benchmark Core into the app is impossible anyway (duplicate @objc(RC...) classes).
• Decision (app tier): iterate via XCUITest terminate/relaunch, with the app exposing its sample through an accessibility element; tests skip (not fail) without BENCH_API_KEY/BENCH_MODE_LABEL so bare scheme runs can never emit mislabeled rows.
• Decision (app tier): headline percentile is the SDK's content-appeared event (paywall_impression / workflows_step_started, via the @_spi(Internal) EventsListener debug API), applied from an adversarial-review finding that the wrapper's onAppear can precede visible content.
  • Rejected: keeping wrapper onAppear as the headline (kept as a secondary paywall_appeared phase).
• Decision (app tier): observe the config path (persist mark, per-blob inline/downloaded events with byte counts) through Purchases.verboseLogHandler, with the parser pinned to RemoteConfigStrings by unit tests so log-copy changes fail tests instead of silently reporting zeros.
  • Rationale: the app links the stock SDK; logs and the events SPI are the only hook-free observables. Logging overhead sits inside the measured window but is identical across variants (documented).
• Decision: UITests run under Release (review finding: Debug frameworks misrepresent shipping numbers), with the script refusing rows from non-Release compiles.
• Decision: runtime variant proof via BENCH_EXPECT_CONFIG_PATH (review finding: cached builds bypassed the build-log flag check); each measured launch reports whether the config refresh ran, and a mismatch with the row's label fails the run.

Files / Symbols Touched

• Tests/Benchmarks/SDKConfigBenchmark/Sources/ — benchmark core
  • Symbols: BenchmarkRunner, BenchmarkSDKStack, SimulatedTransportURLProtocol (+Passthrough/Plans), FixtureServer, BenchmarkPayloadFactory, RCContainerEncoder, BenchmarkMetrics, BenchmarkCommand
  • Review relevance: measurement semantics live in BenchmarkRunner.launch/collectEvents and validateWarmMeasurement.
• Sources/Networking/HTTPClient/HTTPClient.swift, Sources/Caching/DirectoryHelper.swift, Sources/Caching/DeviceCache.swift, Sources/Networking/HTTPClient/ETagManager.swift — gated hooks only
  • Review relevance: confirm every hook is inside #if SDK_CONFIG_BENCHMARK and inert in shipping builds.
• Projects/SDKConfigBenchmark/Project.swift, Workspace.swift — Tuist wiring.
• run-matrix.sh, compare.py, README.md, CONFIG_ENDPOINT_BENCHMARKS.md.
• Tests/TestingApps/SDKConfigBenchmarkApp/ — app-launch tier
  • Symbols: SDKConfigBenchmarkApp/LaunchModel (measuring app), LaunchSample/LaunchClock, AppLaunchBenchmarkUITests (relaunch loop), AppLaunchMetrics (row aggregation, unit-tested), run-app-launch.sh
  • Review relevance: phase semantics in LaunchModel, variant-verification and xcconfig restore in the script.
• Projects/SDKConfigBenchmarkApp/Project.swift — app + UITest targets; Tuist/ProjectDescriptionHelpers/Environment.swift — TUIST_BENCH_API_KEY scheme injection.

Dependencies / Config / Migrations

• New Tuist project with SDK_CONFIG_BENCHMARK + ENABLE_REMOTE_CONFIG compilation conditions; no SPM/CocoaPods changes; no public API changes.

Validation

• Commands run:
  • xcodebuild ... -scheme SDKConfigBenchmarkTests test: 86/86 pass (incl. blob accounting, log-parser + event-type + UserDefaults-suite pins against the real SDK, impression-based row statistics, and cold-config validation)
  • Live CLI config/cold run: blobs_inline_mean 2, blobs_downloaded_mean 0, max_inline_blob_bytes 70,774 (khepri inlining confirmed); simulated run attributes the mirror image (all downloaded)
  • Live app config/cold run (pre-Release-fix): config_persisted_ms_mean ~1219, same 2 inline blobs / 102KB as the CLI
  • xcodebuild ... -scheme SDKConfigBenchmark -configuration Release build: succeeds
  • swift build (plain SPM): succeeds, hooks inert
  • swiftlint: clean
  • run-matrix.sh (simulated, 18 rows) and TRANSPORT=live (4 rows): zero errors
  • Cross-process same-seed runs: loss-free rows byte-identical in counts/bytes
  • run-app-launch.sh (both variants, simulator, live network): exit 0, 4 rows, zero errors, Local.xcconfig restored byte-identically, flag verification ran on real SDK compiles
• Manual verification: live runs against project 5f07e7e3 confirmed real 304/204 revalidation and inline blob persistence (warm prefetched_blobs proof); app tier verified rendering the project's paywall on simulator.
• CI: Not run (branch pushed with this PR).

Validation Gaps

• No CI lane runs the benchmark automatically yet; no checked-in baseline or regression threshold.
• The live CDN blob-download path hasn't executed yet (khepri inlines the currently published blobs); engages automatically once more workflows are published in the project.
• App tier: not yet run on a physical device (script supports DESTINATION); app rows don't carry sdk_commit yet. Release end-to-end verified (4 rows, zero errors, both variant proofs passing) after fixing two real failures the Release run exposed: the SDK's intentional Release+Test-Store-key crash (fixed via its designed BYPASS_SIMULATED_STORE_RELEASE_CHECK opt-out) and SwiftPM's content-hash manifest cache silently reusing the previous variant's flags on cached derived data (fixed via a variant marker comment in Package.swift, removed on exit).
• Review loop: ran to the non-convergence guard (each pass kept finding narrower validity hardenings). Applied across passes: Debug config, wrapper-vs-content timing, cached-variant verification, Release/Test-Store mismatch, config-outcome gating (failed refresh cannot pass as a config row, in both tiers), warm priming validation + strict warm not_modified, failed-run row quarantine, and explicit project labels for externally supplied keys. Rejected: 1 false positive (TEST_RUNNER_ env forwarding, verified empirically); 1 partial (observer-overhead bias re-architecture skipped, microseconds vs hundreds-of-ms deltas, comment corrected instead). Plus a 4-agent simplify pass (cold wipe moved out of the measured window, SDK-copied literals pinned by tests, shared key resolution). Final standard review: clean.
• The 3MB inline budget itself is not yet exercised live: the stress project's published blobs are all far below it. Publishing a >3MB workflow would let min_downloaded_blob_bytes pin the boundary from above.
• CLI tier absolute numbers are macOS CPU; the app tier covers device timings once run on hardware.

Review Focus

• Are the four gated SDK hooks provably inert without SDK_CONFIG_BENCHMARK?
• Does BenchmarkRunner.launch faithfully mirror Purchases.updateAllCaches ordering?
• Is the warm-validation contract (304/204/blob-proof) the right definition of "warm"?
• App tier: is the Local.xcconfig rewrite + build-log flag check a sound variant switch, and is the xcconfig backup/restore safe for developers' local files?

Risks / Reviewer Notes

• Risk: the loss model approximates TCP retransmission, not real packet loss (documented; simulated rows are for A/B comparison, not absolute claims).
• Risk: live cold runs create ~25 subscriber rows per run in the stress project (namespaced benchmark-user-<uuid>-N).
• Rejected findings during the review loop: two claimed compile errors contradicted by passing builds; the hardcoded-keys objection (initially a human decision; later addressed by removing keys from source and history).

[RevenueCat-Danger-Bot]

	1 Warning
⚠️	This PR increases the size of the repo by more than 100.00 KB (increased by 244.93 KB).

Generated by 🚫 Danger