test: add commands tests infrastructure - #1083
Open
pdesoyres-cc wants to merge 16 commits into
Open
Conversation
|
🔎 A preview has been automatically published! If you created the alias to the preview script, you can run this command to download and install this preview: clever-preview update experiment-command-testsYou can also run it from your local repository: ./scripts/preview.js update experiment-command-tests
This preview will be deleted once this PR is closed. |
pdesoyres-cc
force-pushed
the
experiment/command-tests
branch
3 times, most recently
from
April 21, 2026 14:47
689c72a to
bee950b
Compare
pdesoyres-cc
force-pushed
the
experiment/command-tests
branch
from
May 7, 2026 15:43
b7736ed to
a0b45b7
Compare
pdesoyres-cc
force-pushed
the
experiment/command-tests
branch
5 times, most recently
from
May 21, 2026 19:41
809f465 to
93b622c
Compare
|
📊 A code coverage report has been automatically published!
This report will be deleted once this PR is closed. |
pdesoyres-cc
force-pushed
the
experiment/command-tests
branch
2 times, most recently
from
May 22, 2026 15:16
b34c1f6 to
dc4f88f
Compare
pdesoyres-cc
force-pushed
the
experiment/command-tests
branch
from
May 22, 2026 15:42
dc4f88f to
8640490
Compare
pdesoyres-cc
marked this pull request as ready for review
May 22, 2026 15:45
pdesoyres-cc
force-pushed
the
experiment/command-tests
branch
from
May 22, 2026 15:56
8640490 to
a7e7b2a
Compare
Align login with the rest of the CLI by routing the browser launch through the shared `openBrowser` helper instead of importing `open` directly. This keeps the "🌐 Opening …" prefix and debug logging consistent across commands and removes a duplicated integration point with the `open` package.
The success log sat after the try/catch, so it printed even when `addSshKey` threw — the catch block returned silently for any error other than id 505. Move the log inside the try and rethrow unrecognized errors so failures surface to the user.
The check for `invalid-credential` never matched because `responseBody` lives on the error itself, not under `cause` — same access pattern used elsewhere (ssh-keys, emails, k8s). Without this fix the generic error message leaked through instead of the friendly "Invalid credentials" hint.
When ssh fails before printing the start marker (auth failure, connection refused, bad key permissions, …) the noise filter swallowed every byte and the user got a silent exit-255. Buffer stderr too and flush both buffers on non-zero exit when the marker was never reached, so the actual ssh error reaches the terminal.
The plugin always handed isomorphic-git an https.Agent, so requests against plain HTTP servers couldn't reuse the keep-alive agent and effectively bypassed our timeout config. Switching on the URL scheme unblocks the deploy command tests, which exercise the git push path against a local git-http-mock-server served over plain HTTP.
NO_COLOR alone doesn't override FORCE_COLOR — libraries like chalk give FORCE_COLOR precedence, so colors leaked through whenever the caller's environment set both. Clear FORCE_COLOR on the no-color path so the suppression actually takes effect.
Move IDS_CACHE_FILE into the shared config schema alongside CONFIGURATION_FILE and EXPERIMENTAL_FEATURES_FILE, and convert their defaults to lazy factories. Callers in cache.js and features.js now read the path from config instead of resolving it at import time. This lets tests point each file at a temporary location through env vars without having to patch internals, which the upcoming command test suites rely on.
The marker-based stdout filtering used to drive single-command SSH sessions lived inline in the command handler, which made it impossible to cover with unit tests — every path went through a real ssh subprocess and `process.stdout`/`process.stderr`. Lift the logic into `src/models/ssh-marker-protocol.js` with the child process, marker, and output streams injected, so the protocol (marker handshake, pre-marker buffering, failure-mode flush) can be exercised in isolation. No behavior change for the ssh command itself; the handler now just spawns ssh and hands the subprocess to `runMarkerProtocol`.
Picks up the upstream fix "Ignore stdio on Windows when not waiting for process", which is a prerequisite for the incoming Windows unit tests — the old behavior interferes with detached child processes spawned by the test runner.
`validate` calls `typecheck`; defining the dependency before its caller makes the script block easier to scan.
Introduce a CLI runner harness (test/cli-runner.ts) plus mocks (filesystem, git server, redis) and fixtures (app config, profile, IDs, errors) so command modules can be exercised end-to-end without hitting real services. Extend tsconfig to type-check the suite. Adds first-pass coverage for deploy, login, logs, kv, env import, ssh, ssh-keys, tokens, emails, notify-email, profile switch, link, curl, and database backups.
Mainly a Windows optimization: NTFS is very slow at writing the thousands of tiny files `npm ci` produces, so each run paid a heavy install cost. `cache: 'npm'` only caches the npm download cache, so the slow `node_modules` materialization still happened every time. Cache `node_modules` directly, keyed on Node version and `package-lock.json`, so a hit short-circuits install entirely. On miss, run `npm ci` with `--prefer-offline --no-audit --no-fund` to trim the install further.
…/win Pair the new command-level test harness with a CI job that exercises the actual built binary on every supported OS rather than the source tree. The harness reads `CLEVER_BIN`, so packaging regressions (missing files, broken bundling) surface in tests instead of only at release smoke-check time.
The upcoming code-coverage workflow needs to push the full HTML report (hundreds of small files) to Cellar. Issuing them all via Promise.all hits the per-host connection ceiling and can throttle or fail under load, so route both bulk upload and bulk delete through a shared worker pool capped at 8 in flight.
Builds on the test harness work to give reviewers visibility into coverage changes: each PR gets its own report uploaded under a per-branch prefix and a sticky comment summarizing the totals, while closed PRs and pushes to master keep the bucket tidy.
pdesoyres-cc
force-pushed
the
experiment/command-tests
branch
from
July 28, 2026 15:07
a7e7b2a to
9d92711
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The CLI ships without automated command-level coverage — every release leans on manual
smoke testing, so per-command regressions only
surface once users hit them. This branch lays down the infrastructure to drive the CLI
end-to-end against mocked API / KV / Git backends, seeds the first wave of command
tests, and wires both the suite and a per-PR coverage report into CI.
Changes
Test harness
runCliharness (test/cli-runner.ts) with PTY support, interactiveprompt scripting, and per-interaction ANSI-stripped output segments.
test/fixtures/*,test/mocks/*covering filesystem,Git server, Redis, in-process API doublure).
CLEVER_BINswitch so the same suite runs against the source tree locally and againstthe packaged binary in CI — catching bundling / pkg regressions a source-tree-only
suite would miss.
test/tests-checklist.mddocuments the conventions so new command tests stayconsistent.
Custom test reporter
Node's bundled reporters don't fit a large command-level suite well: TAP buffers output
and the
specreporter prints noisy per-file summaries. A small custom reporter attest/reporter/reporter.ts, wired into thetestnpm script, gives:progress is visible while a long run is in flight,
and full stack trace, so diagnostics aren't buried mid-stream.
First-wave command coverage
Tests added for
emails,env import,login,logs,ssh,ssh-keys add,tokens create,kv,link,deploy,curl,database backups download,notify-email, andprofile switch.Fixes uncovered while writing the tests
ssh: surface pre-marker errors instead of exiting silently.ssh-keys: don't report success when the add call fails; remove a leftover debug log.tokens: read the error code fromresponseBodydirectly.cli: clearFORCE_COLORwhen disabling colors.git: pick the agent matching the URL scheme.Refactors that fell out of the test work
ssh: extract the marker protocol into its own module.config: centralize config file paths in the schema.login: use the sharedopenBrowserhelper.CI
tests.ymlworkflow builds + packages the CLI, then runs the suite against thatbinary on Linux, macOS and Windows.
setup-nodeaction cachesnode_modulesso the test job skipsnpm cion hit.comment with the link on the PR.
scripts/lib/cellar-client.jsgains a bounded-concurrencyuploadFileshelper usedby the coverage publisher.
How to review
test/cli-runner.tsandtest/cli-hooks.ts— these are the load-bearingpieces every command test builds on.
test/tests-checklist.mdfor the conventions, then sample one of the per-commandtest files (e.g.
src/commands/database/database.backups.download.command.test.ts)to see them applied.
npm testfor the source tree, ornpm run build && CLEVER_BIN=<built-bin> npm testfor the packaged path.remote — the sticky
Code coveragecomment is where the report link will appear.