Skip to main content

Testing Strategy

X4 has multiple testing layers.

Root CLI Tests​

pnpm test
pnpm test:docker
pnpm test:complex
pnpm test:extension
pnpm test:onboarding
pnpm test:scripts

The default test suite uses temporary local Git repos and does not require real GitHub repos, model tools, cloud resources, or production credentials.

test/onboarding-proof.e2e.test.mjs is the documentation-accuracy proof for the beginner onboarding flow. It creates a temporary four-repo set with an npm app, pnpm API, npm worker, and docs/manual repo, generates and validates a registry, bootstraps the Cursor workspace, runs cross-repo tests across npm and pnpm commands, injects one env file per repo, exercises AI sandbox profile generation through a local fake OpenAI-compatible endpoint, verifies the Docker-less service-sandbox error path, and starts/stops the registry preview command.

test/complex-workspace-proof.e2e.test.mjs is the broader interaction proof. It creates a temporary five-repo mesh with a dashboard, public API, billing service, worker, and observability runbook; verifies generated registry inference for Yarn, pnpm, npm, and Bun-style repos; bootstraps from the worker repo; runs actual cross-repo contract checks across all five repos; proves the AI sandbox prompt includes contracts/jobs/runbook context without env values; injects one env file per repo; starts the dashboard preview command; and verifies cleanup planning.

test/vscode-extension.test.mjs proves the editor-facing part of onboarding: the X4 sidebar and command list exist, bootstrap uses a multi-repo source picker, workspace switching targets generated workspaces, sandbox env selection supports one env file per repo, and extension-spawned CLI arguments match the documented commands.

Unless --skip-ai is passed, the proof runner also creates a disposable four-repo workspace and runs x4 ai sandbox-plan --run against the configured live provider. That step verifies prompt redaction and stores the live model response under the proof evidence directory.

When docs checks are enabled, the proof runner builds the Docusaurus site and verifies .github/workflows/deploy-documentation.yml publishes documentation/build into X4Org/X4Org.github.io:x4/, which is served by GitHub Pages at https://x4org.github.io/x4/.

The repeatable proof runner can also run this onboarding proof on x4-testbox-standard-1 with Docker Compose. The remote step runs both the onboarding and complex workspace proofs, then boots the AI-proposed app/API/worker sandbox, waits for service health, checks printed local URLs, reads Compose logs, and destroys the sandbox.

See Onboarding Proof Matrix for the claim-by-claim evidence table and the remaining Docker-gated scope.

Repository CI​

The repo has a self-hosted GitHub Actions workflow at:

.github/workflows/x4-self-hosted-ci.yml

It runs on pull requests, pushes to main, and manual dispatch. The job uses the self-hosted runner labels x4, x4-actions, x4-dedicated-runner, linux, and x64; enables pnpm 11.1.2; installs Bubblewrap when missing; builds the CLI; exposes the local x4 binary on PATH; sets X4_TEST_TAILNET_HOST from the runner's Tailscale IPv4 address; and runs pnpm test.

Mastra Tests​

cd apps/mastra-orchestrator
npm run typecheck
npm test -- tests/shared/unit/code-assist
npm run build

The Code Assist suite covers planning, implementation, PR follow-up, testing, triage, GitHub event parsing, state store behavior, and Git runner safety.

Remote Smoke Tests​

x4 remote test --write
x4 remote test --execute
x4 control test --execute
x4 control submit test --control x4-control-1 --install eager

Use remote tests to validate control/testbox behavior and capture evidence.

Documentation Checks​

npm run docs:build

The Docusaurus build validates links, routes, and MDX syntax.