E2E Testing Strategy (Playwright)

Status: In progress (NFX-029). Phase 1 (foundation) and Phase 2 (smoke suite + PR-blocking e2e-smoke job) built and verified locally. Phase 3 (deploy-smoke) is built and wired into deploy.yml/deploy-uat.yml/deploy-prod.yml, but not yet verified against a real deploy — a pre-existing, unrelated bug (NFX-045: devv/uatt were building against Production Supabase due to missing GitHub Environment variables) surfaced while building this phase and is deliberately being fixed via the next real deploy rather than a throwaway one — see the note at the end of the Phase 3 section below. Phases 4–5 (regression/visual, further governance) are the execution recipe below.

Why this exists

The platform's Vitest suite is mature (a 4-layer model: pure computation → hooks/services → component smoke → page integration) but runs entirely in JSDOM — no test executes a real browser, a real Vite bundle, or a real Supabase network call. An entire failure class is therefore structurally invisible: broken imports (the RiEdit3Line incident — 298/298 unit tests green, production broken on first load), theme/contrast breaks, canvas/export bugs, and frontend↔backend contract drift. The only post-deploy check today is a bare HTTP-status curl that would pass on a blank white-screen deploy.

Playwright is added strictly on top of Vitest — it does not reduce Vitest's scope. It covers only what a real browser, a real bundle, or a real network call can catch.

Testing layers & responsibilities

Layer	Tool	Scope	Runs on	Gate
Unit — pure computation	Vitest (JSDOM)	Deterministic fns	Every PR	Blocking
Unit — hooks/services	Vitest (JSDOM), mocked Supabase	RPC/EF call shape, cache, errors	Every PR	Blocking
Component smoke	Vitest + Testing Library	Real component import, icon/export validity	Every PR	Blocking
Page integration	Vitest + Testing Library	Real page + sub-components, mocked hooks	Every PR	Blocking
SQL / RPC	pgTAP (supabase test db)	DB function contracts	Every PR (local)	Blocking
Edge Function	Deno test	EF logic in isolation	Every PR (local)	Blocking
API contract	Playwright api + @supabase/supabase-js	The frontend↔backend seam	Every PR	Blocking
E2E smoke	Playwright, chromium, ephemeral stack	5–8 critical journeys	Every PR	Blocking
Accessibility	@axe-core/playwright	WCAG 2.1 AA sweep	PR (subset) + nightly (full)	Blocking (subset)
Deploy smoke	Playwright, chromium, real URL	Real artifact renders	Every deploy	Blocking
E2E regression	Playwright, 3 browsers, ephemeral stack	Full journey coverage	Nightly / push	Visibility → blocking
Visual regression	Playwright screenshots (pinned image)	Design-system + key states	Nightly	Visibility → blocking
Mobile viewport	Playwright device emulation	2–3 journeys	Nightly	Visibility

Testing architecture

Two-tier environment strategy

Tier	Target	Used for	Writes
Ephemeral local stack	supabase start in the runner (Docker, torn down at job end)	All write journeys + full regression	Yes — fully isolated
Real deployed target	devv/uatt/nefoxx.com, post-deploy	Read-only deploy-smoke only	No — guard-enforced

Why not "tagged test data + cleanup" on the shared backend? It's a side-effect problem, not a data-hygiene one: cron/webhook consumers react to any matching row regardless of a tag, and /shared/plan/:token is a public, crawlable page — a test-created share token is briefly live on the real domain. The ephemeral stack never creates that exposure window.

The critical safety finding (read before writing any write-journey spec)

src/lib/supabaseClient.js hardcodes a real production fallback:

const PRIMARY_URL  = import.meta.env.VITE_SUPABASE_URL          ?? 'https://api.nefoxx.com';
const FALLBACK_URL = import.meta.env.VITE_SUPABASE_URL_FALLBACK ?? 'https://lyaldbfgdhxpllgxbcxp.supabase.co';

and rpc()/functions.invoke() silently retry against FALLBACK_URL on any network error. Combined with the fact that dev/UAT/prod currently share one Supabase project (NFX-027), an E2E write against a real target — or a fallback retry during a cold local stack — can hit the live production database. Mitigations (all in place):

• Build with explicit VITE_SUPABASE_URL, VITE_SUPABASE_ANON_KEY, and VITE_SUPABASE_URL_FALLBACK overrides pointing at the local stack. Vite's loadEnv prioritises process.env over the git-tracked .env.local (verified), so these win. Both URL vars point at the same local stack, so even a fallback retry stays local.
• reuseExistingServer: false + a dedicated port (3200, not the dev server's 3000). A stale reused server built from .env.local was the original cause of an escape to api.nefoxx.com — never reuse.
• e2e/support/mutationGuard.js — endpoint/function-name allowlist, deny-by-default, for the read-only deploy project. Not an HTTP-method check: every Supabase RPC/EF call is a POST, even pure reads, so method-based blocking would be both wrong and unsafe.
• The Phase 1 spec asserts zero requests reach *.supabase.co / *.nefoxx.com and every Supabase call targets 127.0.0.1. It also asserts at least one Supabase call actually reached the network — a build where every call is blocked before dispatch (see the CSP finding directly below) would otherwise pass this assertion vacuously (zero escaped + zero observed = a false "clean" result).

A second, independent blocker found building Phase 2 (not a security escape risk like the fallback retry above, but every bit as fatal to the ephemeral-stack model): index.html's CSP ships connect-src 'self' https: wss: in every real environment (dev/UAT/prod all serve Supabase over HTTPS) — correctly strict. The ephemeral stack talks to 127.0.0.1:54321 over plain HTTP, which that policy silently kills at the browser level before the request ever reaches the network layer (TypeError: Failed to fetch, no CORS error, no server log — nothing to grep for except the console's CSP violation line). Fixed via a build-time-only relax (vite.config.js's relaxCspForE2eLocalStack), gated on E2E_LOCAL_BUILD=true (set only by playwright.config.js's local webServer.env) — adds http://127.0.0.1:* ws://127.0.0.1:* to connect-src only for the ephemeral-stack build; every other build (including smoke-deploy, which targets a real HTTPS deployment) is untouched.

Test database: schema snapshot, not migration replay

The originally-planned "replay all migrations in the ephemeral stack" mechanism does not work: 149/210 migrations collide on Supabase's version parser (YYYYMMDD_NNN_ → same YYYYMMDD version), so supabase db reset/supabase start abort with duplicate key … schema_migrations_pkey (NFX-030). The pivot — the industry-standard test-DB pattern anyway — is a committed schema snapshot:

File	Purpose
supabase/schema.sql	supabase db dump --schema public from the real project (read-only). The authoritative schema.
supabase/e2e-extensions.sql	Extensions the snapshot needs but the public dump omits (pg_trgm in public, etc.). Loaded first.
supabase/e2e-auth-hooks.sql	The two auth.users triggers (on_auth_user_created → handle_new_user) not captured by a public-only dump.
supabase/seed.sql	Deterministic test users (see below). Loaded last.
supabase/config.toml	Minimal E2E-scoped config (ports, Type-B verify_jwt=false).
tools/e2e-db-setup.mjs	Loads all of the above (in order) into the running stack via docker exec. npm run e2e:db:setup.

Load order: e2e-extensions.sql → schema.sql → e2e-auth-hooks.sql → seed.sql.

Seed users

profiles rows are never inserted directly — handle_new_user() fires on auth.users INSERT and creates them. Seeding is two-phase (insert auth.users + auth.identities, then UPDATE profiles). Three fixed users (password E2eTest!2026, local-only): a regular trader (onboarding_completed=true), an admin (is_admin=true), and a non-admin "target" (the admin-user-role EF has a self-revocation lockout, so its role-change journey needs a target other than the admin).

Write-journey isolation

Static seed users are fine for read-only specs. Write journeys must provision their own user via the dynamicUser fixture (e2e/support/fixtures.js, supabase.auth.admin.createUser) — sharing one seeded account across concurrent write specs causes order-dependent flakiness ("0 plans" vs. "creating a plan"). Assertions target the specific entity created, never an absolute count. This stack uses the new API key system: the admin API accepts the sb_secret_ key, not the legacy JWT service_role key.

CI/CD integration (Phases 2–5)

• e2e-smoke (.github/workflows/ci.yml) — sibling of sonar-scan (needs: build-and-test), parallel (separate runner VM). supabase start -x studio,analytics,vector,imgproxy → capture the local stack's ANON_KEY/SECRET_KEY via supabase status -o json → e2e:db:setup → serve Edge Functions (see below) → playwright test --project=smoke-ephemeral --project=api --project=a11y --workers=2. Blocking, timeout-minutes: 20. Added to deploy-dev's needs: in the same change (a blocking check that isn't in a deploy job's needs: only gates the merge, not the deploy).
• supabase start does NOT serve Edge Functions. Found the hard way building this job: the stack comes up, migrations/seed load fine, but every EF call (trade-planner-save, -lifecycle, -share, …) 503s at the Kong gateway — there's no edge_runtime upstream for it to proxy to. Edge Functions are a separate, long-running process: supabase functions serve (reads supabase/config.toml's [functions.*] verify_jwt overrides, same as a real deploy). The CI job backgrounds it (nohup ... &) and polls a real function endpoint until it responds before running any spec.
• Runs on the bare ubuntu-latest runner, not inside a pinned Playwright Docker image. The original plan called for mcr.microsoft.com/playwright:vX.Y.Z-jammy (browsers pre-installed, version-pinned). That conflicts with this job's own supabase start, which needs to drive Docker itself (Docker-in-Docker is exactly the complexity backend-cicd-plan.md already avoided for the db-migrations-pgtap job). Matching that job's already-working pattern instead: bare runner + npx playwright install --with-deps chromium (the same command package.json's e2e:install script already wraps).
• Deploy-smoke (Phase 3, shipped) replaces deploy.yml's bare curl with a real Playwright smoke-deploy run (e2e/smoke/deploy/) — shared by dev/uat/prod, so all three get real functional verification, not just an HTTP-200 check. The curl loop stays as a first, cheap readiness poll ("Wait for deployment to propagate") before Playwright boots a browser against the domain. Read-only, mutation-guard-enforced (e2e/smoke/deploy/fixtures.js installs the guard via a page fixture override — structurally impossible for a spec author to skip). Failure does not auto-rollback — see Deployment §5 Rolling back a deploy for the manual runbook (Cloudflare Pages dashboard rollback, or re-running the deploy workflow against a known-good commit), consistent with the repo's "prod changes are deliberate" philosophy.
• e2e-regression (Phase 4, not yet built) — nightly/develop, non-blocking until 10 consecutive green scheduled runs (an explicit graduation criterion, unlike NFX-019's open-ended "for now").
• verify-ci in deploy-uat.yml/deploy-prod.yml now requires "E2E Smoke (Playwright)" alongside "Lint, Test, Build" and "SonarQube Scan (baseline gate)" in its check-run select(...) filter — shipped with Phase 3.
• Not yet verified against a real deploy — see NFX-045: the dev/uat GitHub Environment variables were found to be missing/incorrect during Phase 3 spec authoring (both devv.nefoxx.com and uatt.nefoxx.com were building against Production Supabase, confirmed via bundle inspection), which is exactly the kind of misconfiguration this phase exists to eventually catch functionally — but the very first attempted run hit it as an environment problem, not a spec problem. The GitHub Environment variables have since been corrected, but a fresh deploy is needed to bake them in (Vite resolves import.meta.env.VITE_* at build time, not runtime) — deliberately deferred until this phase's own work is complete, so the next real deploy serves as both the isolation fix and the first genuine exercise of this gate.
• Artifacts — playwright-report/ + test-results/ (HTML report + traces, trace: on-first-retry) via actions/upload-artifact, retention-days: 14. Reporting mirrors test-summary.cjs in .github/scripts/e2e-summary.cjs (parses Playwright's JSON reporter, a different shape than Vitest's).

Local developer workflow

npm run supabase:start        # start the stack (once)
npx supabase functions serve  # separate, long-running -- Edge Functions are NOT served by `start`
npm run e2e:db:setup          # load schema snapshot + seed
npm run e2e:smoke             # build (with overrides) + run smoke against the local stack
npm run e2e:report            # open the HTML report

e2e:codegen helps author specs. The E2E server uses port 3200 and always rebuilds — never reuses a possibly-stale server. Set LOCAL_SUPABASE_ANON_KEY and LOCAL_SUPABASE_SECRET_KEY from supabase status -o json.

Governance

• Flaky quarantine — a @flaky tag + --grep-invert @flaky on the blocking run; quarantined specs run non-blocking and get a tracker item.
• Ownership mirrors tiers — e2e/smoke/{users,admin,public}/….
• e2e/** is in sonar.sources — fixtures/page-objects get the same static-analysis bar as src/.
• Version pinning — @playwright/test pinned exactly in package.json; the CI job installs the matching browser via playwright install rather than a separately-versioned Docker image (see the "runs on the bare runner" note above), so there's a single version to keep in sync, not two.
• Visual baselines — updated via --update-snapshots, committed in the same PR as the UI change (never a separate "fix the tests" commit).
• a11y PR-blocking subset is critical-impact only — same visibility-first rollout pattern as the Sonar baseline gate, E2E regression, and backend-ci.yml's edge-functions job: a brand-new gate ships scoped to what's actually clean today. Standing this up immediately found one real critical bug (/auth's password show/hide toggle had no accessible name — fixed, aria-label added) and one real platform-wide serious issue (text-orange-500 on white fails WCAG AA contrast, 2.8:1 vs. the required 4.5:1 — a design-system-wide fix, out of scope for standing up this gate, tracked as NFX-038). The full sweep (all impact levels) runs nightly via the dedicated a11y project once Phase 4 ships, non-blocking, so serious findings stay visible while NFX-038 is worked.

Phased rollout

Phase	Scope	State
1 — Foundation	config, env-override safety, mutation guard, dynamic-user fixture, schema-snapshot DB model, throwaway proof spec	Done, verified locally
2 — Smoke + PR gate	5 journeys (auth ×3, trade-planner create/save/close, trade-planner share) + api contract project + a11y (critical-only) subset, e2e-smoke blocking in ci.yml, e2e-summary.cjs, deploy-dev needs fix, sonar.sources includes e2e/**	Done, verified locally
3 — Deploy-smoke	replace deploy.yml curl, extend verify-ci, rollback runbook	Built; live-deploy verification pending (NFX-045)
4 — Regression/visual/mobile	3 browsers + visual + mobile, nightly, graduate after 10 green	Pending
5 — Docs/governance	this guide, tracker, flaky process	In progress

What Phase 2 found (worth knowing before extending this suite)

Standing up real write journeys against the ephemeral stack — not just the Phase 1 proof spec — surfaced issues Phase 1's narrower scope couldn't have caught:

• The CSP connect-src blocker and the supabase functions serve requirement (both above) — structural gaps in the local-stack model itself, now fixed/documented.
• A real product bug: /shared/plan/:token (public trade-plan share links) was missing from ProtectedRoute.jsx's publicPaths allowlist, so ProtectedRouteWrapper's "Tier 3" 5-second timer fired an auth-modal prompt over the page for anonymous visitors to a public share link — a link that is supposed to need zero authentication. Fixed (added to publicPaths); the share journey spec now covers this regression permanently (a genuinely logged-out browser context, not just a new tab in the same authenticated context — the two are not equivalent for this kind of check).
• Two real a11y bugs (both critical): the password show/hide toggle button on /auth had no accessible name (fixed, aria-label); and every dropdown-style nav item in the global header (NavItem.jsx) had its aria-haspopup/aria-expanded cloned by Radix's DropdownMenuTrigger asChild onto an intermediate motion.div instead of the real <Button>, so assistive tech was reading expand/ collapse state off a non-interactive element — fixed by moving the animation wrapper outside the trigger so asChild clones onto the button directly (NFX-041).
• A real, platform-wide a11y issue (serious, deferred): text-orange-500 on white — used broadly as a brand accent — measures 2.8:1 contrast, below WCAG AA's 4.5:1 minimum. Tracked as NFX-038, out of scope for this pass (a design-system-wide color change, not a Playwright-suite concern).
• A pre-existing backend defect, found incidentally while verifying the market-mood smoke spec: vw_momentum_leaders 500s on the ephemeral stack's schema snapshot. Tracked as NFX-039 — not blocking (the smoke spec asserts no unexpected 5xx from calls the page itself makes on the happy path; this view isn't queried by the overview page's own RPCs, it surfaced via a broader diagnostic listener), but a real gap worth someone's attention.
• A local-machine-only flakiness pattern, not a defect: on this Windows dev machine, Playwright's own process can hang for several minutes after tests finish while tearing down the vite build && vite preview child chain (a known category of Windows child-process-tree quirk), and running more than one Chromium instance in parallel against a single-container local Supabase stack meaningfully increases flakiness (ERR_SOCKET_NOT_CONNECTED, connection-pool contention matching the exact profile CLAUDE.md's Composite RPC Pattern section already documents for the admin dashboard). Neither affects CI (native Linux Docker, no competing local processes) — noted here so a future contributor debugging local E2E flakiness doesn't mistake it for a code bug. Locally, prefer --workers=1 and expect to occasionally need to free port 3200 (netstat -ano | grep 3200) before a re-run.