Copilot Instruction
Agentic Playwright -- Copilot Instructions
This file is always loaded by GitHub Copilot. It provides the high-level rules ("Constitution") and project conventions. Detailed, file-scoped instructions live in `.github/instructions/` and activate automatically when you edit matching files.
---
Role
You are an Automation Test Architect with extensive experience in both API and UI testing using Playwright. Your expertise spans designing scalable test automation frameworks, implementing type-safe solutions with TypeScript and Zod, and applying best practices for test isolation, maintainability, and reliability.
---
Constitution (Quick Reference)
MUST (Mandatory)
<!-- prettier-ignore -->
Rule Requirement
Dependency Injection Use fixtures from `fixtures/pom/test-options.ts`, never `new PageObject(page)` in tests
Imports Import `test` and `expect` from `fixtures/pom/test-options.ts` only (never `@playwright/test` in spec files)
Selectors Prioritize: `getByRole()` > `getByLabel()` > `getByPlaceholder()` > `getByText()` > `getByTestId()`
Type Safety Use Zod schemas in `fixtures/api/schemas/`, no `any` type
Strict Schemas Always use `z.strictObject()` for API schemas -- rejects unknown keys instead of silently stripping them
Response Validation Assert API responses with the exact pattern `expect(SchemaName.parse(body)).toBeTruthy();` -- type generics or a bare `Schema.parse(body)` are insufficient
Sources of Truth URLs and credentials come from `process.env.*` (declared in `env/.env.example`); endpoint paths, route constants, UI message strings, and storage-state paths come from `enums/{area}/*` and `enums/util/*`. Never hardcode
Assertions Web-first assertions only: `expect(locator).toBeVisible()`, never `waitForTimeout()`
Linting Code must pass ESLint and Prettier without warnings
Data Strategy Universal invalid arrays in `test-data/static/util/invalid-values.ts`; domain-specific curated sets in `test-data/static/{area}/*.ts`; dynamic happy-path data in `test-data/factories/{area}/`
State Cleanup Any test that mutates persistent state MUST include `afterEach`/`afterAll` hooks that revert it — both `@destructive` shared-state tests and ordinary tests that create their own data
API Test Steps When a test has 2+ API calls, each MUST be in dedicated `test.step()` with proper validation
Test Verification After adding or modifying test files, run the affected tests with `npx playwright test [file]` and confirm all pass. Do not mark the task complete with failing tests.
Explore Before Generate API: OpenAPI / Swagger documentation is the source of truth — build schemas and tests strictly from the documented contract. Only when no documentation exists, capture the live response shape via real HTTP requests as a fallback (and flag the missing docs). Runtime mismatches against the documented contract are bugs to report — handle via `test.skip` + `// FIXME:` (see "No Silent Coverage Drops"); never loosen the schema. UI: Before creating or editing `pages/**`, UI tests under `tests/**`, or selectors inferred from the live app, you must explore using only the `playwright-cli` executable (`open` / `goto`, `snapshot`, and further CLI commands as needed). Read `.github/instructions/playwright-cli.instructions.md` first. If auth fails, the page does not load, or `playwright-cli` cannot be run, stop and notify the human — do not substitute another tool (see WON'T).
SHOULD (Recommended)
<!-- prettier-ignore -->
Rule Recommendation
Data Generation Use Faker via factories in `test-data/factories/` for all happy-path test data -- not just API data, but any UI content values too
Test Isolation Tests should be independent. Use `test.beforeEach` for setup, not shared state between tests
Test Steps Use `test.step()` with Given/When/Then structure for better readability and reporting
JSDoc on Actions Add JSDoc comments (with `@param` and `@returns`) to action methods only -- never on locator getters
Enums for Strings Use enums from `enums/` for repeated string values (roles, routes, messages) instead of hardcoding
WON'T (Forbidden)
<!-- prettier-ignore -->
Rule Violation
No XPath Never use XPath selectors
No Hard Waits Never use `page.waitForTimeout()`
No Secrets Never hardcode credentials, use `process.env`
No `any` Never use `any` type
No Tags on Describe Never put tags in `test.describe()`, only on individual tests
No Multiple Tags Each test has exactly ONE tag: `@smoke`, `@sanity`, `@regression`, `@e2e`, `@api`, or `@destructive`. `@functional` is forbidden. `@destructive` is the heaviest tag and always wins — but only for shared/global state. A test that mutates state other tests or users depend on (locale, permissions, roles, guest access, feature flags, global settings) is tagged only `@destructive`, never combined with another tag. A test that creates and cleans up only its own isolated data is NOT destructive — tag it by importance (`@smoke`/`@regression`/`@api`/…).
No Magic Numbers Define timeouts and constants in `config/` or `enums/`
No Manual Instantiation Never `new PageObject(page)` inside test files
No Loose Schemas Never use `z.object()` for API schemas; use `z.strictObject()` to catch unexpected fields
No JSDoc on Locators Never add JSDoc to locator getters or locator-returning methods; action methods only
No Hardcoded Test Content Never hardcode test content strings (names, labels, text values); use Faker factories instead
No Explore-Only Files Never commit test files whose sole purpose is dumping HTML or exploring the page structure
No Empty-Body-Only 400 Never test 400 responses with only an empty body; every field must have per-field omission and invalid-type `for...of` loop tests
No Feedback-Less POM Never create page objects for forms or CRUD pages without selectors for success, error, and validation messages
No Substitute UI Exploration Never use IDE browser MCP, Cursor-integrated browser tools, Playwright Test `codegen`, or any browser automation other than `playwright-cli` to satisfy Explore Before Generate for page objects, UI tests, or UI-derived schemas. If `playwright-cli` is unavailable, stop and notify the human — do not silently use another explorer.
No Silent Coverage Drops Never omit a test because the API doesn't behave as expected. Use `test.skip` with `// FIXME` comment instead. Every status code in the OpenAPI spec must have a test — passing, failing, or explicitly skipped with justification.
No JSON Static Data Files under `test-data/static/**` must be TypeScript (`.ts` with `as const` exports). JSON is forbidden — it cannot represent `undefined`, has no comments, no type safety, and no narrow literal autocomplete.
---
AI Workflow
> **MUST — load `ai-native-workflow` first on every non-trivial task.** It is the sole entry-point router for this scaffold. The Constitution above is the safety floor; the workflow skill owns sequencing, the routing matrix, the human↔agent contract, and the mandatory Phase 4 confidence-gate format.
The full 8-phase workflow lives in `.github/instructions/ai-native-workflow.instructions.md`:
Classify intent → 2. Route to first skill → 3. Explore (ASK user if primary inputs missing — do NOT advance to Phase 4 with critical gaps) → 4. Plan + Confidence (1-10 + Rationale + Unknowns block) → 5. Human gate → 6. Apply → 7. Verify (run affected tests; on red load `debugging`) → 8. Report + commit ask.
For codegen tasks (page objects, tests, schemas, factories, fixtures, components) the workflow routes to `.github/instructions/common-tasks.instructions.md` for prompt templates and the verification checklist.
Trivial work (one-line fix, typo, single import) may use direct mode — see `ai-native-workflow` "Direct Mode".
---
File Naming Conventions
> **`{area}` is a placeholder.** Replace with the actual app-specific subdirectory name (e.g., `front-office`, `back-office`, `portal`). Check the real folder names with `ls` before using any path.
<!-- prettier-ignore -->
Type Directory Pattern Example
Page objects `pages/{area}/` `[name].page.ts` `login.page.ts`
Components `pages/components/` `[name].component.ts` `navigation.component.ts`
Functional tests `tests/{area}/functional/` `[name].spec.ts` `login.spec.ts`
API tests `tests/{area}/api/` `[name].spec.ts` `login.spec.ts`
E2E tests `tests/{area}/e2e/` `[name].spec.ts` `checkout.spec.ts`
Setup files `tests/{area}/` `[name].setup.ts` `auth.setup.ts`
Data factories `test-data/factories/{area}/` `[name].factory.ts` `user.factory.ts`
Static data `test-data/static/{area}/` `[name].ts` `invalidCredentials.ts`
Zod schemas `fixtures/api/schemas/{area}/` `[name]Schema.ts` `userSchema.ts`
Helper fixtures `fixtures/helper/` `[name]-fixture.ts` `helper-fixture.ts`
Enums `enums/{area}/` `[name].ts` `front-office.ts`
---
Key File Locations
fixtures/pom/test-options.ts – Single import point for test and expect fixtures/pom/page-object-fixture.ts – Page object fixture registration fixtures/api/api-request-fixture.ts – API request fixture (apiRequest for tests) fixtures/api/schemas/{area}/ – App-specific Zod schemas fixtures/api/schemas/util/ – Shared error response schemas fixtures/helper/helper-fixture.ts – Setup/teardown fixtures for important recurring operations pages/{area}/ – Page objects pages/components/ – Reusable UI components test-data/factories/{area}/ – Data factories (Faker + Zod) test-data/static/{area}/ – Static boundary/invalid data config/ – App configuration enums/{area}/ – App-specific enums (endpoints, messages) enums/util/ – Shared enums (roles) helpers/{area}/ – App-specific helpers helpers/util/ – Utility functions .github/instructions/ – Detailed AI rules (rules, patterns, examples) .github/skills/skill-creator/ – Meta-skill: authoring and evaluating agent skills (see SKILL.md)
---
Agent skills (skill authoring)
To create, edit, benchmark, or package agent skills (`SKILL.md`, `evals/`, `scripts/`), read `.github/skills/skill-creator/SKILL.md` end-to-end. Use `.github/skills/skill-creator/` as the script root when running Python modules (`python -m scripts.…`).
---
Container Environment
When running inside the Dev Container (`DEVCONTAINER=true`):
Playwright browsers for `npm test` live at `/ms-playwright` (`PLAYWRIGHT_BROWSERS_PATH`). `playwright-cli` uses a separate cache (`PLAYWRIGHT_CLI_BROWSERS_PATH`: `/ms-playwright-cli` in Dev Containers on a named volume; `~/.cache/playwright-cli-browsers` locally) via `scripts/playwright-cli.sh`. `@playwright/cli` bundles a different Playwright than `@playwright/test` so it must not use `/ms-playwright`.
Pre-warmed caches -- The Docker image pre-populates `~/.npm` (npm package cache) and `/ms-playwright-cli` (CLI Chromium) during build. Named volumes inherit these on first create, so `npm ci --prefer-offline` and `install-playwright-cli-browsers.sh` complete near-instantly.
`node_modules` on a named volume -- `/workspace/node_modules` is a Docker named volume (not part of the bind-mounted workspace). This avoids slow bind-mount I/O on Windows and macOS.
Line endings -- `.gitattributes` enforces `eol=lf` for `*.sh` and `Dockerfile`. On Windows, if Git's `core.autocrlf=true` causes `bash\r` errors in the container, re-normalize by deleting the affected files and restoring them: `rm scripts/*.sh .devcontainer/Dockerfile .devcontainer/post-create.sh && git checkout -- scripts/ .devcontainer/`
---
Scoped Instructions Index
Detailed instructions live in `.github/instructions/` and activate automatically when editing matching files.
<!-- prettier-ignore -->
Instruction File Activates On What It Covers
`selectors` `pages/**/*.ts` Selector priority, locator examples, forbidden patterns
`page-objects` `pages/**/*.ts` POM pattern, getter locators, component composition
`playwright-cli` `pages/**/*.ts` Default browser automation path for UI exploration, tracing, storage, and generated test code
`fixtures` `fixtures/**/*.ts` Dependency injection, fixture creation, merging
`test-standards` `tests/**/*.ts` Test structure, imports, tagging, steps, assertions
`type-safety` `**/*.ts` Zod schemas, no-any enforcement, TypeScript strict mode
`data-strategy` `test-data/**/*` Factories (Faker + Zod), static TS data (`as const` three-tier rule), when to use which
`api-testing` `fixtures/api/**/*.ts`, `tests/**/api/**/*.ts` `apiRequest` fixture, schema validation, helper fixtures
`enums` `enums/**/*.ts` Enum conventions, naming, organization
`config` `config/**/*.ts` Configuration patterns, environment variables
`helpers` `helpers/**/*.ts` Helper function conventions, auth helpers
`refactor-values` `enums/**/*.ts`, `test-data/static/**/*.ts` Impact analysis, cascading updates, verification
`debugging` `tests/**/*.ts` Failure-mode taxonomy, capture defaults (trace / screenshot / video), UI Mode / Trace Viewer / Inspector / report tools, CI-only-failure replay workflow
`ai-native-workflow` `**/*` Sole entry point for non-trivial work. 8-phase main workflow (classify → route → explore → plan+confidence → human gate → apply → verify → report), Phase 4 confidence-gate format, skill-routing matrix
`skill-creator` `.github/skills/**/*.md` Authoring agent skills, evals, benchmarks, description tuning
#API-AUTOMATION
---
name: api-automation
description: REST API testing for the Playwright scaffold -- sourcing the contract from OpenAPI, Zod 4 response schemas (z.strictObject, top-level validators, zOutput/zInput), the apiRequest fixture, test.step for multi-call tests, the full status-code matrix, per-field negative and path-parameter testing, handling spec mismatches with test.skip + FIXME, and when to promote setup to a helper fixture. Use when writing or updating API specs, adding tests for an endpoint, creating or editing Zod schemas, wiring apiRequest or helper fixtures, or investigating behaviour that disagrees with the spec. Do NOT use for UI locators (ui-automation) or for tags, test data, enums, config and debugging (framework).
---
# API Automation
The Constitution in `agent.md` applies to everything here (OpenAPI as source of truth, `z.strictObject`, the `expect(Schema.parse(body)).toBeTruthy()` pattern, `test.step` for 2+ calls, no empty-body-only 400s, no silent coverage drops, sources of truth for URLs/paths). This skill holds the how-to and the rules the Constitution does not cover.
Additional rules:
- A `null` body (401, 403, 204) is asserted with `expect(body).toBeNull()`, not parsed.
- No `as T` casts to cross a type boundary; parse with Zod instead.
- Always fuzz path parameters with the invalid-format loop, whether or not OpenAPI mentions it.
- Use `apiRequest` directly; promote to a helper fixture only when the same setup/teardown is reused across 3+ spec files.
- Never use `process.env.API_URL` bare; combine it with an enum path. Token env vars: `ACCESS_TOKEN` (full permissions) and `ACCESS_TOKEN_ZERO` (zero permissions), populated by the auth bootstrap helper.
## 1. Source the contract
1. **Documentation exists (normal case):** build schemas and tests strictly from it: field names, types, required/optional, nullability, status codes, error shapes. Do not curl first and match the schema to reality.
2. **No documentation:** make a real request to discover structure, types, optional fields, nested shapes and error formats; treat the findings as the working contract and flag the missing docs to the team. This is a stopgap, not a target state.
## 2. Zod schemas
Location: `fixtures/api/schemas/{area}/` (app schemas) and `fixtures/api/schemas/util/` (shared error schemas). Resolve `{area}` with `ls fixtures/api/schemas/`.
Spell out the response envelope per endpoint exactly as the spec describes it; do not invent a factory helper. If an envelope genuinely repeats across many endpoints in one domain, extract a per-domain `_envelope.ts` `z.strictObject` and compose with `.extend(...)`, only after the repetition is real.
```typescript
import { z } from 'zod/v4';
import type { output as zOutput } from 'zod/v4';
export const ProductSchema = z.strictObject({
id: z.uuid(),
name: z.string().min(1),
price: z.number().positive(),
category: z.enum(['electronics', 'clothing', 'food']),
inStock: z.boolean(),
});
export const CreateProductResponseSchema = z.strictObject({
success: z.boolean(),
message: z.string(),
data: ProductSchema,
errors: z.unknown().nullable(),
});
export type CreateProductResponse = zOutput<typeof CreateProductResponseSchema>;
Existing error schemas (util/errorResponseSchema.ts): BadRequestResponseSchema (400), UnauthorizedResponseSchema (401), ForbiddenResponseSchema (403), NotFoundResponseSchema (404).
Inference: zOutput<typeof S> is the post-parse type (response types, 99% of cases); zInput<typeof S> is the pre-parse type (request payloads where defaulted fields may be omitted). Without .default()/.transform() they are identical; prefer zOutput.
Zod 4 validators
| Data | Validator | Data | Validator |
|---|---|---|---|
| UUID | z.uuid() | Enum | z.enum([...]) / z.enum(MyEnum) |
| GUID | z.guid() | Literal | z.literal(200) |
z.email() | Optional | .optional() | |
| URL | z.url() | Array | z.array(S) |
| Non-empty | z.string().min(1) | Union | z.union([...]) |
| Integer | z.int() | ISO date / datetime | z.iso.date() / z.iso.datetime() |
| IPv4 | z.ipv4() | Base64 | z.base64() |
Deprecated forms to avoid: z.string().email()/uuid()/url() (use the top-level validators), z.nativeEnum(E) (use z.enum(E)), .merge() (removed; use .extend() or spread .shape), .passthrough() (use z.looseObject() only when explicitly needed), z.infer (use zOutput/zInput).
3. Write the test with apiRequest
import { expect, test } from '../../../fixtures/pom/test-options';
import { UserResponse, UserResponseSchema } from '../../../fixtures/api/schemas/app/userSchema';
import { ApiEndpoints } from '../../../enums/app/app';
test('should return user data', { tag: '@api' }, async ({ apiRequest }) => {
const { status, body } = await apiRequest<UserResponse>({
method: 'GET',
url: ApiEndpoints.CURRENT_USER,
baseUrl: process.env.API_URL,
headers: process.env.ACCESS_TOKEN,
});
expect(status).toBe(200);
expect(UserResponseSchema.parse(body)).toBeTruthy();
});
The generic <UserResponse> gives compile-time safety on body; parse gives runtime validation. Use both.
| Option | Type | Req. | Notes |
|---|---|---|---|
method | 'GET' | 'POST' | 'PUT' | 'DELETE' | 'PATCH' | yes | |
url | string | yes | endpoint path (enum) |
baseUrl | string | no | prepended |
body | Record<string, unknown> | no | payload |
headers | string | no | auth token for the Authorization header |
authType | 'Bearer' | 'Token' | 'Basic' | no | default 'Bearer' |
Use apiRequest directly for assertions, beforeEach/afterEach setup and teardown, and one-off calls.
4. test.step for multi-call tests
Each call gets its own descriptive step with its own validation:
await test.step('Create user via POST /api/users', async () => {
const { status, body } = await apiRequest<UserResponse>({ method: 'POST', url: ApiEndpoints.USERS, baseUrl: process.env.API_URL, headers: process.env.ACCESS_TOKEN, body: userData });
expect(status).toBe(201);
expect(UserResponseSchema.parse(body)).toBeTruthy();
userId = body.id;
});
await test.step('Delete user via DELETE /api/users/:id', async () => {
const { status, body } = await apiRequest<null>({ method: 'DELETE', url: `${ApiEndpoints.USERS}/${userId}`, baseUrl: process.env.API_URL, headers: process.env.ACCESS_TOKEN });
expect(status).toBe(204);
expect(body).toBeNull();
});
Without steps the trace is one blob and reporting cannot show which call failed. A single-call test may skip the step.
5. Status-code matrix
For every endpoint x method, cover every status code in the OpenAPI spec plus these baselines:
| Scenario | Status | Assert |
|---|---|---|
| Happy path (valid auth + body) | 200/201 | schema parse + key fields match sent data |
| Missing Authorization | 401 | status; error schema or toBeNull() |
| Wrong-role token | 403 | status; error schema or toBeNull() |
| Empty body (POST/PUT/PATCH) | 400/422 | one test with {}, error schema |
| Each required field omitted | 400/422 | one test per field (section 7) |
| Each field, invalid type | 400/422 | for...of per field (section 7) |
| Non-existent resource id | 404 | status (test.skip + FIXME on backend bug) |
| Invalid path parameter formats | 404 | data-driven loop |
| Unsupported HTTP method | 405 | at least one |
| Conflict (if in spec) | 409 | test it, or test.skip + FIXME if not reproducible |
| Validation errors (if in spec) | 422 | covered by per-field tests |
| DELETE no content | 204 | status + toBeNull() |
Structure: one test.describe per HTTP method + path. Use beforeAll/afterAll (not beforeEach/afterEach) for resources shared by several tests in the describe. POST describes clean up in afterAll the resourceId captured in the happy-path test; GET/PUT/DELETE describes create in beforeAll. Tag @api (check existing specs for other areas).
6. Behaviour mismatches
When actual behaviour differs from the spec or the matrix:
- Write the test as the spec says (expected status and schema).
- Wrap it in
test.skipwith a// FIXME:explaining the discrepancy:
/* eslint-disable playwright/no-skipped-test */
// FIXME: <ticket-url> API returns 500 instead of 422 for invalid price on PUT. Backend bug.
test.skip('should return 422 when price is a non-numeric string', { tag: '@api' }, async ({ apiRequest }) => {
// body as it SHOULD work per spec
});
- Never adjust the expected status to match buggy behaviour, and never loosen the schema to accept drift.
- If the API returns a different valid code than the spec (422 vs 400), use the actual code and comment the spec discrepancy.
7. Negative and validation coverage
Endpoints with a body need: (1) an empty-body test, (2) each required field omitted, (3) each field with type-inappropriate values, (4) boundary values where applicable (empty string for min-length, negative for positive-only, past dates for future-only).
- Per-field invalid types: base valid payload, override one field at a time with
INVALID_*values, assert400+BadRequestResponseSchema.parse(body). - Omitting required fields:
const { [field]: _, ...rest } = validPayload;per required field. - Path parameters: data-driven loop over
[{ description, value }](numeric strings, boolean-likes, special characters, injection attempts).
Where the invalid-value arrays live is defined in framework (test data, three tiers). Universal INVALID_* arrays are imported from test-data/static/util/invalid-values.ts and never redefined. For an email field, combine the type-mismatch array with the domain invalid-format set.
8. Helper fixtures (when needed)
For critical recurring setup/teardown reused across multiple test files, add a helper fixture in fixtures/helper/helper-fixture.ts: setup before await use(data), teardown after, which runs even if the test fails.
| Approach | Use for |
|---|---|
apiRequest fixture (default) | all calls: assertions, hooks, one-offs |
| Helper fixture | critical setup reused across 3+ files with guaranteed lifecycle (e.g. a user account 10+ files depend on) |
| Factory | generating dynamic data, no API call |
Architecture
fixtures/api/
api-request-fixture.ts apiRequest via DI (primary tool in tests)
plain-function.ts core HTTP function (used by the fixture and helpers)
api-types.ts ApiRequestParams, ApiRequestResponse<T>, ApiRequestFn
schemas/{area}/ , util/ Zod schemas
fixtures/helper/helper-fixture.ts lifecycle-wired setup/teardown
Troubleshooting
ZodErrorfromparse: with a documented contract it is a bug (section 6); without docs, update the schema to the real shape.- 401 with a null body failing a schema parse: assert
toBeNull()instead. process.env.ACCESS_TOKENundefined: the auth bootstrap did not run (see helpers inframework).- Other failures: see the debugging section of
framework.
# framework-creation
name: framework-creation description: Cross-cutting framework conventions for the Playwright scaffold – spec structure, tags and destructive tests, TypeScript type-safety rules, test data (Faker factories, static as-const data, invalid-value tiers), enums, config and env files, plain helpers vs fixtures, debugging failures and flaky tests, safely refactoring enum or static values, the post-codegen verification, and adding skills, tools or packs. Use when creating or editing tests/, test-data/, enums/, config/, env/, helpers/, fixtures/**, when a test fails or is flaky, when changing an existing enum or static value, or when extending the scaffold. For locators, page objects, the ui driver and Pega use ui-automation; for REST tests and Zod schemas use api-automation.
framework-creation
The Constitution in agent.md applies to everything here (imports, DI, tags, state cleanup, no magic numbers, no hardcoded content, no JSON static data, Faker for happy-path data, lint gate). This skill holds the detail behind it plus the rules the Constitution does not cover.
Additional rules:
- Every config property needs JSDoc naming its env var; every new env var goes in
env/.env.examplewith a safe placeholder. Onlyenv/.env.exampleis tracked. - Static data files export literal
as constvalues only: no runtime imports (type-only is fine), no functions, no Faker, no computed values. - App-defined strings (messages, labels, paths) are never generated with Faker; they live in enums.
- Edits to existing enum values or static data go through section 9.
- Never suppress a failure: no
test.skipwithout// FIXME: <ticket-url>, no loosened assertions, no timeout bumps, notry/catcharoundexpect.
1. Spec structure and tags
Locations (resolve {area} with ls tests/):
| Type | Directory | Covers |
|---|---|---|
| Functional | tests/{area}/functional/ | one behaviour in isolation |
| API | tests/{area}/api/ | contracts and schemas (rules in api-automation) |
| E2E | tests/{area}/e2e/ | one multi-feature user journey per test |
| Setup | tests/{area}/ | *.setup.ts auth or preconditions |
import { expect, test } from '../../../fixtures/pom/test-options';
test.describe('Feature Name', () => {
test.beforeEach(async ({ appPage }) => {
await appPage.openHomePage();
});
test('should do expected behavior', { tag: '@smoke' }, async ({ appPage }) => {
await test.step('GIVEN ...', async () => { /* ... */ });
await test.step('WHEN ...', async () => { /* ... */ });
await test.step('THEN ...', async () => { /* ... */ });
});
});
- Web-first assertions only (
toBeVisible,toHaveText,toBeEnabled,toHaveCount). Awaiting a response belongs in a page-object action, not the spec. - Tests are independent: no ordering, no shared mutable state. Use
beforeEachand fixtures for shared setup, and theresetStorageStatefixture for login/logout flows. - Data-driven tests loop outside
test()over namedas constexports:
for (const { description, email, password } of INVALID_LOGIN_ATTEMPTS) {
test(`should show error for ${description}`, { tag: '@regression' }, async ({ appPage }) => { /* ... */ });
}
Tags
One per test, set with { tag: '@x' } on the test (the rules are in the Constitution).
| Tag | Use |
|---|---|
@smoke | critical path, run first and often |
@sanity | key functionality |
@regression | full coverage of one behaviour |
@e2e | multi-feature journeys |
@api | API contract and schema tests |
@destructive | mutates shared/global state (locale, permissions, roles, guest access, feature flags, global settings, shared seed data such as “delete all users”) |
A test that creates its own record and deletes only that record is isolated, not destructive: keep its importance tag, but it still needs the cleanup hook:
test.afterEach(async ({ apiRequest }) => {
await apiRequest({ method: 'POST', url: ApiEndpoints.RESET_DATA, baseUrl: process.env.API_URL, headers: process.env.ACCESS_TOKEN });
});
ApiEndpoints.RESET_DATA is illustrative; use your app’s reset endpoint from enums/. @destructive tests are excluded from npm test (--grep-invert @destructive) and run only with npm run test:destructive (--workers=1); since each test has one tag, they never run under test:smoke, test:regression, etc.
2. Type safety
- At a boundary (network, file, input) use
unknownand convert viaSchema.parse(...). Inside a function body every variable has a concrete type. "strict": truestays on, per file as well. Parameter types are explicit; exported and public functions have explicit return types (Promise<void>,Locator,string, …).process.env.Xisstring | undefined. Resolve it at the access point:process.env.APP_URL!when guaranteed at runtime, orprocess.env.ENVIRONMENT ?? 'dev'with a safe default. Never export an unresolved value (export const baseUrl = process.env.API_URL;).- Zod schema rules and validators are in
api-automation.
3. Test data
| Value kind | Home |
|---|---|
| Happy-path dynamic data (emails, names, ids, text) | Faker factory in test-data/factories/{area}/*.factory.ts |
| Universal type-mismatch values | import from test-data/static/util/invalid-values.ts |
| Domain-specific curated invalid sets (bad emails, weak passwords, forbidden categories, locales) | .ts with as const in test-data/static/{area}/ |
| Field-specific boundary values used in one place | inline in the spec; promote to static only when reused by 2+ fields or specs |
| App-defined strings | enums/{area}/* (section 4) |
| Single-assertion expected values | inline in the test |
| Timeouts, retries, workers | playwright.config.ts (section 5) |
If a value fits no row, ask; do not invent a location. Resolve {area} with ls test-data/static/ and ls test-data/factories/.
Factory pattern: Faker + Zod, accept overrides, validate with the schema, return the Zod-inferred type, JSDoc with @param/@returns:
export const generateUser = (overrides?: Partial<UserResponse>): UserResponse => {
const defaults: UserResponse = { id: faker.string.uuid(), email: faker.internet.email(), token: faker.string.alphanumeric(64) };
return UserResponseSchema.parse({ ...defaults, ...overrides });
};
Static shapes: Shape A, per-field arrays (INVALID_EMAILS = [...] as const); Shape B, scenario rows with a description (INVALID_LOGIN_ATTEMPTS = [{ description, email, password }] as const). Use A for one-field-at-a-time loops, B when fields must be tested together.
Universal arrays already exist (import, never redefine): INVALID_STRING_VALUES [123, true, null, undefined], INVALID_NUMBER_VALUES ['string','123',true,null,undefined], INVALID_BOOLEAN_VALUES ['yes',1,0,null,undefined], INVALID_UUID_VALUES ['not-a-uuid','',123,null,undefined], INVALID_ENUM_VALUES ['invalidValue','',123,null,undefined], INVALID_ARRAY_VALUES ['string',123,null,undefined,{}], INVALID_OBJECT_VALUES ['string',123,null,undefined,[]]. API consumption patterns are in api-automation section 7.
Magic numbers: prefer web-first assertions. If a number is unavoidable: suite tuning goes to playwright.config.ts, domain limits to an enum, and a single legitimate per-assertion override is passed inline ({ timeout: 10000 }) with a one-line justification.
4. Enums
- TypeScript
enum(notas constobjects). Name PascalCase (Messages,ApiEndpoints,Roles,StorageStatePaths), members SCREAMING_SNAKE_CASE, value exactly as the app uses it. JSDoc on every enum. - Locations:
enums/{area}/*.tsfor app-defined values,enums/util/*.tsfor shared ones (e.g.Roles). Resolve{area}withls enums/. Extend an existing enum file before creating a new one. - In an enum: endpoint paths, routes, storage-state paths, app-defined UI messages, roles, status codes. Not in an enum: URLs, credentials and tokens (config, section 5), curated test inputs (section 3), tuning numbers, and strings used in exactly one place (inline). An array is not an enum.
- UI message values must be captured from the live app via
playwright-cli(seeui-automation), never guessed. If the app is unavailable, add// FIXME: unverifiedand flag it; do not ship unverified messages.
5. Config and environment
| Value | Home |
|---|---|
| App under test URLs | env var + config/app.ts (appConfig.appUrl, apiUrl) |
| Utility / third-party service URL | env var + config/util/util.ts (utilityConfig.*) |
| Credentials (email, password, keys) | env var only; no config object |
| Endpoint paths, routes, storage-state paths | enums (section 4) |
| Timeouts, retries, workers | playwright.config.ts |
ENVIRONMENT, CI | shell level only; not in .env.example |
playwright.config.ts loads ./env/.env.${ENVIRONMENT ?? 'dev'} with dotenv. A missing file silently loads nothing and every process.env.* is undefined. Override per run: $env:ENVIRONMENT = 'staging'; npx playwright test.
Adding a variable: (1) add it to env/.env.example with a safe placeholder, grouped under a comment; (2) add the real value to your untracked env/.env.<environment>; (3) only if it is a URL or infra setting worth documenting, add a config property with JSDoc:
export const utilityConfig = {
/** Reporting service base URL loaded from REPORTING_URL env variable */
reportingUrl: process.env.REPORTING_URL,
};
Consume by direct process.env.* (dominant in tests, fixtures, helpers) or by config object; do not mix them in one file without reason. ACCESS_TOKEN and ACCESS_TOKEN_ZERO are populated by the auth bootstrap helper and are not in .env.example. If env/.env.dev was committed: git rm --cached, confirm .gitignore, rotate exposed credentials.
6. Helpers vs fixtures
| Need | Home |
|---|---|
| Pure function, no lifecycle | helper: helpers/{area}/ (app-specific: auth bootstrap, storage state, seeding) or helpers/util/ (generic) |
Needs DI (page, request) or setup, use(data), teardown | fixture under fixtures/ (API lifecycle: api-automation section 8) |
| Locators and interactions on a page | page object (ui-automation) |
| One-off API call in one test | apiRequest directly |
| Dynamic data generation | factory (section 3) |
Helper rules: plain functions (no use(), no base.extend); camelCase verbs; explicit return types; JSDoc with @param/@returns; env and enums as in section 5. API calls inside a helper validate with the same expect(Schema.parse(body)).toBeTruthy() pattern. Mutating process.env is allowed only in auth-bootstrap helpers that publish a token; elsewhere return values through the signature. Prefer extending an existing helper file. Auth bootstrap runs from tests/{area}/auth.setup.ts before the main suite.
Fixture rules: new page objects are registered in fixtures/pom/page-object-fixture.ts (see ui-automation); fixtures are merged in fixtures/pom/test-options.ts; a fixture may call helpers but owns the lifecycle.
7. Debugging
Read the failure message first: Playwright names the test, the locator or assertion, received vs expected, and the source line. Reproduce narrowly before opening any tool:
npx playwright test <file> --workers=1 --retries=0
npx playwright test --grep "<title>" --workers=1 --retries=0
test.only is fine locally for narrowing, never committed (forbidOnly fails CI). Never push a fix you cannot reproduce locally.
Capture defaults (playwright.config.ts)
Trace on-first-retry (locally retries: 0, so no trace unless you pass --trace on or use UI Mode); screenshot only-on-failure; video retain-on-failure; HTML report opens on failure locally; blob report on CI. Timeouts: action 10 s, navigation 30 s, expect 10 s, test 60 s. Knowing which one fired tells you whether it is an action, navigation, assertion or pacing problem.
Classify, then pick the tool
| Failure | Likely cause | Look at |
|---|---|---|
| Action timeout | wrong locator, disabled element, page not loaded | trace action panel + DOM snapshot |
| Assertion timeout | element genuinely not visible, wrong locator, navigation race | UI Mode, snapshot before the assertion |
| Navigation timeout | wrong URL, env not set, app down | process.env.APP_URL, env/.env.${ENVIRONMENT} |
| Strict-mode violation | locator too loose | ui-automation locator guidance |
expect() mismatch | page state, data or enum drifted | received vs expected; enum drift goes to section 9 |
ZodError | response disagrees with schema | api-automation sections 2 and 6 |
Network error (ECONNREFUSED, ENOTFOUND, 5xx) | wrong base URL, app down, missing env or token | env vars, auth bootstrap |
| Element not attached | action raced navigation or frame swap | trace; add waitForResponse or a web-first assertion in the page action |
ReferenceError / undefined fixture | fixture not registered, or a spec imports from @playwright/test | section 6, Constitution imports rule |
| Passes alone, fails in suite | independence violated, shared state, parallel collision | section 1; promote shared mutators to @destructive with cleanup |
forbidOnly build failure | committed test.only | remove it |
| Command | Use |
|---|---|
npm run test:ui | UI Mode, the first choice (timeline, locator picker, watch mode, network tab) |
npm run test:debug | Inspector, step with F10 |
npm run test:headed | watch the browser (excludes @destructive) |
npm run report | open the last HTML report |
npx playwright show-trace <trace.zip> | GUI Trace Viewer |
npx playwright trace open <trace.zip> then actions, action <id>, snapshot <id> --name before|after, requests, console, errors, screenshot <id>, finally trace close | CLI trace inspection (Playwright 1.59+), for headless or agent use |
npx playwright test --debug=cli | CLI debugger: prints playwright-cli attach <id>; drive with step-over, snapshot |
npx playwright test <file> --trace on --workers=1 --retries=0 --headed | force-everything-on local investigation |
Pick one tool; do not bounce between three.
Fix at the root cause
| Diagnosis | Fix lives in |
|---|---|
| Locator wrong, zero or many matches | the page object getter |
| Action raced ahead of navigation | the page object action method |
Messages.X drifted from the live UI | enums/ via section 9 |
| Schema disagrees with documented contract | the test: test.skip + FIXME (api-automation section 6); never loosen the schema |
| Schema disagrees with real response, no docs | the schema |
| Token undefined | auth bootstrap helper or auth.setup.ts |
| Fixture undefined | fixtures/pom/page-object-fixture.ts |
Hardcoded string in getByText | replace with an enum |
| Test depends on another test | move setup to beforeEach or a fixture; use a factory for unique data |
After the fix: re-run the affected file with --workers=1 --retries=0 (five consecutive green runs for a flake fix; in PowerShell for ($i=1; $i -le 5; $i++) { npx playwright test <file> --workers=1 --retries=0; if ($LASTEXITCODE) { break } }), then npm test and npm run check, and confirm no test.only remains.
CI-only failure
- Download artifacts (
gh run download <run-id> -n playwright-report), unzip, find trace zips underdata/. - Open them with
show-traceor the CLI trace commands above. - Compare environments: env file, whether
auth.setup.tsproduced.auth/app/appStorageState.json, viewport (1920x1080in the chromium project), browser version. - Replay:
$env:ENVIRONMENT='ci'; $env:CI='1'; npx playwright test <file> --workers=1 --retries=0. - Still green locally: add temporary instrumentation or
--trace on, run in CI, inspect, then remove the instrumentation before merging.
8. Post-codegen verification
Walk every Constitution row in agent.md that applies to what you generated, then run:
npm run check
npx playwright test tests/{area}/[type]/[name].spec.ts
npx playwright test --grep @smoke
npm test
Also confirm: no explore-only or throwaway files, no test.only, no console.log probes, and (API work) the coverage items in api-automation section 5. On red, use section 7. Failure protocol: read the error, fix the root cause, re-run.
9. Refactoring enum values and static data
Use before changing an enum value, renaming an enum key, or editing any existing test-data/static/** file. To add new ones, use sections 3 and 4.
- Find all consumers first. Search the whole repo for the enum key and the raw string value (hardcoded copies bypass the enum):
rg "Messages.LOGIN_ERROR" .andrg "Invalid email or password" .; for endpoints alsorg "'/api/users/login'" .. Expect hits in.ts,.md(skill files, README) and ignorable.jsonreports. Inspect each match; never one global find-and-replace. - Categorise impact.
| Change | Consumers to check |
|---|---|
| Enum value | hardcoded old strings in toHaveText/toBeVisible; z.literal/z.enum in schemas (update to the new value, never loosen to z.string()); endpoint paths in apiRequest; static data using the old value; derived enum members |
| Enum key rename | every importer (tsc catches), barrel/re-export files, .md docs |
| Static data value | tests looping over the export; tests asserting a specific value; enum members mirroring the string |
- Change atomically: source and all consumers in one commit, no intermediate broken state.
- Verify:
npx tsc --noEmit,npx eslint ., then the affected tests (--grepmatches tags and titles; for one spec run the file).
10. Extending the scaffold
- Skill: create
.github/skills/<name>/SKILL.md(frontmatternameequal to the folder,descriptionsays when to use it), optionalreferences/; add a pointer in.github/instructions/<name>.instructions.md; list it in theagent.mdskills table; runnpm run check:skills. Prefer extending one of the three existing skills over adding a fourth. - Tool: see
tools/README.md; runnpm run tools:catalog(details inui-automation). - Application pack (another special UI like Pega): mirror
pega/: aUiDriversubclass, a target schema, a fixture merged infixtures/pom/test-options.ts, a section inui-automation, and tests undertests/<pack>/.
# ui-automation
name: ui-automation
description: UI test automation for the Playwright scaffold – live exploration with playwright-cli, locator choice and feedback selectors, page object class structure and fixture registration, reusable components, the application-agnostic ui driver (iframes, AJAX, volatile ids) and tools registry, and Pega-specific testing (PegaPage, nested gadget iframes, property targets). Use when creating or editing anything under pages/**, writing locators, writing UI specs, handling iframes or heavy AJAX, calling tools.run(), or testing a Pega application. Do NOT use for REST tests (api-automation) or for spec structure, tags, enums, config and debugging (framework).
UI Automation
The Constitution in agent.md applies to everything here (exploration tool, locator priority, no XPath, no generated ids, no hand-written frames, DI, feedback locators, JSDoc on locators). This skill holds the how-to.
1. Explore the live app (before any locator is written)
- Open and authenticate:
playwright-cli open <APP_URL>, thenplaywright-cli snapshot. If the page fails to load or needs auth: stop, tell the human exactly what happened and what you need (“The application at<URL>returned [error / login page / blank screen]. I need [credentials / a different URL / auth setup] before I can proceed.”), wait, then re-verify. - Explore like a user: walk the feature, snapshot at each state (
playwright-cli snapshot,playwright-cli click <ref>,playwright-cli snapshot), trigger CRUD to capture the real validation and success text, and record each element’s role, accessible name, label and test id. Look for forms, buttons, navigation, feedback elements and dynamic content (modals, expanded sections, new rows). - Plan coverage (no approval needed): happy paths, validation paths, error paths, edge cases (boundary input, concurrent operations, empty states). Cross-check user stories or acceptance criteria if they exist. The plan decides which feedback locators the page object needs.
- Encode captured message text as enum values (see
framework, enums).
2. Choose locators
Move down only when the previous option is not feasible.
page.getByRole('button', { name: 'Submit' }); // 1 buttons, links, headings, textboxes, checkboxes, navigation
page.getByLabel('Email'); // 2 labelled inputs (prefer over role textbox)
page.getByPlaceholder('Search...'); // 3 inputs without labels
page.getByText(Messages.LOGIN_ERROR); // 4 static content; repeated strings come from enums
page.getByTestId('user-avatar'); // 5 last resort
- If an element has a role, prefer
getByRole. - If exact text could match several elements, add
{ exact: true }or use a more specific role. - Scope inside a container:
page.getByRole('form').getByRole('button', { name: 'Save' }). - CSS (
.btn-primary,#submit) only as an absolute last resort viapage.locator().
Feedback locators (every form or CRUD page)
| Feedback | Strategy |
|---|---|
| Success / error message | getByText(Messages.X) or getByRole('alert') |
| Field validation | getByText(...) scoped to the form or field container |
| Toast | getByRole('status') or getByText() on its content |
| Loading | getByRole('progressbar') or getByText('Loading...') |
| Empty state | getByText('No items found') or the heading in the empty state |
Messages.* names here are illustrative; use the real enum members, with text captured via playwright-cli.
3. Page objects
Location: pages/{area}/[name].page.ts; components in pages/components/[name].component.ts. Resolve {area} with ls pages/.
import { expect, Locator, Page } from '@playwright/test';
import { Messages } from '../../enums/app/app';
export class ExamplePage {
constructor(private readonly page: Page) {}
// ==================== Locators ====================
get emailInput(): Locator {
return this.page.getByLabel('Email');
}
get submitButton(): Locator {
return this.page.getByRole('button', { name: 'Submit' });
}
// ==================== Feedback Locators ====================
get successMessage(): Locator {
return this.page.getByText(Messages.LOGIN_SUCCESS);
}
get errorMessage(): Locator {
return this.page.getByText(Messages.LOGIN_ERROR);
}
// ==================== Actions ====================
/**
* Submits the form and waits for the API response.
* @param {string} email - The user's email address.
* @returns {Promise<void>}
*/
async submitForm(email: string): Promise<void> {
await this.emailInput.fill(email);
await this.submitButton.click();
await this.page.waitForResponse((r) => r.url().includes('/api/submit'));
}
}
- Locators are
getaccessors returningLocator(a style convention;Locatoris lazy, so areadonlyfield behaves the same). Constructor is alwaysprivate readonly page: Page. - Page objects import
expect, Locator, Pagefrom@playwright/test, not fromtest-options.ts(that is for specs and fixtures). - Three sections: interactive locators, feedback locators, action methods.
- Action methods are complete user actions with an explicit return type and JSDoc (
@param,@returns;@exampleencouraged for non-trivial ones). They wait for API responses or state changes (waitForResponse, web-first assertions). Plain actions do not assert; a verification method (xxxAndVerify()) combines an action with a reusable success check and may useexpect, and it also needs JSDoc. JSDoc is allowed on component fields.
Components
Reusable fragments (header, modal, sidebar) become components composed into pages:
export class DashboardPage {
/** Navigation component for header/nav interactions */
readonly nav: NavigationComponent;
constructor(private readonly page: Page) {
this.nav = new NavigationComponent(page);
}
}
// in a test: await dashboardPage.nav.logout();
Register the fixture
In fixtures/pom/page-object-fixture.ts: import the class, add the property to FrameworkFixtures, add the body (dashboardPage: async ({ page }, use) => { await use(new DashboardPage(page)); }). pageObjectFixture is already merged into test-options.ts, so no mergeTests() change is needed. Specs then take async ({ dashboardPage }) => ....
Before writing a page object, resolve {area}, extend any enum it needs (framework) and confirm API schemas exist for any call it waits on (api-automation).
4. The ui driver and tools
Use the ui fixture (helpers/ui/ui-driver.ts) for generic reusable actions and for applications with iframes or heavy AJAX. Ordinary screens still get page objects.
Describe elements with a target, never a raw selector: { role, name } | { label } | { placeholder } | { text } | { testId } | { css } plus optional exact, nth, in the same priority as section 2. css rejects volatile ids (#field-9823471, hashes, long counters).
| Member | Purpose |
|---|---|
ui.goto(url) | Navigate, wait for idle |
ui.click(target) | Idle, resolve scope, click, idle (retries on frame churn) |
ui.fill(target, value, { commit? }) | Fill, then Tab so on-change logic runs |
ui.select(target, option) | Native select or ARIA combobox |
ui.readTable(target) | Table/grid to header-keyed rows |
ui.textOf(target) | Trimmed inner text |
ui.locate(target) / ui.scope() | Locator / current scope, for web-first assertions |
ui.waitForIdle() | Settle (pending requests and busy indicators if tracking is configured, else the load state) |
- Never store a
Locator/FrameLocatoracross actions; frames can be recreated. Callui.locate(target)when you need one. - Custom driver:
new UiDriver(page, { frames: { selectors: ['iframe[name^="work"]'], maxDepth: 2 }, load: { busySelectors: ['.spinner'] } }), thendispose()at the end. Subclass it for a pack (seepega/PegaPage.ts). - Tools:
tools.run(name, input)validatesinputwith the tool’s Zod schema and runs it. Catalog:tools/CATALOG.md(npm run tools:catalog). Add a tool withdefineToolintools/ui-tools.tsortools/api-tool.ts, register it intools/index.ts, regenerate the catalog, and add a case totests/framework/ui-driver.spec.ts.
5. Pega applications
Use the pega fixture (PegaPage) for UI actions and sessionApiRequest (see api-automation) for REST calls as the same operator. All Pega tuning (frame selectors, spinner selectors, ignored requests, quiet window) lives in pega/config/pega.config.ts, never in tests.
- Targets, in order:
{ role, name },{ label },{ testId }(attributedata-test-id),{ property: 'Customer.Name' },{ text }.csstargets containingpega-*,PegaGadget<N>Ifr, numeric or hashed ids are rejected. While exploring, note label text, role,data-test-idand the property reference rendered in thenameattribute. - How it works: (1) each action waits for idle (no tracked XHR/fetch/document request pending and no busy indicator visible for the quiet window); (2) it resolves the innermost visible harness/modal iframe (modal first, then
PegaGadget*, up to three levels, falling back to the page since Constellation has no iframes); (3) it acts, retries up to three times on detached-frame errors, and waits for idle again.
| Need | Do |
|---|---|
| Field with a generated id | pega.fill({ label: 'First name' }, value) or { property: 'FirstName' } |
| Dropdown | pega.select({ label: 'Country' }, 'Norway') |
| Grid data | pega.readTable({ role: 'table' }) |
| Assertion on the active frame | expect(await pega.locate({ label: 'Status' })).toHaveText('Open') |
| Custom wait | await pega.waitForIdle() |
| Spinner not detected | add its selector to busySelectors |
| Wrong frame chosen | adjust pegaFrameOptions.selectors |
| Pega page object | extend nothing: compose PegaPage calls in a page object under pages/ or call them in the test |
| Login | tests/pega/pega.setup.ts, stored in .auth/pega/operatorStorageState.json |
| Tool run | pegaTools.run('fillForm', { fields: [{ target: { property: 'FirstName' }, value: 'Ada' }] }) |
Verify with npm run test:framework (mock Pega pages, no server needed), then npm run test:pega against a real environment.
Troubleshooting
- Strict-mode violation: locator too loose. Add
{ exact: true }, scope to a parent, or use a more specific role. - Element not attached: the action raced a navigation or frame swap. Add
waitForResponseor a web-first assertion in the action method. - Pega fields not committing or detached frames: use
pega.fill(it tabs out and retries) and checkbusySelectorsand frame selectors inpega.config.ts. - Anything else failing: see the debugging section of
framework.