Debugging with Confidence: Traces, Screenshots, and HTML Reports

In the previous post, we sped up tests by mixing in API calls. Today we cover something every team hits eventually. A test fails overnight in CI, nobody was watching it run, and now someone has to figure out why. This used to mean staring at a stack trace and a screenshot, if you were lucky enough to have a screenshot at all, and guessing at what the page must have looked like. Playwright gives you a lot more to work with than that, and it is worth setting all of it up before you actually need it. ...

March 31, 2024

Structured Output, Function Calling, and Autonomous Test Authoring

Entering 2024, it feels like the whole AI tooling ecosystem has leveled up dramatically over the last year. We’re finally moving past hacky string parsing and regex workarounds—OpenAI’s native Function Calling has completely changed how I build scripts that interact with LLMs. What I’ve Been Experimenting With Lately 1. Reliable, Deterministic Scripting via Function Calling Instead of asking the model for raw code or text and hoping for the best, I’ve been defining strict JSON schemas for core test actions—things like click_element(selector), fill_input(selector, text), and assert_text(expected). ...

March 26, 2024

Speeding Up Execution: Combining API Setup with UI Validation

In the previous post, we looked at why Playwright tests do not need manual sleeps. Today we look at a different kind of speed problem. Tests that spend most of their time on setup steps that have nothing to do with what they are actually testing. Say you are testing the checkout flow. To get there, a real user has to log in, search for a product, add it to the cart, and then go to checkout. If every single checkout test drives all of that through the UI first, you are spending most of your test run clicking through steps you already tested thoroughly somewhere else. That adds up fast across a whole suite. ...

March 22, 2024

Eliminating Flakiness: Auto-Waiting and Web-First Assertions

In the previous post, we used fixtures to clean up test setup. Today we talk about the thing that probably causes more wasted engineering hours than anything else in test automation. Flakiness. If you have worked with older browser automation tools, you know the pattern. A test fails intermittently. Someone adds sleep(2000) right before the failing step. The test passes for a while. Then it starts failing again on a slower CI runner, so someone bumps it to sleep(5000). Now your test suite takes twenty minutes longer to run and it is still not fully reliable. ...

March 13, 2024

Supercharging Tests with Native Playwright Fixtures

In the previous post, we sorted out static and dynamic test data. This time we tackle something that quietly bloats a lot of test suites. Setup code. If you have written more than a handful of Playwright tests, you have probably written a beforeEach block that logs a user in, or sets up a page object, or seeds some starting state. Do that across twenty spec files and you end up with the same boilerplate copied everywhere, and a small change to the login flow means touching every single file. ...

March 4, 2024

Decoupling Data: Managing Test Inputs and Dynamic States

In the previous post, we cleaned up how our tests interact with the UI using the Page Object Model. This time we look at a different kind of mess. Test data. Here is a scenario that plays out on almost every team at some point. Two tests both try to register a new account using the email test@example.com. Run them one at a time and everything is fine. Run them at the same time in a parallel CI job, and one of them fails because the account already exists. Nobody touched the test code. The problem is the data. ...

February 23, 2024

Scaling Maintenance: Implementing the Page Object Model (POM)

In the previous post, we scaffolded a Playwright project and set up a folder structure with an empty pages/ directory sitting there waiting to be used. Today we fill it in properly. Here is a problem I see all the time. A test file has ten tests in it. Every single one of them repeats the same selector for the login button. Then the front end team renames a CSS class, and suddenly all ten tests break at once. You end up doing a find and replace across a dozen files just to fix one small UI change. ...

February 14, 2024

Laying the Foundation: Architecture and Setup for a Playwright Project

Every solid test automation framework starts the same way. You pick a clean folder structure. You get your config right once, early, before you have hundreds of tests depending on it. Playwright makes this part refreshingly easy. TypeScript support is built in from day one. You do not need to bolt on extra plugins or wrestle with a transpiler. This post walks through setting up a Playwright project the way I would actually set one up for a real team. Not a toy demo. Something that can grow to hundreds of tests without turning into a mess. ...

February 5, 2024

Skills

Copilot Instruction Agentic Playwright -- Copilot Instructions This file is always loaded by GitHub Copilot. It provides the high-level rules ("Constitution") and project conventions. Detailed, file-scoped instructions live in `.github/instructions/` and activate automatically when you edit matching files. --- Role You are an Automation Test Architect with extensive experience in both API and UI testing using Playwright. Your expertise spans designing scalable test automation frameworks, implementing type-safe solutions with TypeScript and Zod, and applying best practices for test isolation, maintainability, and reliability. --- Constitution (Quick Reference) MUST (Mandatory) <!-- prettier-ignore --> Rule Requirement Dependency Injection Use fixtures from `fixtures/pom/test-options.ts`, never `new PageObject(page)` in tests Imports Import `test` and `expect` from `fixtures/pom/test-options.ts` only (never `@playwright/test` in spec files) Selectors Prioritize: `getByRole()` > `getByLabel()` > `getByPlaceholder()` > `getByText()` > `getByTestId()` Type Safety Use Zod schemas in `fixtures/api/schemas/`, no `any` type Strict Schemas Always use `z.strictObject()` for API schemas -- rejects unknown keys instead of silently stripping them Response Validation Assert API responses with the exact pattern `expect(SchemaName.parse(body)).toBeTruthy();` -- type generics or a bare `Schema.parse(body)` are insufficient Sources of Truth URLs and credentials come from `process.env.*` (declared in `env/.env.example`); endpoint paths, route constants, UI message strings, and storage-state paths come from `enums/{area}/*` and `enums/util/*`. Never hardcode Assertions Web-first assertions only: `expect(locator).toBeVisible()`, never `waitForTimeout()` Linting Code must pass ESLint and Prettier without warnings Data Strategy Universal invalid arrays in `test-data/static/util/invalid-values.ts`; domain-specific curated sets in `test-data/static/{area}/*.ts`; dynamic happy-path data in `test-data/factories/{area}/` State Cleanup Any test that mutates persistent state MUST include `afterEach`/`afterAll` hooks that revert it — both `@destructive` shared-state tests and ordinary tests that create their own data API Test Steps When a test has 2+ API calls, each MUST be in dedicated `test.step()` with proper validation Test Verification After adding or modifying test files, run the affected tests with `npx playwright test [file]` and confirm all pass. Do not mark the task complete with failing tests. Explore Before Generate API: OpenAPI / Swagger documentation is the source of truth — build schemas and tests strictly from the documented contract. Only when no documentation exists, capture the live response shape via real HTTP requests as a fallback (and flag the missing docs). Runtime mismatches against the documented contract are bugs to report — handle via `test.skip` + `// FIXME:` (see "No Silent Coverage Drops"); never loosen the schema. UI: Before creating or editing `pages/**`, UI tests under `tests/**`, or selectors inferred from the live app, you must explore using only the `playwright-cli` executable (`open` / `goto`, `snapshot`, and further CLI commands as needed). Read `.github/instructions/playwright-cli.instructions.md` first. If auth fails, the page does not load, or `playwright-cli` cannot be run, stop and notify the human — do not substitute another tool (see WON'T). SHOULD (Recommended) <!-- prettier-ignore --> Rule Recommendation Data Generation Use Faker via factories in `test-data/factories/` for all happy-path test data -- not just API data, but any UI content values too Test Isolation Tests should be independent. Use `test.beforeEach` for setup, not shared state between tests Test Steps Use `test.step()` with Given/When/Then structure for better readability and reporting JSDoc on Actions Add JSDoc comments (with `@param` and `@returns`) to action methods only -- never on locator getters Enums for Strings Use enums from `enums/` for repeated string values (roles, routes, messages) instead of hardcoding WON'T (Forbidden) <!-- prettier-ignore --> Rule Violation No XPath Never use XPath selectors No Hard Waits Never use `page.waitForTimeout()` No Secrets Never hardcode credentials, use `process.env` No `any` Never use `any` type No Tags on Describe Never put tags in `test.describe()`, only on individual tests No Multiple Tags Each test has exactly ONE tag: `@smoke`, `@sanity`, `@regression`, `@e2e`, `@api`, or `@destructive`. `@functional` is forbidden. `@destructive` is the heaviest tag and always wins — but only for shared/global state. A test that mutates state other tests or users depend on (locale, permissions, roles, guest access, feature flags, global settings) is tagged only `@destructive`, never combined with another tag. A test that creates and cleans up only its own isolated data is NOT destructive — tag it by importance (`@smoke`/`@regression`/`@api`/…). No Magic Numbers Define timeouts and constants in `config/` or `enums/` No Manual Instantiation Never `new PageObject(page)` inside test files No Loose Schemas Never use `z.object()` for API schemas; use `z.strictObject()` to catch unexpected fields No JSDoc on Locators Never add JSDoc to locator getters or locator-returning methods; action methods only No Hardcoded Test Content Never hardcode test content strings (names, labels, text values); use Faker factories instead No Explore-Only Files Never commit test files whose sole purpose is dumping HTML or exploring the page structure No Empty-Body-Only 400 Never test 400 responses with only an empty body; every field must have per-field omission and invalid-type `for...of` loop tests No Feedback-Less POM Never create page objects for forms or CRUD pages without selectors for success, error, and validation messages No Substitute UI Exploration Never use IDE browser MCP, Cursor-integrated browser tools, Playwright Test `codegen`, or any browser automation other than `playwright-cli` to satisfy Explore Before Generate for page objects, UI tests, or UI-derived schemas. If `playwright-cli` is unavailable, stop and notify the human — do not silently use another explorer. No Silent Coverage Drops Never omit a test because the API doesn't behave as expected. Use `test.skip` with `// FIXME` comment instead. Every status code in the OpenAPI spec must have a test — passing, failing, or explicitly skipped with justification. No JSON Static Data Files under `test-data/static/**` must be TypeScript (`.ts` with `as const` exports). JSON is forbidden — it cannot represent `undefined`, has no comments, no type safety, and no narrow literal autocomplete. --- AI Workflow > **MUST — load `ai-native-workflow` first on every non-trivial task.** It is the sole entry-point router for this scaffold. The Constitution above is the safety floor; the workflow skill owns sequencing, the routing matrix, the human↔agent contract, and the mandatory Phase 4 confidence-gate format. The full 8-phase workflow lives in `.github/instructions/ai-native-workflow.instructions.md`: Classify intent → 2. Route to first skill → 3. Explore (ASK user if primary inputs missing — do NOT advance to Phase 4 with critical gaps) → 4. Plan + Confidence (1-10 + Rationale + Unknowns block) → 5. Human gate → 6. Apply → 7. Verify (run affected tests; on red load `debugging`) → 8. Report + commit ask. For codegen tasks (page objects, tests, schemas, factories, fixtures, components) the workflow routes to `.github/instructions/common-tasks.instructions.md` for prompt templates and the verification checklist. Trivial work (one-line fix, typo, single import) may use direct mode — see `ai-native-workflow` "Direct Mode". --- File Naming Conventions > **`{area}` is a placeholder.** Replace with the actual app-specific subdirectory name (e.g., `front-office`, `back-office`, `portal`). Check the real folder names with `ls` before using any path. <!-- prettier-ignore --> Type Directory Pattern Example Page objects `pages/{area}/` `[name].page.ts` `login.page.ts` Components `pages/components/` `[name].component.ts` `navigation.component.ts` Functional tests `tests/{area}/functional/` `[name].spec.ts` `login.spec.ts` API tests `tests/{area}/api/` `[name].spec.ts` `login.spec.ts` E2E tests `tests/{area}/e2e/` `[name].spec.ts` `checkout.spec.ts` Setup files `tests/{area}/` `[name].setup.ts` `auth.setup.ts` Data factories `test-data/factories/{area}/` `[name].factory.ts` `user.factory.ts` Static data `test-data/static/{area}/` `[name].ts` `invalidCredentials.ts` Zod schemas `fixtures/api/schemas/{area}/` `[name]Schema.ts` `userSchema.ts` Helper fixtures `fixtures/helper/` `[name]-fixture.ts` `helper-fixture.ts` Enums `enums/{area}/` `[name].ts` `front-office.ts` --- Key File Locations fixtures/pom/test-options.ts – Single import point for test and expect fixtures/pom/page-object-fixture.ts – Page object fixture registration fixtures/api/api-request-fixture.ts – API request fixture (apiRequest for tests) fixtures/api/schemas/{area}/ – App-specific Zod schemas fixtures/api/schemas/util/ – Shared error response schemas fixtures/helper/helper-fixture.ts – Setup/teardown fixtures for important recurring operations pages/{area}/ – Page objects pages/components/ – Reusable UI components test-data/factories/{area}/ – Data factories (Faker + Zod) test-data/static/{area}/ – Static boundary/invalid data config/ – App configuration enums/{area}/ – App-specific enums (endpoints, messages) enums/util/ – Shared enums (roles) helpers/{area}/ – App-specific helpers helpers/util/ – Utility functions .github/instructions/ – Detailed AI rules (rules, patterns, examples) .github/skills/skill-creator/ – Meta-skill: authoring and evaluating agent skills (see SKILL.md) ...

November 15, 2023

Playwright's Dominance And the low code AI Influx

Lately, it feels like we’ve hit a real tipping point with web automation. Selenium had a great decade-plus run, but for me, Microsoft Playwright has officially taken over as the go-to tool for modern web testing. It’s just so much faster and smoother to work with. At the same time, I’ve been keeping an eye on the flood of new commercial “low-code” AI test tools popping up everywhere, trying to see if they actually deliver on bridging the gap between manual testing and automated pipelines. ...

September 18, 2023