Check a local web app with webapp-testing

What the skill does, and what this example adds

Anthropic’s webapp-testing skill is a set of instructions for an agent using Python Playwright. Its pinned source includes a server-lifecycle helper and a reconnaissance-first workflow: inspect the rendered page, choose selectors grounded in that page, then exercise an interaction. It is useful for checking a known user flow in a local application; it is not a substitute for a complete accessibility, security or cross-browser test suite.

This worked example, added on October 2, 2026, uses a small original TechReadly fixture called Reading list lab. It contains three demonstration records, a search input, a category filter and a reset button. The target flow is: load three records → select Office → get Word handoff → enter an unmatched query → see an empty state → reset → recover all three records. The archive week and original publication date are unchanged.

Verification boundary: the fixture’s four JavaScript filtering tests and Python syntax checks passed here. The Python browser suite could not reach page execution: the installed Chromium process failed during launch because the execution environment denied a required socket. No browser screenshot or successful end-to-end result is claimed. The download includes the runnable test, rather than a fabricated pass log.

1. Download the complete fixture

• Download the webapp-testing example ZIP · HTML, filter module, unit tests, Playwright test, setup and verification record
• Open the Reading list lab fixture · A demonstration interface; its automated browser run is still unverified
• Read the actual verification record · Passed unit checks and the blocked browser stage

Extract the ZIP into a disposable working directory. It creates webapp-testing-example/; all commands below run from there. The fixture uses only local files and demonstration data. It does not need credentials, a database or the TechReadly application repository. The Python automation writes its results to a separate artifacts/ directory.

unzip webapp-testing-example.zip
cd webapp-testing-example
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python -m playwright install chromium
node --test catalog.test.mjs

Recorded authoring environment: Debian GNU/Linux 13; Python 3.12.14; Playwright for Python 1.62.0; Node.js 24.19.0. The attempted system browser was Chromium 154.0.8037.57. The requirements file pins Playwright to 1.62.0; its browser-install command selects the matching bundled Chromium. That bundled-browser setup is a reproduction path, not a setup that passed in this environment. On Windows, activate the virtual environment with .venv\Scripts\Activate.ps1.

2. Give the agent a bounded task

Use webapp-testing to inspect the local Reading list lab fixture.
Check the page title and Fixture ready status before acting.
Discover the Search guides, Category and Reset filters controls.
Test Office filtering, a no-match query and resetting both fields.
Run at 1280×800 and 390×844. Capture the empty and reset states,
record console/page errors, and return the result JSON and screenshots.
Report a blocked launch as blocked; do not infer a pass from source code.

This prompt states the observable behavior and the evidence needed. It avoids asking an agent to “test everything,” which can produce a list of clicks without acceptance criteria. The complete smoke_test.py checks the title, ready status, result count, result text, empty message and reset field values. It also records JavaScript page errors and console errors.

3. Start the server and run the assertions

python -m http.server 4177 --bind 127.0.0.1

. .venv/bin/activate
python smoke_test.py

Stop Terminal A with Ctrl+C when finished. If you already have the pinned Anthropic repository checked out, you can use its lifecycle helper instead. First inspect --help, then start a single server through it. $SKILL below must point to skills/webapp-testing at source commit 9d2f1ae187231d8199c64b5b762e1bdf2244733d; the example ZIP does not redistribute the upstream skill.

python "$SKILL/scripts/with_server.py" --help
python "$SKILL/scripts/with_server.py" \
  --server "python -m http.server 4177 --bind 127.0.0.1" \
  --port 4177 -- python smoke_test.py

expect(page.get_by_role('status')).to_have_text('Fixture ready')
results = page.get_by_role('list', name='Matching guides').get_by_role('listitem')
expect(results).to_have_count(3)
page.get_by_label('Category', exact=True).select_option('Office')
expect(results).to_have_count(1)
expect(results.first).to_contain_text('Word handoff')
page.get_by_label('Search guides', exact=True).fill('zz-no-match')
expect(results).to_have_count(0)
page.get_by_role('button', name='Reset filters').click()
expect(results).to_have_count(3)

The archived skill shows a networkidle wait before inspecting a dynamic page. Current Playwright documentation discourages using that condition as a testing readiness signal. This example instead waits for the fixture’s explicit Fixture ready status and uses auto-retrying locator assertions. On a real app, use an equivalent visible ready state; background polling can prevent network idleness without preventing useful interaction.

4. Separate expected output from observed evidence

Expected after a successful browser run: artifacts/result.json has status: "passed" and two passed viewport records; empty-1280.png, reset-1280.png, empty-390.png and reset-390.png show the corresponding states. Two dom-*.html files capture the initial rendered markup. These filenames describe outputs the supplied test will produce on success, not outputs captured in this review.

tests 4
pass 4
fail 0

{
  "status": "failed",
  "stage": "browser_launch",
  "viewports": [],
  "console_errors": []
}

An empty error list in an aborted launch says nothing about page health. No page was opened. The unit tests establish the filter function’s expected data behavior, but cannot establish DOM wiring, reset-button behavior, layout or browser compatibility. Those remain unverified until the browser suite runs successfully and the screenshots are inspected.

Troubleshooting and adapting the test

• “Executable does not exist”: activate the same virtual environment used to install Playwright, then run python -m playwright install chromium. Installing the Python package alone does not install a browser.
• “socket() failed: Operation not permitted”: the browser process was blocked before navigation in this run. Use an environment that permits browser automation; do not treat a source review or a longer timeout as an end-to-end pass.
• Connection refused: check that Terminal A is still running and the URL uses port 4177. If that port is occupied, choose another port and set TEST_URL=http://127.0.0.1:YOUR_PORT/index.html for the test.
• A strict-locator error: inspect the current DOM and narrow the locator by role, accessible name or a containing region. Do not blindly choose the first matching element.
• An empty state never appears: check both query and category, because this fixture combines the two filters. Keep a reset test so a stale filter cannot silently affect the next case.
• A CI browser has missing system libraries: follow Playwright’s OS dependency instructions for your runner. Use matching pinned browser/package versions before changing application code.

For a production app, replace the demonstration records with a stable test fixture, add authentication only through an approved test account, and keep screenshots free of private user data. Add Firefox/WebKit, keyboard-only checks and accessibility tests separately. A passing Chromium smoke test is deliberately a narrow signal.