OpenClaw QA Testing

Use this skill for qa-lab / qa-channel work. Repo-local QA only.

Read first

• docs/concepts/qa-e2e-automation.md
• docs/help/testing.md
• docs/channels/qa-channel.md
• qa/README.md
• qa/scenarios/index.md
• extensions/qa-lab/src/suite.ts
• extensions/qa-lab/src/character-eval.ts

Model policy

• Live OpenAI lane: openai/gpt-5.4
• Fast mode: on
• Do not use:
  • openai/gpt-5.4-pro
  • openai/gpt-5.4-mini
• Only change model policy if the user explicitly asks.

Default workflow

1. Read the scenario pack and current suite implementation.
2. Decide lane:
   • mock/dev: mock-openai
   • real validation: live-frontier
3. For live OpenAI, use:

OPENCLAW_LIVE_OPENAI_KEY="${OPENAI_API_KEY}" \
pnpm openclaw qa suite \
  --provider-mode live-frontier \
  --model openai/gpt-5.4 \
  --alt-model openai/gpt-5.4 \
  --output-dir .artifacts/qa-e2e/run-all-live-frontier-<tag>

1. Watch outputs:
   • summary: .artifacts/qa-e2e/run-all-live-frontier-<tag>/qa-suite-summary.json
   • report: .artifacts/qa-e2e/run-all-live-frontier-<tag>/qa-suite-report.md
2. If the user wants to watch the live UI, find the current openclaw-qa listen port and report http://127.0.0.1:<port>.
3. If a scenario fails, fix the product or harness root cause, then rerun the full lane.

OTEL smoke

For local QA-lab OpenTelemetry validation, use:

pnpm qa:otel:smoke

This starts a local OTLP/HTTP trace receiver, runs the otel-trace-smoke scenario through qa-channel, decodes the emitted protobuf spans, and verifies the exported trace names and privacy contract. It does not require Opik, Langfuse, or external collector credentials.

Matrix live profiles

pnpm openclaw qa matrix defaults to the full all profile. Use explicit profiles for faster CI/release proof:

OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS=3000 \
pnpm openclaw qa matrix --profile fast --fail-fast

• fast: release-critical transport contract, excluding generated image and deep E2EE recovery inventory.
• transport, media, e2ee-smoke, e2ee-deep, e2ee-cli: sharded full Matrix coverage.
• QA-Lab - All Lanes uses explicit fast Matrix on scheduled runs. Manual dispatch keeps matrix_profile=all as the default and always shards that full Matrix selection.

QA credentials and 1Password

• Use op only inside tmux for QA secret lookup in this repo.
• Quick auth check inside tmux:

op account list

• Direct Telegram npm live test secrets currently live in 1Password item:
  • vault: OpenClaw
  • item: Telegram E2E
• That item is the first place to look for:
  • OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN
  • OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN
  • OPENCLAW_QA_PROVIDER_MODE
  • OPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC
• Convex QA secrets currently live in 1Password items:
  • vault: OpenClaw
  • item: OPENCLAW_QA_CONVEX_SITE_URL
  • item: OPENCLAW_QA_CONVEX_SECRET_MAINTAINER
  • item: OPENCLAW_QA_CONVEX_SECRET_CI
• Additional related notes/login items seen during QA credential work:
  • vault: Private
  • items: OPENCLAW QA, Convex, Telegram
• If a required value is missing from those notes:
  • do not guess
  • ask the maintainer/operator for the current value or the current 1Password item name
  • for Telegram direct runs, OPENCLAW_QA_TELEGRAM_GROUP_ID may be stored separately from Telegram E2E
  • for Convex runs, the leased Telegram credential should provide the Telegram group id and bot tokens together; do not require a separate OPENCLAW_QA_TELEGRAM_GROUP_ID
  • for Convex runs, prefer OpenClaw/OPENCLAW_QA_CONVEX_SITE_URL; if that is stale or unclear, ask for the active pool URL before running
• Prefer direct Telegram envs for the npm Telegram Docker lane when available:

OPENCLAW_QA_TELEGRAM_GROUP_ID="..." \
OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN="..." \
OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN="..." \

…