Default workflow

1. Read the scenario pack and current suite implementation.
2. Decide lane:
   • mock/dev: mock-openai
   • real validation: live-frontier
3. For live OpenAI, use:

OPENCLAW_LIVE_OPENAI_KEY="${OPENAI_API_KEY}" \
pnpm openclaw qa suite \
  --provider-mode live-frontier \
  --model openai/gpt-5.4 \
  --alt-model openai/gpt-5.4 \
  --output-dir .artifacts/qa-e2e/run-all-live-frontier-<tag>

1. Watch outputs:
   • summary: .artifacts/qa-e2e/run-all-live-frontier-<tag>/qa-suite-summary.json
   • report: .artifacts/qa-e2e/run-all-live-frontier-<tag>/qa-suite-report.md
2. If the user wants to watch the live UI, find the current openclaw-qa listen port and report http://127.0.0.1:<port>.
3. If a scenario fails, fix the product or harness root cause, then rerun the full lane.

OTEL smoke

For local QA-lab OpenTelemetry validation, use:

pnpm qa:otel:smoke

This starts a local OTLP/HTTP trace receiver, runs the otel-trace-smoke scenario through qa-channel, decodes the emitted protobuf spans, and verifies the exported trace names and privacy contract. It does not require Opik, Langfuse, or external collector credentials.

Matrix live profiles

pnpm openclaw qa matrix defaults to the full all profile. Use explicit profiles for faster CI/release proof:

OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS=3000 \
pnpm openclaw qa matrix --profile fast --fail-fast

• fast: release-critical transport contract, excluding generated image and deep E2EE recovery inventory.
• transport, media, e2ee-smoke, e2ee-deep, e2ee-cli: sharded full Matrix coverage.
• QA-Lab - All Lanes uses explicit fast Matrix on scheduled runs. Manual dispatch keeps matrix_profile=all as the default and always shards that full Matrix selection.

QA credentials and 1Password

• Use op only inside tmux for QA secret lookup in this repo.
• Quick auth check inside tmux:

op account list

• Direct Telegram npm live test secrets currently live in 1Password item:
  • vault: OpenClaw
  • item: Telegram E2E
• That item is the first place to look for:
  • OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN
  • OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN
  • OPENCLAW_QA_PROVIDER_MODE
  • OPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC
• Convex QA secrets currently live in 1Password items:
  • vault: OpenClaw
  • item: OPENCLAW_QA_CONVEX_SITE_URL
  • item: OPENCLAW_QA_CONVEX_SECRET_MAINTAINER
  • item: OPENCLAW_QA_CONVEX_SECRET_CI
• Additional related notes/login items seen during QA credential work:
  • vault: Private
  • items: OPENCLAW QA, Convex, Telegram
• If a required value is missing from those notes:
  • do not guess
  • ask the maintainer/operator for the current value or the current 1Password item name
  • for Telegram direct runs, OPENCLAW_QA_TELEGRAM_GROUP_ID may be stored separately from Telegram E2E
  • for Convex runs, the leased Telegram credential should provide the Telegram group id and bot tokens together; do not require a separate OPENCLAW_QA_TELEGRAM_GROUP_ID
  • for Convex runs, prefer OpenClaw/OPENCLAW_QA_CONVEX_SITE_URL; if that is stale or unclear, ask for the active pool URL before running
• Prefer direct Telegram envs for the npm Telegram Docker lane when available:

OPENCLAW_QA_TELEGRAM_GROUP_ID="..." \
OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN="..." \
OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN="..." \

…

tag-duplicate-prs-issues

帮助团队检索重复的 PR 与问题单，并自动分组同步重复状态。

openclaw-test-performance

用于分析并优化 OpenClaw 测试与插件套件的性能瓶颈、内存占用和覆盖率耗时。