Debug AgentHwTest Failures

End-to-end workflow for building, running, and debugging FBOSS AgentHwTest failures on switches. Drives iterative test-debug-fix cycles.

Prerequisites: Device Access

This skill requires access to a lab switch to deploy and run tests. Read the device access reference (see Reference Routing below) to learn the available commands for your environment (upload, download, run commands on switch).

Invocation Modes

The skill supports two modes:

Mode selection: If the user provides a single test name, run Mode A. If the user provides a file path, read the file and run Mode B.

Test List File Format (Mode B)

One test name per line. Lines starting with # are comments, blank lines are ignored.

# /tmp/test_list.txt
AgentVoqSwitchTest.addRemoveNeighbor
AgentAclCounterTest.AclStatTest

Inputs

Gather these before starting:

Deriving Switch IDs from Config

Switch IDs are platform-specific. Extract them from the materialized config file:

python3 -c "
import json, sys
with open(sys.argv[1]) as f:
    cfg = json.load(f)
for sid, info in cfg['sw']['switchSettings']['switchIdToSwitchInfo'].items():
    print(f'switchId={sid} switchIndex={info[\"switchIndex\"]}')
" /root/<user>/<platform>_multi.agent.materialized_JSON

Known platform switch IDs:

The --switch_id_for_testing flag takes the switch ID (not the switch index).

Build-Before-Run Rule

MANDATORY: Always build binaries on every skill invocation before running any test. Never skip the build step, even if binaries already exist on the switch from a previous session or invocation. Stale binaries produce incorrect results because the source code may have changed.

On every skill invocation:

1. Build the required binaries (mono or multi-switch)
2. Strip and copy them to the switch (strip_and_copy.sh uses md5 dedup — unchanged binaries skip the network copy automatically)
3. Copy the config file and test scripts to the switch
4. Then proceed to run the test

The md5 dedup in strip_and_copy.sh makes this safe and fast — if the binary hasn't changed, the copy is skipped. But the build must always run to ensure the binary reflects the current source.

Debug Loop

Follow this iterative cycle for each failing test:

1. Build the test binary (mono or multi-switch) — never skip this step
2. Load the binary onto the switch
3. Copy firmware if testing on a Broadcom DNX switch (see build-and-load.md)
4. Run the test (cold or warm boot)
5. Analyze output — logs, crashes, vendor diag shell
6. Fix or adjust — code change, feature flag, XLOG insertion
7. Re-run to verify

…