Diagnose Spark failures, session issues, and performance bottlenecks in Microsoft Fabric.
Copy the install command and let the AI configure it · recommended for beginners
Please install the "spark-operations-cli" skill from askskill: 1. Download https://raw.githubusercontent.com/microsoft/skills-for-fabric/main/skills/spark-operations-cli/SKILL.md 2. Save it as ~/.claude/skills/spark-operations-cli/SKILL.md 3. Reload skills and tell me it's ready
Help me diagnose a failed Spark notebook run in Microsoft Fabric. Check the failure cause, driver and executor logs, and summarize actionable fixes.
Returns the root cause, key log evidence, and a list of recommended fixes.
Please triage a Livy session in Fabric that stays in the starting state. Determine whether it is caused by insufficient resources, initialization failure, or dependency issues, and suggest next steps.
Provides a diagnosis of the session issue, supporting status and log evidence, and suggested remediation steps.
Analyze whether this Spark job has executor OOM, shuffle spill, or data skew issues, and use Spark Advisor findings to suggest optimization directions.
Delivers bottleneck findings, likely causes, and practical optimization recommendations.
Update Check — ONCE PER SESSION (mandatory) The first time this skill is used in a session, run the check-updates skill before proceeding.
- GitHub Copilot CLI / VS Code: invoke the
check-updatesskill.- Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
- Skip if the check was already performed earlier in this session.
CRITICAL NOTES
- To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
- To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering
- Skill disambiguation:
spark-operations-cliis for read-only triage and diagnosis of existing jobs and sessions. For creating notebooks, running new jobs, or Spark development, usespark-authoring-cli. For interactive PySpark analysis and Livy session creation, usespark-consumption-cli.
This skill provides diagnostics for Microsoft Fabric Spark job failures, Livy session health, and performance bottlenecks using Fabric REST APIs and CLI tools (az rest). All diagnostic operations are read-only; session cleanup (e.g., stopping zombie sessions) requires explicit user confirmation. For Spark development and notebook authoring, use spark-authoring-cli. For interactive PySpark analysis, use spark-consumption-cli.
The TOC is grouped by purpose. Start at Diagnostic Workflows when triaging an active failure; the earlier sections are foundational references.
| Task | Reference | Notes |
|---|---|---|
| Fabric Topology & Key Concepts | COMMON-CORE.md § Fabric Topology & Key Concepts | |
| Environment URLs | COMMON-CORE.md § Environment URLs | |
| Authentication & Token Acquisition | COMMON-CORE.md § Authentication & Token Acquisition | Wrong audience = 401; read before any auth issue |
| Core Control-Plane REST APIs | COMMON-CORE.md § Core Control-Plane REST APIs | |
| Pagination | COMMON-CORE.md § Pagination | |
| Long-Running Operations (LRO) | COMMON-CORE.md § Long-Running Operations (LRO) | |
| Rate Limiting & Throttling | COMMON-CORE.md § Rate Limiting & Throttling | |
| Job Execution | COMMON-CORE.md § Job Execution | |
| Capacity Management | COMMON-CORE.md § Capacity Management | |
| Gotchas & Troubleshooting | COMMON-CORE.md § Gotchas & Troubleshooting | |
| Best Practices | COMMON-CORE.md § Best Practices |
| Task | Reference | Notes |
|---|---|---|
| Tool Selection Rationale | COMMON-CLI.md § Tool Selection Rationale | |
| Finding Workspaces and Items in Fabric | COMMON-CLI.md § Finding Workspaces and Items in Fabric | Mandatory — READ link first [needed for finding workspace id by its name or item id by its name, item type, and workspace id] |
| Authentication Recipes | COMMON-CLI.md § Authentication Recipes | az login flows and token acquisition |
…
Inspect Fabric alerts, notifications, and automated actions via read-only API calls.
Run authoring T-SQL for Fabric warehouses and SQL endpoints from CLI.
Monitor, inspect, and query Fabric Dataflows Gen2 through a read-only CLI.
Query Fabric Eventhouse data, analyze time series, and monitor ingestion health.
Migrate Databricks notebooks, jobs, and paths into Microsoft Fabric.
List, inspect, and validate Microsoft Fabric Eventstream topology and configuration.
Interactively analyze lakehouse data with PySpark, Livy sessions, and Spark SQL.
Build Fabric Spark workflows, author notebook code, and manage lakehouse resources.
Run stateful local PySpark and SQL sessions for data exploration and validation.
Analyze Fabric warehouse performance via CLI and get optimization recommendations.
Run read-only T-SQL queries on Fabric warehouse and lakehouse data.
Query Spark History Server apps, jobs, stages, and metrics in natural language.