$ loading_
Migrate HDInsight Spark, Hive, and Oozie workloads to Microsoft Fabric.
Copy the install command and let the AI configure it · recommended for beginners
Please install the "hdinsight-migration" skill from askskill: 1. Download https://raw.githubusercontent.com/microsoft/skills-for-fabric/main/skills/hdinsight-migration/SKILL.md 2. Save it as ~/.claude/skills/hdinsight-migration/SKILL.md 3. Reload skills and tell me it's ready
Please migrate this Azure HDInsight Spark code to Microsoft Fabric: remove HiveContext and standalone SparkContext initialization, use the pre-instantiated SparkSession, convert WASB/ABFS paths to OneLake abfss URLs, and explain the reason for each change.
Returns Fabric-ready Spark code with explanations of the key migration changes.
Please migrate the following Hive DDL to Microsoft Fabric Lakehouse: convert STORED AS ORC, external table definitions, and storage paths into Delta Lake-compatible table schemas and locations; also note any Hive metastore semantic differences.
Outputs converted Delta Lake table definitions and lists syntax and storage mapping differences.
I am retiring an HDInsight cluster. Please migrate this Oozie workflow/coordinator to Microsoft Fabric Pipeline: identify matching activities for spark, hive, shell, and sqoop actions, propose schedule triggers, and explain which file and credential operations should be replaced with notebookutils.
Provides a Fabric Pipeline mapping plan, trigger recommendations, and a script replacement checklist.
Update Check — ONCE PER SESSION (mandatory) The first time this skill is used in a session, run the check-updates skill before proceeding.
- GitHub Copilot CLI / VS Code: invoke the
check-updatesskill.- Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
- Skip if the check was already performed earlier in this session.
CRITICAL NOTES
- To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
- To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
- HDInsight has no
mssparkutilsordbutilsequivalent —notebookutilsis net-new capability being introducedHiveContextandSQLContextare legacy Spark 1.x/2.x APIs — Fabric uses Spark 3.xSparkSessionexclusivelywasb://paths are deprecated and require a Storage Account key or SAS — replace with OneLake shortcuts
Read these companion documents before executing migration tasks:
az rest, az login, token acquisition, Fabric REST via CLIFor notebook and Lakehouse creation, see spark-authoring-cli. For Fabric Warehouse DDL/DML authoring, see sqldw-authoring-cli.
| Topic | Reference |
|---|---|
| Migration Workload Map | § Migration Workload Map |
| SparkSession & Context API Changes | § SparkSession API Changes |
| WASB / ABFS → OneLake Path Migration | path-migration.md |
| Hive DDL → Delta Lake / Lakehouse Schemas | hive-to-delta.md |
| Oozie → Fabric Pipelines | § Oozie → Fabric Pipelines |
Introducing notebookutils | § Introducing notebookutils |
| Before/After Code Patterns | code-patterns.md |
| Spark Configuration Differences | § Spark Configuration Differences |
| Must / Prefer / Avoid | § Must / Prefer / Avoid |
| Authentication & Token Acquisition | COMMON-CORE.md § Authentication |
| Lakehouse Management | SPARK-AUTHORING-CORE.md § Lakehouse Management |
| HDInsight Component | Fabric Target | Notes |
|---|---|---|
| Spark cluster (notebooks, scripts) | Fabric Spark (Lakehouse / Notebooks / SJD) | No persistent cluster — Starter Pool or Custom Pool provides on-demand Spark |
| Hive / HiveServer2 | Lakehouse SQL Endpoint + Lakehouse schemas | Delta Lake replaces Hive metastore; schemas provide namespace equivalent |
| HBase | Fabric Warehouse or Azure Cosmos DB (separate from Fabric) | HBase has no direct Fabric equivalent — assess workload access patterns |
| Oozie workflows | Fabric Data Pipelines | Map Oozie actions to Fabric activities; see § Oozie → Fabric Pipelines |
| YARN Resource Manager | Fabric Spark monitoring (Spark UI, Monitoring Hub) | No YARN — Fabric manages compute automatically |
| Ambari | Fabric Monitoring Hub + Admin Portal | Cluster health, capacity, and job monitoring |
…
Inspect Fabric alerts, notifications, and automated actions via read-only API calls.
Run authoring T-SQL for Fabric warehouses and SQL endpoints from CLI.
Monitor, inspect, and query Fabric Dataflows Gen2 through a read-only CLI.
Query Fabric Eventhouse data, analyze time series, and monitor ingestion health.
Migrate Databricks notebooks, jobs, and paths into Microsoft Fabric.
List, inspect, and validate Microsoft Fabric Eventstream topology and configuration.
Migrate Azure Synapse Spark workloads to Microsoft Fabric with code replacements.
Build Fabric Spark workflows, author notebook code, and manage lakehouse resources.
Interactively analyze lakehouse data with PySpark, Livy sessions, and Spark SQL.
Diagnose Spark failures, session issues, and performance bottlenecks in Microsoft Fabric.
Run read-only T-SQL queries on Fabric warehouse and lakehouse data.
Query Microsoft Fabric workspaces, lakehouses, tables, jobs, and dependencies in natural language.