e2e-medallion-architecture

Name: e2e-medallion-architecture
Author: microsoft

Build end-to-end Bronze, Silver, and Gold lakehouse pipelines in Microsoft Fabric.

Stars

★ 772

Source

GitHub

Updated

2026-07-13

// securityLow risk

Prompt only, no code
Open source
Community-trusted· 772

Running security audit…

Credentials
Network egress
Code execution
Data access
Supply chain

// install

Copy the install command and let the AI configure it · recommended for beginners

Please install the "e2e-medallion-architecture" skill from askskill:
1. Download https://raw.githubusercontent.com/microsoft/skills-for-fabric/main/skills/e2e-medallion-architecture/SKILL.md
2. Save it as ~/.claude/skills/e2e-medallion-architecture/SKILL.md
3. Reload skills and tell me it's ready

// download

Download SKILL.mdInstall manifest ↗

// examples

Design a layered lakehouse architecture

Prompt

Design an end-to-end Medallion Architecture for Microsoft Fabric, including responsibilities for Bronze, Silver, and Gold layers, the Lakehouse and Workspace structure for each layer, data flow paths, and suitable PySpark, Delta Lake, and Pipeline components.

Expected

A clear layered lakehouse architecture plan covering responsibilities, resource layout, and technology choices.

Build a Bronze-to-Gold data pipeline

Prompt

Generate a Microsoft Fabric implementation plan for Bronze-to-Silver-to-Gold processing: use PySpark notebooks for raw ingestion, cleansing and standardization, and business aggregation; use Delta Lake for tables; and orchestrate execution with Fabric Pipelines.

Expected

An end-to-end implementation flow from ingestion to analytics outputs, including notebooks, table design, and orchestration.

Add data quality and performance optimization

Prompt

Optimize an existing Microsoft Fabric Medallion Architecture by providing data quality checks, error-handling strategies, and suitable Spark configurations and performance recommendations for the Bronze, Silver, and Gold layers.

Expected

Layer-specific data quality governance and Spark optimization guidance to improve reliability and processing efficiency.

// docs

Update Check — ONCE PER SESSION (mandatory) The first time this skill is used in a session, run the check-updates skill before proceeding.

GitHub Copilot CLI / VS Code: invoke the check-updates skill.

Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.

Skip if the check was already performed earlier in this session.

CRITICAL NOTES

To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering

To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering

End-to-End Medallion Architecture

Prerequisite Knowledge

Read these companion documents — they contain the foundational context this skill depends on:

COMMON-CORE.md — Fabric REST API patterns, authentication, token audiences, item discovery
COMMON-CLI.md — az rest, az login, token acquisition, Fabric REST via CLI
SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, job execution
notebook-api-operations.md — Required for notebook creation — .ipynb structure requirements, cell format, getDefinition/updateDefinition workflow

For Spark-specific optimization details, see data-engineering-patterns.md.

Architecture Overview

Medallion Architecture is a data lakehouse pattern with three progressive layers:

Layer	Purpose	Optimization Profile	Use Case
Bronze (Raw)	Land raw data exactly as received	Write-optimized, append-only, partitioned by ingestion date	Audit trail, reprocessing, lineage
Silver (Cleaned)	Deduplicated, validated, conformed data	Balanced read/write, partitioned by business date	Feature engineering, operational reporting
Gold (Aggregated)	Pre-calculated metrics for analytics	Read-optimized (ZORDER, compaction), partitioned by month/year	Power BI reports, dashboards, ad-hoc analytics via SQL endpoint

Bronze: Schema-on-read — flexible schema, Delta time travel supports audit and rollback
Silver: Schema enforcement — reject non-conforming writes; handle schema evolution with mergeSchema when sources change
Gold: Strict schema governance — curated, business-approved datasets only

Must/Prefer/Avoid

MUST DO

Create a separate lakehouse for each medallion layer (Bronze, Silver, Gold)
Add metadata columns in Bronze: ingestion timestamp, source file, batch ID
Apply data quality rules in the Bronze-to-Silver transformation (deduplication, null handling, range validation)
Use Delta Lake format for all medallion layer tables
Use partition-aware overwrite in Silver/Gold writes to avoid reprocessing unchanged data
Include validation steps after each layer (row counts, schema checks, anomaly detection)
Follow the .ipynb validation + Fabric nuances in notebook-api-operations.md when creating notebooks via REST API — every code cell must include "outputs": [] and "execution_count": null
Default to separate workspaces per layer for governance and access control: one workspace each for Bronze, Silver, and Gold
Complete the full end-to-end flow — do not stop after creating notebooks; always bind lakehouses, execute notebooks sequentially (Bronze → Silver → Gold), verify results, and connect Power BI to the Gold layer unless the user explicitly requests a partial setup