For Spark-specific optimization details, see data-engineering-patterns.md.

Architecture Overview

Medallion Architecture is a data lakehouse pattern with three progressive layers:

• Bronze: Schema-on-read — flexible schema, Delta time travel supports audit and rollback
• Silver: Schema enforcement — reject non-conforming writes; handle schema evolution with mergeSchema when sources change
• Gold: Strict schema governance — curated, business-approved datasets only

Must/Prefer/Avoid

MUST DO

• Create a separate lakehouse for each medallion layer (Bronze, Silver, Gold)
• Add metadata columns in Bronze: ingestion timestamp, source file, batch ID
• Apply data quality rules in the Bronze-to-Silver transformation (deduplication, null handling, range validation)
• Use Delta Lake format for all medallion layer tables
• Use partition-aware overwrite in Silver/Gold writes to avoid reprocessing unchanged data
• Include validation steps after each layer (row counts, schema checks, anomaly detection)
• Follow the .ipynb validation + Fabric nuances in notebook-api-operations.md when creating notebooks via REST API — every code cell must include "outputs": [] and "execution_count": null
• Default to separate workspaces per layer for governance and access control: one workspace each for Bronze, Silver, and Gold
• Complete the full end-to-end flow — do not stop after creating notebooks; always bind lakehouses, execute notebooks sequentially (Bronze → Silver → Gold), verify results, and connect Power BI to the Gold layer unless the user explicitly requests a partial setup

PREFER

• Incremental processing (watermark pattern) over full refresh

…

spark-consumption-cli

通过 PySpark 与 Livy 会话交互分析 Lakehouse 数据，支持高级计算与跨仓查询。

spark-authoring-cli

帮助你开发 Microsoft Fabric Spark 工作流、编写调试 Notebook 代码并管理湖仓与资源。