Medallion Architecture

What is a Medallion Architecture?

The Medallion Architecture, a data design pattern for organizing data in a Data Lakehouse, consists of three layers—Bronze, Silver, and Gold—representing increasing levels of data structure and validation. Transitioning between these layers, the design employs architecture at each level to progressively enhance the quality of the data. Additionally, it is often referred to as a “multi-hop” architecture. In this comprehensive design, Delta Lake emerges as a frequent choice to ensure ACID transactions and versioning.

Utilizing Medallion Architecture to build data pipelines

Databricks provides a robust platform for efficiently implementing Medallion Architecture in data pipelines. Leveraging Delta Live Tables streamlines the development process; furthermore, Apache Spark Structured Streaming ensures real-time updates. Integration with Delta Lake enhances data integrity through ACID transactions and optimized storage. In addition, the ELT methodology prioritizes speed during the Silver layer loading, and the use of optimized storage formats ensures efficient querying. Moreover, Delta Lake’s schema evolution features add adaptability to changing data structures, making Databricks a powerful solution for constructing and maintaining sophisticated and agile Medallion Architecture data pipelines.

Three layers of the Medallion Architecture

The Medallion Architecture, with its Bronze, Silver, and Gold layers, facilitates a systematic and incremental approach to data processing, ensuring data quality and providing a structured foundation for analytics and reporting in a Data Lakehouse environment.

1. Bronze Layer (Raw Data):
   • The initial layer ingests data from external source systems.
   • It stores data in its raw form, largely without validation, enabling rapid Change Data Capture.
   • This layer often adds supplementary metadata, including load date/time and process ID, for historical tracking.
   • Typically stored in a columnar format like Parquet or Delta for efficient storage and querying.
   • Acts as a foundation for the subsequent layers, providing an unaltered historical archive of the source data.
2. Silver Layer (Cleansed Data):
   • The intermediate layer where data undergoes filtering, cleaning, and augmentation.
   • We apply data validation rules to ensure consistency, uniqueness, and correct formats.
   • Enrichment with reference data may take place, and data from different sources is not yet joined.
   • Stored in Delta format, enabling schema enforcement and evolution, ensuring data quality.
   • Serves as a unified “Enterprise view” providing a foundation for self-service analytics, ad hoc reporting, and downstream data projects.
3. Gold Layer (Curated Business-Level Tables):
   • In the final layer, we transform data for specific use cases and apply business-level aggregation.
   • We enforce business rules and may join data from different source systems for comprehensive insights.
   • Represents the presentation layer with consumption-ready “project-specific” databases.
   • Data models in this layer are often denormalized and optimized for reporting, reducing the need for complex joins.
   • Stored in Delta format to take advantage of features like version restoration, ensuring data quality and facilitating reporting and analytics projects.

Use Case: Retail Analytics in a Medallion Architecture

Let’s consider a detailed use case for the medallion architecture in the context of a retail business. The medallion architecture is applied to organize and manage data in a Lakehouse, facilitating the flow of data through different layers (Bronze, Silver, Gold) with the goal of improving data quality and structure progressively.

1. Bronze Layer (Raw Data):
In the Bronze layer, raw data is ingested from various sources, including point-of-sale systems, online transactions, customer interactions, and supply chain systems. The focus in this layer is on capturing data as it is, along with relevant metadata.

-- Example Bronze Layer Delta Table Schema
CREATE TABLE bronze_retail
USING delta
AS
SELECT
  transaction_id,
  product_id,
  customer_id,
  quantity,
  amount,
  transaction_date,
  'POS' as source_system
FROM
  point_of_sale_data;

In this example, raw data from point-of-sale transactions is captured, and additional metadata like the source system is included.

2. Silver Layer (Cleansed and Conformed Data):
In the Silver layer, data from the Bronze layer is processed to create a unified and cleansed view. This layer includes entity resolution, data matching, and conformation processes to ensure consistency and accuracy across the enterprise.

-- Example Silver Layer Delta Table Schema
CREATE TABLE silver_retail
USING delta
AS
SELECT
  t.transaction_id,
  p.product_id,
  c.customer_id,
  t.quantity,
  t.amount,
  t.transaction_date,
  c.customer_name,
  p.product_name,
  'silver_layer' as source_system
FROM
  bronze_retail t
JOIN
  customer_data c
ON
  t.customer_id = c.customer_id
JOIN
  product_data p
ON
  t.product_id = p.product_id;

In this example, the Silver layer combines transaction data with additional customer and product information, creating a unified and cleansed dataset.

3. Gold Layer (Curated Business-level Tables):
The Gold layer is designed for consumption-ready, project-specific databases that support advanced analytics and reporting.

-- Example Gold Layer Delta Table Schema
CREATE TABLE gold_sales_analytics
USING delta
AS
SELECT
  t.transaction_id,
  c.customer_name,
  p.product_name,
  t.quantity,
  t.amount,
  t.transaction_date,
  'gold_layer' as source_system
FROM
  silver_retail t
JOIN
  customer_data c
ON
  t.customer_id = c.customer_id
JOIN
  product_data p
ON
  t.product_id = p.product_id;

In this example, the Gold layer focuses on creating a specific business-level table for sales analytics, incorporating denormalized and read-optimized data models to support efficient reporting.

Business Insights and Benefits

• Historical Analysis:
  Time travel capabilities allow for historical analysis of transactions, helping the business understand trends and patterns over time.
• Unified Customer View:
  The Silver layer ensures a unified view of customer data, enabling personalized marketing and improved customer experiences.
• Inventory Management:
  By integrating product data in the Silver layer, the business can gain insights into inventory levels, popular products, and supply chain optimization.
• Advanced Analytics:
  The Gold layer provides a structured platform for advanced analytics, facilitating the creation of customer segmentation models, product recommendations, and sales forecasts.
• Flexible Data Model:
  The architecture allows for flexibility in adapting to changing business requirements, adding new data sources, and modifying data models as needed.
• Data Management and Auditability:
  Metadata tracking and data lineage support data governance, ensuring compliance with regulations and providing transparency into data transformations.

Advantages of Lakehouse Architecture

1. Simple Data Model
   The architecture employs a straightforward and flexible data model.
2. Ease of Understanding and Implementation
   Simplicity in design enhances the ease of comprehension and implementation.
3. Incremental ETL
   Supports incremental Extract, Transform, Load processes for efficient data updates.
4. Recreation from Raw Data
   Facilitates the recreation of tables from raw data, providing flexibility in data reconstruction.
5. ACID transactions
   Ensures data integrity with ACID transactions.
6. Time Travel
   Offers time travel capabilities for historical data analysis.

For further details, please refer to databricks documentation.