In today’s data-centric landscape, the nature of data has evolved far beyond traditional structured tables. Modern workloads increasingly rely on a diverse mix of data types from JSON logs and real-time streaming events to images, audio, and natural language text, all of which are foundational for AI and machine learning applications.
Originally designed for structured data and static business reporting, traditional data warehouses are increasingly inadequate. They lack the flexibility and scalability. This gap has led to the rise of platforms like Databricks, which introduce the Lakehouse architecture – a mix of a data lake and a data warehouse. This unified approach combines the scalability of data lakes with the reliability and performance of data warehouses, providing a seamless integration of data storage and analysis.
To make things even easier, Databricks has a new feature called Lakeflow. It’s a single, simple platform that helps to manage the entire data process, from ingesting raw data to cleaning, transforming, and integrating it into applications. Lakeflow helps data teams build, test, and launch their data pipelines faster and with more confidence.
In the first part of the Databricks series for testers, we’ll focus on introducing Lakeflow – its definition, how it works and why it’s important for anyone responsible for ensuring data quality.
What is Lakeflow
Lakeflow is a unified data engineering platform integrated into the Databricks Data Intelligence Platform that simplifies the entire process of building and operating data pipelines.
It is an end-to-end data engineering solution that empowers data engineers, software developers, SQL developers, analysts, and data scientists to deliver high-quality data for downstream analytics, AI, and operational applications.
Because the Lakeflow covers ingestion, transformation, orchestration, and monitoring, teams gain a complete solution for designing, running, and managing production data pipelines with only Lakeflow; therefore, there is no need for multiple tools
Key Benefits of Lakeflow
1. All-in-One Data Engineering Platform
Lakeflow seamlessly combines ingestion, transformation, orchestration, and monitoring into a single, unified tool. Through this end-to-end integration, it effectively reduces complexity, enhances reliability, and accelerates development.
2. Low-Code Experience
Lakeflow offers an intuitive drag-and-drop interface, complemented by support for SQL and Python. As a result, both technical and non-technical users can efficiently build, manage, and scale data pipelines with ease.
3. Real-Time Processing
Lakeflow supports both batch and streaming pipelines, enabling up-to-date insights through a single, unified tool. This flexibility allows teams to power low-latency analytics and handle large-scale data processing with ease.
4. Smart Orchestration
Lakeflow streamlines workflow automation by seamlessly managing scheduling, retries, and dependencies. This built-in orchestration reduces the risk of pipeline failures and ensures smoother, more reliable operations.
5. Comprehensive Observability
Lakeflow offers built-in monitoring, lineage tracking, and data quality validation, ensuring transparency and trust across every stage of the data pipeline.
6. Delta Lake Native
Lakeflow is fully integrated with the Databricks Lakehouse and Delta Lake, ensuring governance, performance, and scalability throughout the entire data lifecycle.
7. Scalability and Reliability
Powered by the Databricks Data Intelligence Platform, Lakeflow scales effortlessly to support enterprise-grade workloads, while consistently maintaining high reliability and performance.
Lakeflow Core Components
Core components
Databricks Lakeflow streamlines the entire data engineering lifecycle by combining three core components that work together to simplify ingestion, transformation, and orchestration
1, Lakeflow Connect is the data ingestion layer of Databricks Lakeflow, designed to deliver fast, reliable, and low-code integration of data from diverse sources into the Databricks Lakehouse (Delta Lake).
It offers a user-friendly, declarative interface for ingesting data into the Databricks Lakehouse from a wide range of sources – including relational databases (e.g., SQL Server, Oracle), enterprise applications (e.g., Salesforce, Workday), and cloud storage platforms (Amazon S3, Azure Data Lake Storage (ADLS), Google Cloud Storage).
2. Lakeflow Declarative Pipelines is the transformation engine, which allows users to specify what needs to be done – such as data sources, transformations, targets, and schedules – while the platform intelligently manages how those tasks are executed.
This shifts pipeline design from being imperative (writing step-by-step instructions) to declarative (describing the desired outcome).
3. Lakeflow Jobs is the orchestration hub that automates and monitors your entire data workflow, including tasks created in Lakeflow Declarative Pipelines, notebooks, SQL queries, and machine learning models. It is a framework for building batch and streaming data pipelines using SQL and Python, designed to accelerate ETL development.
While declarative pipelines describe what should happen, jobs are responsible for running those tasks at scale, on schedule, and with reliability.
How Lakeflow Components Interact
Ingestion with Lakeflow Connect
Firstly, Lakeflow Connect uses managed connectors to ingest raw data from various sources, such as databases, enterprise applications, and cloud storage, into the bronze layer of the Lakehouse.
The process is highly automated and supports Change Data Capture (CDC) for real-time, incremental data loading.
Transformation with Declarative Pipelines
Once ingested, data is refined using declarative pipelines that handle cleaning, validation, and enrichment.
These pipelines automatically manage execution logic and dependencies across the bronze, silver, and gold layers, enabling the creation of curated datasets with minimal manual effort.
Orchestration with Lakeflow Jobs
Finally, Lakeflow Jobs orchestrate the entire workflow, coordinating ingestion, transformation, and additional tasks such as machine learning model training or dashboard refreshes.
Jobs ensure correct task sequencing, manage failures gracefully, and provide unified monitoring and health insights.
Why Testers Should Know LakeFlow
By knowing Lakeflow’s components and interactions, testers can:
1. Place the right validation checkpoints (raw ingestion, intermediate transformations, final gold tables).
- Lakeflow Connect (Raw ingestion / Bronze layer): Schema correctness (fields exist, types match); Completeness (all expected rows/records ingested); CDC handling or streaming correctness
- Lakeflow Pipelines (Intermediate transformations / Silver layer): Transformations produce the expected outputs; business rules and aggregations are correctly applied; detect early data anomalies before they propagate downstream.
- Gold tables / final outputs: Confirm correctness, consistency, and completeness; spot any unexpected deviations introduced during upstream transformations
2. Collaborate effectively Across Roles
- Speak the same language as data engineers (pipelines, jobs, CDC).
- Communicate effectively with analysts (Bronze/Silver/Gold datasets, business rules).
- Work alongside ML engineers (feature pipelines, data quality for models).
3. Prevent silent data quality failures
- A subtle schema change may silently corrupt downstream transformations because the pipeline itself doesn’t break; it simply ingests and processes flawed data, creating a false sense of security while the final output is corrupted.
=> Testers should monitor schema changes (e.g. renamed column, change data type, add new fields …) by comparing the source schema vs. the ingested schema in Bronze tables - Missing or delayed CDC events can lead to incomplete Gold tables.
- If an update event for a customer’s address is lost, the Gold table will never reflect the correct address. This results in the data being permanently out of sync with the source.
- An event may arrive out of order. For example, a “delete” event for a record arrives after a new record with the same ID has been inserted. The pipeline might incorrectly delete the new record, leading to data loss and an incomplete table.
=> Tester should validate CDC integrity:- Count checks: Compare the number of inserts/updates/deletes between the source and pipeline for a given period.
- Sequence and timestamp checks: Validate that there are no gaps in the sequence or missing timestamps, updates are applied in the correct order
- Source-to-target validation: Compare the final Gold tables with expected aggregates from the source, considering CDC changes, to ensure that all inserts, updates, and deletes are correctly applied throughout the pipelines.
Conclusion
Lakeflow marks a significant advancement in simplifying and unifying data engineering. By integrating ingestion, transformation, orchestration, monitoring, and governance into a single, cohesive platform, it eliminates the complexity of managing multiple disconnected tools.
By understanding Lakeflow, testers can move beyond basic validations like row counts and begin ensuring end-to-end data quality. This includes verifying the accuracy and completeness of data, the reliability of pipeline logic, the robustness of security controls, and the performance of the system – all while aligning with business outcomes. Lakeflow empowers QA teams to play a more strategic role in the data lifecycle, contributing to the delivery of trusted, production-ready data products.
Reference