What does Data Engineering look like in Databricks?
Data engineering is the practice of designing and building systems that collect, store, and process large amounts of data so organizations can use it to make decisions.
One such system is the Data Pipeline
Data Pipelines are built to ingest(extract) data from a source and transform it before sending it off(load) to a storage destination. You may recognize this process from the acronym ETL (Extract, Transform, Load). It's sibling term, ELT, loads data into storage after extraction before having it transform, a practice becoming more common with the increase of cheaper storage over time.
Databricks is a unified, cloud-based data platform that combines data warehousing and machine learning into a single "lakehouse" architecture.
Lakeflow is Databrick's unified platform for data ingestion, transformation, and orchestration, and consists of the following three parts:
- LakeFlow Connect - Ingestion connectors for enterprise applications, databases, flat files, and the like.
- Apache Spark Declarative Pipelines - A SQL and Python framework building the transformation mechanisms of the data pipeline.
- LakeFlow Jobs - Workflow automation and orchestration
The ELT Process
As Databricks is a Lakehouse, let us take a closer look at its ELT data pipeline components:
LakeFlow Connect
Lakeflow has many connection options from local flat files to SaaS applications and Databases
LakeFlow can handle Batch ingestion, Incremental Batch Ingestion, and Streaming Ingestion
Delta Lake
Delta Lake is a protocol for reading and writing files into cloud storage.
Delta Lake uses Unity Catalogue, Databrick's data governance object model, to handle data ingested by LakeFlow Connect
Spark Declarative Pipelines (SDP)
SDP is the framework upon which Data Engineers write their transformation logic in SQL and Python.
SDP simplifies ETL/ELT development for Data Engineers by abstracting away the mechanics of pipeline execution for simpler notation that focuses on data manipulation.
The Pipeline Concept
Besides the available frameworks and tools upon which the ELT pipeline is built, it is important to consider how the transformation can be executed in a methodical way. The Medallion Architecture is such a methodology that lends itself to a more transparent transformation process.
Medallion Architecture
The idea here is to separate your transformation into different domains. Let us call these domains 'layers':
Bronze layer - The landing version of your data, in its raw original form.
Silver layer - The data, lightly cleaned and shaped into a favorable structure, but maintaining its original granularity.
Gold layer - The data, with business logic applied, its values calculated across and aggregated to an appropriate granularity that is ready for analysis.