A data lakehouse combines the low-cost, open storage of a data lake with properties of a data warehouse: transactions, schema enforcement and fast queries. Data is stored once in open file formats such as Apache Parquet, and reporting, analytics and machine learning work on the same copy. This avoids keeping data twice, but it replaces neither a data model nor the work on data quality. A data lakehouse is a data architecture that brings together two previously separate worlds. From the data lake it takes the storage of any kind of data in open file formats on low-cost object storage. From the data warehouse it takes transactions, an enforced schema and queries fast enough for reporting. A table layer on top of the files makes this possible, for example Delta Lake or Apache Iceberg. The term data lake was coined by James Dixon, then CTO of Pentaho, in 2010: a large store for raw data that only receives structure when it is read. In practice this often led to a two-tier architecture. Raw data lands in the lake, and selected data is copied into a warehouse by load jobs so that reports run quickly and reliably. Every copy needs maintenance, and every copy is somewhat older than its source. On 30 January 2020, Databricks described the lakehouse in a blog post as a response to this. The paper by Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia followed at the CIDR 2021 conference. It names the weaknesses of the two-tier architecture: stale data, reliability, total cost of ownership, lock-in to one vendor and limited support for machine learning. A rough calculation shows what stale data means in practice. Purely as an illustration: if a nightly job loads the previous day’s data into the warehouse at 2:00 a.m., a value measured at 2:05 a.m. appears in the report almost 24 hours later. Averaged over the day, the delay is around 12 hours. If production writes directly into a lakehouse table, the load step disappears, and the delay only depends on how often data is transferred. The numbers are a worked example, not a measurement from a project. A lakehouse is not a single product but a combination of open building blocks. Four mechanisms support the architecture. The sequence is similar in most projects. Which analyses should be created, and which machines, plants and systems provide the data for them? Data enters the first layer unchanged and with its origin. This keeps every later analysis traceable. Remove duplicates, standardise units and timestamps, link measured values to plant, order and shift. Key figures, reports and models are built on the validated layer. Results such as predictions or setpoints can go back towards production. Four distinctions prevent false expectations. These points slow down lakehouse projects in manufacturing most often. A lakehouse solves the problem of keeping data twice, not the problem of unclear data. If you maintain a warehouse and a lake today, you will not automatically have less work after the switch: the effort shifts from copy jobs to table maintenance, permissions and the data model. That effort is better invested, but it remains. For a lakehouse, pronubes handles the way data travels from production and back. The product parts have different tasks. A data lake stores raw data without a fixed schema and without transactions. A data lakehouse adds a table layer on top that enables transactions, schema enforcement and versions. This also makes it suitable for reporting without copying the data into a warehouse. Databricks introduced the lakehouse in a blog post on 30 January 2020. The paper by Armbrust, Ghodsi, Xin and Zaharia was published for the CIDR 2021 conference. No. The lakehouse is an architecture pattern. Open table formats such as Delta Lake and Apache Iceberg are supported by various engines and vendors. For history, analysis and machine learning, yes. For control and reactions within milliseconds, no. High-frequency data should be written in batches so that not too many small files are created. The three layers of the medallion architecture: bronze holds raw data, silver cleaned and linked data, gold analysis-ready key figures and models. pronubes brings machine and process data into Databricks or your Spark cluster without additional middleware, and results back towards production in a controlled way.What is a data lakehouse and what is it used for in industry?
Definition: data lakehouse
Feature
Data warehouse
Data lake
Data lakehouse
Storage
Structured tables, usually in the vendor’s storage
Files of any kind in object storage
Open files in object storage plus a table layer
Schema
Defined before writing
Applied only when reading
Checked on write and can be changed in a controlled way
Transactions
Yes
Not intended
Yes, via the table format
Typical use
Reporting and BI
Raw data archive, data science
Reporting, analytics and ML on the same data
Data format
Mostly vendor-specific
Open, e.g. Parquet, JSON, CSV
Open, e.g. Parquet with Delta Lake or Iceberg
Why the lakehouse emerged
What a lakehouse consists of
Building block
Task
Origin
Apache Parquet
Column-oriented file format for large data volumes
Top-level project of the Apache Software Foundation since 27 April 2015
Delta Lake
Table format with transactions, schema enforcement and access to earlier versions
Released as open source by Databricks on 24 April 2019, a Linux Foundation project since October 2019
Apache Iceberg
Open table format, originally developed at Netflix
Apache Incubator from 16 November 2018, top-level project since 20 May 2020
Query engine
Reads and writes the tables, runs SQL and ML workloads
Several engines can use the same tables, for example Apache Spark
Medallion layers
Organise data by maturity into bronze, silver and gold
Design pattern described in the Databricks documentation
How a lakehouse works technically
Four steps to a lakehouse
Clarify questions and sources
Load raw data (bronze)
Clean and enrich (silver)
Analyse and return (gold)
What a data lakehouse is not
Typical hurdles in practice
How pronubes helps
Frequently asked questions
What is the difference between a data lake and a data lakehouse?
Who described the concept?
Do I need Databricks for a lakehouse?
Is a lakehouse suitable for machine data?
What do bronze, silver and gold mean?
A lakehouse depends on data that arrives with context.
pronubes is a product of inray Industriesoftware GmbH. Over 30 years of industrial software from Germany. Innovative and reliable for manufacturing companies.

