Home / Knowledge / Data Lakehouse

What is a data lakehouse and what is it used for in industry?

Short answer

A data lakehouse combines the low-cost, open storage of a data lake with properties of a data warehouse: transactions, schema enforcement and fast queries. Data is stored once in open file formats such as Apache Parquet, and reporting, analytics and machine learning work on the same copy. This avoids keeping data twice, but it replaces neither a data model nor the work on data quality.

Reading time 6 minutesEditorial team, inray Industriesoftware
Two stores with a copy step compared with one shared lakehouse store

Definition: data lakehouse

A data lakehouse is a data architecture that brings together two previously separate worlds. From the data lake it takes the storage of any kind of data in open file formats on low-cost object storage. From the data warehouse it takes transactions, an enforced schema and queries fast enough for reporting. A table layer on top of the files makes this possible, for example Delta Lake or Apache Iceberg.

Feature Data warehouse Data lake Data lakehouse
Storage Structured tables, usually in the vendor’s storage Files of any kind in object storage Open files in object storage plus a table layer
Schema Defined before writing Applied only when reading Checked on write and can be changed in a controlled way
Transactions Yes Not intended Yes, via the table format
Typical use Reporting and BI Raw data archive, data science Reporting, analytics and ML on the same data
Data format Mostly vendor-specific Open, e.g. Parquet, JSON, CSV Open, e.g. Parquet with Delta Lake or Iceberg

Why the lakehouse emerged

The term data lake was coined by James Dixon, then CTO of Pentaho, in 2010: a large store for raw data that only receives structure when it is read. In practice this often led to a two-tier architecture. Raw data lands in the lake, and selected data is copied into a warehouse by load jobs so that reports run quickly and reliably. Every copy needs maintenance, and every copy is somewhat older than its source.

On 30 January 2020, Databricks described the lakehouse in a blog post as a response to this. The paper by Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia followed at the CIDR 2021 conference. It names the weaknesses of the two-tier architecture: stale data, reliability, total cost of ownership, lock-in to one vendor and limited support for machine learning.

A rough calculation shows what stale data means in practice. Purely as an illustration: if a nightly job loads the previous day’s data into the warehouse at 2:00 a.m., a value measured at 2:05 a.m. appears in the report almost 24 hours later. Averaged over the day, the delay is around 12 hours. If production writes directly into a lakehouse table, the load step disappears, and the delay only depends on how often data is transferred. The numbers are a worked example, not a measurement from a project.

What a lakehouse consists of

A lakehouse is not a single product but a combination of open building blocks.

Building block Task Origin
Apache Parquet Column-oriented file format for large data volumes Top-level project of the Apache Software Foundation since 27 April 2015
Delta Lake Table format with transactions, schema enforcement and access to earlier versions Released as open source by Databricks on 24 April 2019, a Linux Foundation project since October 2019
Apache Iceberg Open table format, originally developed at Netflix Apache Incubator from 16 November 2018, top-level project since 20 May 2020
Query engine Reads and writes the tables, runs SQL and ML workloads Several engines can use the same tables, for example Apache Spark
Medallion layers Organise data by maturity into bronze, silver and gold Design pattern described in the Databricks documentation

How a lakehouse works technically

Four mechanisms support the architecture.

  • Transaction log: The table format records which files belong to which version of a table. Several processes can read and write at the same time without seeing half-finished states, and earlier versions remain available.
  • Schema enforcement: Data that does not match the schema is rejected on write. Schema changes are possible, but deliberate.
  • Separate storage and compute: Data sits in object storage, and compute is only used for queries and processing. Both can be scaled independently.
  • Open formats: BI tools, SQL engines and machine learning libraries read the same files directly, without their own copy.

Four steps to a lakehouse

The sequence is similar in most projects.

  1. Clarify questions and sources

    Which analyses should be created, and which machines, plants and systems provide the data for them?

  2. Load raw data (bronze)

    Data enters the first layer unchanged and with its origin. This keeps every later analysis traceable.

  3. Clean and enrich (silver)

    Remove duplicates, standardise units and timestamps, link measured values to plant, order and shift.

  4. Analyse and return (gold)

    Key figures, reports and models are built on the validated layer. Results such as predictions or setpoints can go back towards production.

What a data lakehouse is not

Four distinctions prevent false expectations.

  • Not a single product: Lakehouse describes an architecture pattern. There are several vendors and at least two widely used open table formats.
  • Not a replacement for control: A lakehouse serves analysis. Control loops at the machine and reactions within milliseconds remain in automation.
  • No automatic data quality: Schema enforcement prevents wrong data types, but not wrong measurements or missing assignments.
  • Not a must for every company: If you only need a few stable reports, a data warehouse is often sufficient.

Typical hurdles in practice

These points slow down lakehouse projects in manufacturing most often.

  • Many small files: high-frequency machine data creates many small files when written often, which slows down queries. Batching and regular compaction help.
  • Timestamps and time zones: controllers, MES and ERP deliver times in different formats.
  • Missing context: a measured value without a link to plant, order and shift is hard to analyse.
  • Permissions: who may read which tables, and who may write?
  • Compute costs: separate compute is flexible, but becomes expensive when queries run unplanned.
The honest part

A lakehouse solves the problem of keeping data twice, not the problem of unclear data. If you maintain a warehouse and a lake today, you will not automatically have less work after the switch: the effort shifts from copy jobs to table maintenance, permissions and the data model. That effort is better invested, but it remains.

How pronubes helps

For a lakehouse, pronubes handles the way data travels from production and back. The product parts have different tasks.

  • pronubes Edge: Edge transfers machine and process data from OPC UA, MQTT, PLCs and SAP into Databricks or your own Spark cluster without additional middleware. Results such as predictions or setpoints return in a controlled way towards production through a configured flow.
  • pronubes Zones: Zones maps sites, production areas, systems and data flows in a clear structure. This helps assign raw data to the right site and area as it is loaded.
  • pronubes Insights: Insights shows whether connections and data flows are running stably in operation. This lets you spot early when data is not arriving in the lakehouse.
Glossary
Data lakehouse
Data architecture that combines open file storage with transactions and schema enforcement.
Data lake
Store for raw data of any kind, structure is only applied when reading.
Data warehouse
Structured database for reporting, schema is defined before writing.
Apache Parquet
Open, column-oriented file format.
Delta Lake
Open table format with a transaction log, a Linux Foundation project.
Apache Iceberg
Open table format of the Apache Software Foundation.
ACID
Properties of reliable transactions: atomic, consistent, isolated, durable.
Medallion architecture
Organisation of data into bronze, silver and gold layers.
Schema-on-read
Structure is applied to the data only when it is read.
Asked and answered

Frequently asked questions

What is the difference between a data lake and a data lakehouse?

A data lake stores raw data without a fixed schema and without transactions. A data lakehouse adds a table layer on top that enables transactions, schema enforcement and versions. This also makes it suitable for reporting without copying the data into a warehouse.

Who described the concept?

Databricks introduced the lakehouse in a blog post on 30 January 2020. The paper by Armbrust, Ghodsi, Xin and Zaharia was published for the CIDR 2021 conference.

Do I need Databricks for a lakehouse?

No. The lakehouse is an architecture pattern. Open table formats such as Delta Lake and Apache Iceberg are supported by various engines and vendors.

Is a lakehouse suitable for machine data?

For history, analysis and machine learning, yes. For control and reactions within milliseconds, no. High-frequency data should be written in batches so that not too many small files are created.

What do bronze, silver and gold mean?

The three layers of the medallion architecture: bronze holds raw data, silver cleaned and linked data, gold analysis-ready key figures and models.

Get started

A lakehouse depends on data that arrives with context.

pronubes brings machine and process data into Databricks or your Spark cluster without additional middleware, and results back towards production in a controlled way.

pronubes by inray

pronubes is a product of inray Industriesoftware GmbH. Over 30 years of industrial software from Germany. Innovative and reliable for manufacturing companies.

inray