Home / Knowledge / Data Fabric

What is a data fabric and what does it mean for industry?

Short answer

A data fabric is an architectural concept that makes distributed data sources usable through a shared layer of metadata, integration and access rules, without copying all data to one place. The data stays in ERP, MES, historians or cloud storage. The fabric knows where which data is, what it means and who may use it. In industry, its value depends on whether production data arrives with context such as asset, line and order.

Reading time 7 minutesEditorial team, inray Industriesoftware
Many individual connections between data sources and consumers compared with a shared data fabric layer

Definition: data fabric

Gartner describes the data fabric as a design concept that serves as an integrated layer of data and connecting processes, based on continuous analytics over existing and inferred metadata. Gartner analyst Mark Beyer put it this way in an article published on 25 August 2021. It is not a single product but an architecture: systems stay as they are, and a layer above them handles discovery, access and delivery.

Characteristic Central data copy Data fabric
Storage location All data is copied into one central store Data stays in the source systems and is copied only when needed
Meaning of the data Hidden in load processes and table structures Recorded in metadata and relationships visible to all users
Access rules Defined again for each store Defined once and applied to every access path
New source New load process into the central store Connection to the fabric and description in the metadata
Typical use Few sources, fixed reporting Many sources across several sites, changing questions

Why the topic matters

Production data is used by more and more functions: controlling, quality, maintenance, energy management and, increasingly, analytics and AI applications. Often every new question gets its own connection or its own copy of the data.

A simple calculation shows how fast this grows. If each of n consumers pulls its own data from each of m sources, up to m · n connections emerge. With 8 sources and 6 consumers, that is 48. With a shared layer, each source and each consumer needs only one connection, m + n in total, so 14 here. The calculation illustrates the principle. In practice not every consumer needs every source, but the pattern of many copies with different states remains the actual problem.

The term itself predates the current trend. Forrester published a market evaluation of the “big data fabric” in the fourth quarter of 2016 (analyst Noel Yuhanna). Gartner listed the data fabric among its “Top Strategic Technology Trends for 2022”.

The building blocks of a data fabric

A data fabric consists of a small number of building blocks. They do not have to come from one vendor.

Building block Task Origin
Metadata catalogue Records which data exists, where it is and who owns it Core of the data fabric according to Gartner, 2021
Active metadata Continuously analyses how data is used and steers delivery and quality accordingly Gartner, Mark Beyer, 25 August 2021
Knowledge graph Represents relationships between data objects, for example asset, order and material Named by Gartner as a building block, 2021
Integration services Deliver data by copy, data stream or virtual access Integration styles according to Hohpe and Woolf, Enterprise Integration Patterns, 2003
Governance and access Define who may see which data in which form Part of every data fabric architecture

How a data fabric works technically

Four mechanisms carry the architecture.

  • Metadata first: Every connected source is described, technically and in business terms. Only then can data be found and interpreted correctly.
  • Several access paths: Depending on the requirement, data is copied, passed on as a stream or queried virtually at the source.
  • Semantics: The same things get the same names. An asset has the same name in the MES, the historian and the ERP, or it is mapped unambiguously.
  • Automation: From how metadata is used, the fabric derives suggestions, for example which data belongs together or where quality problems occur.

Four steps to a data fabric

The sequence is similar in most projects.

  1. Map sources and consumers

    Which systems deliver which data to whom today, through which path and how up to date?

  2. Agree on shared terms

    Asset, line, order and material are named consistently so that metadata matches across system boundaries.

  3. Start with one use case

    A clearly bounded need, such as energy figures per asset, is the first to run through the fabric and is measured.

  4. Extend step by step

    Further sources and consumers follow by value. Duplicate copies and individual connections are replaced deliberately.

What a data fabric is not

Four distinctions prevent false expectations.

  • Not a single product: The data fabric is a design concept. It is implemented with catalogue, integration and governance tools, often from several vendors.
  • Not a data lake: A data lake is a store for raw data. A data fabric can include a data lake or a lakehouse, but it does not replace storage.
  • Not a data mesh: The data mesh, described by Zhamak Dehghani on martinfowler.com in 2019, is primarily an organisational model with data ownership in the business domains. The data fabric is an architectural pattern. The two can be combined.
  • Not machine connectivity: The fabric works with data that already sits in systems. Controllers and sensors have to be connected first.

Typical hurdles in practice

These points slow down data fabric projects most often.

  • Missing metadata: Without maintained descriptions, nobody finds the data and automation has nothing to work with.
  • Production data without context: A measured value without asset, unit and time reference cannot be classified meaningfully in any catalogue.
  • Tool before goal: If a platform is bought first and a use case is sought afterwards, the result is another data collection without users.
  • Unclear ownership: If nobody owns a data source in business terms, descriptions and access rules go stale.
The honest part

A data fabric shifts work, it does not remove it. Instead of many load processes, metadata, terms and access rules have to be maintained permanently. With few sources and fixed reporting, a well-run data warehouse is often the simpler solution.

How pronubes supports

pronubes starts at the beginning of the chain: where production data is created and gets its context. The product components have different tasks.

  • pronubes Edge: Edge runs in the plant and connects machines, controllers and IT systems via protocols such as OPC UA, MQTT, REST and SQL. Data is structured where it is created and reaches the data fabric or an existing data store with information on origin and unit.
  • pronubes Zones: Zones maps sites, production areas and systems as a structure. This structure provides the order that metadata and terms in the fabric can refer to.
  • pronubes Insights: Insights makes data flows from production visible during operation. If a source fails or delivers late, this becomes apparent before analyses rely on outdated data.
Glossary
Data fabric
Architectural concept for uniform access to distributed data through metadata, integration and access rules.
Metadata
Data about data, such as origin, meaning, unit, freshness and ownership.
Active metadata
Continuously analysed metadata that steers delivery and quality.
Knowledge graph
Model that represents data objects and the relationships between them.
Data virtualisation
Querying data at the source without copying it first.
Data mesh
Organisational model in which business domains own their data as products.
Data lake
Central store for raw data in its original format.
Governance
Rules and responsibilities for the quality, protection and use of data.
Asked and answered

Frequently asked questions

What is the difference between a data fabric and a data mesh?

The data fabric is an architectural pattern that makes distributed data accessible through metadata and a shared integration layer. The data mesh is primarily an organisational model in which business domains own their data as products. Many companies use elements of both.

Does a data fabric replace the data warehouse?

No. A data warehouse or a lakehouse can be part of the fabric. The fabric governs how data from these and other sources is found, described and delivered.

Do you have to store all data centrally for a data fabric?

No, that is precisely not the approach. The data largely stays in the source systems. It is copied only where performance, availability or analysis require it.

What role does production data play in a data fabric?

It is among the most valuable and most difficult sources. Machine data is created at high frequency and without business context. To be usable in the fabric, it has to be connected in the plant, structured and assigned unambiguously.

How do you start with a data fabric?

With an inventory of sources and consumers, consistent terms for central objects and a clearly bounded use case. The choice of tools follows from that, not the other way round.

Get started

A data fabric starts in production.

pronubes connects machines and systems in the plant and delivers production data with context to your data platform.

pronubes by inray

pronubes is a product of inray Industriesoftware GmbH. Over 30 years of industrial software from Germany. Innovative and reliable for manufacturing companies.

inray