Data lakes
« Back to Glossary Index
A data lake is a centralized repository that stores data in its raw, native format — structured tables, semi-structured logs and JSON, and unstructured files such as images, audio and documents — without requiring it to be modeled before it lands. Structure is applied when the data is read rather than when it is written.
That single design choice explains both the appeal and the risk. Ingestion becomes cheap and fast because nothing has to be designed up front, and the same property is what allows a lake to fill with data nobody can identify or trust.
What Is a Data Lake?
A data lake stores data as files in low-cost object storage — Amazon S3, Azure Data Lake Storage, Google Cloud Storage — and keeps it in the form it arrived. There is no requirement to define a schema, fit the data to an existing model, or decide in advance what questions it will answer.
This is described as schema-on-read, in contrast to the schema-on-write approach of a traditional warehouse. In a warehouse, data is transformed into a defined model before loading, so anything that does not fit is either reshaped or rejected. In a lake, the data lands as-is and interpretation happens at query time.
The consequences are significant. Storage is cheap and effectively unlimited. Data science teams can work with raw detail rather than a pre-aggregated view built for reporting. Data that has no defined use yet can be retained rather than discarded. The cost is that nothing enforces meaning — two files can describe the same entity differently, and only a person reading them will notice.
Data Lake vs. Data Warehouse
The two were designed for different problems, and most enterprises end up running both.
A data warehouse stores structured, modeled data optimized for analytical queries. Definitions are enforced, quality is controlled, and performance for reporting is excellent. It is the appropriate home for numbers the business relies on. The constraints are cost, and the lead time to add a new source because it must be modeled first.
A data lake stores anything at low cost with no modeling requirement. It suits exploration, machine learning and retaining raw history. It is poorly suited to being the direct source of governed business reporting, because nothing guarantees consistency.
The common pattern is a lake as the landing and retention layer, with curated, modeled data promoted into a warehouse for reporting. The friction with that pattern is duplication — data stored twice, pipelines maintained twice — which is the problem the lakehouse was created to address.
What Is a Data Lakehouse?
A lakehouse keeps data in open file formats on object storage, as a lake does, while adding the guarantees that made warehouses trustworthy — ACID transactions, schema enforcement, versioning and time travel, and performant SQL access.
The enabling technology is the open table format: Delta Lake, Apache Iceberg and Apache Hudi. These add a transaction log over the files, so concurrent writes behave predictably, schema changes are controlled, and a table can be queried as it existed at an earlier point.
For most organizations building new, the lakehouse has become the default because it removes the duplication of the lake-plus-warehouse pattern while keeping data in open formats rather than a vendor’s proprietary store. It does not remove the need for modeling and governance — it simply means one copy of the data can serve both exploration and reporting. Designing that layout is core modern data architecture work.
Data Lake Architecture
Well-run lakes organize data into zones by degree of refinement, commonly labeled bronze, silver and gold.
Bronze, or raw, holds data exactly as ingested, unmodified. It is the reproducible record — if a downstream transformation is later found to be wrong, everything can be rebuilt from here.
Silver, or cleansed, holds data that has been validated, deduplicated, typed and conformed. This is where most data engineering effort goes and where analysts should generally start.
Gold, or curated, holds business-level aggregates and modeled tables aligned to agreed definitions. This is what reporting and dashboards consume.
Alongside the zones, three components determine whether the lake stays usable: a catalog so datasets can be found and understood, access controls at table and column level, and lineage showing how each dataset was derived. A lake without a catalog is a filesystem, and searching it does not scale.
Why Data Lakes Become Data Swamps
A data swamp is a lake that has accumulated data nobody can find, interpret or trust. It is the predictable outcome of treating cheap ingestion as the goal.
The pattern is consistent. Ingestion is easy, so many sources are connected quickly and the programme looks successful. Because landing data requires no modeling, cataloging and documentation are deferred. Ownership is never assigned, so nobody is accountable for whether a dataset is still correct or still needed. Over time, users cannot tell which of several similar datasets is authoritative, so they stop trusting all of them and revert to extracts from source systems.
Avoiding it requires a small number of non-negotiables applied from the start: every dataset has an owner, nothing enters without catalog metadata, the zone model is enforced rather than optional, and datasets nobody queries are retired on a schedule. These are governance decisions rather than technical ones, which is why the problem recurs across every generation of platform.
When a Data Lake Is the Right Choice
A lake or lakehouse fits when there is significant semi-structured or unstructured data, when machine learning needs raw detail rather than reporting aggregates, when data must be retained before its use is known, or when source volume and variety make up-front modeling impractical.
It is the wrong answer when the actual requirement is governed reporting on a well-understood set of structured sources. In that case a warehouse is simpler, cheaper to operate and easier to trust — and choosing a lake adds engineering effort that produces no benefit.
The decision is usually made during a platform move, where it sits alongside choices about cloud migration approach and target architecture. Deciding it deliberately at that point is considerably cheaper than discovering it afterward.
Frequently Asked Questions
A data lake is a centralized repository that stores structured, semi-structured and unstructured data in its raw native format on low-cost object storage, applying structure when the data is read rather than when it is written.
A warehouse stores structured, modeled data optimized for reporting, with definitions and quality enforced before loading. A lake stores anything in raw form at low cost with no modeling requirement. Warehouses suit governed reporting; lakes suit exploration, machine learning and raw retention.
A lakehouse keeps data in open formats on object storage like a lake, while adding warehouse guarantees — ACID transactions, schema enforcement, versioning and performant SQL — using open table formats such as Delta Lake, Apache Iceberg or Apache Hudi.
Schema-on-read means data is stored without a predefined structure and interpreted at query time. It makes ingestion fast and flexible, but nothing enforces consistent meaning, so two datasets can describe the same entity differently.
Bronze holds raw ingested data unmodified as the reproducible record. Silver holds validated, deduplicated and conformed data where most analysis should start. Gold holds curated business-level tables aligned to agreed definitions for reporting.
A data swamp is a data lake that has filled with data nobody can find, interpret or trust, because cataloging, ownership and lifecycle management were deferred while ingestion continued. Users eventually revert to extracting from source systems.
Assign an owner to every dataset, require catalog metadata before anything lands, enforce the zone model rather than making it optional, maintain lineage, and retire datasets nobody queries on a schedule. These are governance decisions, not technical ones.
Use a lake when you have significant unstructured or semi-structured data, when machine learning needs raw detail, or when data must be retained before its use is known. Use a warehouse when the requirement is governed reporting on well-understood structured sources.
