What is a Data Lake ?

Publish Date: 2026-07-12   7 min read

A data lake is a central repository where you can hold and store all your data in its native format, until it’s needed for analytic applications — from dashboards and visualizations to big-data processing, real-time analytics, and machine learning — in order to guide better decisions.

Data lakes usually store data in files on object storage. The object store keeps metadata and a unique identifier alongside each object, which makes data easier to locate and retrieve across regions and improves performance — giving teams far more flexibility over how data is managed, stored, and used.

Why data lakes exist: the flexibility problem

Classic data warehouses are brilliant at answering known questions over clean, structured data — but they force you to decide the schema before you load anything (schema-on-write), and they get expensive fast at big-data scale. That’s a poor fit for the messy reality of modern data: clickstreams, JSON event logs, IoT telemetry, images, audio, PDFs, and tables that all need to live somewhere before anyone knows exactly how they’ll be queried.

A data lake flips the model. You land everything — structured, semi-structured, and unstructured — in its raw form, cheaply, and you impose structure only when you read it (schema-on-read). Nothing is thrown away for lack of a predefined table, and one copy of the data can feed many different workloads.

The building blocks

1. Object storage is the foundation

The storage layer is almost always cloud object storage — Amazon S3, Azure Data Lake Storage (ADLS Gen2), or Google Cloud Storage — or an on-premises equivalent such as HDFS or MinIO. Object storage is attractive because it decouples storage from compute (you pay for each independently and scale them separately), it’s cheap and effectively limitless, and it’s durable and globally accessible.

2. Open file formats do the heavy lifting

How you write the files matters enormously for cost and speed. The workhorses are:

• Apache Parquet — a columnar format that is the de-facto standard for analytics. Because it stores data column-by-column, a query that touches 3 of 50 columns reads only those 3, and compression is excellent.
• Apache ORC — another columnar format, common in the Hive/Hadoop world, with strong compression and indexing.
• Apache Avro — a row-based format that shines for streaming ingestion and record-by-record writes, and carries its schema with the data.

A simple rule: row formats (Avro) for writing/streaming, columnar formats (Parquet/ORC) for reading/analytics. Pair the right format with sensible partitioning (for example, laying files out by year/month/day) so engines can skip whole folders they don’t need.

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# Read raw JSON events landed in the lake, apply a schema on read...
events = spark.read.json("s3a://my-lake/raw/events/2026/07/")

# ...then write them back as partitioned Parquet for fast analytics
(events
    .write
    .partitionBy("event_date")
    .mode("append")
    .parquet("s3a://my-lake/curated/events/"))

Data lake vs. data warehouse

These aren’t rivals so much as different tools; many organizations run both. The essential differences:

The danger: from data lake to “data swamp”

A data lake’s greatest strength — dump anything, decide later — is also how it dies. Ingest data with no governance and the files pile up without documentation, ownership, or quality checks. Six months later nobody knows what final_v2_REAL.csv contains, whether it’s trustworthy, or who owns it. That’s a data swamp: plenty of data, no meaning.

Organizing the lake: the medallion (bronze / silver / gold) architecture

The most widely adopted pattern for keeping a lake usable is the medallion architecture — sometimes called a multi-hop architecture — which progressively improves data quality as it flows through three zones:

• Bronze (raw): data ingested exactly as received, in its original format, with no cleanup. This is your immutable source of truth — it lets you reprocess and audit later.
• Silver (cleansed): validated, de-duplicated, conformed and joined into a clean “enterprise view” of your key entities — the layer most analysts and data scientists actually work from.
• Gold (curated): business-level aggregates and features, optimized for a specific consumption pattern — the dashboards, reports, and ML models that drive decisions.

The evolution: from data lake to lakehouse

For years the trade-off was stark: lakes gave you cheap, flexible storage but no transactions, no reliable schema enforcement, and no easy updates; warehouses gave you reliability but at cost and rigidity. The lakehouse closes that gap by adding a metadata/transaction layer on top of the files in your lake — so you get warehouse-grade guarantees (ACID transactions, schema evolution, time travel, efficient upserts and deletes) directly on cheap object storage.

That layer is delivered by open table formats, and this is where the field has moved fastest:

• Apache Iceberg — has emerged as the industry-standard interchange format, now supported by essentially every major cloud, query engine, and platform. Its snapshot-and-manifest design excels at pruning huge datasets.
• Delta Lake — the Spark/Databricks-native format built on a transaction log; its UniForm layer now lets Delta tables be read as Iceberg for interoperability.
• Apache Hudi — specialized for record-level updates and streaming ingestion, where heavy upserts and deletes are the norm.

Newer entrants like Apache Paimon (streaming-first) and DuckLake (which keeps table metadata in a plain relational database) show the space is still evolving — but the headline is clear: a modern data lake with an open table format is a lakehouse.

When does a data lake make sense?

Reach for a data lake (or lakehouse) when you have large volumes of varied data — especially semi-structured or unstructured — that you want to keep cheaply and explore later; when you’re feeding machine learning or data science, which thrive on raw, granular data; when you need to decouple storage cost from compute; or when you want one governed copy of data to serve BI, analytics, and ML at once. If your needs are purely fast, structured, well-understood reporting, a warehouse alone may still be the simpler answer.

Conclusion

A data lake, at its core, is a simple and powerful idea: store everything cheaply, in open formats, and decide how to use it later. The catch is that flexibility without discipline turns a lake into a swamp — so the real craft is in the governance, the zoning (bronze/silver/gold), and the open table format that together turn a pile of files into a trustworthy, query-ready platform. Master those, and the lake stops being a storage bill and becomes the foundation your analytics and ML actually stand on.

Designing or untangling a data lake and not sure where to start? I work on exactly this — feel free to get in touch.

References & further reading

• AWS — What is a Data Lake?
• Databricks — What is the Medallion Architecture?
• Apache Iceberg — open table format
• Delta Lake — the Linux Foundation project
• Apache Hudi — streaming data lake platform
• Apache Parquet — columnar storage format
• Apache Polaris — open lakehouse catalog