From Data Swamps to Lakehouses: Architecting Modern Data Storage
David LazarevicQuick summary of this article
- Traditional databases are too rigid and expensive to easily scale for massive, varied data streams (like IoT data).
- A Data Lake is a cost-effective, scalable solution for storing raw structured, semi-structured and unstructured data on object storage using an ELT (Extract, Load, Transform) approach, without requiring strict schemas upfront.
- Because they lack strict schemas, Data Lakes suffer from slower query performance, lack row/column-level governance, and can easily devolve into unmanageable "Data Swamps."
- Data Lakehouses are the modern upgrade that fix these issues by introducing Open Table Formats (e.g., Apache Iceberg) and Metadata Catalogs on top of the Data Lake.
- Lakehouses bring warehouse-level reliability (ACID transactions, time travel, strict security) to cheap object storage, often organizing data via the Medallion Architecture (Bronze for raw, Silver for cleaned, Gold for aggregated).
- Because the architecture is fully decoupled, teams can mix and match specific tools for ingestion, storage, processing, and orchestration, avoiding vendor lock-in.
Storing data used to be a simple affair: you would spin up a database, define your schemas and call it a day. But as both the volume and the variety of the data we produce keep growing, so does the pain of squeezing all of it into rigid, predefined tables.
In this article, we will go over how the industry solved this problem, first with the Data Lake and then with its “next generation”, the Data Lakehouse, covering the architectural layers they are built on, the advantages and disadvantages they come with and the tooling used to build them.
The Problem
Let’s say that you are storing a lot of structured data and wish to integrate streaming IoT data into your database. After doing some research, you realize that it will be a pain to do so because:
- You will have to define schemas in order to store the data
- Updating and maintaining these schemas will become more and more complex over time
- You will have to implement a strict cleanup strategy
- It will require a lot of compute to process
- While it works for now, doing this can be a scalability bottleneck
And then you think to yourself:
“I wish there was a solution that allows me to store all types of data for cheap!”.
Thankfully, there is a solution like that, and it’s called a Data Lake!
What is a Data Lake?
A Data Lake is a cost-effective way to store large volumes of structured, semi-structured and unstructured data, with the end goal of making the process of ingesting, managing and querying the data for machine learning, business intelligence, data analytics and other use cases easier and more secure.
| Data Type | File Types |
|---|---|
| Structured | Database Tables (SQL), Spreadsheets,… |
| Semi-Structured | JSON Files, XML Files, Logs, YAML Files,… |
| Unstructured | Text Files, Emails, Images, Audio Files, Videos,… |
But how is a data lake any different from a traditional database or a data warehouse? Well, the main difference lies in the schema: databases and data warehouses operate on schema on write, meaning that the data has to fit a predefined schema before it can be stored, while a data lake operates on schema on read, storing the data in its raw format and enforcing a schema only once the data is being queried.
This, along with the other main differences between these data stores, can be seen in the following table.
| Feature | Database | Data Warehouse | Data Lake |
|---|---|---|---|
| Definition | Handling day to day transactions (OLTP) | Handling complex queries on large volumes of historical data (OLAP) | Cheaply storing a large volume of raw data |
| Data Structure | Highly structured | Highly structured | Raw, structured, semi-structured and unstructured |
| Schema Type | Schema on Write | Schema on Write | Schema on Read |
| Mainly Used By | IT administrator, application end-users, software developers | Business analysts, data analysts | Data scientists, data engineers, AI/ML |
| Storage | Stores data on Block Storage (Block Devices) | Stores data on Block Storage (Block Devices) | Stores data on inexpensive Object Storage |
| Speed and Latency | Fast single-row lookups/updates | Fast high volume and complex historical aggregations | Slower for regular queries but highly scalable for massive batch processing |
| Security and Governance | Average, User/Role Accounts | Advanced, RBAC support | Complex to implement and very basic |
Instead of doing ETL (Extract, Transform and Load), where we would be taking data, processing it so that it fits our predefined schema and then loading it into a database, a data lake relies on ELT.
ELT (Extract, Load and Transform), is what allows a data lake to easily scale and handle large volumes of both batch and streaming data which is made possible through the process of ingesting data from a source, storing it in its raw, unmodified format on object storage and then processing the data when doing queries on it (schema on read).
The ELT approach also comes with the added benefit of flexibility, as there is no schema enforcement, and since we are storing raw data we can store all types of it.
Besides allowing for flexibility and scalability, having the possibility to go back to the raw data and query it is a great addition, for example, if some data got corrupted during the processing step, this can turn out to be extremely useful.
Data Lake Architecture
A data lake is constructed of multiple layers, each serving a specific purpose, which can be seen in the previous example.
| Layer | Use Case |
|---|---|
| Ingestion | Fetching data, both streaming and batch, from various sources and then loading it raw (unmodified) into the storage layer. |
| Storage | Object storage holding the raw, unmodified data. |
| —————– | |
| Processing | Transforming the data so that it fits the schema enforced when reading/accessing it. |
| Access | Querying and analyzing the data stored within the data lake. |
| —————– |
Even though these layers serve different purposes, some of them can be done by one tool. For example, the access and the processing layer can be done by a query engine like Trino.
At the same time, it is common to use different tools for these purposes, which we will go over in the Lakehouse Tooling section.
The Ups and Downs of a Data Lake
Like any solution, a data lake comes with its own set of advantages and disadvantages, of which some will be talked about in the current section.
Cost Reductions
Depending on your specific situation, choosing the appropriate data store can save you a lot of headaches in the long run. To be able to pick the correct data store, you need to know which data you are going to be storing, but also the advantages and disadvantages of each data store, some of which were shown in the table found in the What is a Data Lake? section.
As seen in that table, compared to a traditional database or a data warehouse, both of which store their data on block storage (block devices), a data lake stores its data on object storage.
Block storage splits your data into fixed-size blocks and stores them on a device directly attached to a machine. This makes it extremely fast, but it also makes you pay for the capacity that you provision (and not for the data that you actually store), and scaling it usually means attaching bigger and faster disks.
Object storage, on the other hand, stores your data as objects, along with their metadata, and makes them accessible over HTTP. While slower, it is “infinitely” scalable, and you only pay for the data that you actually store on it.
And compared to block storage, object storage allows you to drastically reduce costs as you store more data on it, as seen in the following example.
Along with the cost reductions, a data lake running on object storage allows us to leverage core object storage features such as bucket replication, metadata support, object lock and versioning and many more.
These features have a direct impact on the lake itself: bucket replication gives you disaster recovery basically for free, versioning and object lock protect the raw data (your source of truth!) from accidental deletions and overwrites, and metadata support makes the life of the tooling running on top of the lake that much easier. In other words, features that we would otherwise have to implement ourselves are inherited straight from the storage layer.
This, in collaboration with the high volume capabilities of object storage and the compression capabilities of columnar file formats such as parquet, allows you to store way more data, more efficiently, and all of that while saving money.
Of course, while this is not an end all be all, it is extremely useful in streaming data, data analytics, BI or machine learning.
Flexibility & Scalability
In a data lake, both flexibility and scalability can be done on different levels, and both of these depend on the choices made during its implementation. For example, if you decide to go with Trino as the query engine, you will be able to scale the data lake further and query its contents faster due to it being a distributed query engine and you not being limited by its capabilities.
As you can observe in this chart, on a single node query engine, as time passes and data ingestion increases, the backlog on the node also increases drastically. Even if the engine is well optimized, being a single node engine means we’re limited by the hardware configuration of that one node. This results in a growing backlog, as the engine simply can’t keep up with the pace of data ingestion.
Here, however, you can observe that the engine keeps up with ingestion, and at times, its throughput even surpasses the rate of data being ingested. Even if the hardware configuration of individual workers is less powerful than that of the node running the single-node engine, as the distributed engine scales the number of workers to match the incoming data, throughput scales right along with it. And naturally, as throughput demand decreases, the number of workers scales down too, helping you save on infrastructure costs.
Of course, this does not mean that you are permanently stuck with the choice that you made, which is one of the reasons why a data lake is “flexible”. You are always able to swap out a component and introduce a new one, as if you go with open source solutions, there is no vendor lock-in holding you back. The other reasons include the data lake’s innate ability to ingest and query different types of data from various sources, and, of course, the ability to query this data using different query engines and tooling, as long as they are compatible with object storage.
By this point, the case for data lakes is (hopefully) compelling enough to serve as the perfect excuse to try building one.
Before doing so, however, you must know that with all of these great things that can come from a data lake, there are also disadvantages.
Lackluster Governance
So you just deployed a data lake for your company and everything is (or at least, seems to be) working great. The data ingestion works “flawlessly”, and everyone can access the data!
And then you come to the realization:
Wait a minute, everyone can access the data?
Yes! But it’s a little bit more complicated than that.
Data lakes struggle with data governance, meaning that even if you implement IAM policies so that certain users may query only the data that is necessary for them, you will “never” achieve column and row-level governance that you are used to in your databases.
“Mediocre” Performance
Compared to a traditional database or a data warehouse, a data lake can’t even begin to compare in a pure contest of speed.
While not having a predefined schema does mean that the data can be ingested faster, with minimal human effort, it also means that a schema will have to be defined when reading the data.
This, in turn, comes with the consequence of either requiring a lot of compute when doing queries or the queries taking more time to complete.
Difficult Maintenance & Data Swamps
While ingesting various files and storing them in your data lake is a great option to have, it can also cause difficulties trying to find the data you are looking for.
Getting to this situation is not uncommon, and a data lake that gets into this situation is called a Data Swamp.
Being un-navigable is one thing, however, in a data swamp we can also get into a situation where we are not sure whether the data quality is adequate and combining this with the lackluster governance “provided” by a data lake makes misuse extremely easy.
When viewing a data lake implementation as an iceberg (pun intended) you can think of the ingestion as the layer that sits above the water, as the most difficult parts that ensure that a data lake doesn’t turn into a data swamp are visible only after taking a deeper look into it.
With all of this in mind, however, you should now have a clearer picture of what a data lake is, what it can bring to the table and some of the drawbacks that it brings as well.
However, even for all of the aforementioned drawbacks, there exists a (relatively) simple solution: A Data Lakehouse!
What is a Data Lakehouse?
A Data Lakehouse is the “next generation” of a data lake, that fixes a lot of the downsides of one, and even adds new, highly useful features to it.
At its core, a lakehouse is still a data lake: the same raw data, stored on the same inexpensive object storage. The difference is the metadata layer sitting on top of it, consisting of open table formats, such as Apache Iceberg, and a catalog, which is what allows for advanced, warehouse-like features such as ACID Transactions, Time Travel Capabilities, Schema Evolution and Data Partitioning among other features.
Being a mix between a data warehouse and a data lake, it has the flexibility of a data lake with the performance and transactional capabilities of a data warehouse, making it an even better fit for the industry. And again, like a data lake, one of the key features of a Data Lakehouse is its ability to adapt to your specific use case in order to get the most out of it.
Data Lakehouse Architecture
A data lakehouse typically consists of the medallion architecture, however, this is a choice that can be made and comes with the disadvantage of the data taking up more storage.
The medallion architecture, a concept coined by Databricks, logically “splits” the storage in a lakehouse into three different layers.
Raw, unmodified data is stored in the Bronze layer and it is then processed and transformed into an OTF-compatible file format such as parquet or avro while validating and cleaning the data, in order to store it in the Silver layer.
From the silver layer, if needed for certain applications, data can be further aggregated, processed and sent into the Gold and final layer of the architecture, usually used by and optimized for BI, reporting and ML.
While taking up the highest amount of storage, this approach also speeds up the queries and simplifies access control, as we have the logical layer separation on where to point the users.
Of course, this is an architectural choice, and it is also possible to either not apply this architecture or apply your own version of it, for example, by storing raw data in the parquet format and then processing/aggregating it if needed.
On the other side, the medallion architecture allows for processing the raw data into multiple formats rather than just keeping the files in a single file format like parquet, orc, avro or others.
Having the ability to process data from its raw files into a silver bucket where they are transformed into a table format allows for using a table format best suited for that data.
For example, if you need fast analytical read queries, you would want to transform your data into a columnar format like parquet. Conversely, if you have write-heavy streaming data, a row-based format like avro might be a better fit.
For more information about the differenes between these file formats, you can check out this article from datacamp.
And for those of you with a keen eye, you may have noticed that two layers were not named in the table found in the Data Lake Architecture section. Metadata & Cataloging and Security & Governance layers were obfuscated due to them being present “only” in the architecture of a data lakehouse.
| Layer | Use Case |
|---|---|
| Ingestion | Fetching data, both streaming and batch, from various sources and then loading it raw (unmodified) into the storage layer. |
| Storage | Object storage holding the raw, unmodified data. In a lakehouse, this is often structured using the medallion architecture. |
| Metadata & Cataloging | Allows for schema evolution, makes finding the data “easy” and prevents the data lake from turning into a data swamp. |
| Processing | Transforming the data so that it fits the schema enforced when reading/accessing it. |
| Access | Querying and analyzing the data stored within the data lake. |
| Security & Governance | Allows for row or column level access control and ensures data privacy and compliance, and even allows for easier auditing. |
Both of these layers are done by a single tool, crucial for a lakehouse, called a catalog, which is basically a big inventory, containing references to all of the data in the lakehouse. It points the access layer to the metadata of the data being queried, enforcing the governance by ensuring that the user has the access rights necessary to query it.
A lot of these catalogs store the references to the metadata in a small relational database, as the catalogs themselves can be distributed so that they can serve the highest amount of users.
Besides ensuring that the data lake doesn’t turn into a data swamp, catalogs can enforce column and row level access control through RBAC and even allow for authentication through the use of IdPs.
IdPs - Identity Providers, are a centralized service, that usually generate a bearer token for a user accessing a connected service (service provider), like a catalog. The catalog then reads the token and uses the role inside of it in order to authorize the user, ensuring governance.
For more information on IdPs you can take a look at Authentik or Keycloak.
In addition to the aforementioned layers, two additional (unofficial) layers can be incorporated into the lakehouse architecture to create a fully functional, production-ready data lakehouse.
| Layer | Use Case |
|---|---|
| Orchestration | Orchestrates and automates the flow of the data, from the ingestion layer to the processing layer. |
| Data Quality | Ensures data quality in combination with the processing step, allowing for better governance of the data through making sure that it was modified properly and contains valid data. |
Lakehouse Table Formats
Open table formats (OTFs) are the core of a data lakehouse and, in combination with catalogs, offer a variety of features that take a lakehouse to the next level.
In short, an OTF is a metadata layer that sits on top of the raw data files (like parquet) in object storage, turning a “simple” folder of files into a proper table, with a schema, a history and transactional guarantees.
Time travel is one of these features, allowing you to query historical versions of your data at any given time. This is made possible through the metadata attached to the data files, which tracks changes and preserves previous states without duplicating the underlying data.
Another feature that relies on metadata is schema evolution. Schema evolution allows you to modify the schema of your tables (e.g., adding, dropping, or renaming columns) without needing to rewrite the underlying data files, saving you a massive headache in the long run.
Optimizing query speed across extremely large volumes of data is achieved through partitioning, which allows for grouping similar data based on specific attributes like timestamps, dates, or regions.
As for OTFs, the previously mentioned Apache Iceberg is the current “star of the show” in data engineering. However, there are also other established table formats such as Delta Lake and Apache Hudi.
While all of these formats have their own trade-offs, they can all be used to build a modern lakehouse, however, with different levels of ecosystem compatibility, community support, and specific features.
Since we talked about the industry giants, it’s also important to mention up-and-coming formats like DuckLake. DuckLake is a new format created by the founders of DuckDB that simplifies the lakehouse architecture by storing metadata directly in a standard relational database (like PostgreSQL or DuckDB itself), eliminating the need for thousands of complex metadata files while keeping the actual data safely in Parquet files on object storage.
Lakehouse Tooling
As previously talked about, one of the most exciting and beautiful things about data lakes (and lakehouses) is their flexibility. This, however, comes at a cost of there being simply too many tools that are made for a specific layer and use case, which can be seen in the following table.
| Layer | Tool |
|---|---|
| Ingestion | Airbyte (ELv2), Apache Kafka, Apache NiFi, Debezium, DLT, Meltano |
| Storage | Amazon S3, Azure Blob Storage, Exoscale SOS |
| Metadata & Cataloging | Apache Gravitino, Apache Hive Metastore (HMS), Apache Polaris, Databricks Unity Catalog, Project Nessie |
| Processing | Apache Flink, Apache Spark, Dask, dbt-core, DuckDB, Presto, Ray, SQLMesh, Trino |
| Access | Apache Flink, Apache Spark, ClickHouse, DuckDB, Presto, Trino |
| Security & Governance | Apache Gravitino, Apache Polaris, Databricks Unity Catalog, Project Nessie |
| Orchestration | Airflow, Dagster |
| Data Quality | dbt-expectations, Great Expectations, soda-core (ELv2), SQLMesh |
When building the lakehouse architecture, it is crucial to know which data you will be ingesting and the interval you will be ingesting it on.
Armed with this knowledge, you may then properly decide which tools to incorporate, for example, if you are dealing with a lot of streaming data, you might want to go with Apache Kafka and Apache Flink for your ingestion and processing layers, or, if you’re already deep into the Databricks ecosystem, you might want to opt for Delta Lake instead of Apache Iceberg as your OTF.
Most of the tools in the aforementioned table are completely open source, meaning that they are either under an MIT or an Apache license.
Tools under different licenses, such as the ELv2, have that mentioned next to them.
Conclusion
By this point, you should have a pretty clear picture of what data lakes and lakehouses are, the architectural layers they’re built on, and exactly what they can bring to the table.
While storing all types of data for cheap on object storage sounds amazing on paper, nobody wants to end up dealing with the headaches of an un-navigable data swamp or lackluster governance. That is exactly why the data lakehouse has become such a major player. It gives you that low-cost flexibility while keeping the performance and transactional capabilities you actually need in production.
If you want to see how to actually get your hands dirty building a lake from scratch, take a look at the step-by-step guide here: Building a Data Lake on Exoscale.
