Fundamentals of Data Architecture

Introduction

Someone on r/dataengineering asked if there’s a standard for modern data architecture. With 20 years of experience, they had a batch pipeline pulling 200MB files into a data lake and a new source as a serialized Kafka stream. They wondered if streaming was the new paradigm or if they were becoming dinosaurs.

The best answer cut to the confusion: the question asked about architecture, but the follow-up discussed tools. They are on different levels. Data architecture has little to do with tool selection.

Data architecture refers to decisions on data flow, including its storage, transformation, single source of truth, and how downstream products access it. The tools used are interchangeable; the structure is enduring.

This distinction matters because most “architecture” arguments are tool arguments in disguise. If a discussion about data architecture starts with “what do you think about dbt?”, it misses the point. Tools are some of the least important decisions.

What this is (and isn’t): This article explains the durable parts of data architecture, the layered model that has held up for over 30 years, why a system of record anchors correctness, and how ingestion style (batch or streaming, pull or push) fits into that model. It doesn’t teach a specific tool, and it won’t tell you which vendor to buy. It gives you a mental model so you can evaluate any tool against it.

Why data architecture fundamentals matter:

• Stability over churn - The layered model survives tool migrations, so you redesign less often.
• A clear source of truth - Knowing where truth lives stops reports from disagreeing with each other.
• Faster onboarding - A predictable structure means new engineers know where any piece of data should sit.
• Better tool decisions - When you understand the structure, you can judge whether a tool serves it or fights it.

Understanding these fundamentals shifts you from chasing trends to designing systems that adapt to new sources and tools without rewrites.

This article outlines a basic flow that every analytical system follows:

1. Land the raw data without changing it (staging).
2. Integrate it into one trusted version (core).
3. Shape it into purpose-built products (semantic).
4. Serve those products to consumers.

Type: Explanation (understanding-oriented).
Primary audience: beginner to intermediate engineers and analysts who want a durable mental model for how data systems are organized.

Prerequisites & Audience

Prerequisites: Basic familiarity with databases and SQL, and some exposure to moving data between systems. If pipelines are new to you, read Fundamentals of Data Engineering first, since it covers the mechanics this article assumes.

Primary audience: Engineers and analysts who can build pipelines but want to understand the structure those pipelines should serve. Experienced practitioners may use it to align vocabulary with a team.

Jump to:

• Architecture Is Not Tools • The Three-Layer Standard • The System of Record
• Inmon and Kimball • How Data Arrives • Contracts and Delivery Guarantees
• Common Mistakes • Common Misconceptions • When Not to Over-Architect
• Laws, Bias, and Fallacies • Future Trends • Glossary

If you are new to the topic, read from top to bottom. If you already know the layered model and only care about the streaming question, jump to How Data Arrives.

TL;DR – Data Architecture in One Pass

If you remember one thing, make it this: architecture is the shape of the flow, not the tools that carry it.

• Three layers (staging, core, semantic) so each stage has one job and you always know where data belongs.
• A system of record so “correct” has a definition: the warehouse is right when it balances back to source.
• Truth in the core so every data product starts from the same integrated base and reports agree.
• Ingestion is a detail so batch, streaming, pull, or push are all just ways to get data into staging.

The flow looks like this:

Learning Outcomes

By the end of this article, you will be able to:

• Explain why data architecture and tooling are different decisions made at different times.
• Describe why the staging, core, and semantic layers each exist and what belongs in each.
• Explain why a system of record defines correctness, and how data balances back to it.
• Decide when to model the cbore with Inmon-style normalization versus Kimball-style dimensions.
• Explain why the ingestion style (batch, streaming, pull, push) is a staging-layer detail, not an architecture change.
• Recognize the contracts and delivery guarantees that any data exchange depends on.

Section 1: Architecture Is Not Tools

The core idea is that architecture describes structure, and data tools execute it. Mixing the two is the most common reason data conversations go in circles.

Think of building a house. The architecture is the floor plan: how many rooms there are, how they connect, which walls carry the load, and how water and power flow. The tools are the brand of drill, the lumber supplier, and the contractor you hire. You can swap every tool and still build the same house. You cannot swap the floor plan halfway through without tearing things down.

Understanding the Distinction

Architecture decisions are slow to change and expensive to reverse. Where does raw data land? What is the single source of truth? How do products read from it? These shape everything downstream.

Tool decisions are fast to change and cheap to reverse, at least relative to architecture. Spark or Flink, Snowflake or BigQuery, Airflow or Dagster. Each does roughly the same job in roughly the same place in the flow.

When you treat a tool choice as an architecture choice, you tie your structure to a vendor. When the vendor disappoints you, the structure goes with it.

Why This Works

A good architecture is tool-agnostic on purpose. The layers and the source of truth stay fixed while tools come and go underneath. You replace orchestrators, query engines, and storage formats inside the same architecture without the analysts noticing, because the shape of the flow never moved.

This is also a useful filter for design reviews. The acid test from that Reddit thread holds up: if someone describes their architecture entirely in product names, they are describing a toolbox, not a design.

Trade-offs and Limitations

Staying tool-agnostic has a cost. Generic structures sometimes ignore a tool’s best feature, and an abstraction layer that hides the tool can add complexity. The goal is not to avoid tools; it is to avoid letting a tool dictate the structure.

Quick Check: Architecture vs Tools

Before moving on, test your understanding:

• Is “the team uses dbt for transformations” an architecture statement or a tool statement?
• If you replaced Spark with Flink tomorrow, would your layer structure change?
• Where in your current system does the single version of truth live?

Answer guidance: Ideal result: You can name the layers of your system without naming a single product, and you can say which decisions would survive a full tool swap. If you can only describe your system through product names, start by drawing the flow instead.

Section 2: The Three-Layer Standard

The durable standard for analytical data has three layers: staging, core, and semantic. It has been around for over 30 years, and it has lasted because it works.

Picture a factory. Raw materials arrive at the loading dock (staging). The factory floor turns those materials into a finished product (core). Packaging wraps the product into ready-to-use units for specific customers (semantic). Each stage has one job, and you never confuse the loading dock with the shipping department.

Staging: Land the Data Untouched

Staging is where data lands when it first enters the warehouse. The rule here is simple: do not mess with the data when it arrives. Staging is your main link back to the source system, so it should look as much like the source as possible.

Many teams aim for a 1:1 mapping between each source table or file and its staging table. The data is raw and untouched. You can also use staging as a scratch pad while you prepare data for the next layer, but the landed copy stays faithful to the source.

Core: The Integrated Version of Truth

Core is where the integrated truth lives. Every input ends up here, and every data product starts here. Because all consumers read from a common base, reports built on the core have a much better chance of agreeing.

The core needs to serve every purpose, which means it should favor none. A well-designed core is purposeless: it stores integrated facts without bending them toward any single report. The purpose gets added later, in the semantic layer.

Semantic: Where Data Gets Its Purpose

Semantic is where data products live. These can be views on the core, materialized views, star schemas, or wide tables. This is where the purposeless core data gets a purpose, a grain, and a shape tuned for how people will actually query it.

The Medallion Rebrand

If “staging, core, semantic” sounds unfamiliar but “bronze, silver, gold” rings a bell, you already know this model. The medallion naming popularized by Databricks maps almost directly onto the older layers:

The medallion names are friendlier, and they have helped a lot of teams adopt the model. But the concept is not new. Knowing the older vocabulary keeps you from believing a rebrand is a breakthrough.

Trade-offs and Limitations

Three layers add storage and processing cost, since each stage copies and reshapes the data. Small systems may not need the full separation. Some teams also add an intermediate sub-layer between core and semantic to split complex transformations into atomic, testable steps. The number of layers can flex; the principle that each layer has one clear job should not.

Quick Check: The Three Layers

Test your understanding:

• Which layer should never modify the data, and why?
• Why does putting truth in the core help reports agree?
• What is the relationship between “semantic” and “gold”?

Answer guidance: Ideal result: You can describe staging as the faithful landing zone, core as the integrated and purposeless truth, and semantic as the purpose-built product layer, and you recognize medallion as the same model with newer names.

Section 3: The System of Record and What “Correct” Means

A system of record (SoR) is the authoritative source for a given piece of data: the application or database where that data is created and owned. It defines what “correct” means for your warehouse.

This gives you a precise, testable definition of correctness: the warehouse is correct when it balances back to the systems of record. If a number in a report is wrong, you do not patch it in the warehouse. You fix it in the source system and let the corrected data flow back out.

Why This Works

Patching data inside the warehouse feels faster, but it breaks the link to the source. The next time data flows, the patch is overwritten or, worse, it conflicts with the source, and nobody can tell which number is real. Fixing at the source keeps one authoritative answer.

This is why staging matters so much. Because staging is a faithful copy of the source, it is the evidence you use to prove the warehouse balances. If you “clean” staging on the way in, you lose your audit trail.

When the Source Is Shared

A harder case, raised in that same thread, is a system like a customer relationship management (CRM) tool where ownership is split and data syncs in both directions. When two systems can both write the same record, “the source of truth” is no longer a single place. You then need an explicit rule for which system wins for which field, and a reconciliation process when they disagree. The principle holds, but you have to declare the SoR field-by-field rather than system-by-system.

Quick Check: Systems of Record

Test your understanding:

• Where should you fix a wrong value that appears in a report?
• Why does cleaning data during staging weaken your ability to balance back to source?
• What changes when two systems can both write the same field?

Answer guidance: Ideal result: You fix errors at the source, you keep staging faithful so it can serve as evidence, and you recognize that shared ownership forces a field-level source-of-truth decision.

Section 4: Inmon and Kimball – Where Each Belongs

Bill Inmon and Ralph Kimball gave the field two influential approaches to warehouse modeling. The long-running “which is right” debate mostly dissolves once you place each one in the right layer.

• Inmon favors a normalized, enterprise-wide core, often modeled in third normal form (3NF). Integrate first, then build marts from the integrated base.
• Kimball favors dimensional models (star schemas with facts and dimensions) built for how the business asks questions. Model for the query, then integrate through shared dimensions.

Why This Works as Layers, Not Rivals

Lean Inmon for the core and Kimball for the semantic layer, and the two stop fighting. The core integrates everything in a normalized, purposeless form so it can answer any future question. The semantic layer reshapes that core into dimensional products tuned for specific questions.

Third normal form (3NF) is a normalization level that removes redundant data so each fact is stored once. It suits the core because it keeps the integrated truth consistent. A star schema, by contrast, deliberately denormalizes for fast, intuitive querying, which suits the semantic layer.

Trade-offs and Limitations

A fully normalized core is more work to build and slower to query directly, which is exactly why you do not expose it to end users. Some smaller teams skip the normalized core and model dimensionally from the start; that trades long-term flexibility for speed today. Neither approach is wrong, but choosing one without knowing which layer you are in usually is.

Quick Check: Inmon and Kimball

Test your understanding:

• Which approach fits the core layer, and why?
• Why would you not expose a fully normalized core directly to analysts?
• How do shared dimensions help separate marts stay consistent?

Answer guidance: Ideal result: You see Inmon and Kimball as complementary choices for different layers rather than competitors, and you can justify normalization in the core and dimensional shaping in the semantic layer.

Section 5: How Data Arrives – Pull, Push, Batch, and Streaming

Ingestion is how data gets into staging. This is the part of the Reddit question that caused real anxiety, so it deserves a clear answer: the way data arrives is a staging-layer detail, not a change to your architecture.

There are two axes that people tend to tangle together.

• Pull vs push: Does your system reach out and fetch the data (pull), or does the source send it to you (push)? Pull often needs only credentials, which is why many engineers prefer it. Push frequently needs the owning team to add you as a consumer, which can mean months of meetings.
• Batch vs streaming: Do you move data in scheduled chunks (batch), or as a continuous flow of events (streaming)? Batch favors throughput and simplicity; streaming favors low latency.

Pub/Sub Is Not New

When a team says their data is only available on a Kafka topic, they are describing a publish-subscribe (pub/sub) architecture. The publisher owns the cadence of when messages are emitted, and subscribers decide when to consume them. Kafka is one way to build pub/sub; it solves throughput and replay well, at the cost of new infrastructure.

This pattern is decades old. You can build pub/sub on a database table, a distributed file system, or UDP multicast, each with its own trade-offs. Streaming events into a landing zone is not a new paradigm. It is an old idea with good tooling.

Streaming Into a Batch Data Lake

Here is the part that reassures the 20-year veteran: you do not have to rebuild your batch lake to consume a stream. A common pattern is to bridge the stream into your existing batch flow.

If you only need the data once a day, you can run a batch job on your own schedule that reads from the topic, records its position (offset), and writes the raw events into staging. You do not have to become a streaming shop to read from a stream.

The Small-Files Problem

The veteran’s specific fear was real: a billion rows a day arriving as tiny events could produce millions of kilobyte-sized files, which wrecks read performance on a data lake. The fix is to control file size at write time. Set a flush size so each landed file holds many events (for example, batches of thousands), or run a compaction step after landing to merge small files into large ones. The deserialization from the wire format into something queryable happens in a downstream layer, not at the moment of landing.

Quick Check: Ingestion

Test your understanding:

• Why is streaming from Kafka a staging detail rather than an architecture change?
• How can you read from a streaming topic without running a continuous streaming pipeline?
• What causes the small-files problem, and how do you avoid it?

Answer guidance: Ideal result: You can place ingestion in the staging layer, describe a scheduled batch pull that commits its offset, and explain that flush size or compaction keeps files large enough to query efficiently.

Section 6: Contracts and Delivery Guarantees

Any data exchange between teams rests on contracts and delivery guarantees. These matter most at the staging boundary, where data crosses from someone else’s system into yours.

The Two Cross-Team Contracts

In a pub/sub exchange, two agreements do most of the work:

• The message format - the schema and serialization of each message so that you can read it.
• The retention policy - how long messages stay available, which sets your window to recover from a failure.

Serialization means turning structured data into bytes for transport. Kafka itself is agnostic to format; it moves bytes from one place to another. That means you have to deserialize on your side, which requires knowing the format. Plain JSON carries its own field names. Compact binary formats like Avro or Protobuf are smaller but need the schema, often pulled from a schema registry, to decode.

Delivery Guarantees

When a source delivers data as events, you inherit a choice about how often each event can arrive:

• At most once - events may be lost but never duplicated.
• At least once - events are never lost but may be duplicated, so consumers must handle duplicates.
• Exactly once - events arrive once and only once, which is the most expensive to guarantee.

Most systems land on at-least-once and make downstream processing idempotent, meaning reprocessing the same event does not change the result. If event order matters to you, that is a separate property you have to ask for explicitly.

Why This Works

Contracts turn a fragile integration into a durable one. With a fixed format and a known retention window, the subscriber owns its own errors: if a load fails, you have until the retention window closes to replay the messages and recover. Without those contracts, every source change is a surprise outage.

Quick Check: Contracts and Guarantees

Test your understanding:

• What two contracts does a pub/sub integration depend on?
• Why does at-least-once delivery push you toward idempotent processing?
• What does a schema registry give you that raw JSON does not?

Answer guidance: Ideal result: You can name format and retention as the core contracts, explain idempotency as the answer to duplicate delivery, and describe a schema registry as the source of the schema needed to decode compact binary formats.

Section 7: Common Data Architecture Mistakes

Most architecture failures are not exotic. They come from a handful of repeated mistakes that blur the layers or break the source of truth.

Mistake 1: Treating Tool Selection as Architecture

Choosing a tool first and calling it a design is the most common error. The structure ends up shaped by the tool’s defaults rather than by your data flow, and a future migration becomes a rewrite.

Instead: Draw the layers and the source of truth first. Choose tools to fit that flow last.

Mistake 2: Transforming Data in Staging

Cleaning or reshaping data as it lands breaks the faithful link back to the source. You lose the evidence you need to prove the warehouse balances.

Instead: Land raw and untouched. Do transformations on the way to the core, where they are visible and reversible.

Mistake 3: Building Products Straight From Staging

Skipping the core and building reports directly on raw staging data means every report integrates the data in its own way. Reports then disagree, and nobody can say which is right.

Instead: Route everything through an integrated core so all products share one base.

Mistake 4: Patching the Warehouse Instead of the Source

Fixing a wrong number in the warehouse creates a value that the source system will never reproduce. The next load either overwrites your fix or contradicts it.

Instead: Fix at the system of record and let the correction flow downstream.

Quick Check: Common Mistakes

Test your understanding:

• Why is staging the wrong place to clean data?
• What goes wrong when reports are built directly on staging?
• Where should a data correction originate?

Answer guidance: Ideal result: You can tie each mistake back to a broken principle: layer separation, the faithful staging copy, or the system of record.

Section 8: Common Misconceptions

A few persistent misconceptions make data architecture harder to reason about than it needs to be.

• “Modern architecture means new tools.” Tools change constantly; the layered structure underneath has barely moved in 30 years. New tooling is not new architecture.

• “Medallion is a fresh approach.” Bronze, silver, and gold are friendlier names for staging, core, and semantic. The model predates the names by decades.

• “Streaming replaces batch.” Streaming and batch are different ingestion styles for different latency needs. Most systems run both, and a stream can feed a batch lake.

• “Inmon and Kimball are rivals you must choose between.” They fit different layers. Normalize the core, model the semantic layer dimensionally, and the debate mostly disappears.

• “The warehouse is the source of truth.” The warehouse reflects truth; the systems of record own it. The warehouse is correct only when it balances back to them.

Section 9: When Not to Over-Architect

The full three-layer model is not mandatory for every project. Forcing it onto a small problem adds cost and ceremony with no payoff.

A single small data source feeding one dashboard may need only a direct query or a thin staging copy, not a normalized core.

Early-stage exploration where requirements are still moving rarely benefits from a heavily modeled core; you will remodel it before it stabilizes.

Low data volume that one machine can process comfortably does not need distributed processing or layered lakes.

A throwaway analysis that answers one question and gets deleted does not need an architecture at all.

Even when you skip the full model, keep the cheapest principle: land a faithful copy of the source before you transform it. That single habit preserves your ability to start over when the small project unexpectedly grows.

Laws, Bias, and Fallacies

How people reason about data architecture shapes the decisions as much as the technology does. A few named effects and thinking traps show up again and again.

Laws and Named Principles

• Conway’s Law - Systems mirror the communication structure of the organization that builds them. Gist: org charts leak into architecture. How it shows up: the push-versus-pull standoff in the Reddit thread was an org problem (the other team was too busy) wearing a technical mask. Data boundaries often follow team boundaries.
• Goodhart’s Law - When a measure becomes a target, it stops being a good measure. Gist: Optimizing a proxy corrupts it. How it shows up: teams that chase “freshness in seconds” as a headline metric sometimes degrade the correctness that actually matters.
• Inmon and Kimball as named approaches - Two foundational warehouse philosophies. Gist: normalize for integration, denormalize for querying. How it shows up: placing each in its layer ends a debate that has consumed careers.

Bias

• Appeal-to-novelty bias - Treating newer as better. What goes wrong: a rebrand like Medallion gets adopted as a breakthrough, and teams discard hard-won vocabulary. What to do instead: ask what concept a new term maps onto before assuming it is new.
• Availability bias - Overweighting whatever you have seen most recently. What goes wrong: an engineer who just read about streaming assumes streaming is the standard everywhere. What to do instead: separate what is common in your feed from what fits your latency needs.
• Sunk-cost bias - Sticking with a tool because of past investment. What goes wrong: a structure stays welded to a fading vendor because replacing it feels wasteful. What to do instead: judge the tool against the flow it is supposed to serve, not against what it cost.

Fallacies

• False dichotomy - Framing a choice as either-or when both can coexist. Scenario: “batch or streaming?” When most mature systems run both, a stream can feed a batch lake.
• Appeal to authority - Accepting a claim because a large vendor said it. Scenario: adopting a layering scheme purely because a platform’s marketing named it, without checking it against your own data flow.
• Moving the goalposts - Redefining success mid-argument. Scenario: a tool debate that started about maintainability ends up about raw speed once the first claim is challenged.

Building Data Architecture That Lasts

The pieces fit together into one coherent picture: a faithful landing zone, an integrated source of truth, purpose-built products, and a clear definition of correct.

Key Takeaways

• Architecture is structure, not tools - Design the flow first; choose tools last.
• Three layers, one job each - Staging lands, core integrates, semantic shapes.
• The source of record defines correct - The warehouse is right when it balances back to the source.
• Ingestion is a staging detail - Batch, streaming, pull, or push all fill staging.
• Contracts make integrations durable - Format, retention, and delivery guarantees turn surprises into routine.

How These Concepts Connect

Staging exists to preserve the link to the system of record, which is what lets you define correctness. The core integrates those faithful copies into one truth, which is why reports built on the semantic layer agree. Ingestion style decides only how staging gets filled, so a new Kafka source is a staging change, not a redesign. Each idea protects the next.

Getting Started with Data Architecture

If you are designing a system, start narrow and let it grow into the model:

1. Identify the systems of record for the data you care about.
2. Land raw copies in a staging area without transforming them.
3. Integrate those copies into a small core that favors no single report.
4. Build one product in a semantic layer on top of the core.
5. Write down the contracts for any external source: format, retention, delivery guarantee.

Once that loop feels routine, the same shape absorbs new sources, including streaming ones, without a rewrite.

Self-Assessment – Can You Explain These in Your Own Words?

Before moving on, see if you can explain these out loud:

• Why is data architecture a different decision from tool selection?
• What is the job of each layer, and why does the order matter?
• How does a system of record give “correct” a precise meaning?
• Why is consuming a Kafka stream a staging concern rather than an architecture change?

If you can explain these clearly, you have internalized the fundamentals.

Future Trends

Data architecture vocabulary keeps churning, but the underlying structure changes slowly. A few trends are worth tracking without mistaking them for new foundations.

Data Mesh and Decentralized Ownership

Data mesh pushes ownership of data products out to the teams that produce the data, rather than centralizing everything in one warehouse team.

What this means: The source-of-truth and contract questions become organizational, not just technical. Conway’s Law moves to center stage.

How to prepare: Get rigorous about contracts now. A decentralized model only works when each producer publishes a clear, stable interface.

The Lakehouse and Open Table Formats

The lakehouse blends the cheap storage of a data lake with the structure and transactions of a warehouse, often through open table formats.

What this means: The boundary between “lake” and “warehouse” blurs, but the staging, core, and semantic layers still apply inside it.

How to prepare: Learn the layered model first. It tells you where any new format belongs, rather than the format dictating your design.

Glossary

Data architecture: The set of decisions about how data flows through a system: where it lands, how it changes shape, where the single version of truth lives, and how products read from it. Distinct from the tools that execute it.

Staging layer: The landing zone where data first enters the warehouse, kept as a faithful, untouched copy of the source. Often a 1:1 mapping to source tables or files, used as the link back to the system of record.

Core layer: The integrated layer where the single version of truth lives. Every input lands here and every data product starts here, modeled so it favors no single purpose. Also called silver in medallion terms.

Semantic layer: The layer where integrated core data is reshaped into purpose-built data products (views, materialized views, star schemas, wide tables) tuned for how people query. Also called gold in medallion terms.

Medallion architecture: A bronze, silver, gold naming for data lake layers popularized by Databricks. Maps almost directly onto the classic staging, core, and semantic layers rather than introducing a new model.

System of record (SoR): The authoritative source where a piece of data is created and owned. It defines correctness: a warehouse is correct when it balances back to its systems of record.

Inmon approach: A warehouse design philosophy favoring a normalized, enterprise-wide core (often third normal form) that integrates data first, then builds marts from the integrated base.

Kimball approach: A warehouse design philosophy favoring dimensional models (star schemas of facts and dimensions) built for how the business asks questions, integrating through shared dimensions.

Third normal form (3NF): A normalization level that removes redundant data so each fact is stored once. Suits the core layer because it keeps integrated truth consistent.

Star schema: A dimensional model with a central fact table joined to surrounding dimension tables. Deliberately denormalized for fast, intuitive querying, which suits the semantic layer.

Publish-subscribe (pub/sub): A messaging pattern where publishers emit messages on a topic and subscribers consume them independently. The publisher owns cadence; subscribers decide when to read. Kafka is one implementation.

Batch processing: Moving data in scheduled chunks rather than continuously. Favors throughput and simplicity over latency.

Streaming: Processing data as a continuous flow of events rather than scheduled chunks. Favors low latency at the cost of added infrastructure and complexity.

Serialization: Turning structured data into bytes for transport, and deserialization the reverse. Consumers must know the format (for example JSON, Avro, or Protobuf) to decode the bytes.

Schema registry: A service that stores message schemas so consumers can decode compact binary formats like Avro or Protobuf, which do not carry field names inline the way JSON does.

Delivery guarantee: The promise about how often each event arrives: at most once (may lose, never duplicate), at least once (never lose, may duplicate), or exactly once (once and only once).

Idempotency: A property where processing the same event more than once produces the same result. The common answer to at-least-once delivery, since duplicates then cause no harm.

Compaction: Merging many small files into fewer large ones to keep read performance high on a data lake. An alternative to controlling flush size at write time.

Data mesh: An organizational approach that decentralizes data ownership to the teams that produce the data, each publishing data products through clear, stable contracts.

Lakehouse: An architecture that blends the cheap storage of a data lake with the structure and transactions of a warehouse, often via open table formats. The layered model still applies inside it.

References

Related Articles

Related fundamentals articles:

Data and storage: Fundamentals of Data Engineering covers the pipeline mechanics that move data through these layers. Fundamentals of Databases explains the storage and retrieval concepts the core layer depends on. Fundamentals of Data Analysis shows how the semantic layer’s products get used. Fundamentals of Data Structures grounds the shapes data takes inside these systems.

Infrastructure: Fundamentals of Distributed Systems explains how large-scale ingestion and processing spread across machines. Fundamentals of Networking underpins how data moves between source systems and your staging zone.

Production systems: Fundamentals of Monitoring and Observability is how you detect when a layer stops balancing back to source. Fundamentals of Reliability Engineering helps you set expectations for pipeline freshness and recovery.

Foundational Sources

• Inmon, W. H. (2005). Building the Data Warehouse. Wiley. The classic argument for a normalized, integrated core.
• Kimball, R., & Ross, M. (2013). The Data Warehouse Toolkit. Wiley. The definitive guide to dimensional modeling for the semantic layer.
• Kleppmann, M. (2017). Designing Data-Intensive Applications. O’Reilly Media. Deep coverage of batch, streaming, and the trade-offs between them.

Practical Resources

• Apache Kafka Documentation for delivery guarantees, retention, and consumer offsets.
• Confluent: Delivery Guarantees, for a clear explanation of at-most-once, at-least-once, and exactly-once semantics.
• Databricks: Medallion Architecture, for the bronze, silver, and gold framing and how it maps to the classic layers.

Note: Data architecture vocabulary evolves faster than the underlying structure. Treat new names with curiosity, but check them against the layered model before assuming they are new.