Single Source of Truth Across 10+ Systems: How a Logistics Operator Unified Its Data in a Databricks Lakehouse

A principal data architect’s field account of how a logistics operator collapsed ten-plus
operational systems into one governed Databricks Lakehouse — turning multi-hour reporting into
minutes that operations and finance both trust, establishing a single source of truth Databricks Lakehouse.

Few industries generate as much operational data as logistics — and few struggle as much to
turn it into a single, trusted answer. One shipment touches order management, a transportation
management system (TMS), a warehouse management system (WMS), telematics on the trucks, fuel cards,
customs brokerage, EDI feeds from partners, and the finance ledger that must eventually reconcile all
of it. When each of those systems answers the same question differently, the business stops trusting
its own numbers, and decision-making slows to the speed of the next reconciliation meeting.

This is a field account of how one mid-sized logistics operator moved from ten-plus contradictory
systems to a single, governed Databricks Lakehouse. The headline outcome is easy to state and hard to
earn: report generation fell from hours to minutes, and operations and finance began
working from one shared, trusted view, creating a single source of truth Databricks Lakehouse. The engineering discipline that delivered it is the same one
our team applies on every data
engineering engagement — and it generalises well beyond logistics.

A word on method before the build. Consolidation projects fail most often not for want of tooling,
but because teams start moving data before they agree on what the data means. The approach below
inverts that: definitions first, immutable raw data second, conformance and governance as first-class
features, and performance engineering last. Applied in that order the layers compound; applied as a
rush to a dashboard, they quietly drift back into the fragmentation they were meant to cure.

Implementing a single source of truth Databricks Lakehouse can drastically improve data accuracy and trust across an organization.

01Start with definitions, not pipelines

The operator had grown by acquisition and by adding best-of-breed tools as needs arose — a
normal way to scale that nonetheless left more than ten systems each with its own schema, refresh
cadence, and definition of basic terms. Operations measured “on-time delivery” from the
TMS event log; finance measured it from invoiced milestones. The two numbers were never the same, and
review meetings dissolved into arguments about whose spreadsheet was correct.

The first deliverable on the project was therefore not code. It was a signed-off glossary: a single
authoritative definition for “active shipment,” “cost per mile,” “on-time,”
and “customer,” agreed jointly by operations and finance. Without that, any pipeline simply
encodes one team’s assumptions and guarantees the other will distrust the result. A
single source of truth is a governance achievement before it is a technical one.

02Land every system in Bronze, unchanged

With definitions agreed, each of the ten-plus sources lands in a Bronze layer as
raw, append-only Delta Lake tables, with
ingestion metadata — source system, load timestamp, batch identifier — attached to every
record. Bronze is deliberately faithful to the source: no business logic, no cleaning. This gives a
complete, replayable history, so when a downstream definition changes the platform can be reprocessed
from raw without re-extracting from production systems. Databricks
Auto
Loader handles incremental file ingestion with schema inference and evolution built in.

(spark.readStream
   .format("cloudFiles")
   .option("cloudFiles.format", "json")
   .option("cloudFiles.schemaEvolutionMode", "addNewColumns")
   .load("/landing/tms_events/")
   .withColumn("_source_system", lit("tms"))
   .withColumn("_ingested_at", current_timestamp())
   .writeStream
   .option("checkpointLocation", "/chk/bronze/tms_events")
   .option("mergeSchema", "true")
   .toTable("logistics.bronze.tms_events"))

03Conform and resolve identities in Silver

The Silver layer is where fragmentation is actually defeated. Here we standardise
data types and time zones, deduplicate, and — most importantly — resolve the same
real-world entity across systems. A carrier identified one way in the TMS and another way in the
finance ledger is mapped to a single conformed key. The glossary from step one is encoded here, once,
in version-controlled transformation logic rather than in a dozen private spreadsheets. A
MERGE
keeps Silver idempotent, so replays and late-arriving corrections never produce duplicates.

MERGE INTO logistics.silver.shipments AS tgt
USING (
  SELECT
    xref.conformed_carrier_id,
    b.shipment_id,
    CAST(b.pickup_ts AS TIMESTAMP) AS pickup_ts,
    b.status,
    b._source_system
  FROM logistics.bronze.tms_events b
  JOIN logistics.ref.carrier_xref xref
    ON xref.source_carrier_id = b.carrier_id
   AND xref.source_system     = b._source_system
) AS src
ON  tgt.shipment_id = src.shipment_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

04Publish the single source of truth in Gold

The Gold layer exposes curated, business-ready tables aligned to how the
organisation actually asks questions: shipments, lanes, cost per mile, on-time performance, and revenue
by customer. The decisive point is that operations and finance now read from the same Gold
tables. When both teams pull on-time delivery, the number is identical because it is computed once,
from conformed data, against the agreed definition. That — one definition, one number — is
what “single source of truth” means in practice.

CREATE OR REPLACE TABLE logistics.gold.on_time_performance AS
SELECT
  date_trunc('day', s.pickup_ts)          AS ship_date,
  s.lane_id,
  COUNT(*)                                 AS shipments,
  AVG(CASE WHEN s.delivered_ts <= s.promised_ts
           THEN 1 ELSE 0 END)              AS on_time_rate
FROM logistics.silver.shipments s
WHERE s.status = 'DELIVERED'
GROUP BY 1, 2;

05Match the ingestion pattern to each source

Unifying the estate was as much an integration exercise as an analytics one, and the ten-plus
sources fell into three categories, each handled differently:

• Transactional databases (order management, WMS, finance) ingested via change
  data capture, so updates flow in near-real-time without loading reporting queries onto the production
  systems they depend on.
• SaaS and API sources (TMS, fuel cards, customs brokerage) pulled through
  scheduled, incremental extracts with watermarking and retry.
• Streaming and semi-structured feeds (vehicle telematics, partner EDI) landed
  continuously, with schema-on-read handling for the messy, evolving payloads these sources are known
  for.

Moving analytical load off the transactional databases was a quiet but significant win in its own
right: it improved the stability of the very systems that run the day-to-day business. This is the kind
of work our data lake and lakehouse engineering
practice does repeatedly.

06Govern for trust with Unity Catalog

A single source of truth is only as valuable as the trust placed in it, and trust requires
governance. The platform standardised access control, lineage, and auditing through
Unity
Catalog, delivering three things the business genuinely needed: consistent role-based access across
every table; end-to-end lineage so any Gold metric can be traced back through Silver and Bronze to the
originating system; and a documented data dictionary so the agreed definitions have one authoritative
home. When a number is questioned, the answer is evidence, not opinion.

-- Governed, role-based access on the Gold schema
GRANT USAGE ON SCHEMA logistics.gold TO `finance_analysts`;
GRANT SELECT ON TABLE logistics.gold.on_time_performance
  TO `finance_analysts`, `ops_analysts`;

-- Lineage is captured automatically and is queryable
SELECT source_table_full_name, target_table_full_name
FROM system.access.table_lineage
WHERE target_table_full_name = 'logistics.gold.on_time_performance';

07Engineer the lakehouse for minutes, not hours

The final layer is performance. The morning operations summary used to consume an analyst’s
half-day of exporting CSVs and stitching them in spreadsheets. By doing the heavy joining and
conforming in the pipeline — and by physically laying out the Gold tables for the query patterns
that matter — the same report now refreshes in minutes. Liquid clustering (or
ZORDER
on older runtimes) co-locates the columns reports filter on, and regular
OPTIMIZE
keeps file sizes healthy for fast scans.

-- Co-locate on the columns dashboards filter by
ALTER TABLE logistics.gold.on_time_performance
  CLUSTER BY (ship_date, lane_id);

-- Compact small files; keep scans fast
OPTIMIZE logistics.gold.on_time_performance;

The outcome: hours to minutes, one shared view

The business results were concrete and durable. Report generation dropped from hours to minutes,
because the expensive work now happens in the pipeline rather than in a last-minute spreadsheet
scramble. Operations and finance stopped reconciling against each other and began making decisions
against the same numbers, shifting review meetings from “whose data is right” to
“what do we do about it.” Production systems were relieved of ad-hoc reporting load, and
new questions became cheap to answer — a query against governed Gold tables rather than a fresh
integration project each time.

Key takeaways

One platform, one engineering discipline: agree, land, conform, govern, then optimise.

• Agree definitions before pipelines. The signed metric glossary prevents most
  reporting disputes before any data moves.
• Keep raw data immutable. Faithful Bronze makes every later decision reversible
  and every number explainable.
• Conform once, in version control. Identity resolution and shared definitions
  belong in Silver, not in private spreadsheets.
• Treat governance as a feature. Lineage and access control are what turn a fast
  platform into a trusted one.
• Design for the messy cases first. Late data, schema drift, and backfills are the
  daily weather in logistics, not edge cases.

Frequently asked questions

The Importance of a Single Source of Truth Databricks Lakehouse

MinervaDB Inc. · San Francisco Bay Area
· 24×7 global data operations.
Data Analytics ·
Data Lakes ·
Remote DBA ·
Database SRE.

Databricks and Delta Lake are
registered trademarks of Databricks, Inc. MinervaDB is not affiliated with, endorsed by, or sponsored
by Databricks, Inc. All other trademarks are the property of their respective owners. Copyright
© 2010–2026. All Rights Reserved by MinervaDB®.