Moving from Teradata to Databricks (Without Regretting It Later): A Dose of Reality

At some point, someone in your organization will stand up in a meeting and say something like:

“We’re moving off Teradata. Databricks is modern. It’s cheaper. It’s faster. And we can finally stop worrying about all that old-school data warehouse discipline.”

This usually sounds very confident. It is also usually partially wrong.

What follows is not a story about technology failure. Databricks is an impressive platform. Teradata is a mature one. Both do exactly what they are designed to do. Failures happen because people assume they are philosophically the same thing. They are not.

Teradata is a full-featured enterprise data warehouse. Databricks is a unified data platform that can behave like a data warehouse, but only if you deliberately make it do so. That “if” is where most migrations go off the rails.

Databases encode assumptions. When you change platforms without understanding those assumptions, you don’t get clean modernization. You get what I like to call a Lakehouse Liability: something shiny, expensive, and possibly untrustworthy.

Let’s talk about why.

Built-In Discipline vs. Earned Discipline

Teradata enforces a great deal of discipline for you, whether you like it or not.

Declare a primary key? Uniqueness is enforced.

Define a SET table? Duplicates are rejected.

Create triggers? They fire exactly when rows change.

Use views? You can insert, update, and layer them in surprisingly powerful ways.

Databricks does none of this by default.

Primary keys are informational.

Uniqueness is a suggestion.

Triggers exist, though not in the way Teradata users expect.

Views are read-only abstractions.

This is not a flaw. It is a design choice.

Databricks assumes you will manage correctness through pipelines, patterns, and governance. Teradata assumes correctness is the database’s job. If you don’t notice this difference early, you will notice it later, usually during a reconciliation meeting where nobody is smiling.

This is earned discipline, not optional discipline.

Example: Primary Keys

Teradata

CREATE TABLE customer (

  customer_id INTEGER PRIMARY KEY,

  name VARCHAR(100)

);

Duplicate insert:

INSERT INTO customer VALUES (1, ‘Alice’);

INSERT INTO customer VALUES (1, ‘Bob’);

Result: Error. Constraint enforced.

Databricks

CREATE TABLE customer (

  customer_id INT,

  name STRING,

  CONSTRAINT pk_customer PRIMARY KEY (customer_id)

);

Same duplicate insert:

INSERT INTO customer VALUES (1, ‘Alice’);

INSERT INTO customer VALUES (1, ‘Bob’);

Result: Both rows succeed.

The constraint is informational. It helps tools. It does not protect your data.

So you enforce uniqueness in pipelines:

MERGE INTO customer t

USING staging_customer s

ON t.customer_id = s.customer_id

WHEN MATCHED THEN UPDATE SET *

WHEN NOT MATCHED THEN INSERT *;

Discipline moves from engine → pipeline.

If you don’t redesign for that, duplicates will accumulate quietly until someone runs a reconciliation.

Example: Duplicate Rejection

Teradata SET tables reject duplicates automatically.

CREATE SET TABLE orders (

  order_id INTEGER,

  amount DECIMAL(10,2)

);

Databricks equivalent behavior must be simulated.

CREATE OR REPLACE TABLE orders_clean AS

SELECT DISTINCT *

FROM staging_orders;

Or enforced in streaming pipelines.

df.dropDuplicates([“order_id”])

Collation: The Quiet Way to Break Trust

Collation is one of those topics everyone skips because it sounds boring.

Until joins start dropping rows.

Collation defines how text is compared and sorted: case sensitivity, accent handling, and language rules. Teradata gives you extensive control over this at the database, user, and column levels. Databricks largely relies on UTF-8 binary comparisons: fast, simple, and unforgiving.

Teradata often behaves case-insensitively. Databricks often does not.

If you don’t explicitly decide how text comparisons should behave, you will still get a decision, just not one you made consciously. The result is dashboards that look like they “randomly” change, distinct counts that drift, and analysts who start adding workarounds downstream.

Once trust is lost, performance improvements stop mattering.

Example: Case Sensitivity

Teradata

SELECT *

FROM customers

WHERE name = ‘alice’;

May match Alice, ALICE, alice.

Databricks (default)

SELECT *

FROM customers

WHERE name = ‘alice’;

Matches only exact case.

So this join:

SELECT *

FROM a

JOIN b

ON a.name = b.name;

can silently lose rows after migration.

Mitigation pattern:

SELECT *

FROM a

JOIN b

ON LOWER(a.name) = LOWER(b.name);

But now you’ve introduced:

• CPU overhead
• semantic drift
• inconsistent behavior across teams

The correct solution is not sprinkling LOWER() everywhere. The correct solution is a collation strategy and documented comparison rules.

Data Types: “It Loaded” Is Not the Same as “It’s Right”

Teradata’s type system is strict, expressive, and full of edge cases people forgot they were relying on.

BYTEINT.

INTERVAL.

High-precision DECIMAL.

TIMESTAMP WITH TIME ZONE.

Databricks supports many of these concepts, but not always in the same way, and not always as first-class citizens. Types get promoted. Precision limits matter. STRING becomes the universal escape hatch.

The classic failure looks like this:

You migrate a fact table.

The data loads cleanly.

The rollups run.

The totals are off by a very small amount.

That “very small amount” will cost you credibility.

Floating-point math behaves differently. Implicit casts behave differently. Rounding behaves differently. None of this is obvious until someone compares old numbers to new ones and asks the most dangerous question in analytics:

“Which one is correct?”

If you don’t want that conversation, build a type-mapping matrix before you migrate and test the ugly cases deliberately.

You don’t test averages.

You test pathological rows:

• Maximum precision
• Negative values
• Null combinations
• Extreme timestamps

The Data Model: The Discipline You Have to Rebuild on Purpose

Teradata explicitly forces you to think in terms of a data model. Not because it’s philosophical, but because if you don’t, things break. Keys matter. Relationships matter. Table design has consequences.

When teams move to Databricks, one of the first invisible casualties can be the data model, not because anyone deletes it, but because the platform no longer insists on it. The engine will happily accept loosely related tables, duplicate identifiers, drifting semantics, and “temporary” structures that quietly become permanent.

Nothing may explode immediately. That’s what makes it dangerous.

A data model is not just a diagram created once for documentation. It is the explicit statement of what the business believes is true. What is a customer? What makes an order unique? Can an account exist without an owner? What does “active” mean?

Teradata encodes these beliefs structurally. Databricks assumes you will encode them socially, through conventions, pipelines, and shared understanding.

That assumption only works if you are deliberate.

Without a formalized model, you don’t have a lakehouse. You have a very fast junk drawer.

Ontology: When the Problem Isn’t Structure, It’s Meaning

A data model answers: How is the data structured?

An ontology answers: What does the data mean?

Two teams can share the same schema and still disagree about reality.

An ontology is a shared vocabulary that defines concepts, relationships, and intent. It sits above the schema and prevents meaning from drifting as the system grows.

Same table name. Different ontology. The schema didn’t fail. The shared meaning did.

Lakehouse platforms amplify this problem because they remove central bottlenecks. More teams can publish data faster. That’s good for velocity. It’s terrible for semantic consistency if you don’t define a common language.

Databricks Unity Catalog does not currently provide native capabilities for capturing and managing a full ontology layer. It robustly manages technical governance (access control, auditing, lineage, and physical schema), but business meaning requires an additional solution.

You are not adding bureaucracy. You are preventing semantic entropy.

Transactions: ACID Is Not a Yes/No Question

Teradata users are used to wrapping sequences of statements in transactions and expecting all-or-nothing behavior.

Databricks provides ACID guarantees — but typically at the level of a single operation.

That distinction matters.

If your pipeline:

• Updates a dimension
• Inserts facts
• Writes audit rows
• Publishes a completion flag

And step three fails, you may have just published a half-truth.

Design for atomicity explicitly: publish-from-staging patterns, idempotent pipelines, and controlled promotion.

Time Zones: Decide Early or Pay Forever

Time zones are exhausting. Everyone knows this. That’s why teams postpone decisions about them.

Then daylight saving time hits.

You must decide what your timestamps mean:

• Are they instants? Store UTC.
• Are they business wall-clock times? Store local time explicitly.
• Are they mixed? Store both the instant and the original offset.

Assuming “timestamp is timestamp” guarantees pain later. This is not a technical nuance. It is a semantic contract.

SQL Differences: The Death of a Thousand Paper Cuts

Most Teradata SQL ports cleanly. The rest may make you question your life choices.

Volatile tables

Macros

Triggers

Update-with-join patterns

Stored procedures

Each one is manageable. Collectively, they demand a systematic approach. Decide early how procedural logic will be handled — Databricks SQL scripting, notebooks, dbt, or something else. Inconsistency here is how maintenance costs quietly explode.

Don’t Just Rebuild Teradata on Databricks

One of the most common mistakes is recreating the Teradata warehouse exactly as it was, just on new infrastructure.

Same schemas.

Same layering.

Same mental model.

You can do this. You just won’t get much benefit from it.

The lakehouse does not eliminate the need for discipline. It removes the guardrails that forced it.

Final Thought

Teradata spent decades teaching organizations hard lessons:

• Typing matters.
• Semantics matter.
• Governance matters.
• “It runs” is not the same as “it’s right.”

Databricks does not invalidate those lessons. It simply gives you a much larger platform on which to relearn them, faster, louder, and at cloud scale, if you are not careful.

The devil is not in the technical details. The devil is in the assumptions and decisions you didn’t realize you were making.

Learn how Ness helps enterprises modernize Teradata environments and build scalable lakehouse architectures with advanced data engineering and cloud platform expertise.