Schema Validation in PySpark, Enforcing Expected Structure

The failure mode this actually prevents

Here is a situation that happens more often than people expect. A source system team adds a new column to their database, or renames an existing one, without telling anyone on the data team. Your pipeline, which has no schema validation, just keeps running. It does not error out, it does not send an alert, it just quietly starts either dropping the new column's data entirely, or worse, mapping data into the wrong field because column positions shifted.

Nobody notices for two weeks, until someone on the business side asks why a report looks off, and by then you are tracing back through two weeks of silently corrupted data trying to figure out exactly when things went wrong. Schema validation exists specifically to prevent this scenario, by catching structural changes the moment they happen, rather than letting them flow through silently.

The difference between schema enforcement and schema validation

These two terms get used somewhat interchangeably, but it helps to separate them. Schema enforcement, which Delta Lake provides natively, prevents data with a mismatched schema from being written into an existing Delta table at all, rejecting the write outright. Schema validation is a broader concept, checking that incoming data actually matches expectations before you even attempt to write it, which gives you more control over how to handle a mismatch, rather than just having the write fail.

Both matter, and they work well together. Let's look at each.

Delta Lake's built in schema enforcement

By default, Delta Lake actually rejects writes that do not match an existing table's schema, which is a real safety net worth understanding.

existing_table_path = "/mnt/datalake/silver/orders/"

# assume the existing table has: order_id, customer_id, order_amount, order_date

df_new = spark.createDataFrame(
    [(1001, 501, 4500.0, "2026-08-15", "extra_unexpected_field")],
    ["order_id", "customer_id", "order_amount", "order_date", "unexpected_column"]
)

df_new.write.format("delta").mode("append").save(existing_table_path)
# This will raise an AnalysisException, since unexpected_column
# does not exist in the target table's schema

This built in protection is genuinely useful, since it means a malformed or unexpectedly structured dataset cannot silently corrupt an existing table just by writing to it. But relying on this alone has a downside, since the failure happens at write time, often deep inside a pipeline, rather than earlier where it would be easier to catch and handle gracefully.

Explicitly validating schema before writing

A more proactive approach is comparing the incoming DataFrame's schema against an expected schema explicitly, early in the pipeline, so you can catch and handle a mismatch on your own terms, rather than waiting for Delta Lake to reject the write.

from pyspark.sql.types import StructType, StructField, IntegerType, DoubleType, StringType

expected_schema = StructType([
    StructField("order_id", IntegerType(), True),
    StructField("customer_id", IntegerType(), True),
    StructField("order_amount", DoubleType(), True),
    StructField("order_date", StringType(), True)
])

def validate_schema(df, expected_schema):
    actual_fields = set((f.name, str(f.dataType)) for f in df.schema.fields)
    expected_fields = set((f.name, str(f.dataType)) for f in expected_schema.fields)

    missing = expected_fields - actual_fields
    extra = actual_fields - expected_fields

    if missing:
        print(f"Missing expected fields: {missing}")
    if extra:
        print(f"Unexpected extra fields: {extra}")

    return len(missing) == 0

is_valid = validate_schema(df_new, expected_schema)

if not is_valid:
    raise ValueError("Schema validation failed, halting pipeline before write")

This gives you a clear, controlled failure point, with a specific, readable message about exactly what changed, rather than a generic Delta Lake exception buried inside a write operation. It also gives you the option to handle mismatches differently depending on the situation, like logging a warning and continuing for a genuinely harmless extra column, versus halting the pipeline entirely for a missing critical column.

Allowing schema evolution deliberately, rather than accidentally

Sometimes a schema change is expected and legitimate, like a source system genuinely adding a useful new column that should now flow through the pipeline. Delta Lake supports this through explicit schema evolution, but it needs to be turned on intentionally, not left as an accidental side effect of a permissive write.

df_new.write.format("delta") \
    .mode("append") \
    .option("mergeSchema", "true") \
    .save(existing_table_path)

The key word here is deliberately. Using mergeSchema as a default setting on every single write essentially disables the protection schema enforcement was giving you in the first place, since now any new or changed column just gets silently absorbed into the table without anyone reviewing whether that change is actually correct or expected. A better pattern is keeping mergeSchema off by default, and only enabling it explicitly during a planned schema change, after confirming with whoever owns the source system that the change is intentional.

Validating data types, not just column names

A schema mismatch is not always about missing or extra columns. Sometimes a column that should be numeric starts coming through as a string, often because of an upstream change in how a source system formats its export, and this kind of mismatch can be sneakier since the column name itself has not changed at all.

def validate_data_types(df, expected_schema):
    mismatches = []
    actual_dict = {f.name: str(f.dataType) for f in df.schema.fields}
    expected_dict = {f.name: str(f.dataType) for f in expected_schema.fields}

    for column, expected_type in expected_dict.items():
        actual_type = actual_dict.get(column)
        if actual_type and actual_type != expected_type:
            mismatches.append((column, expected_type, actual_type))

    return mismatches

type_mismatches = validate_data_types(df_new, expected_schema)

if type_mismatches:
    for column, expected, actual in type_mismatches:
        print(f"Type mismatch on {column}: expected {expected}, got {actual}")

Catching a case where order_amount silently shifted from a DoubleType to a StringType is exactly the kind of thing that, left unnoticed, causes every downstream aggregation on that column to either fail outright or produce quietly wrong numbers, depending on how the rest of the pipeline handles it.

Where this fits in a real pipeline

Schema validation logic like this typically runs right at the boundary between the Bronze and Silver layers, as one of the first steps applied to newly ingested raw data, before any real transformation logic runs on top of it. Catching a schema issue this early means the rest of the pipeline never even attempts to process data it was not built to handle, which keeps failures clear and traceable, rather than manifesting as confusing downstream errors several steps later.

Interview angle

If asked how you would handle schema changes in a source system, describing both Delta Lake's built in schema enforcement and an explicit upfront validation step shows a more complete understanding than mentioning just one. Bringing up the distinction between accidentally allowing schema drift through a blanket mergeSchema setting, versus deliberately enabling schema evolution only when a change is confirmed and expected, tends to be the detail that shows genuine hands on experience rather than surface level familiarity with the concept.

Quick recap

Schema validation protects a pipeline from silently absorbing unexpected structural changes from a source system, whether that means new or missing columns, or a column's data type quietly changing. Delta Lake provides built in schema enforcement that rejects mismatched writes by default, but explicitly validating schema earlier in the pipeline gives clearer, more controlled error handling. Schema evolution through mergeSchema should be used deliberately for confirmed, intentional changes, not left on as a default that quietly disables the protection schema enforcement was meant to provide in the first place.