Writing a transform that gives the same answer on pandas and Spark¶

A transform written with Narwhals runs on pandas and on Spark unchanged. Running is not the same as agreeing. This page is for anyone writing a transform that must give the same answer on both. It lists the places where the same line of code gave different numbers, each found by running a real workload on both engines and comparing the run records with ubunye prove. Each has a fix that holds on both. To check your own task, jump to How to know. Why it works this way: ADR 005.

Rounding: round halves yourself¶

.round(4) rounds a half up on Spark and to the even neighbour on pandas. A monthly mean of 921.03125 became 921.0313 on Spark and 921.0312 on pandas; 43 of 6,366 monthly food prices differed this way in the food prices example.

def round4(expr):
    return (expr * 10000 + 0.5).floor() / 10000   # half up, on every engine

Expr.floor needs Narwhals 2.9 or later.

Means and sums of floats: add integers¶

A sum of floating-point numbers depends on the order it is added in, and engines add in different orders (pandas pairwise, Spark partition by partition). The results differ in their last bits, which is invisible until a value sits on a rounding boundary: after fixing the rounding, 7 monthly prices still differed by 0.0001. Turn the values into whole units first; an integer sum is exact in any order:

micro = (nw.col("price") * 1_000_000 + 0.5).floor().cast(nw.Int64)
monthly = prices.with_columns(micro=micro).group_by(...).agg(
    total=nw.col("micro").sum(), n=nw.len(),
).with_columns(mean_price=round4(nw.col("total") / nw.col("n") / 1_000_000))

Money in integer cents from the start (as in the proving workload C01) avoids the question altogether.

Casts inside a group-by: cast after it¶

On pandas, Narwhals 2.26 dropped a .cast(nw.Int64) written inside a group-by that also computed a mean: the counts came out as float64, and the schema no longer matched Spark's bigint (reported to Narwhals as narwhals#4005). Cast after the aggregation:

agg = df.group_by("k").agg(n=nw.len(), m=nw.col("x").mean())
agg = agg.with_columns(nw.col("n").cast(nw.Int64))

Replacing text: `replace_all`, literal¶

str.replace is not implemented for Spark in Narwhals; it raises NotImplementedError. str.replace_all(" KG", "", literal=True) works on both.

Casting text to numbers: filter first, then cast¶

"25 KG" to 25 is a cast of text to a number. When some rows cannot be cast, pandas raises, Spark 3.5 quietly makes them null, and Spark 4 (ANSI mode) raises. Keep only the rows that can be cast, then cast:

weights = df.filter(nw.col("unit").str.contains("^[0-9.]+ KG$").fill_null(False))
weights = weights.with_columns(
    kg=nw.col("unit").str.replace_all(" KG", "", literal=True).cast(nw.Float64)
)

Time: say which zone¶

Cutting a timestamp into a day or an hour depends on a time zone. Every Ubunye backend uses UTC unless the task sets spark.sql.session.timeZone. In older versions, local Spark used the machine's own zone without saying so. A task that means local days sets the key. Why it works this way: ADR 007.

Days between two dates: count with whole numbers¶

Narwhals cannot subtract two timestamps on Spark (dt.total_seconds and dt.timestamp are not implemented there), and on pandas dt.ordinal_day() fails on a column with a null in it, which a delivery date often has. The Olist example turns each date into a day number with whole number arithmetic on the year, month and day, which every engine does alike, and subtracts:

def day_number(ts):
    month = ts.dt.month().cast(nw.Int64)
    y = ts.dt.year().cast(nw.Int64) - nw.when(month <= 2).then(1).otherwise(0)
    march_based = nw.when(month >= 3).then(month - 3).otherwise(month + 9)
    day_of_year = (march_based * 153 + 2) // 5 + ts.dt.day().cast(nw.Int64) - 1
    return (y * 365 + y // 4 - y // 100 + y // 400 + day_of_year).cast(nw.Int64)

Two smaller traps met on the way: pandas 2 has no modulo (%) on Arrow backed integers (Ubunye's pandas backend reads into Arrow), and Narwhals has no when().when(): nest a second when inside otherwise.

Null group keys: keep them on purpose¶

Spark keeps rows whose group key is null as one group; plain pandas drops them. Narwhals follows Spark only when asked: group_by(..., drop_null_keys=False).

How to know¶

Run the task on both engines with --lineage and compare:

T="-d pipelines -u shop -p orders -t clean"
ubunye run $T --backend pandas --lineage && ubunye prove observe --workload my-task --env pandas-local $T -o evidence
ubunye run $T --backend spark  --lineage && ubunye prove observe --workload my-task --env spark-local  $T -o evidence
ubunye prove report evidence --workload my-task --reference spark-local

prove observe reads the task's latest run record (--run-id picks another), so observe each run right after it.

A data FAIL with schema and rows PASS is usually one of the traps above: compare the two outputs row by row on the columns that are computed.