pandas vs Polars vs DuckDB on 3.5 Million Taxi Rides. DuckDB Won Every Query, and the Timings Were the Least Interesting Part.

Every "X is 10x faster than pandas" post uses a different dataset, different hardware, and different queries, which is why none of them agree with each other. So I ran the obvious benchmark myself: one CSV, 3,475,226 NYC yellow taxi trips from January 2025, the same four operations in pandas 3.0.6, Polars 1.44.2, and DuckDB 1.5.6, each timed as the median of three runs in a fresh process on a 2-CPU, 7GB RAM machine.

The setup

The dataset is the NYC Taxi and Limousine Commission trip record data for January 2025: 3,475,226 rows, 20 columns, 388MB as CSV. Real data, with real messiness: nulls, a store_and_fwd_flag column that is mostly one value, and fare columns that occasionally disagree with each other.

Four operations, the kind that make up 90% of exploratory data work:

1. Read the CSV into the engine's native structure.
2. Group by payment type: trip count and average fare per group.
3. Filter to long trips (over 10 miles) with a positive fare, then average the tip percentage.
4. Join the trips against a 7-row rate-code lookup and sum fares per rate name.

Each engine ran in its own fresh process. Peak memory is the process peak RSS, so it includes the CSV read plus all three queries. All three engines agreed on every correctness check: the same group counts, the same filtered tip mean to six decimals. The numbers below are timings, not opinions.

The code, in full, for the filter query, the only one with any subtlety:

# pandas
f = df[(df["trip_distance"] > 10) & (df["total_amount"] > 0)]
tip_pct = (f["tip_amount"] / f["total_amount"]).mean()

# Polars
tip_pct = (
    df.filter((pl.col("trip_distance") > 10) & (pl.col("total_amount") > 0))
    .select((pl.col("tip_amount") / pl.col("total_amount")).mean())
    .item()
)

-- DuckDB
SELECT AVG(tip_amount / NULLIF(total_amount, 0))
FROM trips WHERE trip_distance > 10 AND total_amount > 0;

The results

Operation	pandas 3.0.6	Polars 1.44.2	DuckDB 1.5.6
Read 388MB CSV	7.74s	8.18s	4.61s
Group by payment type	0.10s	0.03s	0.02s
Filter + tip %	0.08s	0.04s	0.04s
Join (7-row lookup)	0.67s	0.68s	0.03s
Peak memory (RSS)	2,082MB	1,714MB	1,035MB

All three engines agreed on every answer: 6 payment-type groups, the same filtered tip mean (0.10489) to six decimals, 7 rate-code groups.

What the numbers actually say

1. DuckDB swept. Fastest read, fastest groupby, fastest filter (tied), a 22x margin on the join, and half the memory of pandas. On a small box with zero tuning, the SQL engine was simply the most efficient at everything I asked it to do.

2. The query timings barely matter. Look at the absolute numbers: the groupby, filter, and join all finish in under a second on every engine. The read is more than 90% of total runtime. Anyone optimizing their groupby while still reading CSVs is polishing the wrong step.

3. Memory is the real gap. pandas peaked at 2,082MB to process a 388MB file, over 5x the file size. DuckDB did the same work in 1,035MB. On this 7GB box, pandas spent nearly a third of available RAM before doing anything clever. Scale the file to 4GB and pandas is the engine that does not finish.

4. Polars lost the read, and the reason is the most useful finding in this benchmark. Polars' CSV reader infers column types from the first rows of the file. The fee columns looked like integers in the early January rows, then values like 0.75 and 0.30 showed up later in the file, and the read failed. Twice: the default sample and a 10,000-row sample both broke. The fix was infer_schema_length=None, a full-file scan, at the cost of the slowest read in the test (8.18s). This is the tax on messy real-world CSVs that synthetic benchmarks never show you: the reader has to guess the schema, and guessing wrong costs more than any query optimization saves. On Parquet, where the schema is stored in the file, this entire problem disappears.

5. pandas did nothing wrong. It finished every query, agreed with both challengers to six decimals, and its read was middle of the pack. The knock on pandas was never correctness. It is memory: 5x the file size in RAM is the price of the ecosystem and the API everyone knows.

The verdict

Pick DuckDB when the job is analytical queries over files, especially when memory is tight or you are comfortable in SQL. It was the fastest reader, the fastest querier, and the lightest on RAM here, with no tuning at all.

Pick Polars when you are building a pipeline in Python and want the speed without leaving the dataframe API. Just budget for CSV schema inference on messy files, or better, convert to Parquet once and never think about it again.

Pick pandas when the ecosystem matters more than the clock: the libraries, the answers, the code your team already reads. Budget 5x the file size in RAM and move on.

And regardless of engine: convert CSV to Parquet as a first step. It removes the single slowest and most fragile step in this entire benchmark.

Key takeaways

1. On a small untuned box, DuckDB was fastest on all four operations and used the least memory. The sweep matters less than the margins: reads dominate, and every query was sub-second on all three engines.
2. The read is the benchmark. It took more than 90% of total runtime everywhere. Optimize the read before anything else.
3. Memory gaps beat speed gaps. pandas needed 5.4x the file size in RAM; DuckDB needed 2.7x. On a constrained machine, that decides which engine finishes.
4. Polars' CSV schema inference failed twice on real-world messy columns and needed a full-file scan. Messy CSVs punish guessing readers; Parquet removes the problem.
5. All three engines agreed on every answer to six decimals. The days of worrying whether the fast engine is the wrong engine are over. Pick on memory, API, and ecosystem.

What is the largest CSV you have opened in pandas before it fought back, and what did you switch to?

Related on Everyday Data Science: Stop Eyeballing Your Agent's Answers. A 100-Line Eval Harness Catches the Regressions You Miss., the same instinct applied to AI agents: stop eyeballing, measure.

Sources: NYC TLC Trip Record Data · pandas documentation · Polars documentation · DuckDB documentation