Quantizing Half the Attention Hurt. Quantizing Both Paths Barely Did.

A familiar optimization instinct says: change as little as possible.

If reducing numerical precision introduces error, apply it only where it buys the most speed. Keep the rest of the computation untouched.

A new paper on tabular foundation models finds a case where that instinct fails.

The researchers tested FP8 attention in TabPFN-v3, a transformer-style model that predicts from rows of training data presented in context. When they quantized only the train-to-train attention path, performance fell by 31.8 Elo points. Quantizing only the test-to-train path cost 20.3 Elo points.

Then they quantized both.

The drop shrank to roughly 1 Elo point in one configuration and 1.3 points in another—small enough to be nearly lost inside normal preprocessing variation. With a practical size gate, the final system achieved up to 1.7× faster attention and end-to-end inference without a relevant accuracy loss across the reported benchmarks.

Less precision was not the surprising part.

The surprising part was that consistent error was safer than partial accuracy.

First: what is a tabular foundation model?

Most working data scientists still train a new model for each table. You receive a customer-churn dataset, split it, fit gradient boosting or a neural network, tune it, and evaluate it.

A tabular foundation model changes that workflow. It has already learned broad statistical patterns from many synthetic or real tabular tasks. At prediction time, you provide the training rows, their labels, and the test rows. The model uses those examples as context and produces predictions without conventional task-specific training.

TabPFN is the best-known family in this category. The appeal is obvious: strong results with less pipeline work, especially on small and medium tabular datasets.

But the serving pattern is different from a language model.

An LLM processes a sequence of tokens and generates new tokens. A tabular foundation model may need to compare large numbers of dataset rows with one another. As the training table grows, that row-wise attention can become the dominant cost.

The new paper, “Attention Quantization for Tabular Foundation Models,” asks whether lower-precision arithmetic can reduce that cost. (arXiv)

The bottleneck is attention, not the weights

Quantization discussions around LLMs often focus on weights and the KV cache. That makes sense when the model contains billions of parameters and autoregressively stores key-value states for long sequences.

Tabular foundation models have a different profile.

The models are comparatively small, so compressing their weights offers limited benefit. Their expensive operation is attention across rows.

Let N be the number of training rows and M the number of test rows. In the in-context learning stage, the model performs two related attention operations:

1. Train–train attention: N training queries attend to N training keys.
2. Test–train attention: M test queries attend to the same N training keys.

The work grows roughly as N² + MN. Weight-matrix computation grows only linearly with N + M.

At large row counts, optimizing the attention arithmetic attacks the part of the system that is actually growing fastest.

That is the first practical lesson: quantize the bottleneck you measured, not the component everyone else talks about.

What FP8 changes

The baseline uses 16-bit floating-point arithmetic for the important attention matrix multiplications. The proposed kernel converts queries, keys, and values to FP8—an 8-bit floating-point format—and uses hardware instructions designed to execute those matrix multiplications faster.

Lower precision creates approximation error. Values must be rounded into a smaller representable range, and many distinct 16-bit values can collapse into the same 8-bit value.

The researchers scale each attention head dynamically using its largest absolute value, then quantize into the FP8 e4m3 format. They implement the fused operation as a Triton kernel and compare it with a FlashAttention-2 baseline.

The obvious question is whether the speedup is worth the predictive damage.

The less obvious question is where the approximation must be applied.

The ablation that changes the story

The authors evaluate four variants on 51 TabArena datasets with three preprocessing seeds.

FP8 placement	Relative error change	Elo change
Train–train only	+2.81%	-31.8
Test–train only	+2.38%	-20.3
Both paths, separate query scales	+0.01%	-1.0
Both paths, shared train-query scale	+0.02%	-1.3

If you saw only the first two rows, you might conclude that FP8 attention is too destructive for this model.

But applying the approximation to both related computations nearly removes the degradation.

Why?

The model does not merely need each representation to be individually precise. It needs the training examples and test examples to inhabit a compatible internal geometry.

If the training rows are processed one way and the test rows another, the relationship between them shifts. The test example asks its question in one coordinate system while the training context answers in another.

When both paths receive the same kind of approximation, the absolute representations are noisier—but the relationship is preserved.

This is not just a quantization trick

The researchers test whether the result is specific to FP8 rounding. They replace quantization with Gaussian noise added to queries, keys, and values.

The same pattern appears.

Perturbing only one attention call damages performance more. Perturbing both calls in a coordinated way is much safer, especially when the two calls consume the same perturbed versions of the training keys and values.

That matters because it points to a broader mechanism.

The result is not simply “this FP8 format happens to work.” It is evidence that the architecture depends on alignment between its train-context and test-context computations.

The model can absorb a shared distortion. It struggles when only one side moves.

A concrete analogy: changing the ruler

Imagine measuring a room twice.

For the first measurement, your ruler is slightly short. For the second, you use a perfectly accurate ruler.

Each set of numbers may look reasonable on its own. But comparing them directly creates systematic disagreement because the unit changed.

Now use the same slightly short ruler for both measurements. Every length is imperfect in absolute terms, yet the relationships between the measurements remain coherent.

This is not a license to corrupt data. It is a way to understand why consistency can matter more than local precision inside a comparative computation.

The relevant question is not only:

How much numerical error did quantization add?

It is also:

Did quantization change both sides of the comparison in the same way?

Why the optimization turns on at 8,192 rows

Low precision is not free.

The system must calculate scaling factors, convert tensors to FP8, and manage the fused kernel. On small matrices, that overhead can cost more than the faster multiplication saves.

The paper finds that the quantization cost is amortized at around 8,192 training rows. The final implementation therefore uses a gate:

• below 8,192 rows, keep the original 16-bit path;
• above 8,192 rows, activate FP8 attention.

This is good production engineering.

A benchmark optimization should not become a tax on the workloads that do not benefit from it. The gate converts a research result into a conditional policy based on the size of the actual problem.

It also prevents an easy reporting mistake. “Up to 1.7× faster” does not mean every dataset becomes 1.7× faster. Small datasets remain on the baseline path, and end-to-end gains depend on how much of total runtime attention occupies.

What the final evaluation shows

The final configuration shares the query scale calculated from training rows with test queries. This keeps predictions invariant to the exact composition of a test batch—a small but important operational property.

It is evaluated on:

• TabArena: 51 datasets, with FP8 active on 18;
• BeyondArena: 142 datasets, with FP8 active on 56, including datasets with up to one million training rows;
• TabPFN-v3 as the main model;
• TabICLv2 as an additional architecture check.

The custom kernel reaches up to 1.72× kernel speedup and 1.67× end-to-end speedup in the reported measurements. On the gated benchmark runs, performance stays effectively level with the 16-bit baseline, with observed differences smaller than ordinary variation from TabPFN preprocessing seeds.

The authors also implement a 16-bit version of their Triton kernel. It performs similarly to the FlashAttention-2 baseline, supporting the claim that the gain comes from FP8 computation rather than merely a better-written kernel.

“No relevant accuracy loss” needs careful reading

The paper does not claim mathematical identity.

Without the 8,192-row gate, even the coordinated variants show a tiny negative Elo shift, and the authors’ one-sided sign tests suggest that some small regression may be detectable.

With the gate, the practical effect across the reported benchmarks is within seed noise. That is a stronger and more honest statement than “lossless.”

It means:

• predictions can differ;
• some datasets can move slightly up or down;
• the aggregate quality change is small relative to normal experimental variation;
• the reported speed benefit is substantial on sufficiently large datasets.

In production, whether that trade is acceptable depends on your metric, tolerance, hardware, and deployment cost.

The lesson for ordinary data pipelines

Most readers will never write a Triton attention kernel. The principle still travels.

Consider feature preprocessing. Your training pipeline standardizes values with one mean and variance. Your serving pipeline recomputes them from a different window. Both transformations are individually sensible. Together they create skew.

Or category encoding. Training maps “unknown” to one index, while serving maps it to another. The individual rows remain valid. Their relationship to the learned model does not.

Or embeddings. A document index is built with one model version, while queries are encoded after an upgrade. The new model may be better in isolation. Retrieval can still get worse because documents and queries no longer occupy the same vector space.

Or time-series joins. Features are computed with one cutoff rule in training and another online. Each table passes its unit tests, but the comparison has changed.

In all these cases, local correctness does not guarantee relational consistency.

A checklist for low-precision inference

Before adopting a quantized path, ask:

1. What operation dominates runtime at the sizes we actually serve?
2. Does quantization overhead erase the gain on small inputs?
3. Which branches later interact or compare representations?
4. Are all sides using compatible scales, versions, and transformations?
5. Can test batching alter predictions?
6. Is the speedup kernel-only or end-to-end?
7. Was accuracy evaluated per dataset, across seeds, and with a practical equivalence threshold?
8. Does the result generalize to our hardware and model architecture?
9. Can we gate the optimization to workloads where it helps?
10. What is the rollback path if a particular dataset regresses?

That checklist is more useful than the slogan “FP8 is fast.”

What would make me wrong?

This is a fresh preprint from researchers at Prior Labs, the company behind TabPFN. It has not yet accumulated independent replications.

The main kernel experiments use an NVIDIA RTX Pro 6000 Blackwell GPU. The paper includes additional results for an NVIDIA L4 and different head dimensions, but hardware-specific kernels do not transfer uniformly. Your accelerator, compiler version, batch sizes, and memory behavior may produce different break-even points.

The central ablations focus on TabPFN-v3, with TabICLv2 results in the appendix. That is useful evidence across two model families, not proof that coordinated quantization will protect every architecture.

The evaluation covers many datasets, but benchmark averages can hide a regression on the one dataset that matters to your business. “Within seed noise” at the suite level is not a safety guarantee for an individual medical, credit, or fraud model.

The 8,192-row gate is empirical, not universal. A different GPU or kernel may move it substantially.

Finally, the claimed cost reduction follows from faster execution on the tested workload. Real serving cost also includes data movement, preprocessing, orchestration, idle capacity, and utilization. A 1.7× kernel speedup is not automatically a 1.7× lower cloud bill.

So the paper provides a strong mechanism and a promising implementation—not a universal deployment rule.

The deeper idea

We often evaluate approximation locally.

How many bits did we remove? How much error entered this tensor? How far did one embedding move?

But machine-learning systems are built from relationships between computations. Sometimes the crucial property is not the precision of either side. It is whether both sides still agree on what their numbers mean.

That is why the most memorable result in this paper is not the 1.7× speedup.

It is the failed halfway optimization.

Quantizing one attention path preserved more exact arithmetic overall—and hurt the model much more.

Quantizing both paths introduced approximation everywhere that mattered—and preserved the relationship.

The model did not need perfect coordinates. It needed a shared map.

Key takeaways

1. Tabular foundation models have a different inference bottleneck from LLMs: attention across data rows can dominate while weight quantization offers limited value.
2. In TabPFN-v3, quantizing only one of two related attention paths caused a 20–32 Elo drop.
3. Quantizing both paths consistently reduced the drop to roughly 1 Elo point before gating.
4. A gate activates FP8 only above 8,192 training rows, where conversion overhead is amortized.
5. The reported implementation reaches up to 1.72× kernel and 1.67× end-to-end speedups.
6. The practical lesson extends beyond FP8: when two representations will be compared, consistent transformation can matter more than local precision.
7. The results are promising but hardware- and architecture-dependent, and they come from a new preprint that needs independent replication.

Primary source

• Jonas M. Kübler, Benjamin Jäger, Klemens Flöge, Noah Hollmann, and Frank Hutter, “Attention Quantization for Tabular Foundation Models”, arXiv
  .13031v1, submitted September 11, 2026.