Polars 2.0

TL;DR

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Polars has released version 2.0, making its streaming engine the default for LazyFrame queries and enabling initial spill-to-disk support by default. The project also promotes SQL as a first-class interface and reports strong TPC-H and TPC-DS benchmark results, while noting limits in the benchmark and current out-of-core support.

Polars has released version 2.0, changing how lazy queries run by default: calling collect on a LazyFrame now uses the streaming engine, and initial spill-to-disk support is enabled by default. The release also expands SQL support and adds performance and data-type changes, which matter to users running analytical workloads that must balance execution speed, memory use and familiar query interfaces.

The Polars team says the streaming default can bring memory and performance improvements on many queries. It also changes a behavior users may rely on: operations including joins, group-bys and unpivots do not guarantee observable row order by default under the streaming engine. Users who need that order can set maintain_order=True for supported operations, according to the release post.

Polars 2.0 enables its initial out-of-core, or spill-to-disk, support by default. The engine begins spilling at about 80% of available RAM, a threshold the team says may need tuning, and has a default disk budget of 64 GB. The currently supported operations include sorts, window functions and many expressions; joins and group-bys are not yet included in this support.

The release also treats SQL as a first-class interface, with improvements to the optimizer and engine including join reordering, common-subplan elimination and dynamic predicates or bloom filters. A new Map dtype represents Arrow MapType data directly, with dictionary-like key and value operations. The team also describes stricter handling of data types and explicitness as a way to provide faster feedback.

Streaming Changes Query Behavior

The most consequential change for existing users is that streaming is now the default, rather than a separate execution choice for lazy queries. That may help workloads constrained by memory, but users should check whether their results depend on row order in operations where the release says streaming does not preserve it by default. The option to request order provides a route for those cases, though users will need to make that requirement explicit.

Spilling supported work to disk can let some queries proceed when their data exceeds available memory. The current limits matter: not all operations spill yet, and the default disk budget is finite. For teams evaluating Polars for large joins or group-bys, the release is an incremental step rather than a claim that every high-memory workload can now run out of core.

SQL support and the reported benchmark results may interest analysts and organizations comparing query engines. But performance varies by workload and machine, and the published measurements are the project’s own benchmark results, not independent confirmation of a general ranking.

How Polars Tested SQL Performance

To support its performance claims, the Polars team compared its SQL engine with DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0 on TPC-H and TPC-DS data generated in Parquet format. The tests ran on two Amazon machine types: a c7a.4xlarge with 16 vCPUs and 32 GB of memory, and a c7a.metal with 192 vCPUs and 384 GB. Queries were run five times in a hot setting, and the best run was used for comparisons based on both total time and geometric mean.

Polars says it was fastest on all but one of the reported benchmarks by default. It also reports a scaling overhead on the 192-thread machine that harms small-data queries; limiting Polars to 32 cores was competitive or faster across the benchmarks, according to the team. The comparison excluded certain queries for all engines after DataFusion timed out on TPC-DS queries or ran out of memory on a TPC-H query on the smaller machine. The team has published a repository for others to replicate the tests.

Limits of the New Engine

The release post does not provide independently verified benchmark results. Its performance conclusions reflect a particular set of machines, datasets, query runs and comparison engines; readers may get different results with other workloads or configurations. Polars also says its high-thread-count overhead affected small queries and that it hopes to address the issue in a later release.

Out-of-core support remains partial: the post identifies sorts, window functions and many expressions as supported, while joins and group-bys are planned for future work. The release material does not establish when those operations will be added. Users should also test whether the streaming default changes order-dependent results in their own pipelines and review the disk threshold and budget for their environment.

Join and Group-By Spill Support

Polars says it plans to extend out-of-core execution to joins and group-bys, which would broaden the workloads able to spill data to disk. The team also says it has diagnosed the scaling overhead seen on the 192-thread machine and hopes to fix it in the next release, but gives no firm date or release number in the supplied post.

For now, users can examine the project’s benchmark repository and run comparisons against their own queries and hardware. Teams moving to 2.0 should check order-sensitive operations, confirm which expressions are supported by spilling, and account for the configured disk budget before relying on the new defaults for production workloads.

Key Questions

What is the main change in Polars 2.0?

LazyFrame.collect now defaults to the streaming engine. The release also enables initial spill-to-disk support by default and expands SQL capabilities.

Does streaming preserve row order?

Not for some operations by default, including joins, group-bys and unpivots, according to Polars. Users who require observable order can set maintain_order=True where supported.

Which operations can currently spill to disk?

The release post lists sorts, window functions and many expressions. Joins and group-bys are not yet supported for out-of-core execution and are on the team’s roadmap.

Are the Polars 2.0 benchmark results independent?

No independent verification is described in the release material. Polars reports its own TPC-H and TPC-DS tests and has shared a repository intended to help others replicate them.

Source: hn