I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.
I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
That's how I thought semantic versioning worked
I don't know how to read this sentence other than "there are breaking changes we want to make"
https://docs.pola.rs/development/versioning/
> Polars adheres to the semantic versioning specification:
And it does have breaking changes in 2.0. The original asker presumably missed that.
E:
On the other hand, that whole page on versioning seems inconsistent.
Can there be deprecated cruft without new features? :-D
All new shiny new features shouldn't have waited for the (N+1).0 version, they should already have been part of the (N).(M) version.
In practice, the removing the deprecated cruft will remove blockers for some new features, but that should be rare.
Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly around column types and missing values. It's very hard to know if you've tested all the edge cases. The only way to test your code is to throw all variations of data at it. Fine if you're sitting at a notebook and have the patience to validate and "clean" the data on its behalf. Not so fine if you get paged at 3am because your data pipeline failed when it expected an int column but got float.
Polars is more strict by default and front-loads costs through its planner. The resulting apps are noticeably more stable in production. You can test code and reasonable assurance that it will work on data in the wild.
I don't really have any interest in the API ergonomics or syntax - both are fine. It's all about how they deal with data variation at runtime. Can you write general code that doesn't break on variants? Pandas, not a chance. Polars, absolutely!
Bonus round: polars has a Rust API too, the compiler can effectively prove that your program handles every edge case. It's common to write rust polars apps that run unattended for years.
I do not think of a dataframe as a set, but an ordered collection of rows. My source csv had the rows in this order and I want that maintained unless I choose maximum performance.
I'm not sure if I agree that "hidden setting actually keeps your data correct" is something that should be the default.
[1] Aha, now I see why language models use this so frequently and why it might be overrepresented in the data. This is a perfect way to move the blame from the person you're responding to, if they're mistaken. They probably have a super, super overtuned "politeness" gym using sentiment analysis that tries to reword answers to not blame the misunderstandings of the person. Then this blame shifting unfortunately gets re-used as this super, super common phrase.
> Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.
How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
maintain_order=False
?You seem to suggest they did it for benchmarking reasons only. They could use the option there themselves without changing the default so that is unlikely to be the motivation.
We recently added a Polars backend to GFQL (cypher graph queries on dataframes, no DB needed), both CPU and GPU mode, and super impressive. Noticeable improvements vs pandas/cudf, and enabled GFQL to beat out popular systems on more categories like low-latency, not just big datasets: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...
But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS
After using pandas for 10 years, I favor SQL now, for some reason.
- much faster, multithreaded by default. Read in a big csv with it and see how it feels.
- no index/MultiIndex. Pandas special treatment of index always felt like more trouble than it was worth, so no need to reset_index() everywhere.
- expressions are very portable. At first using pl.col everywhere feels like a bit much, but you can define them anywhere and then apply them to a dataframe whenever you want.
- once internalized, the syntax makes much more sense and is far more consistent compared to pandas.
Of course all depends on what your use cases are. If performance is important then I'd strongly recommend trying it out. If you just use it to have a look at the odd dataframe, maybe not worth your time as much
This is not a criticism. As someone who doesn’t use Python, I simply found it amusing.
The duckdb python api is okay, but it is a bit limited, no ctes, no as of join, and it can be slow at bind/interpretation time when you do stuff like unioning multiple relations in a loop (I think that becomes O(N^2), but I might be wrong). Most issues can be worked around, but Polars is designed from the ground up to be used from python.
Pandas is a mess though.
I think on basic queries, SQL is really nice, but when stuff gets more complex, with a bunch of CTEs, let alone functions requiring loops, it becomes pretty obtuse.
df.select(
pl.col("x"),
(pl.col("w")/pl.col("z")).alias("y")
with
df |> select(x, y = w/z)
ggplot vs matplotlib
dplyr vs pandas
And I loved that everything in RStudio was so easily inspectable. Have a huge dataframe? Just look at it right in your IDE.
`df.select("x", y=pl.col.w/pl.col.z)`
Unfortunately, polars does not support parameterized queries, so the risk of SQL injection is extremely high.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
import polars as pl
# 1. Base Dataset
lazy_df = pl.LazyFrame(
{
"store_id": ["S01", "S02", "S03", "S04", "S05"],
"revenue": [5000.0, 2400.0, 15000.0, 900.0, 3200.0],
"margin": [0.45, 0.30, 0.60, 0.15, 0.50],
"tx_count": [120, 45, 300, 20, 85],
"returns": [5, 12, 45, 2, 8],
}
)
# 2. Define Layer Abstractions
def get_kpi_layer() -> list[pl.Expr]:
return [
(pl.col("returns") / pl.col("tx_count")).alias("return_rate"),
(pl.col("revenue") / pl.col("tx_count")).alias("avg_order_value"),
]
def get_threshold_layer(thresholds: dict[str, list[float]]) -> list[pl.Expr]:
return [
(pl.col(col) > limit).alias(f"is_{col}above{int(limit)}")
for col, limits in thresholds.items()
for limit in limits
]
def get_interaction_layer(numeric_cols: list[str]) -> list[pl.Expr]:
return [
(pl.col(a) / (pl.col(b) + 1e-5)).alias(f"ratio_{a}per{b}")
for i, a in enumerate(numeric_cols)
for b in numeric_cols[i + 1 :]
]
def get_segmentation_layer() -> list[pl.Expr]:
return [
pl.when(pl.col("margin") > 0.4)
.then(pl.literal("High"))
.otherwise(pl.literal("Low"))
.alias("margin_profile")
]
# 3. Consolidate and Execute Single Graph Pass
thresholds = {"revenue": [1000.0, 5000.0, 10000.0], "tx_count": [50, 100, 200]}
numeric_cols = ["revenue", "margin", "tx_count", "returns"]
expr_pool = [
*get_kpi_layer(),
*get_threshold_layer(thresholds),
*get_interaction_layer(numeric_cols),
*get_segmentation_layer(),
]
final_df = lazy_df.with_columns(expr_pool).collect() WITH raw_data AS (
SELECT * FROM (
VALUES
('S01', 5000.0, 0.45, 120, 5),
('S02', 2400.0, 0.30, 45, 12),
('S03', 15000.0, 0.60, 300, 45),
('S04', 900.0, 0.15, 20, 2),
('S05', 3200.0, 0.50, 85, 8)
) AS t(store_id, revenue, margin, tx_count, returns)),
base_data AS (
SELECT
store_id,
revenue,
margin,
CAST(tx_count AS DOUBLE) AS tx_count,
CAST(returns AS DOUBLE) AS returns
FROM raw_data
)
SELECT
store_id,
revenue,
margin,
CAST(tx_count AS BIGINT) AS tx_count,
CAST(returns AS BIGINT) AS returns,
-- KPI Layer
returns / tx_count AS return_rate,
revenue / tx_count AS avg_order_value,
-- Threshold Layer (matching original alias names)
revenue > 1000.0 AS is_revenueabove1000,
revenue > 5000.0 AS is_revenueabove5000,
revenue > 10000.0 AS is_revenueabove10000,
tx_count > 50 AS is_tx_countabove50,
tx_count > 100 AS is_tx_countabove100,
tx_count > 200 AS is_tx_countabove200,
-- Interaction Layer (preserving exact numeric formula & aliases)
revenue / (margin + 1e-5) AS ratio_revenuepermargin,
revenue / (tx_count + 1e-5) AS ratio_revenuepertx_count,
revenue / (returns + 1e-5) AS ratio_revenueperreturns,
margin / (tx_count + 1e-5) AS ratio_marginpertx_count,
margin / (returns + 1e-5) AS ratio_marginperreturns,
tx_count / (returns + 1e-5) AS ratio_tx_countperreturns,
-- Segmentation Layer
CASE WHEN margin > 0.4 THEN 'High' ELSE 'Low' END AS margin_profile
FROM base_data;And now write it such that all the conditions and transformations are injected into the string (somehow) rather than written in explicitly. Much worse.
The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).
The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
(Or a more general question: What is the best resource for me to read about how the streaming engine and cache work?)
In that specific case I use a Polars wrapper in Elixir (called Explorer) all week long, and I am very happy they are giving us early hints.
On the plus side I would spend all day hearing I am "absolutely right" from a superior being.
(But I agree that "land" is fine here, and the rest of TFA doesn't strike me as obviously AI-written. And I'm not a fan of the "look, they did one thing that AIs often do! Must be AI and therefore bad!" thing in any case.)