Pandas is terrific, yet even its original author has noted inherent shortcomings [1], and there exist alternatives.
Polars seems to be the most prominent competitor in the Python DataFrame space, and DuckDB appears to pursue an approach similar to SQLite, but columnar.
I am personally working on a solution to a broader problem, which can also be viewed as an alternative to Pandas [2].
For your link [1], many of those issues have been addressed with pandas 2.0 (which I believe Wes Mckinney [pandas' original author] contributed to). So it's a bit disingenuous to point to that post and say "See? Even Wes disowns it!"
That being said, if I were to start a new project requiring that kind of work today, I would probably try Polars first. Their greenfield implementation allowed them to get rid of many of the crusty edges of pandas.
Build this as an interactive tool for our popular 101 Pandas exercises. The code runs entirely in local in your browser. Would love feedback on the ease of use and the editor UX.
You'll get a lot of responses saying Polars is better than Pandas. I argue those people are missing the point and don't understand Pandas' real strength or why people choose Pandas today.
Pandas was never meant to be a technologist's tool. It was meant to be a researcher's tool and was unfortunately coopted to be a technical solution as well. It has not well escaped it's roots.
Pandas is fantastic for doing iterative and interactive research on semi-structured data. It has a lot of QoL facilities and utility functions for seamlessly dealing with exploratory timeseries analytics for in-core data. Data that fits into memory.
For example, I can take two time series and calculate their product:
ts3 = ts1 * ts2
This one line does a huge amount of heavily lifting by automatically aligning the timestamps and columns between the two inputs so that I'm not accidentally multiplying two entries that have the same ordinal but not the same timestamp or column label.
Can I do the same with Polars? Yes, but it comes with exponentially more cognitive overhead. And this is just one example.
Pandas is ultimately a flawed product as it's origin's go back more than a decade where R's dataframe was cutting edge. A lot of innovation happened since then and the API and internals of Pandas mean that certain choices that were made early on are nontrivial to change.
This doesn't change the fact that Pandas is still immensely useful. Eventually perhaps Polars will come close to it, but so far the focus wasn't on interactive use ergonomics unfortunately.
As it stands, I use pandas for research and polars for production systems.
Dope. I've just started using Pandas in some personal projects, and am quickly hitting my knowledge ceiling. I think this will be useful. I'll check it out properly after work.
The author posted a Polars version in the comments and almost nobody noticed. Meanwhile the top comments are still asking for it. Building something useful and having people ignore what you made to request what you already made is a special kind of frustration.
33 comments
Polars seems to be the most prominent competitor in the Python DataFrame space, and DuckDB appears to pursue an approach similar to SQLite, but columnar.
I am personally working on a solution to a broader problem, which can also be viewed as an alternative to Pandas [2].
[1] https://wesmckinney.com/blog/apache-arrow-pandas-internals/
[2] https://github.com/ronfriedhaber/autark
That being said, if I were to start a new project requiring that kind of work today, I would probably try Polars first. Their greenfield implementation allowed them to get rid of many of the crusty edges of pandas.
Pandas was never meant to be a technologist's tool. It was meant to be a researcher's tool and was unfortunately coopted to be a technical solution as well. It has not well escaped it's roots.
Pandas is fantastic for doing iterative and interactive research on semi-structured data. It has a lot of QoL facilities and utility functions for seamlessly dealing with exploratory timeseries analytics for in-core data. Data that fits into memory.
For example, I can take two time series and calculate their product:
ts3 = ts1 * ts2
This one line does a huge amount of heavily lifting by automatically aligning the timestamps and columns between the two inputs so that I'm not accidentally multiplying two entries that have the same ordinal but not the same timestamp or column label.
Can I do the same with Polars? Yes, but it comes with exponentially more cognitive overhead. And this is just one example.
Pandas is ultimately a flawed product as it's origin's go back more than a decade where R's dataframe was cutting edge. A lot of innovation happened since then and the API and internals of Pandas mean that certain choices that were made early on are nontrivial to change.
This doesn't change the fact that Pandas is still immensely useful. Eventually perhaps Polars will come close to it, but so far the focus wasn't on interactive use ergonomics unfortunately.
As it stands, I use pandas for research and polars for production systems.