Thank you. I can always xargs grep for a function name, at worst. Dir() in python at a debugger for other things. With YAML, kubernetes and other devops hotness, I frequently can’t even find the relevant scripts/YAML that are executed nor their codebases.
This also happens with configuration based packaging setups. Python hatch in particular, but sometimes node/webpack/rollup/vite.
Have you tried polars? It’s a much more regular syntax. The regular syntax fits well with the lazy execution. It’s very composable for programmatically building queries. And then it’s super fast
I found the biggest benefit of polars is ironically the loss of the thing I thought I would miss most, the index; with pandas there are columns, indices, and multi-indices, whereas with polars, everything is a column, it’s all the same so you can delete a lot of conditionals.
However, I still find myself using pandas for the timestamps, timedeltas, and date offsets, and even still, I need a whole extra column just to hold time zones, since polars maps everything to UTC storage zone, you lose the origin / local TZ which screws up heterogeneous time zone datasets. (And I learned you really need to enforce careful manual thoughtful consideration of time zone replacement vs offsetting at the API level)
Had to write a ton of code to deal with this, I wish polars had explicit separation of local vs storage zones on the Datetime data type
I think pandas was so ambitious syntax wise and concept wise. But it got be a bit of a jumble. The index idea in particular is so cool, particular multi-indexes, watching people who really understand it do multi index operations is very cool.
IMO Polars sets a different goal of what's the most pandas like thing that we can build that is fast (and leaves open the possibility for more optimization), and clean.
Polars feels like you are obviously manipulating an advanced query engine. Pandas feels like manipulating this squishy datastructure that should be super useful and friendly, but sometimes it does something dumb and slow
From a brief look, I think he's trying to position it for "Exploratory DOM Analysis". The intro demo looks scheme/smalltalk like in that you are creating the structure and primitives as you go (as demonstrated by changing 'ivory' to 'red' and watching the syntax highlighting change in realtime).
I understand how we got here, but it's a shame that javascript frameworks and libraries aren't easier to just play with in the browser. It's just JS, you should be able to play with it quickly in a lightweight environment. This approach excels at that. This approach brings back the whimisical possibilities of HTML/JS. I'd love to see more stuff like that and less TS, rollup, webpack,...
edit:Actually after reading a bit, this is being proposed for data analysis. I think that's a poor fit for this approach
To be specific, the suit against the mother was against the mother's estate, since the mother was murdered by the shooter... like right away. The suit was settled by the estate.
The suit against Remington ended in a settlement, probably because Remington didn't want a chance in hell to set any legal precedent. The fact that the families got settlements is really a symptom of how unsettled the issue of gun control is in America. Like it's completely inane that it's fully legal to manufacture and sell AR-15 rifles to basically anyone, BUT that somehow marketing them to civilians is inappropriate. Remington settled because they just don't want any possibility of the status quo moving against them.
This is not even remotely true. I have done a decent amount of shooting, some dedicated training, and own multiple firearms of different types including AR style rifles. Your sort of rhetoric is at best disingenuous and not even remotely true.
If you have ever trained with any rifle you will quickly realize that while there are hunting oriented semi-automatic rifles out there, the minimized recoil, the high rate of fire, the lightweight nature, and all the ergonomic accessories make AR style rifles incredibly fast and easy to shoot. Using a red dot site you can fire two rounds to the chest and one to the head at 25 yards in under 2 seconds with a small amount practice and training. Minimally trained people can do the same with iron sites in under 3.
I am a big fan of the AR platform because of these reasons. They are not unique to the AR, but they are unique to a class of gun that is designed with these characteristics in mind. These are not the characteristics of hunting rifles.
Honesty is important, even if it works against your beliefs!
> I don’t know exactly what compensation they should get, but this does not seem like a healthy or sustainable way for our society to deal with tragedy.
I don't know if it's healthy or sustainable, but it definitely sounds healthier than ignoring the tragedy altogether.
Agreed. It doesn't seem like a long-term solution, but it is the best way we have _right now_ to visit consequences on people/orgs that enabled the tragedy. If our society sees everything in cost/benefit, then increasing the costs of actions that lead to tragedies like this is one of the best things we can do.
They did, if even indirectly. Just like how McDonald's holds some responsibility for the obesity epidemic.
The company that makes rifles makes them to be sold. It is in the company's best interest that as many mass shootings happen as possible. By providing guns, they DID contribute to the tragedy. We can tell, because if they had never produced that gun then it would've never shot anyone.
This doesn't even touch on the fact that the reason gun laws are so lax is because these companies lobby for it to be so. Again, they are incentivized to cause as many people to die as possible. Incentives matter. If mass shootings were the next blue jeans, these companies would quickly overthrow Apple.
Blame is very hard and tricky, but any institution or system in place is responsible for an intuitional failure. And that's what mass shootings are - an institutional failure.
> Not suing others for millions or billions and spreading misery. Nothing can bring those kids back.
> Maybe the government could have offered education and employment guarantees to the families?
The lawsuit wasn't about responsibility or compensation for the school shooting. It was about the years of harassment and death threats that the families of those killed had to endure from people who believed the lies that Alex Jones repeatedly told about them.
> Not suing others for millions or billions and spreading misery. Nothing can bring those kids back.
How about not slandering the parents of the victims causing Jones' followers harass and threaten them? He could have admitted he was wrong (which he only did finally at trial and under oath - far too late), but chose to double down. What about that misery?
Jones is not a victim here. He chose greed, but got owned. The motives of the families, lawyers, etc are whataboutism at best. You're essentially arguing that if somebody throws a punch at another person, said person has no right to hit back because hitting back won't take away your black eye.
> but this does not seem like a healthy or sustainable way for our society to deal with tragedy
I don't know, this, to me, is the proper set of incentives. Nobody wants to lose money, so you better do everything you can to prevent these tragedies. If we just sob a little and move on, the systems in place will not change.
I am working on a somewhat similar open source project. I intend to add heuristic data cleaning. With the UI I want to be able to toggle between different strategies quickly - strip characters from a column to treat it as numeric, if less than 2% or 5% of values have a character, fill na with mean, interpret dates in different formats - drop if the date doesn't parse. The idea bing that if it's really quick to change between different strategies, you can create more opinionated strategies to get to the right answer faster.
Happy to collaborate and talk tables with anyone who's interested.
Yes I do have plans for data preprocessing using DuckDB WebAssembly (I have upcoming features secion in this blog: https://kengoa.github.io/software/2024/11/03/small-software....) but this will require SQL which some of the target audience might not be familiar with. I'm thinking of something like visual query builder from metabase.
> With the UI I want to be able to toggle between different strategies quickly - strip characters from a column to treat it as numeric, if less than 2% or 5% of values have a character, fill na with mean, interpret dates in different formats - drop if the date doesn't parse
Those are really good examples and I can make those predefined preproccesing techniques available as toggles in the dataset tab. Thanks for the feedback!
not quite what you're describing, but I open-sourced a fuzzy deduplication tool last week: https://dedupe.it
Would be interested in expanding it to deal with data cleaning more broadly
i just had a silly idea. around 100 years ago there used to be these devices you'd put a scroll in and it would scroll it[0]. the use i saw for it was maps, in cars. you'd insert like interstate 10 scroll #N for whatever section you were on, and as you passed mile markers or exits you'd scroll it. i hope this explains it.
So the idea is to replicate that and use receipt paper, and thermal or what, dot matrix print onto the roll of paper your tweet stream. then you get something like those plastic M&M bottles, pill bottle, 35mm film bottle (boy i am full of ancient tech ideas)...
if you make some 3d printed cheap compliant mechanism[1] that snaps together and everything fits in a small tin or box...
Curious to hear about other people's experience with Tanstack. I built a complex app [1] around AG-Grid and here are my thoughts:
I would recommend AG-Grid for most data apps until a well supported alternative comes out. My understanding is that Tanstack Table is aimed at generic apps that happen to have a table. Perspective's [2] table component is most likely more capable than AG-Grids, but it needs significantly more documentation and examples.
AG-Grid has a particular way of building things that seems to map to well to pulling data from a database or basic webservice. AG-Grid works very well for this use case. The AG-Grid enterprise features are built around this type of data model too.
If your data is small enough to fit into memory (less than 50 MB), it works very well. AG-Grid doesn't have a connector to arrow or parquet (I will eventually write my own). For larger datasets JSON generation and parsing performance bottlenecks. Additionally Parquet/Arrow offers much better types.
Configuring AG-Grid is tricky, especially when you want config to be declarative. If you want to define your own custom rendering components, getting them to work properly per column when you want different renderers for pinned rows is tricky (my solution [3])
Support for the community version of AG-Grid is practically non-existent. I understand that they have an enterprise business model that charges for support. They don't seem to be interested in answering community bug reports or comments, even for generically relevant bugs.
I have a decent amount of trust in AG-Grid's community offering [4], I don't see them cannibalizing community features to put them in their enterprise offering. Their enterprise features are the main development focus of the team, particularly the charts. The charts offering isn't impressive compared to bokeh, plotly, or vega. It would work for line of business apps, but not for a userbase familiar with better analytic platforms.
All that said. AG-Grid has excellent documentation and examples. They examples are swappable between Javascript, Typescript, React, and Angular.
Would Perspective work for a similar use case as Causal's where the data tables have a large amount of interactivity, tree-based information and/or master-detail style UI concerns?
We're building a product that is more oriented around the interactive DAG concept and less about big data. The snappiness of those examples is very impressive.
This also happens with configuration based packaging setups. Python hatch in particular, but sometimes node/webpack/rollup/vite.
reply