Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> You should be using a database to keep track of experiment results, model lineage etc. even if it's just a sqlite file.

You should be keeping track of those things, but I see little benefit and lots of downsides in using an SQL database to do so.



What are the downsides? Do you work in a team?


Hard to make it visible to others. Can't use standard VCS/diff/etc., so you don't get history unless you store it manually. Limited and cumbersome types, e.g. you can't use sets in a natural way (I think sqlite may now have some sort of support for JSON columns but they're clunky).


Fair enough if it’s not your tool of choice, but I think you may have missed a lot of what SQLite can do these days.

https://antonz.org/sqlite-is-not-a-toy-database/


I don't think any of that contradicts anything I said?


You can import and export CSV, JSON, etc (including parquet via an extension) which gets you VCS diffs for history. Set operations are built in (though I admit they may be as “natural” as you like), and there’s been decent (though, again, your idea of cumbersome may be different than mine) JSON blob support for some time.

It’s a tool that consistently gets underestimated at the same time it’s also consistently improving.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: