Depends on your definition of "ML practitioner", "building" and "ML". Look at the section on optimization - some people have an extremely good grasp of this and it helps them mentally iterate through possible loss functions and possible ways to update parameters and what can go wrong.
Some people see studying as a chore and want to learn the minimum to get the job done. Others find it insightful and fun and enjoy doing problems and reading material.
Both approaches make contributions and can lead to success, but in different ways.
I am curious about the same thing. I worked as a ML engineer for several years and have a couple of degrees in the field. Skimming over the document, I recognized almost everything but I would not be able to recall many of these topics if asked without context, although at one time I might have been able to.
What are others' general level of recall for this stuff? Am I a charlatan who never was very good at math or is it just expected that you will forget these things in time if you're not using them regularly?
To be fair, I think most of the TSA agents are not, many are just doing their job and trying to bring some dignity to a tense situation. I travel a lot and have met some very kind TSA agents.
But as an organization, they clearly have a culture that allows or even encourages people to openly abuse and harass travelers, and punish people for exercising their rights. When I was being sexually harassed by a TSA agent, the other agents standing nearby allowed it to happen and said nothing.
From “big program, small data” to “big data, small program” seems like a useful way to summarize the main shift from elaborate rules in the first generation, to huge piles of data today.
Yes, classes are data structures. A data structure is the construct by which you organize, process, retrieve, store, etc data. A traditional "struct" or "table" is how people think of a data structure, as a sort of dumb layout of 1s and 0s in a file or memory somewhere. But how that data is separated, accessed, what constitutes "valid" data, how it is presented to you, what you can do with it, etc is all part of what constitutes the "structure" of the data, because the "structure" enforces what that data looks like, how it behaves, etc. A class fits that purpose aptly.
If water is data, then a straw, a vase, a kettle, a pool, a water pump, even a hydroelectric dam, are all data structures. They enforce the form of the water, what goes in, how you access it, how it comes out, etc. You might say "but doesn't a function do the same thing?", and, well.... yes. A data structure is kind of like a tightly-bound collection of functions that take in data and put out data in particular ways.
A data structure in memory or on disk, without program logic attached to it to enforce the structure, is a "data format". The hope is that a program will do the right thing with the data according to its intended structure.
IMHO - “class” definitions are really just a data structure (abstractly, a possibly-nested hash) with associated functions, and that’s it.
This is extremely apparent in Perl, where that’s literally what a “class” is - a hash data structure “blessed” with a class name. JS is almost identical (a class is a hash where some of the values are functions), and “under the hood” both Ruby and Python are similar (although in very different ways - consider that the first arg in a member function in Python is the instance).
Database schema is (usually/mostly) just that structure without those functions; then you maybe use an ORM to provide them again.
So, IMO, on a theory and design level, both the database table and the data part of a class definition are “data structures” (note that you should include in considerations the relationships between data structures)
To take this a step further, IMO a big issue with how OOP is taught & practiced is that people approach it from the wrong direction.
As classes provide both data and functionality, it's better to first model your data and then think about the functionality to provide on those data. But people tend to think about relationships & actions (i.e. functionality) first and model their data to that.
The two approaches will tend to produce very different data structures. I'd agree with the sentiment from the OP that being data-first is ideal.
Like others say, usually just go for XGBoost, which is increasingly even shown (proven?) in literature, eg:
https://arxiv.org/abs/2106.03253
You can also start with scikit-learn's `LogisticRegressionCV`, if a linear model is more palatable to your audience.
The bigger challenge is to reliably estimate how good your model is.
It's not about getting the best performance – it's about getting the "real" performance. How will your model _really_ do on future unseen datasets?
The answer to this challenge is cross-validation. But what are the questions?
There are 2 very different questions for which the answer is cross-validation.
One is, which hyper parameters to use with your model?
The second is, what is the generalization performance of the model?
This requires 2 separate applications (loops) of cross-validation. The authors of this book talk about this in terms of having a "validation set" and a "test set" (Sometimes, these terms are switched around, and there is also "holdout set". It's critical to know how you, and the rest of your team, are using these terms in your modeling)
A robust way to implement these 2 CV's is with nested cross-validation, readily available in many packages – and also, should be "fast enough" using modern computers.
One exercise that remains is: with nested CV, which model do you pick as your "production" model?
> One exercise that remains is: with nested CV, which model do you pick as your "production" model?
Why even pick one, just make an ensemble out of all of them. Unless you need your model to remain blind to one test or validation set for some reason, but then you should probably choose at random anyways to avoid bias.
Of course this being HN somebody is going to mention some case where they only have the CPU or memory budget for one model. But in general making ensembles of multiple XGBoost models is a great way to get better model performance.
What makes you think I don't shitpost in the #engineering channel?
And heuristics don't even scratch the surface of the bigger problem where it's trained on people who aren't great at their jobs but type a lot of words on slack about circling back on KPIs.
Through regularization techniques, data augmentation, loss functions, and gradient optimization, ensuring the model focuses on meaningful patterns and reduces overfitting to noise.
It’s not obvious how any of those would do anything but better approximate the average of a noisy dataset. RLHF might help, but only if it’s not done by idiots.
