12 years of experience doing backend systems, including five at various robotics startups doing data pipelines and automated deep learning training pipelines. Completing my Masters of Science in Robotics Engineering; actively looking for a new role in robotics, AI, or other interesting spaces. Currently investigating language enriched models and AI w/ reinforcement learning!
I've worked with this poster before for a few years before moving on to break into a different industry. Fantastic engineer, great person to work for. I highly recommend sending your resume their way.
Have been using go for 8 years now, and started with this. And to be quite honest, I still refer to this sometimes for quick lookup on certain things related to syntax, pools, waitgroups or channels. Beautifully done tutorial of Golang, can be done in a few hours and gets you a good base knowledge of the language.
12 years of experience doing backend systems, including five at various robotics startups doing data pipelines and automated deep learning training pipelines. Completing my Masters of Science in Robotics Engineering; actively looking for a new role in robotics, AI, or other interesting spaces. Currently investigating language enriched models and AI w/ reinforcement learning!
Innovator's Dilemma demonstrated wonderfully right here.
If VR is the next big thing, but will take years of niche smaller profit margin products to launch it, established tech companies will struggle to justify the products existence wherein a smaller player could go all in, achieve roughly equivalent results, and be quite content.
Yes. Interviewed someone that the team loved. We said definitely hire.
HR said they didn't accept the offer and went elsewhere. What they didn't know is that the person LinkedIn messaged me saying they enjoyed the team, and were disappointed we passed, respectfully asking what I would recommend to work on for their interviewing.
Opinions on whether to reach out after a rejection aside, it highlighted what I already suspected; the job was never going to be filled, and even as the team lead they were lying to me and the team.
When there was yet another round of layoffs months later, the spot remained, unsurprisingly, unfilled.
That is wild. Let me get this straight - the company actively lied to the team in an effort to get their hopes up that help was coming... by actually giving the extra overhead of sourcing a new hire?
Were you guys actively trying to fill the role (screen dozens of candidates, several went through final rounds, etc), or was this just some passive thing where recruiting tried to push someone who looked great?
We ranked candidates out of four, and this candidate was a four across the board. Experience in our tech stack, demonstrated competency, passed the lunch test, etc. The team was eager to get them started because we were desperate for more hands on. The team was at one point 6 and we were short handed, but through people resigning we were down to 4, with me taking the role of senior engineer and team lead.
This was an unequivocal HR lied and kept the mirage of "help is coming don't worry!".
Yes there's overhead of us spending time doing the interviewing, but a few hours here and there once a month is enough to keep the charade going. The applications site would go first to HR, then they'd put resumes in front of me. I always wondered why I couldn't just see everyone that applied and why it had to be filtered first; after this event I understood why. I suspect there was plenty more applicants than I was told of.
At my current job, we had a position that an absolutely stellar candidate supposedly ghosted HR on. Same thing; candidate reached out to me asking about next steps, and I was like, "let me check". Went and asked HR, and was told they'd handle the rest, and not to talk to candidates outside of HR-scheduled interviews. Never heard back again.
That was 3 months ago (and 3 months into the position posting), and the position is still unfilled.
Same question - were you and are you continuing to actively interview candidates?
That's the mystery for me, why put the team through a charade that actively harms their productivity?
The only thing I can think of is the candidate disclosed something that would be a legal red flag and can't really divulge that is an issue without threat of a lawsuit. Or something came up in a background check, but I imagine it didn't get to that point if the candidate reached out to you directly.
The "why" is because people quit environments that are toxic, but hold out if there are signs of improvement. People don't want to quit - it's a large investment of time and investment to job hunt. It's annoying as hell. So if you can string together promises:
- We'll hire more people, it's just so hard to find candidates!
- We're doing bonuses right after evaluations, but we're doing a new evaluation system that's taking a bit longer than we expected this year
- We're just finalizing talking to a customer we'll have them signed next month, we swear
That delays people looking, delays that large investment, and you can extract engineering time with little investment.
If you ran a company that couldn't afford another engineer for a team that desperately needed another engineer, are you going to paint the bleak picture for your team and risk losing those engineers that you have, or lie? If you have morals and say you'd be upfront, great; but that's unfortunately not a universal answer.
Harming productivity only matters for hourly staff (in the minds of bad executives leaders). Salaried folks have to finish the with no matter how long it takes, and it doesn't cost the company more.
This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track.
During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks.
The result of this is that each token has essentially even probability of being routed to one of the sub models, with the underlying logic of why that model is an expert for that token being beyond our understanding or description.
I heard MoE reduces inference costs. Is that true? Don't all the sub networks need to be kept in RAM the whole time? Or is the idea that it only needs to run compute on a small part of the total network, so it runs faster? (So you complete more requests per minute on same hardware.)
Edit: Apparently each part of the network is on a separate device. Fascinating! That would also explain why the routing network is trained to choose equally between experts.
I imagine that may reduce quality somewhat though? By forcing it to distribute problems equally across all of them, whereas in reality you'd expect task type to conform to the pareto distribution.
Computational costs, yes. You still take the same amount of time for processing the prompt, but each token created through inference costs less computationally than if you were running it through _all_ layers.
It should increase quality since those layers can specialize on subsets of the training data. This means that getting better in one domain won't make the model worse in all the others anymore.
We can't really tell what the router does. There have been experiments where the router in the early blocks was compromised, and quality only suffered moderately. In later layers, as the embeddings pick up more semantic information, it matters more and might approach our naive understanding of the term "expert".
It doesn't perform better and until recently, MoE models actually underperformed their dense counterparts. The real gain is sparsity. You have this huge x parameter model that is performing like an x parameter model but you don't have to use all those parameters at once every time so you save a lot on compute, both in training and inference.
Here's my naive intuition: in general bigger models can store more knowledge but take longer to do inference. MoE provides a way to blend the advantages of having a bigger model (more storage) with the advantages of having smaller models at inference time (faster, less memory required). When you do inference, tokens hit a small layer that is load balancing the experts then activate 1 or 2 experts. So you're storing roughly 8 x 22B "worth" of knowledge without having to run a model that big.
Maybe a real expert can confirm if this is correct :)
Would it be analogous to say instead of having a single Von Neumann who is a polymath, we’re posing the question to a pool of people who are good at their own thing, and one of them gets picked to answer?
12 years of experience doing backend systems, including five at various robotics startups doing data pipelines and automated deep learning training pipelines. Completing my Masters of Science in Robotics Engineering; actively looking for a new role in robotics, AI, or other interesting spaces. Currently investigating language enriched models and AI w/ reinforcement learning!
Yes and no. LLMs are definitely struggling with long horizon planning in complex situations, and can hallucinate/misunderstand environmental states such that they can recommend nonsensical or impossible steps. But we are starting to see more research that "grounds" LLM answers in a set state. Where they shine in the planning stage is they provide a sense of contextual heuristics we didn't have available before without a lot of expert domain programming.
LLMS in their current state are certainly not the answer, but point to some exciting possibilities that we couldn't do before because of sheer complexity.
I absolutely adore thess kind of DIY projects, where it becomes easily expandable and gets a good community empowering a feedback loop of better and better features.
Remote: Preferred unless it's a great fit
Willing to relocate: Yes, for the right offer
Technologies: PyTorch, Tensorflow, Python, Golang, AWS, GCP, Kubernetes, Robotics, LLMs
Resume/CV: https://github.com/hlfshell https://github.com/hlfshell/resume
Site/Portfolio: https://hlfshell.ai
Email: kchester@gmail.com
12 years of experience doing backend systems, including five at various robotics startups doing data pipelines and automated deep learning training pipelines. Completing my Masters of Science in Robotics Engineering; actively looking for a new role in robotics, AI, or other interesting spaces. Currently investigating language enriched models and AI w/ reinforcement learning!
reply