On the latency of the first request - How is the CFG cached?
Is it done at the API Key + schema level? Meaning that for a given API key, the latency penalty for a new schema is only paid one time, regardless of how far apart requests are? Or is cached with less duration, e.g. each session, conversation thread, etc?
I've read that SFT is good for "leveraging existing knowledge" gained during initial pretraining, and helpful in changing the way that the model responds, but not useful for teaching it new knowledge. In your experience is that true?
For example, changing the way in which it responds could be:
- debate me
- brainstorm
- be sarcastic
Which also seems like something that could be accomplished with a system prompt or few shot examples, so I'm not sure when SFT is the more appropriate approach or what the tradeoffs are.
Alternatively, gaining new knowledge would be training it on a dataset of e.g. sports trivia to make it highly effective at answering those types of questions.
P.S. nice username... Irving Fisher would approve.
Books in the little series are all about "learning by doing", and teach using the Socratic method (question/answer). Each concept is introduced with small problems that build upon each other. You are forced to get your hands dirty from the beginning, passive learning is not really an option. For most people, I think this helps facilitate a much deeper understanding than just reading a text.
I always had trouble with recursive functions when I was new to programming, and many recommended working through "the little schemer" to solve that problem. It was a tough read for me, but the investment was well worth it and it did for me what it said on the tin. I didn't have nearly as much trouble with recursion after that book, but an unfortunate side effect was developing an affinity for lisps which I haven't yet shaken.
Tough to answer since I've only read the two intro chapters currently available. I'm still in the early stages myself with with Deep Learning but have years of experience with programming and lisp, so I'm coming in with a background that might bias my views.
If you have never used lisp, you need to be patient with the notation and resist the urge to balk at what is unfamiliar. I remember grumbling quite a bit when I first went through TLS, but that phase was over pretty quickly and within days I had no trouble following the code.
Are there better ways to introduce deep learning and neural networks? Maybe.. but I like "little" books and there is no better way to learn something than by building it yourself. For that reason alone I'd recommend the book sight unseen (but with the knowledge of prior little books). I do think choosing what is likely an unfamiliar language for most may be somewhat of an impediment for an already challenging topic, but scheme is a simple language and allows the author to focus on the ML concepts.
I'm highly confident you'll learn a lot, if you put in the work. For DL as a separate topic, the best resources I've found are:
- 3 blue 1 brown Neural Net playlist
- Karpathy's "The spelled-out intro to neural networks and backpropagation"
Not sure on relative popularity. If you haven't already, you should try it! Here's why I like it:
- Very consistent, easy to keep the whole language in your head
- Stable, core rarely changes ("accretive only" mindset).
- REPL
- Focus on a handful of data structures which will solve 95% of the problems you face, rather than a deluge of complicated abstractions.
- Excellent concurrency primitives
- Access to Java libraries if a Clojure equivalent doesn't exist
- Macros
The only blemish for me was horrifically bad error messages.. other than that it's near perfect.
Is that not all programming languages. Practically have identical syntax for the problem they are solving. The thing which differs is “standard” libraries, which is not part of the language but a side effect of.
- async (I think async is a workaround for getting more done with one thread because of the GIL). Pointless on machines with increasing number of cores. Just get multithreading right (and I want to learn Clojure's model - focusing on immutability) and you will not need async. Python can barely share memory across threads. I'm not sure you can get by without serialization.
My last job was mostly C++, my current job is mostly Clojure but I still do a lot of C++ stuff on the side as it is appropriate for some of my side projects.
Clojure is so much nicer to work with and this really resonates with me; I'm always still having to check language docs for C++ stuff but only very rarely for Clojure. Being able to keep almost all of it in my head is a huge benefit as I don't have to break my "flow" nearly as much.
I'm not sure I understand the point being made here. Are you simply saying that higher rates means a higher monthly payment, and that the amount that prices have fallen is still not enough to offset the increase in the mortgage payment? That's fairly obvious.... for now.
I don't see how home prices can really fall enough to keep the cost of housing equal as interest rates rise. Because don't higher rates also disincentivize sellers and create more of a supply shortage as well, by taking more homes off the market, apart from disincentivizing new construction? I'm admittedly a little daft at economics but... e.g. I've got a fixed rate mortgage around 4%. Suppose I wanted to sell and I think we're at the top of the housing market... my options would be to buy something else now at the top of the market and pay a higher interest rate, or rent for a year until prices come down and probably still pay a higher rate.
Certainly, low mortgage rates drove home prices way up... but does that mean high rates could drive them down? I mean, maybe over a long period of time as existing mortgages get paid off (?) but for now while the vast majority of mortgages out there are lower than the going rate, won't that just keep constraining the supply and prevent the price of houses from dropping significantly?
This is pretty close to the most common counter-argument I've seen to the "this is another 2000s housing bubble" claim - the bubble burst when people had to sell or got foreclosed on, people taking out the mortgages today, or purchasing in cash, appear to be in much better financial shape to weather things and not have to sell.
You'll still have sales - some people will die, some people will have to move for various reason, etc - but how much it will fall seems like a very hard to predict question.
Purchasing cash wasn't really straight handing over a fat stack to the seller. A lot of "all cash" purchases were people who got loans through non-traditional means. So instead of getting a regular mortgage, they got financing through an institution who offered cash to the seller but is still a loan to the buyer.
A TON of investment properties were purchased over the last few years, so those may be sold as investors need to balance out the huge losses they took on stocks, crypto, etc.
With rising rates, the counter pressure on prices from buyers may not exist in large enough supply to prevent the investors selling from driving prices down.
Just anecdotally, a friend who took an enormous shellacking in crypto told me the other day he's glad he'd bought two rental houses before getting into DeFi. He seems totally disinclined at this point to make any high risk moves with what remains of his savings. I suppose if he had to sell that would be another thing, but I don't think most even-marginally-sane people in that position are going to sell property and chase higher returns in the market.
Rents aren’t dropping - so the investment properties are likely going to be better off held as inflation helps push down the cost of those mortgages vs rental income.
How does saying that demand increased more than supply reduced in 2020 contradict the idea the the overall size of supply in in-demand parts of the country is lower than the overall size of demand in those areas? Like, if you look at 50-year price histories in SF or LA, how can you conclude that supply has kept up with demand?
I think we've now just seen that same pattern catch up with more places. I know people in Austin and Houston who were complaining that prices were already getting too high before 2020. That's where the folks priced out of CA were going even then.
I don't know the LA market, but there is a scenario where both supply and demand could remain constant and yet housing prices could still rise, and that's if the income of housing buyers rises. And if the income rises disproportionally, then buyers with relatively lower income can become displaced by higher income people buying additional homes to rent. Even though base demand remained the same, and supply in the sense of actual houses remained constant, a relative scarcity was created by the inequality in purchasing power between the wealthiest buyers and people lower on the income scale. And this can happen even if the lower income tier has actually increased its income over time, simply because it hasn't increased at the same rate as the wealthier segment.
If I'm following, this scenario is basically "as the higher income/wealth people invest more of their money into more property, prices and rent can both continue to increase even if demand is fixed until the lower-income people are spending 100% of their disposable income on rent"?
Which is not entirely dissimilar to the general discussion around gentrification - those who can afford to pay more can displace those who can't - but a sort of special case that doesn't require migration.
Seems like there would be some sort of game theory aspect - this only works if something close to every landlord is pushing prices up?
I think what we're really seeing is more the high-migration "gentrification" case yet applied to populations that normally aren't who you think of as being the ones on the receiving end of gentrification: the 100M-dollar-buyers are pushing out the 10M-dollar-buyers are pushing out the 1M-dollar-buyers from California and then they go to Texas and push out the 500K-dollar-buyers... and it continues to flow out to touch everyone else too.
I think the point is that given typical demand, the amount of supply would have been sufficient to not cause meteoric price increases. Demand exploded due to incompetent monetary policy driving a speculative mania, and that demand was what gave the appearance of a supply constraint. Until recently housing starts were at a 10 year high, and they are still on the higher-end of the past decade: https://fred.stlouisfed.org/graph/?g=Rzky
It doesn't contradict that. The paper looks at the national housing market, which ignores individual market dynamics. Supply constraints still exist, but they don't explain the national increase in housing prices over the last 18 months.
Their argument is that there are fewer houses being built. In the long term, this will constrain supply far more than rising interest rates will push down prices.
15 years ago was a different animal because the money largely stayed in the banking sector, as excess reserves. This time, a substantial portion of it was deposited directly into consumer's accounts (stimulus checks) - and that money makes it's way into the real economy, pushing prices up.
Out of curiosity (not much experience with Actor systems), do these systems solve a different problem than having disparate processes communicate by sticking a distributed message queue between them? From my naive point of view it seems like alot of the scaling and fault tolerance concerns nowadays can be solved in any language, by having a queue be the interface between two services.
Same question applies to Erlang, which I realize is not exactly an actor system as per the below comment, and has a much more sophisticated error recovery story with supervisors, but the general question holds.
So I think the Erlang distinction is a little unrelated (I mean, it's technically accurate, but not because of the error recovery; that's an implementation detail. The original Actor formulation Carl Hewitt proposed was heavily influenced by Smalltalk semantics; whereas Smalltalk was truly OO, in that everything is an Object that passed messages, one issue it had was that objects did not equate to threads of execution. The Actor model, then, said each object executed independently. But in the same way that Smalltalk did not have primitives; even integers are always Objects, the Actor model stipulates there are no primitives; even integers are Actors). But I'll take a stab at answering what I think you're asking.
If your services don't deal with internal mutable state, nor high degrees of concurrency then there isn't much gain to be had with an actor system. That said, that begs the question of what the queue is for; just create more instances, since there's no internal state to share.
As soon as you start having internal mutable state and high levels of concurrency, that's where the actor model applies. Queues don't exist for concurrency (you don't need them; just create more executors), they exist for imposing sequence where it is needed (an obvious case; you have a DB connection, you want to only have one query at a time. So every desired query goes into a queue, and the process at the end that owns the DB connection pulls from it). Internal mutable state gets stored inside of an actor; updates and reads get serialized on that actor.
At the highest level, I would describe the actor model as taking a 'successful' model for distributed computing, and making it the only model you use, even locally.
In, let's say Java, for instance, using standard concurrency approaches, it matters where a process lives. My way of operating/communicating to another thread of execution (unit of concurrency) is very, very different than my way of operating/communicating to another machine. Locally I have threads and locks and need to be very mindful. When communicating to another machine, I send a message and that's it (maybe I expect a response, and timeout if I don't get one, but that's really just the same thing, the other machine sending a message).
I don't actually need a queue involved for theoretical correctness unless I need to process messages in sequence (after all, I could have multiple copies of the other process, and send a message to each of them). Now, in the real world I do, simply because if my concurrency gets too large it can't be handled by what the units of concurrency already available (instances, threads, whatever), and scale up takes time, but that's really just a special case of why I need to impose a sequence on messages (handle these first, then handle these, rather than handle all of them concurrently).
The actor model makes this the local model of communication (and so makes the impedance mismatch negligible between local and distributed; so much so that some languages it's actually irrelevant whether you're sending a message to a local actor, or a remote one). Scaling concurrency up internally just means spin up a new actor. When you need to serialize, you send messages to the same actor, where it ends up in a queue.
So it's not solving a different problem, exactly, if that problem is "how do we write systems that can do multiple things at once", but the specifics, complexity, etc, tend to be pretty different. The problems it's solving are a bit more subjective than simply "can we handle this problem", and more "how well do our tools and mental model lend themselves to the problem we're trying to solve".
I always found it strange that I really like salmon sushi/sashimi, but don't really care for cooked salmon. For some reason cooked salmon tastes too... heavy. I realize that's probably the fat content, but for whatever reason it doesn't bother me raw.
Is it done at the API Key + schema level? Meaning that for a given API key, the latency penalty for a new schema is only paid one time, regardless of how far apart requests are? Or is cached with less duration, e.g. each session, conversation thread, etc?