Statements like this are hard to evaluate without knowing the denominator: the total value of new mortgage bonds in each period.
All too often, an author who should know better throws the reader a scrap like the following sentence:
> The market for unconventional home loans is still tiny compared with the rest of the mortgage market as well as its precrisis past, when unconventional borrowing peaked at more than $1 trillion.
But this still doesn't convey what percentage of the loans are to sub-690 FICO borrowers.
I see this all the time and wonder to what extent it has contributed to mistrust of the traditional media.
If we find out that the percentage in 2019 is 1% but in 2007 it was 56%, that casts the entire story in a different light.
'[X market/stock] slumps as [thing related to X] [does something]'
Yet when you go and look at the long-term graph, it's well within normal variance. There's no evidence they're connected at all. I'm sure it happens with other media providers too.
Why? Because they didn't have the numbers right, or just didn't check at all. Well, if I'd written a statement like that in an essay during my schooling, I'd be marked down for unsubstantiated claims at best or admonished for plagiarism at worst.
I find it irresponsible that the news rarely cites its sources beyond admitting they bought it from AP/Reuters. I believe it should be enough to prevent them being cited as a trustworthy secondary source until they at least have their justifications to a level that would be considered adequate by a high school history class. It's the only reason citogenesis happens on Wikipedia.
As you've noted, financial news is post-hoc analysis. It is narrative-based, and not fact-based. Sometimes the narratives and facts coincide, though.
> "Buyers outnumber sellers as equity markets post gains"
And those statements aren't even necessarily true.
Of course, there's still the matter of actually validating that hypothesis, and examining whether said hypothesis actually holds true or the coincident timing really is just coincidence and nothing more (or, indeed, if the timing really was coincident between the two events, e.g. whether or not the stock move happened before the event that allegedly "caused" it).
And of course, even if the event did cause the stock move, that's still a separate concern from the long term impact of that stock move. It could very well just be a temporary blip, or it could be a harbinger of some more significant change.
What a completely mathematically and financially illiterate thing to say. Whether or not a market move is within “normal” variance has nothing to do with whether or not it is clearly attributable to a particular economic event. If Jerome Powell says something about rates or Trump tweets something about tariffs and the market moves 1% in the exact minute that happens you can pinpoint the cause of the move with pretty high confidence even if 1% is not significant relative to the long term variance.
I’ve seen this bizarre sentiment propagated on HN before. A lot of people seem to think it’s never possible to identify the causes of market movements (even when obvious market moving news is released).
That's not a phrase I used. I'm sorry the sentiment upsets you; please don't conflate it with mine.
My spouse works in e-commerce as an expert in international payments systems. Her undergrad degree was an Art/Asian Civilizations double major (with honors work in historic Chinese painting), and her master's degree is in Chinese pedagogy.
By your standards, she's not qualified for the field she has worked in for the past 15 years. Then again, you can't exactly get a college degree in international payments.
The other problem is that for-profit media would never have an incentive to hire journalist with quantitative skills. There's very little demand for it outside of trade and professional presses and I would bet media companies would have to pay quant journalist at least twice as much as regular ones.
So why should media companies hire and publish quant journalists? There's no good reason.
And that's also why journalism programs don't teach serious quantitative methods - there's not going to be demand for journalists with those skill sets.
The obvious exception to this is trade and professional publications, but those are often subscribed to by businesses with a vested economic interest in the information.
So that you can take them seriously. Then again, modern journalism is such a joke I don't know if this is a worthy endeavor
Now that many of the writers aren't anonymous anymore, is there anybody still saying that it's all young 20-somethings?
The specific example I used above is actually perfectly terrible because I think we do still have accountability in a lot of the "trades" since so many people work independent of large firms and feel empowered to call out bullshit, but that's sort of what I wanted to highlight - it's when we're talking about an industry that is largely consolidated or centralized (like say banking) where everyone who knows what's going on is employed either by the primary party or a friend of the primary party that we get a breakdown of the truth.
1. I.e. Those iraqis have weapons of mass destruction because intel and this mustard coloured powder.
(Anecdata, but why not?) In the run-up to 2008 I was having regular random conversations with non-technical people about the fact that the bubble was about to pop. It was obvious to them that valuations were divorced from all common sense.
But most expert economists and/or bankers were insisting the valuations were correct, and there was no cause for concern.
In this kind of context - and there's a very long history of it in finance and banking, so it's hardly a one-off - it seems a little strange to be picking on journalists for alleged innumeracy.
Put crudely, it isn't journalists who do the damage.
because college has moved away from being a luxury of broadly educating landed gentry to being a signaling of "i can be employed and responsible" the train a generalist approach doesn't really work.
To be specific, there are two types of journalists/writers
1. domain experts that are good writers/communicators
2. good writers/communicators with an ability to pick up domains quickly
at some point the 2nd type of person will be out of their depth. but the 1st type of person is expensive.
Why are Matt Levine and Adam Minter (even Krugman sometimes) such great journalists? All were domain experts first.
Great journalists that were not domain experts still exist, but they are more of the investigative variety. the ones that win Pulitzers.
many hacks are also the first type. see degrasse tyson, jared diamond, malcolm gladwell, arguably pinker. very little substantive base, but great storytellers
activists reduces to the second type imo. if you have domain expertise, you usually hold a more nuanced view and don't come off as an activist even if you are very active. those with a preconceived notion are typically not a domain expert and just use their superior communication ability to prove their axe
The best journalists are aware of this and also remember the other lesson from English class, which is to consider counterarguments. But even then, there is never enough space for all the counterarguments, so those are cherrypicked too.
In 2015-16, there were 1.9 million bachelors degrees awarded, of which about 1.1 million were in business, engineering, science, math, psychology, or health fields, all of which would involve significant math/statistics. A further 160,000 were in social sciences, which typically involve a significant statistics component these days. So only a minority, 35-45% graduate in fields like communications and journalism, where math would not necessarily be part of the curriculum.
Words matter. They shape worlds. Ironic, huh.
Seriously, what is it going to matter what a few semi-anonymous comments on the internet are going to say? The news media has all the exposure and presumption of telling the truth, whereas people writing blogs and posting comments do not.
Words matter. If we're going to misuse words then we deserve the (intellectual) abuse we get from those we are allowing to run wild.
The Orwellian use of the word journalist / journalists is one of the most ironic quirks of the 21st century.
If we want to user the word journalism / journalist properly and fairly then the easy way to do is to think of it as being a verb - like leadership. Leader isn't a title. It's actions you take. Just like journalism.
Wouldn't the news there be that the total number of mortgages have grown dramatically? Unless you have a narrative to push that is.
The article title and first sentence are: Mortgage Market Reopens to Risky Borrowers. Strict lending requirements that were put in place after financial crisis are starting to erode. The risky mortgage is making a comeback.
Supporting facts are:
- "Borrowers took out $45 billion of these unconventional loans in 2018, the most in a decade, and origination is on track to rise again in 2019."
- "Unconventional loans are largely being extended by nonbank mortgage lenders. But big banks have found another way in ... Some $2.5 billion worth of subprime loans, those with FICO credit scores below 690, ended up in mortgage bonds in the first quarter of 2019. That is more than double a year earlier and the highest level since the end of 2007."
Then there's the single anecdote about a borrower with a sub-690 FICO. Is sub-690 arbitrary? Yes. Is this borrower an example of an egregious loan? No. But is this someone who could not have gotten a loan previously? Yes. It supports the article.
The article is fair in its assessment:
- "Big banks’ mortgage arms are still avoiding riskier borrowers, leaving them to nonbank lenders. Still, the increase in unconventional loans shows that lenders are looking farther afield for customers."
Let me put it another way: at what point should the WSJ write an article like this? The 2008 financial crises cratered the economy after previous lending standards became lax. We put regulations in place to prevent that from happening again. Those regulations are starting to be weakened again. I want to know that.
Let's say Peak 2006, Sub-Prime Loan was $600B
Let's say the average, natural Sub-Prime Loan is $100B
After the crash, may be the market over-corrected way to much and slowly crawling back to it's natural $100B.
With this perspective, the narrative becomes totally different. The lending standard is still too tight and still way below what the natural / average economy support.
Remember there has to be a balance between exuberance and over-cautious. A journalist should find out, what the happy medium is
That's why numerical literacy is important
> Some $2.5 billion worth of subprime loans, those with FICO credit scores below 690, ended up in mortgage bonds in the first quarter of 2019. That is more than double a year earlier and the highest level since the end of 2007.
Although the phrasing makes me wonder if more subprime loans are being originated or if more are just being securitized as opposed to being held on balance sheet.
To illustrate why this matters. My modern FICO score is something like high 700s, low 800s (depending on credit data vendor). My V2 score is like 690.
How did that happen? Turns out when I moved out from my last house (5 years ago) and canceled my internet with Time Warner they failed to charge me $31 which was always set to auto-play. I actually settled 5 months after the move when TW sent me to collections. The shady collection agency promised to remove it off the record if I paid (they didn't). Instead, it shows up as a 5 year old, $31 late payment, paid in full.
How did I find out, while looking to refinance at these current rates. I'm getting this sorted out now. Both TransUnion and Experian got it fixed in a few business day.... fucking Equifax cannot get their shit together 3 weeks later.
For me it's a hassle, annoyance and wasted time. But as you can see it can impact real people and the score methodology is pretty dumb.
It's insane that a 5 year old, paid in full debt for $31, that's not even my fault drags my credit score down ~100 points (that's what it is once corrected) and prevents from getting a refi. Doesn't matter that all my other credit cards are always paid in full, no late payments on mortgage, car, insurance ... which all add up to several magnitudes more then $31 over the 5 years.
Also, this is a TERRIBLE example of a pending subprime crisis. A guy borrows $675K on a $1.1 million property.
Well isn't the point that you/they are making that determination on equity alone? As we saw in 2008 all that equity can disappear in a blink of an eye. Equity has no bearing on ability to repay. So before deciding if this is a good or bad loan wouldn't we need to know his outstanding debt to income ratio?
And neither Equity, or ability to repay (debt/income ratio) has anything to do with subprime lending or a subprime lending crisis. Subprime generally means enticing debtors with interest rates below prime, with a loan that will adjust to above prime (and many times even into a single balloon payment at the end which statistically almost no American can make). No one should qualify for such loans on the basis "you may be able to refinance again at the end to avoid the balloon payment", if you don't have the cash on hand to pay the balloon payment, you shouldn't qualify for these types of loans.
No, subprime lending refers to riskier loans, usually with commensurately worse terms, offered to people with less-than-prime credit status. It has nothing to do with the prime rate, and I have no idea where you got that idea. I mean, as folk etymology goes it's superficially plausible, but it's one of those things that I think wouldn't survive even the most casual contact with the use of the term in the wild.
Everything you say is true, but its also true over 90% of subprime loans were ARMS (adjustable rate mortgages), leading to the subprime mortgage crisis, where the interest rates started off below prime and gradually increased to over prime to compensate for the risk of the borrower.
2. Subprime refers to the class of borrower, typically based on their less-than-ideal credit scores. The term does not mean that the bank is offering a rate below the prime interest rate to "trick" potential borrowers into taking on debt.
Its not a trick, just something the Borrower's en mass did not understand. They just understood the initial low payments, anyway as I replied above, leading to the mortgage crisis over 90% of subprime loans were ARMS that started off below prime and gradually increased. I did use the word "generally" because its not all, but I think 90%+ is a good use of generally.
In theory the banks didn’t put a proper interest rate on the loan, since interest rates are mainly to compensate for the risk of lending to the borrower. Although it doesn’t matter since the banks bundled the mortgages together, sold them to each other, and were bailed out.
Yes, but most people are not entering legal agreements that require them to understand algebra. One would think, especially with the most uneducated and highest risk borrowers, it would be necessary to understand the initial payment under ARMs are temporary for 3/5 years (depending on the terms, but 3/5 year were the most common) and thereafter the monthly payments will go up (statistically by 75% - so your $1,000/month mortgage payment will go up to $1,750).
>In theory the banks didn’t put a proper interest rate on the loan
Yes, that was one part of the problem (obviously the most uneducated and riskiest borrowers should not have been eligible for these complex adjustable rate mortgages), why on Earth would the riskiest borrowers have gotten loans with temporary interest rates below prime? These were the most likely borrowers to not understand adjustable rates. Other major issues were of course stated income (no proof of income required) and 100% to even 103% financing (no downpayment, banks will even pay your closing costs).
>Although it doesn’t matter since the banks bundled the mortgages together, sold them to each other, and were bailed out.
That is exactly why it matters, because laws were passed to prevent these kinds of loans, and here we are full circle, and the banks are offering these loans again.
No, that is exactly what it means.
> Subprime generally means enticing debtors with interest rates below prime, with a loan that will adjust to above prime (and many times even into a single balloon payment at the end which statistically almost no American can make). No one should qualify for such loans on the basis "you may be able to refinance again at the end to avoid the balloon payment", if you don't have the cash on hand to pay the balloon payment, you shouldn't qualify for these types of loans.
No, those are teasers with a low-fixed rate for 2,3 or 5 year, with the rate resetting to a very high adjustable rate after the teaser period. Teasers technically are 30 year amortizing mortgages, but you are correct in that they should be treated like balloons, because the payment after the reset is higher than the borrower can typically make.
In an environment of rising home prices, it should be easy to refi on similar terms at the end of the teaser period, but if prices are not rising, you can see what happened in 2008.
Not "teasers" but ARMS, and 90%+ of subprime loans were ARMs leading to the mortgage crisis.
Is anyone's experience that "non traditional media" or news sources are more mathematically rigorous?
I understand that all media has bias, lots of content is written by non experts, and presenting data (especially statistical data) is really hard and rarely done well. But any "alternative" or "non mainstream" news source I've seen is _way_ worse at those things than traditional mainstream sources like NYT, WP, Economist, Foreign Policy, NPR, 538, etc. Even while those sources still make noticeable blunders.
(Curious because I've seen this statement before.)
I'd say it's more likely that people are mistrustful of the media because the media will say things they either don't like or don't agree with personally, rather than a specific concern about articles lacking certain context on complicated topics.
Actually, this article differs in no way from similar articles written since printed news was invented. Which of course means that all the other criticisms, including imprecision, are still valid. It just means that it is not the source of the recent spike in media dislike.
And the number of defaulted loans/foreclosures was tiny compared to the total toxic (sorry, not letting them re-brand toxic loans as unconventional) assets. However, that tiny number of defaults/foreclosures, was enough to overturn the entire economy...but for Bush's $1T+ bailout followed by Obama's $1T+ bailout, banks and wall street would have crumbled, instead the got to consolidate with their new taxpayer cash on hand.
The banks and wall street recovered, of course homeowners and American working class taxpayer never did, so its time to run the whole scam again and get those devious working class American's who escaped 2008 unharmed.
I am financially ready to purchase my first home, currently living in the bay area, but I think right now just looks like a bad time.
- A lot of housing price growth is seemingly "priced in" since rents for condos/apartments significantly lower than total monthly ownerships costs (mortgage+hoa+insurance+taxes+etc.), even with 20% down.
- The major high paying tech companies are expanding mostly outside of the area. The fresh batch of big tech companies, outside of Airbnb, have an unclear path forward.
- Subprime lending, interest rates, the current PEs of REITS etc. seem to indicate an impending housing contraction. So interestingly, dumb money (the consumer lending market) is pricing in growth while the smart money is indicating a contraction...
- A lot of VC backed companies are looking bubbly, so if you are at an actually profitable tech company in the area, your relative purchasing power for housing would likely improve if the bubbly companies went under.
That said, the bay area market has been slowing the past year or so
I'm not sure your first point is particularly valid; I think that's basically always true here. Rent is cheaper in areas where assets are expected to appreciate quickly; rent is relatively expensive in slow/stagnant markets (where, for example, maintenance could be a significant deterrent to ownership, versus price appreciation).
I'm not sure at all about your second point, that sounds highly speculative.
But it sounds like you've done your homework on the last two.
Obviously, past results don't necessarily predict future returns, but it is starting to feel a little like 2005-2006 again.
So in other words, you think that more expensive homes value changes at the same rate as a basic home? Did you look at any data to make that claim or just an educated guess? I'm thinking about getting data from the local property appraiser to check that out.
In spite of the high wages possible equity it still seems risky/unfeasible to enter Seattle's housing market right now. First I would have to get lucky that a 400k house/condo would appear on the market. Then I would have to come up with 40k for small down payment on the cheapest house/condo in the metro area. Assuming I wasn't outbid by a developer or someone with more money than myself, I'd then have a nearly 2k a month mortgage - in addition to any maintenance/repair costs we might(will) incur.
If I or my wife lost either of our jobs that would be a stretch, and we have pretty good jobs. Our family is all 2500 miles back east. Too much to lose.
I eventually made the decision that my house is not an investment, it's a home, which will I will use for shelter, and that I do not plan to sell for 20+ years. So long as I can afford to make the mortgage payments and not be house poor, I will be reasonably happy and this should work out.
Through that lens I figured even if the housing market were to collapse, I will have 20 years to make up the difference.
Right or wrong, this helped me frame it in a way to so I remove the timing aspect.
I think that for the amount of property I could afford now, I would probably want to trade up for a larger property after about 5-15 years once I have kids. Or I might take a job in NYC around then as well. So my outlook is a bit more short term, and the flexibility of renting carries a bit of a premium.
The benefit of your approach is that you will probably weather any short term decrease in market value of your property and make up for it over time. That definitely works if you are willing to really commit to an area for a long time
As usual, if you wait too long then you will miss out on potential gains. Starting as soon as possible tends to be more important than the specific time you start investing.
Now, if you expect an economic down turn, there might be less people working and people who can't afford their mortgage, but imho that's not going to dent house prices in areas like these much, unless the downturn is for something like a decade. However, cities with sprawl or are effected heavily due to downturn will have a more variable price. A notable example is Las Vegas, where the majority of the industry is funded through tourism.
Not putting a probability on that, but we’ve never seen a teleconferencing platform with eye contact and body language before. I have no idea if that’s the limiting factor, but it seems at least plausible. FB, Google, and Apple will almost certainly all be selling $500 devices that can do those two things inside of 5 years.
Is that enough to disrupt physical offices? I don’t know.
But the probability is not 0. Personally I doubt ‘p(by 2030 one tech giant allows most workers to choose to work in VR)’ is < 0.1.
I’m going to price that into any Bay Area property valuation I make. How many people are only here because their office demands it?
Personally I think it's more likely that some lawmakers eager to disrupt the status quo find a reason to break up big tech, and a bunch of the baby facebooks, baby googles, etc, relocate to cheaper cities when executives decide margins need to improve. Not sure if I see this happening by 2030 though, maybe by 2050.
One of the signals that the analysts used to correctly predict the housing bubble existing in 2007 was growth rates across mortage payments and rental prices. As mortgage payments grew faster than rental prices, and diverged, you have evidence of a bubble.
Currently in the bay area, the cost to rent and the cost of a mortgage payment for the same type of property are so drastically different that it is a signal we are still in a bubble. So many people jump into owning, only to become "house poor" because of it.
That said, prices are dropping so just be ready to pull the trigger if you find an opportunity.
Basically, once you own your house you are no longer at risk with regards to the overall housing market. It may crash, and yes you will lose some dollar-value, but so will other houses, so you will still be able to sell/buy another comparable house.
If you don’t buy however, You may buy a better home in a few years (if there is a crash), but if the price continues to go up, you may need more money to buy the house you could buy today...
If the answers are yes your probably in good shape. However, a lot of folks rationalize their way around the second condition; dragons be there.
If you want to buy in the Bay Area, I'd wait a year and see what happens. I don't think the price increased will outpace your savings by very much if you hold out and continue to add to your down payment fund.
But I'm just a guy on the internet, so don't blame me if I'm wrong. :)
I'm looking to buy a 2br condo somewhere between San Mateo and Santa Clara (inclusive) which should probably be hit the hardest, however even 10% seems not enough - not from an affordability standpoint, but from a value standpoint, I think even with some appreciation it would be better to keep renting. And there must be a limit to how much 1-2 br condo in the area can appreciate, since I assume it will probably always be pegged to at most somewhere around what a local software engineer salary with ~3-10 years of experience could pay for a mortgage. And they are already right around that level.
Google, Facebook, Apple are all expanding to Seattle, some in multiple locations
Amazon is building an office in NoVa
Apple is building in Austin. Looks like Amazon, Apple, and Google are too
Google, Amazon, and Netflix are building in LA
I'm sure there are many more expansions occurring too, anyone may feel free to list more or correct me
What were the locations of small, nascent local software industries perhaps bootstrapped by some ancient incumbents are now becoming real hubs. The cycle will continue as the cities with small/no local software industries grow and those in turn become hubs. What's clear is that the more stable, hyper-profitable software companies are not expanding as much in the bay area as they used to
They aren’t always bad, they aren’t always crazy balloon loans or predatory instruments. Sometimes they allow someone who has an alternative life circumstance to not be excluded.
If the rents are high relative to expected mortgages/recurring costs in your area, it will pay itself off very quickly (minus the risk of default you open yourself off to). And it allows you to lock in a monthly payment for potentially decades, which has a massive amount of value by itself.
It also makes a lot of sense if you have reasonable expectation of a large amount of income growth / expense decrease that will free up cash flow. For example, if a household member is finishing up school or in training for a more lucrative field (like an MD in residency) it might make sense to be house poor for a small period of time to lock in rates, take advantage of a nice buying opportunity, etc. Or if you are about to be released from child support / settlement obligations, and thus will experience some increased cashflow, could also make sense.
It's not fair to assume that everyone with poor credit or lack of capital for a downpayment is a bad person to lend to, HOWEVER, in the general case it's probably true that in the absence of impending changes in household finances, an inability to get good credit or accumulate savings is a negative signal. And unless you are a small lender, your subprime loans probably aren't discriminating against people who are about to increase income/decrease expenses while building better credit, and people who just suck with money.
There is a huge gulf between those who have unconventional sources of income and those who lack capital for a downpayment or who have poor credit. Poor credit is caused by things like not paying off previous loans / carrying high balances. Not having a down payment is a sign you have not yet demonstrated an ability to save enough, or don't have enough income to save enough, both of which seem reasonable criteria off of which to base a lending decision.
The parent commenter described a circumstance -- self-employment -- that is wholly unrelated to both poor credit and an inability to save a down payment. You made the leap between the two. We need to end this kind of thinking.
> a nice buying opportunity, etc
Equal housing laws mean that -- in the absence of very particular circumstances -- any opportunity for this hypothetical student would also be available to more reliable debtors.
Well, that might be a good thing when house prices are appreciating at a crazy rate, but it also locks you in when the housing market and/or the economy crashes. If you're renting, you lose your job and you're stuck with your lease for maybe another six months (assuming you don't break it). Once you buy a house, you're stuck with it until you can sell it or foreclosure, even if the price goes down or you lose your job.
Grumble grumble... we will have to stay disciplined and be honest with ourselves as to what the home is truly worth to us.
I ended up putting in offers on 8 houses before I didn't get out bid.
The neighborhood is predominantly aging Boomers, so it's possible in a decade there will be increasing sales volume, providing a better opportunity to get in.
Save even harder now :)
Few years ago, used to be have a 820+ credit score, earning £30k a year, had one barely used credit card that was always fully paid off, I would struggle to get approval for loans or overdraft, etc.
My depression and anxiety came back, got into reckless behaviour and drugs for a while, three credit cards, payday loans, random holidays on credit for no reason, down to 400s or so now. My banking app has started aggressively suggesting mortgages to me since last month, every time I open the app.
To those worried I'm turning things around, cut down on drugs quite a bit, payday loans all paid off, £2k left in credit card debt and will start earning £60k a year once I start my new job next month.
Or maybe everyone is just fooling themselves that this time will be different
Somewhat tangential comment but feels like it is in line with OP article...
Nope, not there yet.
Perhaps the data exists on this, but to my unresearched mind there's a lot of "fudging" potential within it all by itself. For instance, maybe the current 'borrower state' that gets you a 690 previously would have given you a 725; or maybe it's the opposite.
I understand how averages work, but is there any oversight or auditing done on the rating agencies and the ratings themselves? It just seems ripe for 'gaming' at that level IMO; if banks need to sell financial products and they buy the ratings from the agencies, surely they can lean on an agency to 'fudge' numbers one direction or the other (I'm speaking in aggregates here)?
Anecdotally, every time I look at my credit score I cringe -- I've never carried much debt, have always paid it off on time or early, and yet my credit score stays somewhat low, seemingly because I largely don't participate in "the system". Furthermore, every time I've ever applied for credit I end up with scores (and offers of credit) far in excess of the numbers I see when I either get my 'free credit report' from the 3 agencies, and they're similarly different than the numbers I see if I pay to see my reports from the 3 agencies.
I don't know, it just seems like a "chicken-egg" problem to me, and one where the rating agencies are perhaps incentivized in a way that does not align with the "economy" as a whole -- and is instead much more closely with their high-value customers (lenders). I also feel like there's so much "missing data", which could probably tell quite the tale, for instance, if someone such as Google was to begin a 'ratings agency'. Half of my finds that to be a scary thought, the other half of me thinks that we'd probably end up with objectively better outcomes, both economically and probably, in many cases, even individually too.
You, as a lender, can't call Equifax (a CRA) and say "Hey you said 690 but I think really 725." Their information says what it says regarding tradelines (the fact that they were delinquent on a CC 2 years ago is a fact on record that you can't alter), and you have no easy interface to modify that (though you can, and you do, encourage customers in marginal situations to do their own work on their credit).
You ask why don't banks defraud the GSEs on a retail level for marginal loans. The short and perhaps unsatisfactory reason is that if the GSE is dissatisfied with a particular loan they'll put it back on you (you have to repurchase rather than securitizing) and that if you make a practice of this you're going to go to jail.
Anecdotally, every time I look at my credit score I cringe The video game player in me understands that being told one has a score one wants to increase it, but, do you care about this number? Why? If you don't anticipate buying a house in the next 6 months it is all but irrelevant to you. If that is in your near term plans, then you have a fairly straightforward path to increasing it. (Stop paying off earlier than the reporting date, which can result in tradelines looking like they're dormant and therefore not counted towards your score.)
Furthermore, every time I've ever applied for credit I end up with scores (and offers of credit) far in excess of the numbers...
Yep, because the offers of credit are an actual fact about the world and the credit score is a point-in-time snapshot of a model which predicts a model that partially influences a decision that might be relevant to your interest. In the case where reality and your model differ, reality takes precedence, so why do you care what the model outputs?
What oversight is there on the ratings agencies themselves?
How do we know the distribution of scores is the same today as 10 years ago (and proportionate to credit-worthiness)? Is there anything that prevents credit-ratings companies from under/over valuing the credit-worthiness for large swaths of the population?
Is there historic data available to correlate "consumer credit score" to distressed notes -- and can that be done in a forward-looking manner, to exclude credit score 'fudging' as a factor, and to better be able to know that it's apples-to-apples comparison historically speaking. Otherwise, could it not simply be that borrowers previously known as higher-quality borrowers are simply being rated more poorly/cautiously, for instance? I'm not saying that is the case, but I've always felt that the incentives seemed strange.
Are credit rating agencies paid (by lenders) based upon their prediction accuracy?
When your business is extending credit (typically very low margin) you walk a very fine line between being overly permissive in lending and not lending enough. Minor movements of that line can be the difference between profitability and not.
If a ratings agencies algorithm was shown to be changing to be too aggressive (either historically or against its peers) the lenders will dial back its importance in their own decisions. In the extreme to the point of not paying for it anymore.
This internal competition does a pretty good job of making sure the algos represent the current state of the world.
There is a question of about this system causing future systemic risk to be underpriced because no one is incentivized to investigate it fully and add it to their algorithm.