* The useful part of the project is doing the rail connections across the mountains, which is unfortunately the expensive part. For various reasons, the project started with building the section in the Central Valley, which is the most useless part of the project by itself. If the project is killed without connecting to either SF or LA, then it really will be a useless white elephant.
* The best routing is far from obvious. The only easy way out of LA is roughly I-10 to Palm Springs, but that's the wrong direction for anything other than Phoenix and maybe Las Vegas. This means the decision is going to be contentious, and every person drawing the transit fantasy maps is going to be upset at it, thus spending more time bickering and less time supporting the project.
* The cities aren't very well set up for mass transit to take advantage of HSR. This is particularly true for non-SF/non-LA--Sacramento, San Jose, San Diego, Las Vegas, Phoenix (latter two aren't in CAHSR's plans, but they're easily close enough to have branches and are constantly suggested as useful extensions--see above point). Even LA and SF are fairly bad in where the "downtown" station ends up relative to major business districts,
* As a megaproject, CAHSR ends up attracting lots of little related projects that balloons the cost. Not only is it about creating a grade-separated electrified rail from SF to LA, it's about building an entirely new underground rail station, etc. The project becomes about trying to justify why you need to spend billions more on other desiderata rather than thinking about how to make do without those add-ons.