
I think Catalina 10.15.4 broke SSH - chmaynard
https://feed.tyler.io/so-uh-i-think-catalina-10154-broke-ssh/
======
neonate
[https://archive.md/lvCKA](https://archive.md/lvCKA)

------
floatingatoll
To summarize OP:

Their 10.15.4 macOS built-in ssh terminal command is unable to reach hostnames
when a port number higher than 8192 is used.

EDIT:

Comments differ; one indicates issues SSH'ing to lower than 8192 ports,
another indicates _no_ issues SSH'ing to higher than 8192 ports.

~~~
Apofis
chaos:1111

------
slovette
Catalina is broken in many ways.

This complain and Remote Access in (so I can SSH to my $4k MacBook) disables
itself anytime the computer is restarted.

But more importantly, I’ve still not found a Thunderbolt Display that doesn’t
routinely crash screen manager services upon idle user activity. 3 x $300
thunderbolt3 dock solutions later and not a one hasn’t crashed this computer.
All main brands, two of which sell accessories in the Apple store.

Problem also existed with a top of the line 13” MacBook Pro.

I’ve just gotten used to the shoddy-ness that is Catalina. Figure if I go to
the bathroom, upon return I have a fresh, new clean desktop environment.
Feature not a bug. Yay!

~~~
kerakaali
This is why I went back to Mojave. Apple has had a history of breaking dev
environments on release for people who don't code under their ecosystem of dev
tools (well on second thought, they make life difficult at times even for
people that do), and I don't see that trend changing in the future.

Eventually every new release has stabilised, but it seems that doesn't hold
true for Catalina.

~~~
ImprovedSilence
haha oh boy. I was actually just about to install Catalina today, figuring I'd
put it off long enough and everything has to be smooth by now (and system
update bugs me about it often enough)... But lo and behold, I log into HN and
see this thread....

~~~
eslaught
Just FYI, you can disable that notification:

[https://www.macworld.com/article/3447396/how-to-stop-
getting...](https://www.macworld.com/article/3447396/how-to-stop-getting-a-
reminder-to-update-to-catalina-in-macos.html)

It does not prevent the red notification dot on the System Preferences app,
but it does mean at least you don't get the notifications pop up on your
screen.

------
makecheck
So, some quick debugging here...

In his screenshot the bad login hangs at "Connecting to clickontyler.com port"
(noting that no port number appears and no period at the end).

While I can’t be sure exactly which "ssh" patch Apple may have, this seems to
be the relevant file and logging code (starting at line 448):

[https://github.com/openssh/openssh-
portable/blob/master/sshc...](https://github.com/openssh/openssh-
portable/blob/master/sshconnect.c#L488)

In that code, the only thing that can set the "strport" value that is used in
the log is a call to getnameinfo().

If that string is corrupted in any way, e.g. not terminated or perhaps has
invisible characters that trigger bad terminal behavior (such as
invisibility), the act of logging it might produce the apparent hang seen
here.

Again, a guess but it is possible that getnameinfo() is not necessarily
processing the record correctly (for whatever reason). One such example is in
the "getnameinfo" man page at the end, under CAVEATS, where they show an
example of not simply trusting the result of the first call.

~~~
tylerhall
Good sleuthing, but the missing port number is simpler than that. I just
blacked it out of the screenshot. I know very well that running sshd on a non-
standard port has no benefits security-wise, but it does lessen the length of
my log files from dumb script kiddies. I redacted the port in the screenshot
for that reason.

~~~
bo1024
A port is mentioned in this line, you may want to redact it. Where I put X's
below, is a port number.

> So, I tried ssh ip-address -pXXXXXXXXX

~~~
tylerhall
Thanks, but that's not the port number :-) That was just for illustrative
purposes.

~~~
bo1024
Ok great.

------
ProAm
"It just works" \-- Is Apple too large now? Is this a QA problem, product team
problem? Management? Catalina is still stumbling and Im surprised to be honest
after the past 4 years.

~~~
rvz
Well no business end-user or any typical Mac user is going to be bothered
about something technical like 'SSH' breaking their system. Only actual devs
here would care.

For those business users, it just still works. For developers it's a problem.

~~~
quantified
Apple's made huge inroads with developers over the last few years, partly
coasting off of a social dislike for Microsoft. There's enough Apple fandom
out there that they can probably annoy developers a good deal more without
affecting the inroads. After all, exactly what can a dev do about it anyway?

~~~
IggleSniggle
Switch to OpenBSD ;-)

------
JdeBP
Here's an actual bug report:

* [https://openradar.appspot.com/radar?id=4931259776106496](https://openradar.appspot.com/radar?id=4931259776106496)

From that and the discussions.apple.com. post, hyperlinked elsewhere in this
discussion, it appears that the >8192 condition varies according to what the
hostname actually is.

The bug report is datelined 2020-04-26, interestingly. There might be a bug in
the bug reporting system. (-:

~~~
oefrha
> The bug report is datelined 2020-04-26, interestingly. There might be a bug
> in the bug reporting system.

No, you can type whatever date you want. The "add a new radar" screen is just
a bunch of text input boxes:
[https://i.imgur.com/nNf457J.png](https://i.imgur.com/nNf457J.png)

~~~
JdeBP
The ability to type whatever date one wants is often considered to be a bug.
The ability to post-date reports a month into the future sometimes is, too.
(-:

~~~
oefrha
You can not only type whatever date, but also whatever non-date. The site just
assumes good intentions and is working as designed.

------
0x0
I can't reproduce this. macOS 10.15.4, ssh'ing to a very high (5digit) port
with a hostname no problems.

~~~
ajphdiv
I can't either, in fact all 10+ of the hosts that I routinely access have
ports higher than OPs issue.

~~~
andai
Offtopic but why are people using high port numbers? Additional security due
to a nonstandard port? If so, does that go together with anything additional
like port knocking? Or is it multiple hosts on the same IP, but different
ports?

~~~
vbezhenar
Some people think that it adds to security. Some people want to reduce noise
in logs.

------
vgene
I had the same problem on a MacBook after upgrading to 10.15.4. However, I
wasn't using a port number higher than 8192, the socket was 75 with a
hostname. The problem was solved when I replaced the hostname with its IP or
plugged in an Ethernet Cable. I tried to restart mDNSResponder and flush the
dns cache and switch to a different DNS server. Nothing works so far.

------
colechristensen
I experienced a similar issue with a git repository hosted on a high port,
`brew install openssh` fixed it even though the homebrew `ssh` was not first
on my $PATH. Didn't bother to investigate further.

~~~
moonchild
Possibly that overwrote the config file for the system ssh?

~~~
colechristensen
I was thinking homebrew's git perhaps had a different $PATH (or was using
shared objects?) that used the different openssh. Just guessing, didn't seem
worth my effort at the time.

------
acdha
I suspect this is due to a feature being enabled for canonicalization and that
the key part is the presence of the colon rather than the port number. On a
10.15.4 system, I see a line in the debug output which is not present in the
screenshot:

> debug1: resolve_canonicalize: hostname example.org:7999 is an unrecognised
> address

If instead I use `-p` or a config-file option, everything works as expected.

~~~
lilyball
hostname:port is not a valid destination according to ssh syntax. A
destination may either be [user@]hostname or a URI of the form
ssh://[user@]hostname[:port].

------
KiDD
"I don’t want to end up on Hacker News again bitching about Catalina." Pretty
sure that guarantees getting to the front page :D

~~~
phlakaton
This is truly the darkest timeline for that poor blogger. :-P

------
derefr
Is macOS /user/bin/ssh just upstream OpenSSH, or does Apple maintain a fork?
If no fork, this would be an upstream OpenSSH problem, no?

~~~
paxswill
Apple includes a customized version of OpenSSH. From what I recall from the
last time I looked at it, the changes were mostly integrating the key
retrieval mechanisms with the rest of macOS. For example, Apple's ssh-add can
store key passphrase in Keychain with the -K option, and then later access
those passphrase with the -A flag.

~~~
skoskie
If using the upstream version there is one line to add to a startup script or
to your zshrc (et. al.) file ...

ssh-add -A > /dev/null

... and one default value to place in your ssh config file...

AddToKeychain Yes

... to get around this issue. It works fine after that.

(On mobile. Sorry for formatting)

------
stock_toaster
Maybe a weird ControlMaster/ControlPath config? I have had issues with the
ControlPath result being too long with certain hostname/port combinations in
the past -- which resulted in ssh to ip working but ssh to hostname not
working. As a result, I haven since started using %C instead of %l%h%p%r in my
ControlPath config.

------
_-___________-_
If you have Homebrew or something similar, I recommend installing openssh
through there -- you get a newer version to boot.

~~~
Doctor_Fegg
Oh god no. Homebrew managing openssh has been the cause of more command-line
instability and forced reinstalls than anything else I’ve encountered in the
last few years of OS X (sorry, macOS). I’ve started installing stuff from
source again just to prevent a cascade of Homebrew upgrades breaking
everything.

~~~
rswail
Why don't people use MacPorts instead? I've never had any problems with it.

Homebrew wants to screw around in /usr, Macports installs itself in /opt and
doesn't interfere with things in the MacOS world.

Set your PATH to have /opt/local/{bin,sbin} and everything Just Works.

~~~
saila
What is the practical difference between _/ usr/local_ and _/ opt_ or _/
opt/local_? I don't think macOS puts anything in _/ usr/local_.

------
judge2020
In case it's slow for you too: <removed since they wanted it taken down>

~~~
saagarjha
It’s been taken down on purpose:
[https://news.ycombinator.com/item?id=22738841](https://news.ycombinator.com/item?id=22738841)

------
dimtion
I'm surprised nobody noticed this bug at Apple before the release. Is there
nobody there that connects by hostname to a ssh server with a port > 8192?

~~~
pwg
I'm not. Not all 'testers' actually try to test edge cases. The /good/ testers
do try edge cases, but for every /good/ tester you have, you'll have hired
100+ testers who do little more than check that the standard happy-path works
correctly and sign off as "passes tests".

The good testers all tend to fall into what Bruce Schneier calls the 'Security
Mindset' way of thinking:
[https://www.schneier.com/blog/archives/2008/03/the_security_...](https://www.schneier.com/blog/archives/2008/03/the_security_mi_1.html)

~~~
Hamuko
> _Not all 'testers' actually try to test edge cases._

Yeah, but surely macOS devs are eating their own dog food.

~~~
bangonkeyboard
"I've learned that Apple engineers have internal tools which allow them to
delete macl xattr as well as to bypass other Catalina privacy and sandbox
protections without rebooting and disabling SIP.

"Inside Apple they don't suffer the same problems as external users and
developers."

—
[https://twitter.com/lapcatsoftware/status/121929275891082854...](https://twitter.com/lapcatsoftware/status/1219292758910828544)

------
oefrha
One more data point: just tried to repro with hostname and port 8193 and
failed, so the issue is probably more intricate than described.

(Guest in my test: OpenSSH 7.6p1 on ubuntu bionic, stock config other than
sshd port.)

~~~
zimpenfish
Tried with dropbear and sshd on Arch, port 9022 from 10.15.4 Beta (19E258a) -
no issues with a variety of hostnames that end up at the same host.

~~~
zimpenfish
Just updated to 10.15.5 Beta (19F53f) and still no issues with dropbear or
sshd.

------
nathell
I've been a Linux user for the last ~15 years, and now I need to do some iOS
development so just a few days ago I've ordered a Mac Mini. I guess I'm in for
a bumpy ride. Oh well.

~~~
redsymbol
I used linux on all my laptops/work machines for about 7 years, then switched
to macbook pros 5 years ago. Definitely some things you have to adapt to (I
_still_ miss focus-follows-mouse). But for the most part, you'll find it's a
smooth ride. When bumps like this happen, they tend to push out a fix quickly
- especially when it gets traction on HN like this.

~~~
quesera
For me, focus-follows-mouse is most useful for terminal windows. It is a
feature you can enable in iTerm2.

This doesn't help across applications of course, and there's a reasonable
argument that the inconsistency is worse than the absence -- but for me,
iTerm2's FFM feature helps.

~~~
redsymbol
Didn't know iTerm2 did that. Thanks, I'll check it out - might help a lot.

------
hazebooth
I'm on mac OS 10.15.4. SSH has worked on all betas and the official release
from Apple.

~~~
beervirus
Using a port above 8192 and connecting by hostname?

~~~
hazebooth
Yes, but maybe I've mucked around with my ssh too much.

------
Zelphyr
As an aside; is there a reason to host SSH on a non-standard port? I recently
came across a system that had it listening to a really high port number. I
dismissed it as security through (bad) obscurity but is there a valid security
reason to do this?

EDIT: Thanks to everyone who answered my question! It makes sense to me now
why one might do this.

~~~
carlisle_
There's actually a good reason to not use SSH on a non-privileged port: It
allows an unprivileged user to bind their own binary to the port when SSH
restarts or otherwise stops listening.

~~~
nickodell
That unprivileged user will not have the SSH host key, which will create a
warning for any user who connects, just as though someone had conducted a man-
in-the-middle attack.

Of course, there are plenty of privileged ports to choose from.

[https://www.google.com/search?q=random+number+between+1+and+...](https://www.google.com/search?q=random+number+between+1+and+1023)

------
di3goleite
I faced the same problem two weeks ago with the previous version of Catalina
(I don't remember the correct number but was a previous on 10.15.4) and git (I
use SSH to authenticate with the server). So I did a report to Bitbucket with
a solution that worked for me after investigate more about the problem:
[https://twitter.com/di3goleite/status/1239596891471581189?s=...](https://twitter.com/di3goleite/status/1239596891471581189?s=20)

Thank you about that clarification. Also your website seems to be down
actually.

------
anongraddebt
"I’m not even going to go into it. I don’t want to end up on Hacker News again
bitching about Catalina."

+1

------
chmaynard
Apple's stance is that it didn't happen unless someone reports it using Radar
(internal) or bugreport.apple.com (external). Unfortunately, they don't
believe in Linus's Law, which states that "given enough eyeballs, all bugs are
shallow."

~~~
saagarjha
Things that end up on the front page of Hacker News get fixed.

------
LilBytes
This is an anecdote but the latest update forced me to rebuild my Mac from a
hard/factory reset. My Dell D6000 on my 2019 MacBook Pro no longer charges the
laptop. I've tried.

My Mac's resources were getting gobbled up by an internal process I coudln't
terminate and my keychain was borked and I couldn't log in after a reset (to
try and get around the resource hogging). Recovery didn't get me any where so
I used Recovery over the Internet to do a clean install.

I'm running 10.15.4, no issues as of yet. And this all occurred after the
security update. I'm running on the version prior for now but will make sure
I've got a good backup and give it another go.

~~~
gangstead
I had a d6000 and had to update the firmware (with a borrowed windows machine)
before it would charge my mbp.

~~~
LilBytes
Hadn't considered that, I'll give it a go. Thanks!

~~~
LilBytes
If anyone comes across this comment, this 100% worked.

------
supernintendo
I've moved to Linux for 99% of my computing but still use macOS for some audio
production work. Catalina is unusable for me personally (most of the software
I need just silently crashes) so I disabled the upgrade prompt:

    
    
      sudo softwareupdate --ignore "macOS Catalina"
      defaults write com.apple.systempreferences AttentionPrefBundleIDs 0
      killall Dock
    

Apple should really slow down on major releases of macOS or stop altogether in
my opinion. macOS Mojave is a great OS and it's basically feature complete.
Just stick with that, introduce bug fixes and security patches as needed and I
think people will be happy.

------
stinos
_At this point I’m thinking maybe the permissions on my local private key got
screwed up. So, I blow away ~ /.ssh and recreate all of my keys from a backup_

Is that a common thing to do, or any reason why the OP would do that? Doesn't
ssh reject your key, saying it does that if there's such a problem? And even
if not wouldn't it be advisable to at least look at the permissions; I mean
suppose they're not -rw------- or so, wouldn't you want to know that, and also
_why_ they are not ok?

------
zeveb
> Am I and this one other forum poster just doing something totally bizarre
> yet the same?

One might uncharitably suggest that using macOS and expecting standard
decades-old Unix behaviour is itself bizarre … but that's also true of using
Linux with systemd (viz., nohup no longer nohups, or systemd-resolved, or
innumerable other broken bits).

It's almost as though no-one cares about quality anymore.

------
ulkesh
I agree, that’s an annoying bug/feature.

However, there is an amazingly easy workaround, assuming the IP and port don’t
change often: create a ~/bin shell script that connects via IP and port, make
it executable, and add ~/bin to PATH.

This workaround doesn’t excuse Apple of doing something so egregiously stupid,
but it’s so easy that you may as well do it and move on.

------
teilo
Well, the description of this bug is not generally reproducible, so whatever
is causing it, it's not as simple as using a high port with a server name.

I tested this specifically on a number of servers that I run with port numbers
> 10000, using /usr/bin/ssh on macOS 10.15.4, with and without IP addresses.
Nothing broke for me.

------
yasp
Catalina also broke `apropos`.
[https://apple.stackexchange.com/questions/374025/errors-
from...](https://apple.stackexchange.com/questions/374025/errors-from-whatis-
command-unable-to-rebuild-database-with-makewhatis)

------
neilwilson
Catalina seems to have bust Wifi monitor mode on tcpdump on my MacMini 2018,
yet it works fine on my Mac Air.

Still not sure if that is my machine, or a general fault - but the lack of
monitor and promiscuous mode is playing havoc with IPv6 multicast packets from
VMware Fusion VMs.

------
steve1977
"I don’t want to end up on Hacker News again bitching about Catalina. I just
hope I’ve stuffed this post with enough keywords so that anyone else searching
on Google might come across the answer." Ok. And did you actually report it as
a bug to Apple?

~~~
wyattpeak
Whether or not he did, there's absolutely nothing wrong with posting about a
problem in case others come across the same. In fact it's damned helpful.

------
tkubacki
It is time to move to Linux or Windows desktop. Really if you are not hostage
of Apple ecosystem then decent desktop is much more reliable in my experience
(got old MacBook which is ok too but eg can't connect to old vpn on it)

------
epiphanitus
What’s the backstory to why apples bash has to be different than gnu bash?

I love having the Linux kernel with a nice UI but there are some useful
commands that are missing.

There are ways to get them set up but in any case it’s kind of a pain

------
buildbot
Wow, just the other day this started happening to me as well with one of my
serversfrom my MacBook. It used to work fine, but now only that laptop can’t
connect to it. iIt’s on a high port too.

------
jki275
I'm using ssh to a named server online using a port >8192 from a 10.15.4
machine right now.

Have had no issues with it at all.

------
vgaldikas
>I’m not even going to go into it. I don’t want to end up on Hacker News again
bitching about Catalina.

Whoops

------
ThePowerOfFuet
> Next, I ssh into a different server and then hop to the problematic one. It
> connects without any trouble. At this point I’m thinking maybe the
> permissions on my local private key got screwed up. So, I blow away ~/.ssh
> and recreate all of my keys from a backup. Still can’t login.

Someone should have paid more attention to that verbose SSH output first.

------
liquidify
Are you talking about the new apple feature?... non working ssh? It's all the
rage.

------
ipv6ipv4
In my experience the breakage is with IPv6. Try forcing IPv4.

'ssh -4 <hostname>'

------
StreamBright
Thank god I did not upgrade. Software upgrades are the best way to waste time.

~~~
rovr138
They're the best way to stay secured and receive new features.

The issue with them is lack of testing before deploying them.

~~~
anonymou2
I think proprietary software and staying secure are contradictory ideas.

------
Elrac
Am I the only one who thought this article was about Apache Tomcat?

------
viburnum
If I'm still on Sierra, which version should I upgrade to?

~~~
v64
I have a MacBook Pro (Retina, 13-inch, Mid 2014) and Mojave runs very well on
it. I have no plans to upgrade to Catalina.

------
geuis
Looks like the post was just deleted.

------
krzysztofeng123
_laughs in Linux_

~~~
dang
Please don't do this here.

------
awinder
“I’m not even going to go into it. I don’t want to end up on Hacker News again
bitching about Catalina.”

Welp.

~~~
karol
With this foresight he sadly didn't enable CloudFront and the website went
down.

~~~
tylerhall
Yeah, it's a $5/month DigitalOcean box with only my blog on it and nothing
else. All assets come off a CDN and Varnish is sitting in front of WP, but
looks like that still wasn't enough this time. It worked fine for my previous
two HN'ings earlier this year.

~~~
asveikau
I am still able to access it. So I don't think it's doing so bad.

------
m_a_g
>I don’t want to end up on Hacker News again bitching about Catalina.

 _Ends up on Hacker News again bitching about Catalina._

~~~
jbverschoor
I don't wanna pay taxes.

Paying taxes anyway

------
tambourine_man
>I don’t want to end up on Hacker News again bitching about Catalina.

~~~
davidkuhta
To be fair, full context tempers that sentence:

> I’m not even going to go into it. I don’t want to end up on Hacker News
> again bitching about Catalina. I just hope I’ve stuffed this post with
> enough keywords so that anyone else searching on Google might come across
> the answer.

------
deeblering4
> I changed the server to listen on standard port 22 and tried connecting via
> the hostname once again. Holy crap, it worked.

Shocking!

Granted high ports shouldn't be broken, but running SSH on a non-standard port
is security (read obscurity) theater at best.

There's really not much benefit, unless you need multiple sshds on the same
IP, but at that point I'd question the sanity of the approach.

~~~
lukevp
Security through obscurity shouldn’t be used as an edict that something is not
effective. You are talking about the fact that it doesn’t increase the
security of the protocol itself or the passphrases/keys used. This is true.
However, there are tons of bots out there that scan 22 and try to exploit
common logins. There are presumably quite a few less that are port scanning
every machine for every possible high port and attempting to handshake ssh and
then try logins. Do you disagree with that? If not, this is not security
through obscurity, it has a very real impact on the volume of bots that have
knowledge that this service is running and are actively exploiting it. It’s
just a different type of security, it’s discoverability of the service.

Here’s another example. Say you have a web server running that is only for
internal employee use. But you want to expose it externally so that they can
reach it without a VPN. Even if you follow proper security protocols, why
would you not turn off search engine indexing on this page, and limit the
pages that link to it? It will not increase the inherent security of the
protocol or the user accounts, but it will drastically lower the # of bots
using up CPU and iptables entries trying to fail2ban or blacklist them.

Security is a spectrum and you want to have defense in depth. Moving ssh to a
nonstandard port is a security best practice and you shouldn’t be advising
people not to use it. But should they also have good key setting, fail2ban, ip
whitelisting/blacklisting, etc? Of course they should.

~~~
deeblering4
> Moving ssh to a nonstandard port is a security best practice

For whom? Could you please cite this?

