We are making [NTRU Prime + x25519 key exchange the default] now (i.e. ahead of cryptographically-relevant quantum computers) to prevent "capture now, decrypt later" attacks where an adversary who can record and store SSH session ciphertext would be able to decrypt it once a sufficiently advanced quantum computer is available.
"Formal complaint regarding 8 June 2021 incident - 2021.06.15, Daniel J. Bernstein..."
"Executive summary. A week ago Dr. Daniel Apon from NIST publicly accused me of professional misconduct. Specifically, he accused me of initiating private contact with NIST so as to provide false information to NIST regarding the timing of an upcoming announcement relevant to NIST’s ongoing decisions..."
It is unfortunate that this disfunction has a practical impact upon OpenSSH.
Is there anywhere I can learn (or learn to learn) about NTRU Prime? I have a fairly decent knowledge of x25519 but in know nothing about lattice-based cryptography. I tried reading the NTRUP paper but most of it goes over my head.
It isn't, you can always brute force the symmetric key used for traffic if you wanted. The handshake itself can also in theory be brute forced if you know all input parameters except the input key.
It's true in theory, but not in practice, since the time needed is on a cosmic scale. Quantum computers are practical devices, also they use dedicated algorithms, not brute force.
The scp protocol is being removed, but the scp command you find convenient isn't going anywhere. So that convenience ought not to change, except that now the quoting does what you probably expect, because it's just a local command and so nothing weird is happening.
I'm somewhat of an old fart, but rsync(1) has worked well for me. It integrates well with ssh on both sides, using ssh channels to execute the binary on the server side and transporting data nicely back and forth.
It isn't suited for millions of files, but neither is scp.
It handles as many files as I've ever thrown at it - often in the millions. The great thing about rsync is it is restart-able and the source and destination don't even have to be local.
the biggest weakness of rsync is that it's single-threaded. Single-thread-copy with millions of small files is painfully slow, Rsync is as good as it can be but you just need more threads.
I think that's what a sibling is getting into with the "better to tar files and send them over ssh in some cases" thing. And yes, you can hack it in after-the-fact with xargs/etc but it's clunky compared to just having native multithreading like rclone/etc.
Got any benchmarks/write ups on the subject? I did a bit of testing myself a long time ago and basically the answer just ended up being to use rsync because any differences were marginal. That said, I didn't test with millions of files.
I think perhaps this was a bigger issue back in the day when we were using rotating harddisks. In those day doing a seek would be a lot slower than doing a write.
Today seeks are mostly instants, so maybe my experience isn't valid anymore.
Your experience is valid today but for a different reason: if you're comparing millions of tiny files there's a lot of back and forth. If you're streaming a single archive, it only checks if that single file has been modified.
Like everyone here I've no benchmarks but have got burned trying to rsync around too many small files.
We've put rsync in a HPC scheduler and used it, or some tooling ontop of it really, to copy billions of files for a large-ish HPC compute cluster with many P of data.
Also, rsync can create more faithful copies than scp. E.g. scp can't copy a symlink as a symlink, instead it will follow the symlink and copy the file or directory.
I use "gtar" on the command-line in that case; it either fails because gnu-tar is missing, or it works as expected.
"-H pax" is also an important item to specify on the side creating the tar file, as pax support is much older than pax as a default, and the pre-pax default (gnu or USTAR?) has a short enough maximum path length as to have caused me problems in the past.
the only snag about this plan is that the old `scp` tool works with any server, whereas the new tool will require additional configuration on the server to enable SFTP.
It's not enabled by default in some distributions and many "how to secure your server" guides recommend disabling it/keeping it disabled due to the best practice of only enabling necessary services.
This will break `scp` for many corporate cases (where arguing that sftp is necessary is an impossible fight)
The fact scp "worked" illustrates that such "How to secure your server" guides are... at best... poorly thought out. If "remote file access" is not in fact a "necessary service" then you can't very well allow scp to work, and thus you can't allow basic ssh service.
Actually a more sane setup, which I've used, is only SFTP is enabled, no shells. You can fetch files, you can send files, but you can't... for example... run a race exploitation shell script to seize root permissions.
This also has the advantage that if we can only send and receive files, the "server" we're connecting to needn't really exist, like an HTTP virtual host it's just a bunch of "files" and could actually be backed by a database with some blob tables.
There's a lot of scp usage in a typical corporate environment that should actually be files living in a shared filestore anyway, I've seen way too many files which ought to actually be:
* A wiki entry
* Checked into a git repository
* Shared in some cloud service e.g. OneCloud
... but instead they live on Steve's F:\Documents and if Steve is off sick well, here's the copy I made last week, I hope it isn't missing any important updates and somebody make sure Steve gets this when we change it...
The file living in /usr/project/ on testserver03 instead of F:\Documents on Steve's laptop is not a real improvement.
You're actually making the opposite argument that you think you're making: if the scp command stops working, from a "corporate" perspective, from all intents and purposes, it means that "our server no longer works with scp due to an update". Regardless of implementation details, the opposite will then happen: whoever is in charge will request for "SCP to be fixed", which will actually mean turning on SFTP support.
Removing options from "corporate" users usually results in saner choices being made, because you stop being able to bikeshed the implementation details. People have a server and they need a way to transfer files. If you have two protocols that can be used, arbitrary decisions will be made. If the readily available tools all break, there is no more decision to be made – whatever needs to be enabled will be enabled.
if regular ssh is available, nothing stops people from building pipes with tar on both ends. This will be integrated into additional tools (future "show HN - scp replacement written in $FANCY_LANGUAGE that works even when sftp is disabled") and will be the "new way" of doing things because it won't require any configuration change on the server.
Never underestimate the inertia of server configurations.
I guess you can call it a (application) protocol. However, when implementing this I've not needed to anything to make scp works as opposed to anything else. Just supporting ssh executions requests and connecting the channels as you do with any remote invocation.
The protocol is being deprecated not the scp command. It's been reimplemented over sftp. They have had to handle wildcard/glob expansion, and how scp tried to deal with it.
TL;DR it's just the same command, mostly, for now.
scp and sftp make sense, but is that really the case for rsync?
As far as I remember, rsync does a few neat tricks like hashing files on both sides of a connection to determine whether it's worth transferring them over a potentially slow connection, which would not work with pure sftp.
You're correct, rsync operates over SSH (or its own native protocol) but runs a copy of the rsync binary on the remote side rather than using the SFTP subsystem.
That would make us confederacy of dunces hawking free lifetime subscriptions to dupery, no less. I think you’re making the right call here, on the whole.
One nice thing about recent versions of SSH is that they include support for signing arbitrary messages with your ssh key. I know you're not supposed to re-use keys but there have been situations where I needed to prove my identity to people who only had my ssh public key. This would have been super useful in the past.
The OpenSSH arbitrary signing comes with a rationale for why this is actually safe. Basically they know these keys are intended for SSH, for which they will sign messages with a particular structure, and so they can choose the structure signed for the "arbitrary data signing" feature so that the two are necessarily disjoint - so no possible "arbitrary data" signature can be confused for a "I'm signing in to this SSH server" signature, and you're at no risk.
Thus, even if you allow some adversary to literally pick the arbitrary data, when you'll sign that data and so on, your signing it cannot possibly allow them to impersonate you to a SSH server.
This is easier to pull off than the rationale for key re-use in encryption because signing opaque blobs is a neutral action your SSH client already does, if adversaries could learn your keys or whatever from seeing you sign such a blob, then SSH authentication itself would be unsafe already. In contrast decrypting data an adversary sent to you might reveal something, especially if you can be persuaded (as happened for HTTPS with older TLS versions and most popular implementations) to tell the adversary what happened when you tried so this will usually be dangerous and a rationale for why it's safe must be thorough if we want non-experts to do it.
> In contrast decrypting data an adversary sent to you might reveal something, especially if you can be persuaded (as happened for HTTPS with older TLS versions and most popular implementations) to tell the adversary what happened when you tried so this will usually be dangerous and a rationale for why it's safe must be thorough if we want non-experts to do it.
What exactly happened for HTTPS with older TLS versions? Sounds like you’re alluding to some sort of oracle attack.
A sort of oracle, yes. I believe there's a vulnerability with servers that are willing to attempt RSA kex (which no longer exists in TLS 1.3 and has been a bad idea for many years), where you can (at least in principle) arrange for them to do an operation which either reveals some bits of their private RSA key (extremely bad) or is equivalent to RSA signing a message of your choosing. RSA kex involves the server attempting to RSA decrypt messages chosen by an adversary.
It's fuzzy, perhaps somebody will remind me of the specifics.
> there have been situations where I needed to prove my identity to people who only had my ssh public key
Just wondering what kind of situation can this be where someone has only your ssh public key? Usually people know each other via say, github user names or irc handles but ssh public keys? (genuinely curious)
Every time I pay enough attention to an OpenSSH release that I trick myself into reading some of the code, I'm left smh in disbelief that I actually rely on this software still.
Can you tell why that is? Did you find the OpenSSH source code especially bad? I am under the belief that OpenSSH is a really high quality, security focused software project.
If you think this [0] is quality workmanship, I never want to work on your team.
I haven't even exhaustively looked through this code, only took a quick glance at clientloop.c [1] and was curious about how the channels were integrated into the poll loop.
This is a mess, and I'm pretty sure I already see a bug.
Edit: I'm not looking at a future CVE, but I've definitely found a bug. For those of you tossing up low-effort straw men, actually go grok the code. The way they've integrated the poll loop into all the actual handling of the returned events is a sprawling mess, and the bug I've found is a direct result of that.
Have you created a new issue ticket for it on GitHub? If I'd spotted a bug, and felt strongly enough to comment on HN about it, opening a new ticket would be my next move.
So I cloned the repo to start looking more closely at the code in a real editor with an eye towards making a PR.
Fortunately the nature of the bug I've found seems unlikely to be a security issue, it's a busy-loop CPU-burning bug in clientloop.c's poll() integration.
Good point on the /issues redirect. They have a separate bug tracker it seems [0]. Thanks for taking the time to look into it. This sort of thing is what makes open source software great.
P.S. I'm not Greg (added it to my profile->about section to clarify).
Out of curiosity, can you explain what's wrong with that code? I dabbled in C a few years back but I'm nowhere near competent enough to find anything wrong with it.
If p is < 0 or >= npfd (the number of elements in pfds), this should read out of bounds of the pfds array, and the check happens too late.
This is similar to a famous Linux kernel bug from ages ago, where a !NULL check was optimized away in such a situation, leading to an exploit: https://lwn.net/Articles/342330/
In this case I'm not aware if something like that happens, or if it happens to be harmless.
The first line is not actually dereferencing anything - it's taking the address of the pointer, and returning another pointer with offset p from the original one.
As far as I know, C allows arbitrary pointer arithmetic (including pointing to an out-of-bounds address) as long as you don't read/write into the invalid memory. If the p check fails, nothing will be done with pfd.
Unfortunately you are not correct. Pointer arithmetic is only valid if the resulting calculated address is within the bounds of the same array as the original pointer, or calculates an address that is just past the end of the array. Otherwise it is our old friend undefined behavior. See section 6.5.6 of the C11 spec.
Given that the function fd_ready is static, it is limited to calls within this translation unit. It looks like the function which calls fd_ready has already confirmed that p is a valid index.
`pfds + p` is only defined if both the original pointer and the result pointer are pointing at elements of the same array or one past the end of that array. Note that executing `pfds - 1` when `pfds` points at the first element of an array is undefined behaviour and may fail on some platforms.
It's not undefined behavior, but it is implementation-defined behavior [0]:
Indirection through an invalid pointer value and passing an invalid pointer value to a deallocation function have undefined behavior. Any other use of an invalid pointer value has implementation-defined behavior.
cppreference.com hosts references for both the C and C++ languages. My quote is from the C language part of the site.
Furthermore, we are not discussing operations on an invalid pointer value. We are talking about the addition operator operating with a pointer value and when its behaviour is defined.
Lastly, it is your reference here that is to a C++ language draft, not the C language.
What is the difference between "operations with an invalid pointer value" and "addition operator operating with a pointer value"?
EDIT (since I can't reply yet):
Official C standard documents are surprisingly hard to come by. The only place I've found one was genesis. Here's a direct quote about pointer arithmetic from ISO/IEC 9899: 1990, page 47:
---
When an expression that has integral type is added to or subtracted from a pointer, the result has the type of the pointer operand.
If the pointer operand points to an element of an array object, and the array is large enough, the result points to an element offset from the original element such that the difference of the subscripts of the resulting and original array elements equals the integral expression. In other words, if the expression P points to the i-th element of an array object, the expressions (P)+N (equivalently. N+(P) ) and (P)-N (where N has the value n) point to, respectively, the i+n-th and i-n-th elements of the array object, provided they exist.
Moreover, if the expression P points to the last element of an array object, the expression (P)+1 points one past the last element of the array object, and if the expression Q points one past the last element of an array object, the expression (Q)-1 points to the last element of the array object.
If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined.
Unless both the pointer operand and the result point to elements of the same array object, or the pointer operand points one past the last element of an array object and the result points to an element of the same array object, the behavior is undefined if the result is used as an operand of the unary * operator.
---
Maybe I just suck at reading technical documents, but I can't figure out which part says that storing an invalid pointer is undefined behavior. Would you care to point it out for me?
If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined.
This is that part that says the result of addition is undefined unless both the original pointer and the result point to the same array object (or one past the last element).
In case it's not obvious how this can happen, it's because the promotion rules can turn them into signed ints, and if they're big enough the product can overflow that. For more details, see https://cryptoservices.github.io/fde/2018/11/30/undefined-be...
In our example here the argument `pfds` to the addition operator in the expression `pfds + p` is (presumably) a perfectly valid pointer. And `p` is a perfectly valid `int`. So there are no invalid pointers passed as arguments to the addition operation. You might think the result of the addition expression is an invalid pointer, but the result is simply undefined behaviour.
Typically an invalid pointer is created by deallocating some memory, which transforms previously valid pointers to the memory into invalid pointers without the pointer values themselves changing their value.
Personally I'd still want to do the check before, because someone could still be adding something in-between the assignment and the check, but if that's the order it's resolved in it seems to be working now.
(I'm guessing they declare and assign the variable first because of C version constraints - wasn't that something that changed in C99?)
This is how I would have written it (I exclusively write C99), but the OpenSSH developers target a lot more platforms that may not have reliable C99 compilers, and are understandably reticent to rely upon the GNU extensions to C89 that would also let them do this.
It's perfectly valid C89 to declare a variable at the start of a nested compound statement no matter where it appears in the outer block, so in practice you can declare variables wherever you want so long as you don't mind the extra clutter.
For consideration, "declare variable" is just that; one could still choose to only assign to it after having done the validation (assuming fatal_f causes control flow termination)
> I am no security guru, but as I understand it, all three have a well deserved security reputation, and the core maintainers run a tight ship.
Among us security gurus, OpenSSL has a really bad reputation. It's a giant pile of spaghetti code with a huge history of critical system vulnerabilities stemming from trying to be everything to everybody. Bitcoin is in the process of writing replacement functionality in order to remove OpenSSL entirely from its dependency tree because they got burned by major bugs multiple times in the past. A lot of modern alternatives like libsodium list "not OpenSSL" as a selling point. The situation is so bad that multiple organizations have forked OpenSSL to create stripped down alternatives (BoringSSL, LibreSSL) that are more likely to be secure.
I know less about the OpenSSH code base specifically, except that it came out of the same culture and the same developers. I wouldn't use it in new projects unless I had no other choice.
It's one of the most deployed pieces of software on the planet, and for that reason has a million eyes on it, yet there doesn't seem to be any alarming concern about the source quality. Maybe it's all right after all...
Maybe. You've expressed the same sentiment as you'd have seen for OpenSSL just a few months before everybody decided OpenSSL was unacceptable and they'd fork it (producing BoringSSL, LibreSSL and so on).
OpenBSD has some very peculiar preferences, some of which are in step with modern thinking (e.g. they like sandboxes, privilege dropping and defence in depth very much) but some very much are not (they like C despite its lack of Type Safety, Memory Safety, and generally just Safety)
SecSH (the SSH v2 standard) is a fairly clean modern encrypted protocol design. You do a key exchange, you bring up an encrypted link and then everything else happens inside this, alleviating many of your security concerns, the same way TLS 1.3 works (but TLS 1.3 is much more recent).
But the OpenSSH codebase dates back to even before SSHv2, it's basically the original SSH codebase, from back when encrypted remote shells was an entirely new idea, the mid-1990s, the SSH codebase went proprietary and OpenSSH is the fork. That's some crufty old code, and in C that's not an insignificant problem because problems in code you never read has potentially drastic consequences for unrelated code in the same binary.
> You've expressed the same sentiment as you'd have seen for OpenSSL just a few months before everybody decided OpenSSL was unacceptable and they'd fork it
I remember recurring complaints about the state of OpenSSL for years before e.g. mBed TLS and LibreSSL happened, but granted it's a decent example.
> it's basically the original SSH codebase
Would you be able to prove that by showing a bisection of the current sources and one of the releases from 22 years ago? I'd be surprised if not almost every single function in the code base hadn't seen significant changes.
That would be an obvious reason why their SSH implementation is in C. But for example their ACME implementation (to get certificates from e.g. Let's Encrypt) is also in C, even though various more famous ACME clients that were started earlier and have better features are in more modern languages like Python. That C ACME client complies with OpenBSD's other preferences, it is privilege separated resulting in a bunch of separate programs running to handle different privileges for example, but it is written in C and lacks some obvious security features that aren't platform specific.
Thus on OpenBSD a separate program handles your private key data from the one talking to the network. But, if you used a more capable ACME implementation for other platforms written in say, bash, a completely separate piece of software even on another machine can be talking to the network since that's just how the certificate process works. OpenBSD does not support this (last time I looked).
