For instance, if you download the Swift 4.1 tarball, uncompress it, and start up the REPL, it doesn't really work. The first thing you will do is probably "import Foundation" (because the Swift standard library is really minimal, so Foundation is where most of the interesting stuff is). Then the REPL will complain:
error: repl.swift:1:8: error: missing required module 'CoreFoundation'
sudo chmod a+r /opt/swift/usr/lib/swift/CoreFoundation/*
There are a bunch of little issues like this, that make it clear that Linux isn't yet a first-class platform for Swift. All this stuff is gonna work fine on your Mac.
OTOH though, the Foundation library itself has made amazing, amazing progress. It's super useful, mostly finished, and its pretty clear when stuff that works on Apple platforms isn't yet ready on Linux (because those bits use NSUnimplemented() which just crashes the program). That's important because without Foundation, which contains a lot of the basic building blocks for programs, Swift won't be competing with Go or Rust.
Swift Package Manager is also fully cross-platform and awesome.
Apple also recently released an interesting low-level cross-platform asynchronous event-driven network application framework, called swift-nio. This is super-interesting, but also causes more churn; for instance, once swift-nio was released, the team behind the most popular Swift web application framework, Vapor, started frantically rewriting their v3 release to leverage it.
So... yeah, I would say "server side Swift" is becoming less niche, but... it's still pretty niche. On Linux, it's far less mature than Go or Rust, and you need to expect to have to bing a bunch of annoying little issues to get things working.
: just joking! haha ;-D