> In this talk Julian will outline his work as sysadmin, systems and security architect for the climate and environmental defense movement Extinction Rebellion. Responsible for 30 server deployments in 11 months, including a community hub spanning dozens of national teams (some of which operate in extremely hostile conditions), he will show why community-owned free and open source infrastructure is mission-critical for the growth, success and safety of global civil disobedience movements.
I am on board with this stack 100% and have been recommending Mattermost myself. Imagine my disappointment when I found out that Mattermost, even the self-hosted one, is spyware.
They call it "Diagnostics" to hide its true purpose, but really it's phone-home. Silently and with no notification, on f/oss self-hosted software; it's really a letdown.
To disable it, you must use the following entirely undocumented environment variables:
I go the further step of using a small Dockerfile that contains the following to patch the binary itself:
FROM mattermost/mattermost-team-edition:latest
RUN sed -i 's#api.segment.io#xx.example.com#gI' /mattermost/bin/mattermost
RUN sed -i 's#securityupdatecheck.mattermost.com#xxxxxxxxxxxxxxxxxxxxxx.example.com#gI' /mattermost/bin/mattermost
You might want to consider Matrix, where admittedly we do have phone-home stats, but you have to explicitly opt in to them during installation if you want to participate. (https://youtu.be/dDddKmdLEdg?t=605 in the video in the original post here)
If you use the Matrix identity server, which is required to have federation, the 3rd party identity server operated by the Matrix organization retains a list of your usernames. They don't tell you up front about this, either, and I think silently leaking a username list is pretty bad. You have to really pay attention during setup to realize that the federation technology relies on a bastion operated by matrix.org.
The identity server is optional and you can use your own, but you will lose the federation that Matrix is so proud of, and the instructions to set up the reference home server don't make it clear that this is necessary in order to avoid a leak of your users' identities.
> Matrix identity server, which is required to have federation,
The identity server is not required to have federation to work. All it does is let you optionally discover users on Matrix by their email address or phone number.
> 3rd party identity server operated by the Matrix organization retains a list of your usernames.
Not sure what this means, but the identity service does not retain a "list of your usernames". All it does is keep track of email->matrix ID mappings for users who have published them. When you look up an email address (or phone number), a hashed representation is sent to the service, and even then, they're not retained.
> They don't tell you up front about this
We do; to use the identity service you have to click through a very explicit GDPR terms of use which explains precisely how it works. You only get prompted with this when you actually use the identity service though (i.e. when inviting someone by email address) which might be why you've never seen it, however.
> You have to really pay attention during setup to realize that the federation technology relies on a bastion operated by matrix.org.
Again, Matrix federation does not depend on identity servers (and I kinda wish we'd never even implemented the feature, given how confused and upset people get about them).
> All it does is keep track of email->matrix ID mappings for users who have published them
This is what I mean by "it leaks the userlist." Matrix (the organization) stores the email addresses of my users, along with some mapping that could allow Matrix the organization to correlate email addresses with my server. To me, as a server operator, this is a deal-breaker, even if it was just email addresses with no mapping. I see this as a privacy violation against my users who trust me to hold their information privately and securely. My understanding is that you cannot join another Matrix homeserver server with an identity established on a homeserver disconnected from the vector.im identity server, which effectively forces the homeserver operator to use the vector.im centralized identity server if you want, as an end user, to actually take advantage of federation. I do not know how a user is supposed to take their login from one homeserver to log into another one if the first homeserver is not connected to vector.im.
Please correct me if the above is wrong.
Additionally, when I set up Synapse I was not presented with any kind of GDPR info, and it wouldn't make sense that I would be, because the GPDR is for end users, not site operators. Maybe this is presented to new users who connect to the public reference Synapse instance using Riot.im or something, but I'm not talking about this issue from the perspective of an end user, I'm talking about it from the perspective of a homeserver operator. I got about halfway through the homeserver setup before I realized that vector.im was necessary for identity lookup and I realized it only by carefully following the docs. This was long before the 9/27/2019 blog post was published, so I guess maybe this has been addressed somewhat. I have been following Matrix now for the better part of a decade.
If federation is possible without identity mapping done on a central server, then I too wish that identity mapping was never implemented.
> My understanding is that you cannot join another Matrix homeserver server with an identity established on a homeserver disconnected from the vector.im identity server
This is not true. The identity server is an optional feature, which users can use if they want to try to discover a user's matrix ID based on their email address. Matrix itself operates using matrix IDs to federate and establish conversations.
A good analogy is using LDAP as an address book in an email client. LDAP addressbook lookups are very clearly optional, not relevant to all people, and don't stop email itself working.
> Additionally, when I set up Synapse I was not presented with any kind of GDPR info, and it wouldn't make sense that I would be, because the GPDR is for end users, not site operators.
Because the identity server is an optional feature for users (just like a user, not a sysadmin, would configure LDAP lookups in Thunderbird), the GDPR terms of use are shown to users if they try to use an identity server to make sure they understand what they're doing.
Well then I'm glad that the blog post linked above was written, because obviously this situation was confusing when I set up Synapse a couple years back. I might not be a genius but I'm not stupid, either, and I'm obsessed with chat systems (I trialed every available self-hostable chat server at the time), so I guarantee if this confused me, it confused plenty of perfectly intelligent individuals.
I hope the team has clarified this in the documentation.
I'd like to see this idea implemented at compile time. Don't opt-in and the code that does it doesn't even exist in the binary. That way there's no worry a logical error in the code could accidentally ignore the choice.
"The following information [a boolean, not the event details] is sent when the specified event occurs:
- Sign-in Error
- Account creation
- Login succeeded
- reset password
- updated password
- Joined a channel
- Created, edited or deleted a message
- searched for a term"
… and the list goes on: https://docs.mattermost.com/administration/telemetry.html
You are licensed to use compiled versions of the Mattermost platform produced by Mattermost, Inc. under an MIT LICENSE
- See MIT-COMPILED-LICENSE.md included in compiled versions for details
You may be licensed to use source code to create compiled versions not produced by Mattermost, Inc. in one of two ways:
1. Under the Free Software Foundation’s GNU AGPL v.3.0, subject to the exceptions outlined in this policy; or
2. Under a commercial license available from Mattermost, Inc. by contacting commercial@mattermost.com
You are licensed to use the source code in Admin Tools and Configuration Files (templates/, config/default.json, model/,
plugin/ and all subdirectories thereof) under the Apache License v2.0.
We promise that we will not enforce the copyleft provisions in AGPL v3.0 against you if your application (a) does not
link to the Mattermost Platform directly, but exclusively uses the Mattermost Admin Tools and Configuration Files, and
(b) you have not modified, added to or adapted the source code of Mattermost in a way that results in the creation of
a “modified version” or “work based on” Mattermost as these terms are defined in the AGPL v3.0 license.
It is absolutely open source. AGPL for source, MIT for the binaries. Both of those are free software.
Their enterprise version has a different license; the source for that one is not available and the binaries are released under a much more restrictive license, but that's nothing to do with the normal one.
Disclosure: I run Zulip, a Mattermost competitor that has intentionally avoided taking venture funding in part because we don't want to be forced by investors to be Open Core.
Venture-funded Open Core software like Mattermost is an awkward place for the terminology. The free version is distributed under an open source license, as so there is a thing that is "open source", but the clear intent of Open Core licensing policies is primarily to achieve the goals of:
(1) Being able to market the software as open source.
(2) Maximizing the portion of users who buy the paid/proprietary version.
While in theory open core can be run responsibly, the incentive structure is to intentionally not include features important for typical use cases in the "open source" version of the software that any similar community-driven open source project would have considered an essential, early feature.
FOSS means different things to different people, but my personal perspective is that venture-funded open core software like Mattermost feels like FOSS to me about as much as proprietary software with a free plan like Slack or GitHub does.
For Mattermost in particular, this thread is a good reference: https://news.ycombinator.com/item?id=21820583. They made changes to the specific items mentioned there in response to the community pressure, but that doesn't change their fundamental business strategy.
I think this telemetry should be opt-in for self-hosted, but the data they send doesn't seem to push it to the level of spyware (i.e., no message contents or PII as far as I can tell). It's still much more private than Slack.
Instead of generating the certs with prosody (there was some issue since my system uses p11-kit), I found it easier to just generate them all with certbot. update-ca-trust doesn't seem to correctly add them to the Java keystore and then you'll encounter problems. Certbot does. If you're on a debian based distro you shouldn't have to worry, however.
All you really have to do is copy/paste configs and then also change the url in the config.
Any suggestions on simple auth methods to avoid running an open Jitsi server?
Last time I tested it, it seemed to be very open by default, letting anyone create meetings. I got lost when digging deeper.
If I install-and-forget, I want to avoid situations where strangers are using my Jitsi server and overloading the system, or pretending to be our company. Last I checked, it was not possible to have simple auth, or monitor/list calls.
I also run an Asterisk VoIP server with a WebRTC bridge (because most Linux SIP clients have terrible usability). That can make one pretty paranoid :)
Yes, I installed Jitsi Meet over the weekend and enabled auth so you need credentials to start a new conference, but anyone with the link (and optionally password too) can join unauthenticated.
I've been experimenting with Asterisk WebRTC (video & audio) and developing a SIP + P2P webphone for some time now. Quite close to something that's useable imho. See https://github.com/garage11/ca11 in case you're interested
my university' setup has something asking for "host credentials" before starting the session, so I think it's possible. If it's just for internal use, one could just put basic HTTP(s)-auth in front?
We regularly run it with 40-50 users, and it's fine... as long as you limit the number of displayed video streams to 12-15 or fewer. This tends to happen organically with people muting video, or otherwise you can configure Jitsi to limit it to show video for the last 12 people who spoke.
Otherwise you risk overloading people on devices which can't render >12 simultaneous video streams without melting. You can push the limit higher if you know everyone is on a fast machine however.
One thing worth noting is that if a one or more user connects via Firefox then quality degrades for everyone - but fixes for this look to be in flight over at https://github.com/jitsi/jitsi-meet/issues/4758
What server resources to you need for those 40-50 users with Jitsi/Matrix? I haven't seen estimates of required resources when I've looked, but I assume there's some doc somewhere.
(Thanks for the good work.)
Roughly speaking we're seeing Jitsi serve around 1000 concurrent streams (i.e. 25x 50-user conferences) on a typical 4 core box with 8GB of RAM. However, it's worth noting that Jitsi is pretty low resource - all it's doing is forwarding streams of data around the place. All the heavy lifting is done by the clients when displaying all the concurrent videos, so it's the clients which tend to be the bottleneck.
Jitsi uses about 5.5Mbps per connected Chrome user. This math doesn't line up, but if Jitsi isn't transporting the 720p video 10Mbps might work for 6 users.
1:1 video uses P2P and doesn't require video going through the Jitsi bridge.
Tbf, the bitrate depends entirely on the res constraints you've specified. We see 1Mbps when requesting 854x480, which seems to be quite a reasonable resolution when on a multiway conference.
Tangentially, I wanted to run Jitsi Meet for some meetings and created an account on Digital Ocean, only for the account to be promptly locked with no access to a human for support...just automated replies rehashing the same text again and again for tickets saying that I could provide more information if I believed that was in error (with no responses after providing additional information). Now I'm looking at trying Linode. Any other provider recommendations are welcome.
I am using a hetzner.cloud box for €5.88/month for a personal Jitsi instance without problems. There servers are in Europe though, so if you are somewhere else on the globe, it might be better idea to look for a provider which is geographically closer to you
Depends on the instance I guess, but a few dozen users on a cx21 seem to be no trouble at all. As someone else already mentioned in this thread, Jitsi does no transcoding server-side and is pretty low on resources.
I am by no means an expert on that topic, but I would assume that while a big cloud instance could go far with regards to CPU and memory, network bandwidth/traffic could quickly become an issue with several 1000 users.
Depending on the amount of work you are willing to put into this, you could even start with a load-balanced setup were one or more web servers, jicofos and prosodys are distributing the load to several videobridges. That would make it easier to scale the system up by adding additional instances.
Start small. Failure is cheap. Worst case, you seek out local sponsors for a dedicated server in donated colo somewhere nearby. Even if it doesn’t take off, you’ll have learned from the experience.
That's a good suggestion as well. If the plan is to run this instance as a non-profit for local users, it might also be possible to ask local ISPs to sponsor a box. That could help keeping traffic local and latencies low.
What does matrix add to jitsi? Jitsi is already easy to run over https. Does this make it so you can't randomly enter rooms if you know the name and there is no password set?
Matrix adds featureful decentralised e2e-encrypted chat alongside the voice/video conferencing, and makes it possible to coordinate the location of a given conference for a given room. It doesn't impose additional auth currently to the conferences (but it could). It also maintains your displaynames & avatars for you inside the Jitsi :)
I'm running Mattermost and Jitsi for my employer on a $20 droplet (2 vCPU/4GB RAM) and it works like a champ. Meetings of 5-10 are common, but we've run meetings larger than that too. As a facilitator of 1:1 adhoc meetings, it's basically zero overhead, because the server just coordinates a P2P connection then gets out of the way.
I have an old R710 with 24 cores and 32GB I got on ebay for $200 in my dining room with 10mbits upload/100 download and it works great. bought my .me domain for $5. It's a wonderful time to be alive.
I’ve run 10-12 users with video and desktop screen sharing on a 6 core azure vm with between 20-30% cpu load with spikes to 46%. It also depends a bit on the adaptive bit rate. Image quality of FaceTime is higher than Jitsi.
I hope those things are much better than they used to be. Last I looked, they were a bunch of APIs glued together to look like a media server. No hard features; no guarantees. Almost a mockup of what a media switching server should look like on the outside; nothing inside.
I tried it a while ago and message delivery took minutes between me and a friend. It also won't do well on Android versions that kill background tasks and effectively force you to use Firebase Cloud Messaging for push (although you can't really blame it for that).
This was deliberately the Debian package flavoured installation. We'll probably do a Docker one too (which will likely be a lot faster, but also a lot more mysterious as to what's actually going on :)
Has anyone done a recent comparison of Jitsi and Janus? The only benchmark I can find is from 2018 and seemed a bit shallow. I’d be interested in seeing which one could deliver the best performance while running on a really cheap EC2 box. My guess would be Janus, since it’s C and Jitsi is Java, but maybe Jitsi has something about its architecture that gives it an edge.
Hi guys and thanks for this tutorial.
Has I'm new to docker and synapse but I would like to setup the same thing using docker but I'm not finding a clean step by step tutorial on how to do this.
My goal is to set in way I could use docker swarm in future.
Any advises or links? thanks in advance
I have had a matrix instance on one of my servers running but integrating Jitsi was a real pain. And even when the integration is done correctly the user experience of using it within Matrix is at least weird. It appears as an "attachment" in the conversation and is very non-intuitive for everyone.
Jitsi is much easier to add in the latest Riot-web version. No longer do you need to set up your own integration manager. You just add the jitsi URL to Riot's config.json and you're done.
youtube and other steaming platforms use a lot of servers to handle the load, so no jitsi can't be youtube itself unless you have a beast of a connection and even then you'd cap your connection after a hundred users. CDNs like cloudflare does the same but for websites.
I already have a lot of servers. I was asking if jitsi is server side software that can handle the encoding / delivery to users. The answer is no. Jitsi is apparently completely unscalable.
I haven't tried it myself but according to the FAQ:
"Jitsi offers a telephony interface that allows users to dial into a conference or for placing dial-out reminder calls. You can try this for free on meet.jit.si. Self-installed Jitsi Meet deployments will need to setup and configure Jigasi with a SIP provider to connect to the phone network. "
if you go to the jitsi.org site and start a meeting, there is a share box that can be popped up and it shows a dial-in number for your conf. So it must have sip integration.
Many people do just that, but it really depends on what you use it for.
Synapse resource usage is dependent on the complexity of the rooms that it participates in, not the number of users.
So if you intend to use it just to talk to a few friends, you'll have no problems at all. If you want to join rooms with 1000s of other servers participating then it will be hungrier.
It’s still a ways to go for non technical users. We tried Jitsi to mixed success. Some people had it work flawless the first time, others had to switch browsers, still others couldn’t use it at all, probably because of some privacy or cookie blocking extension.
Open source software needs to be as easy to use and configure as the alternative if they really hope to gain wise adoption.
That's interesting by brother and I have both worked in IT (he still does) and came independently to the same conclusion, that for domestic offerings meet.jit.si was the least friction - you literally just go the website and so far it just works.
Yesterday my kids used it for a 4hr call, no interruptions though audio quality dropped on occasions - I think it was the remote iPad's multiplexing that was struggling but couldn't be sure.
Open source software doesn't owe you anything and you can use the alternative if that doesn't satisfy you.
People want self hosted, free software with privacy BUT also all other features that big companies add to their software. I'm sorry but you have to make compromises. Most people decide to compromise their freedom and privacy.
If you really want the software to improve the best way is to contribute (or donate if that's an option), complaining things could be better on the other hand don't help much.
Just join the chats from your server, they will exist on both matrix.org and your server. Then you may leave with your old matrix.org account.
As for history, you could just import your key backup, if the room history is set to visible. Truly decentralized accounts will likely come at a later point, especially with all the work surrounding p2p matrix, where each p2p client is a server.
I've tested this with a couple of test accounts. By default the e2e rooms history are set like this:
Who can read history?
Members only (since the point in time of selecting this option)
but this option doesn't seem to work for me. The history didn't synchronize when i added my own homeserver account and verified it with other accounts - do i need to import the keys from the old account first for it to show up or am i misunderstanding how this option works?
As of recent Riot/web versions old messages that can't be decrypted in rooms will not be shown (saves the user from being blasted with Unable to Decrypt errors).
So the messages are likely there, but you won't see them until you import your e2e keys.
Jitsi despite its frequent re-occurence here is a nightmare to configure with so many bells & whistle to setup just one basic functionality. Try setting it up with word-to-word instructions for setup & later SSL certs to work on your own iOS app of Jitsi meet without ripping your hair out.
Super easy for me. I followed the instructions to setup on Ubuntu and it was ten minutes with zero confusion. Just copy the steps from the guide. Even the let's encrypt script installed certbot and configured whatever was needed with the existing web server.
Then I just hit the url and it worked perfectly.
The biggest surprise was when I tried to access that same page from an Android phone. It prompted me to install the jitsi app. After I installed it, it directed me to my jitsi server.
For me it was flawless and even better that I expected. It's a strong competitor to zoom because of the fact that it works right inside the browser really well.
Android isn't the problem, I mentioned iOS. I know WebRTC works and all starting iOS 11, but the problem remains with the way Jitsi configures SSL in the nginx conf. iOS Safari is simply unable to establish a secure connection despite the server having a valid LetEncrypt cert.
Hey there, saghul from Jitsi here. Have you reported that to us? I have deployed several self-hosste instances with Lets Enccrypt and haven't seen this, but there might a bug lurking somewhere.
There was a version of nginx that broke TLS for Safari when HTTP/2 is enabled. This was a number of years ago now and I’m sure isn’t happening on new versions.
Same experience here. I had a VM up and running in about 30 minutes. (Having said that, I never got test meetings to work reliably. My clients, both on the LAN and on the Internet, would repeatedly "disconnect" and "reconnect" every few seconds. I didn't spend too much time on it because the company ended up making a "let's standardize on WebEx" decision the next day.)
It not my experience at all. In fact it was the easiest install I have ever done. It knew I was using nginx as a reverse proxy and inserted itself correctly as a subdomain. The only issue I had was webmin was running on port 10000 which created a conflict because it also wanted to bind to 10000.
After reading your message, I tried installing it on archlinux, and here we are 3 hours later and I still didnt figure out to make it work, so I'm sadly giving up
I've been running it on a 4 core, 8GB ram droplet at DigitalOcean with no problems. We typically have 3-6 people in multiple sessions at the same time. In the test install, I did it with a $10 droplet, and it stuttered with 30 people in one conference, but didn't drop anyone.
> In this talk Julian will outline his work as sysadmin, systems and security architect for the climate and environmental defense movement Extinction Rebellion. Responsible for 30 server deployments in 11 months, including a community hub spanning dozens of national teams (some of which operate in extremely hostile conditions), he will show why community-owned free and open source infrastructure is mission-critical for the growth, success and safety of global civil disobedience movements.