This is a great guide but from my experience, even if you configure it 100% correctly, email services like Gmail may still classify your emails as spam for no apparent reason while not being on any IP or domain blacklist. I tried for hundreds of hours to get around it with no avail, and my emails to Gmail always went to spam unless it was a response to an email from a Gmail address. Had to go back to a 3rd party hosted service (iCloud) because of it.
Came here to say this, plus add a little personal insight to the future of email.
I've run 5 or 6 different mail servers over the past 10 years. Originally before O365 I was an exchange admin, then postfix, iRed, mailcow, mail gun, you name it. Hosted on every cloud provider, even in our colo with part of a private /24 allocation with good reputation (built since 1997, gawdamn). Every sort of header combination, tls setup, and no blacklists. Always 100% alignment, including strict rejection policy (best results even over quarantine).
Does not matter, if you're sending from custom domain not handled through a big name, expect the spam box with Gmail. Yahoo and Outlook are fine, but Gmail is the bane.
I've spent maybe 100 hours of my own over this last year and know what I realized? Nobody cares about email anymore, except for automated account management stuff (login, PW reset). Businesses pay the $3 /mo / seat for fastmail and don't think twice.
But the current trend is toward social chat (discord or Whatsapp) and most the people who own an iPhone just use their apple ID email for everything.
Although I am a fervent supporter of open protocols and believe email (with pgp signing) is an awesome long form communication format... Face it, it's going the way of the fax machine.
Email as a communication method with your friends/family — absolutely, this has been dead for over a decade I’d say.
However, email has basically evolved into the way you communicate with “systems” and I’m kind of happy about it. Communication with companies outside your network, e-commerce accounts/purchases, communication with government systems, schools, banking, airlines, concerts/events, restaurants, etc. Hell, even RSS is now basically in email — newsletters are growing fast as a medium, not shrinking.
You just book a hotel in Nairobi? It’ll be in your email. No other communication method even comes close for this use case.
Social/chat apps will never unseat this because they’re social. Like nightclubs, the trendy ones come and go. Come back when you’ve set up an interoperable network of virtually every person on earth. Then we’ll talk about email being dead.
Guilty until proven innocent, an excellent initial position.
I’ve had to relax my SPF record to include the entire mail pool of my ISP to be able to send to anything hosted by Microsoft. I tried to liaise with them directly, and through Linode, but they refused to exclude the IP from their opaque blocklist. Their proposed solution was to change the IP of the VPS, but that’s just agreeing to play whack-a-mole with a bad faith actor.
There should be a path to greater transparency and accountability from the SMTP cartels, but I’m at a loss as to how that can manifest.
Did you send out mails directly from your ISP? (Than I could unterstand Microsoft.) Only your MX must send out mails. And only it should be in the SPF record.
It would be crazy of Microsoft to look at all Received-Hearders and want everything mention there in the SPF record. If that is what really happens, than you should exclude this information from the header.
(mask-src on Opensmtpd. Pretty sure Postfix has that option to, but haven't used it in a decade, so I can't tell you the syntax.)
I originally had only my MX hosts listed in my SPF record, and configured to accept them exclusively. Microsoft unilaterally blocked the entire /23 my VPS resides within, and refused to exclude my configured IPs from their block.
The only way I’m currently able to deliver anything to domains hosted by Microsoft is to expand my SPF record to include my ISP’s mail hosts, and route delivery through them.
Microsoft, and the other SMTP cartel thugs, are undermining the protections these protocols were designed to provide.
I have had a Gmail account from the days when it was invitation only. The inbox contains spam and my test emails and nothing else!
I've run tiny smtp systems for 25 years or so. It can be done. I am based in the UK but at least one of my domains is a .net jobbie, so nominally American. That one still works fine and it is my (ltd) company domain, so all good. The MX records etc have moved around a bit but always very carefully.
It all starts around the IP address you are using. Is it "tainted"? is it in a tainted block? If it is then you need to either go elsewhere or clean it up and that takes a bit of time. By clean it up I mean apply for removal from the usual suspect's blocklists - Spamcop (lol), Spamhaus and all the rest that you can find.
Now setup PTR records. That has to be done by your ISP. If they can't do it for you, then find a new ISP. If you can't get PTR records to match A records then you may have to give up. One of the first checks an anti spam system will do is reverse look up an incoming IP address and compare it. Also that should match the HELO/EHLO announced by the SMTP MTA:
SMTP connection from IP address 12.13.14.15
HELO (my name is) smtp.example.co.uk
Receiver will check: smtp.example.co.uk == 12.13.14.15 AND 15.14.13.12.in-addr.arpa == smtp.example.co.uk.
Everyone gets their knickers in a twist about SPF, DKIM and DMARC but if you do not get the prior basics of IP -> A -> HELO -> PTR sorted out first then you will fail sooner or later. I also recommend that you ensure your MX records (receiving) match up too with your sending records. It means you can use mx is SPF, for example.
If you have multiple internet connections and IPs then be absolutely certain that your inbound and outbound IPs for SMTP match up.
Sorted all that? Cool, now proceed to SPF.
Most people fail at the PTR stage. If your ISP will not do PTR for you then you are probably screwed for self hosted SMTP. If you cannot change ISP to one that will, then you are really screwed. Sorry. In that case you will have to engage a service that will route SMTP on your behalf. It won't cost much but you won't own it and you will have to pay someone to do it. Soz.
Reverse pointer is pretty easy with some hosting (Linode) and painful with others, but that's pretty basic knowledge. Same with managing IP reputation. Heck, mail gun helps warm up IPs for you (but if you're not email marketing it's ridiculous to maintain that).
What really gobbles my bobble is BIMI. Even without the paid-for certificate ($1500 is absurd), you can set it up to show your logo, and works on some providers (like yahoo). But careful, you have BIMI without the cert set up? Gmail spam-cans it.
Same with pgp, if you include your signature a lot of providers will immediately increase it's spam rating, usually high enough to land in spam (+7 pts usually), even though I doubt any spammer or scammer is inviting you to encrypted chats.
Email is broken because we all signed up for Gmail and didn't know better at the time.
It's getting pretty expensive to rent one IPv4 address per domain these days. You also don't always control every address in a block, which means there may be nothing you can do about your reputation no matter where you go.
You only need one IP per MTA not per domain. I have a "vanity" email system that I run at home. I run it for my mates too. I have around 10 domains inbound. It all works fine.
SMTP and SIP are often held aloft as fucked up. My Dad's home telephony runs off a RPi and a Yealink DECT station and a dynamic DNS.
The modern internet might look a bit fucked up if you only look at the X/Facebook/webby wankery stuff but the real internet is functioning quite happily.
The current draft would cover every kind of service that allows people to exchange information so that every DM you send on reddit, twitter, discord, steam, ... would be have to be scanned. Not even the most totalitarian governments on this planet have tried to implement something like this. Also it sounds extremely illusory that the people exchanging CSAM wouldn't simply switch to private services knowing their messages on public services are scanned.
"... As services which enable direct interpersonal and interactive exchange of information merely as a minor ancillary feature that is intrinsically linked to another service, such as chat and similar functions as part of gaming, image-sharing and video-hosting are equally at risk of misuse, they should also be covered by this Regulation. "
> Not even the most totalitarian governments on this planet have tried to implement something like this.
Arguably North Korea since their RedStar OS had a kernel module that scanned all files and text looking for keywords like 'torture'. And if you're being compared to one of the most brutal and isolated dictatorships on Earth, things are not good.
>Also it sounds extremely illusory that the people exchanging CSAM wouldn't simply switch to private services knowing their messages on public services are scanned.
The justification is obviously a lie anyway. If CSAM were such a huge concern, you wouldn't have member states where distributing CSAM is about as severe of a crime as theft, which is the case in Germany.
Surely the first step would be to have actual significant criminal charges for these crimes in all member states.
page 46. "... measures shall be ... targeted and proportionate in relation to that risk, taking into account, in particular, the seriousness of the risk as well as the provider’s financial and technological
capabilities and the number of users; ..."
.
.
It's a big framework to push the industry to have more "parental controls".
Everything is covered, but there the actual requirements make sense. See page 45.
It's still bad, because it's extremely tone-deaf (and playing with fire is bad), but it's written by and for policy idiots, who live in Word documents, and (un)fortunately rarely have contact with the outside world.
Wouldn't client side scanning prior to E2EE circumvent this issue? If WhatsApp or iMessage scan your messages on device it doesn't really matter if they are then encrypted during transmission.
I remember trying to deploy it a couple years ago and it was fairly complicated. Granted, I was trying to deploy a multi-node mnesia cluster, which is probably what caused all my issues.
But for non-production single nodes I just created a Dockerfile and deployed it because the other options were too much hassle. Can't remember the details, but in my (in)experience, deploying was always the hard part with Elixir.
I will need to try it again, it's been a while and the ecosystem has improved dramatically in the meantime ... and it was already excellent before. Good stuff!
My view on Elixir deployment is that for a given complexity of setup, it's no harder than anything else. For a basic single node webserver, `mix release` makes it like releasing anything else (often more easily). For complex multi-node setups, yes there's some hard work to do, just like there with on any other platform. The difference is that the development experience of getting to a working stable multi-node setup is so much easier on Elixir than anything else that you notice the difficult deployment more.
I've been building my startup 100% fullstack in elixir, and it's been the most wonderful technology I've ever worked with. I'm evangelising all my serious tech friends about how great it is.
Now it would be awesome if rabbitMQ and its client would run on OTP 27, would love to upgrade :(
A news aggregator (and premium news chatbot) that indexes and analyses around ~150.000 new articles a day (http://im.fo)
I'm absolutely certain the real time processing would be unfeasible in any other technology in terms of complexity and the minimal compute resources it's running on.
Modules like broadway, ash, oban, phoenix liveview ... make it not just a pleasure to work with but insanely performant.
With over 20 years of programming experience, I can say with certainty that there is no language that makes me as productive as elixir. It's at least 10x my python productivity (despite being at an expert level in python as well).
For "straight line" single thread number-crunching, other languages will often be faster. With the new JIT, I doubt they are 10 times faster, but you're right there is a difference.
That's often not the limiting factor though. Elixir makes it very easy to have excellent parallelism on your work so you actually take full advantage of processor. The design of the BEAM means that things are naturally quite low latency, and often you lose performance due to just waiting for things (this characteristic is why webservers on the BEAM are pretty fast).
The other key aspect is NX - like ML libraries in Python, Elixir is just orchestrating and the number crunching is done in C libraries or on GPU etc.
I don’t know enough about news aggregators to evaluate the claim, but presumably that is why parent mentioned both complexity and compute, not just performance.
If you need per-core number crunching performance you'd reach for Nx, similar to how you would do the crunching in Python. With OTP it's then (almost) trivially concurrent.
Compared to squeezing performance out of multiple cores with the JVM it's absurdly convenient and consumes way less RAM. I have two reasons for still working with the JVM, multiplatform desktop GUI and high quality PDF libraries that support rather low-level aspects of the standard that I need. It's kind of obvious why these things aren't readily available on the BEAM, though.
Very cheap in C# and Go. I assume Java has now closed the gap with its Green Threads implementation.
(Spawning an asynchronously yielding C# task is ~100B of allocations depending on state machine box size, with very small overall overhead and threadpool handling millions of them, they are cheaper than Elixir tasks which make different tradeoffs (and are subject to BEAM limitations), you can try this out on your machine by running the examples from this thread: https://news.ycombinator.com/item?id=40435220)
if you want to see successful "machine learning based financial statement analysis", check out my paper & thesis. its from 2019 and ranks #1 for the term on google and gs because it is the first paper that applies a range of machine learning methods to all the quantitative data in them instead of just doing nlp on the text. happy to answer questions
Same at first it was like "Cool, no thoughts head empty, I don't feel anything anymore" but then it was "fck no thoughts head empty, I really don't feel anything anymore".
I got them prescribed by my psychiatrist (~4mg/day). And getting off of them was like going through hell, I really just wanted to stop taking them (I tried it for one day...), but I needed to reduce it by 0.5/week to not completely go berserk. But even with slowly reducing it, it wasn't a pleasent experience.
Yes, I wasn't addicted (as it didn't feel good to take it) and after reducing it step by step, I was off of it (took some time). Though after that my other stuff (the reason why I got benzos prescribed) was there again, but I had no urge to take Benzos again.
