The number of websites that try reject my email address with a + in it, ugh!
Surprisingly, the validation is often done 100% client-side anyway, and simply modifying the incorrect regex lets my email address through... If I wrecked havoc on your back-end, then it's your fault for sucking ;)
Caring what characters are in the password heavily implies that the site is not hashing the plaintext password in any way, and scarier still, may just be storing the plaintext password as plain text.
Why: Because if they were (at least) hashing it the output from the hash would be a binary string in which case they would have to be 8-bit clean through to the DB column where the hash output resided, and then there would be no reason to care what character was in the input.
Caring specifically about a + in a password also implies that their authentication might be setup internally as a http endpoint with URL encoding of a "password=" form variable (because + is used as the escape for hex encoded characters in URL encoding).
Both, of course, imply a lack of proper secure design in their password handling.
> Caring what characters are in the password heavily implies that the site is not hashing the plaintext password in any way, and scarier still, may just be storing the plaintext password as plain text.
I don't think that is true at all.
I may very well want to put a few simple rules I validate serverside, such as
1) No username in password
2) No email in password
3) No list of 100 most common passwords in password
All of which require me to look at the text for your password, none of which mean I am storing it in plaintext.
I've run into a couple of sites that reject passwords containing ', %, and other special characters that suggest there may be some truth to it.
If you're scrubbing input as if it's about to be insert into a database via SQL, then there's really only two possibilities. Either a) you're running legacy code that still does the check and does blind escaping (which has its own set of implications) or b) you really truly are storing passwords as plain text.
Hah, good point. Although I do have to wonder: When does it turn into filtering by force of habit? After all the legacy cruft has long been forgotten and is no longer maintained?
Because the other monkeys will chew you out if you start doing it differently all of a sudden. Nobody knows exactly why we're doing it the way we are doing it, but it is complex, and changing it might break something.
Let's use the "filtering chars from password" example above. You can't put some special chars in password field, and you want to change that so it's doing normal hashing where special chars don't matter.
In a larger org, even changing a practice like that so that it "makes more sense" can have a big ripple effect.
You have to
* explain to someone else on the team who came up with the original process that it's flawed (and why)
* explain to other dept that they need to update their testing process (and why)
* get support dept to change their language/process
* change outbound messaging in all affected places (perhaps with code you can't touch, involving other teams)
* possibly have a flag that deals with 2 versions of data
Even if your change brings you in to line with normal/safe practices, you may have to fight multiple inane battles, spend loads of time and political capital, and at the end of the day, you'll be able to also accept a !"@+'$ in a password field? Most people will not grasp the bigger issue at play.
At a small business convincing colleagues isn't that hard. At a larger corporation, getting the rest of the team on-board is the job of whoever is in charge of defining security policies and such. Not allowing characters present on all keyboards and input devices (!"@+'$) statistically increases the risk of people picking weaker passwords then possible, and he/she will guide that change through the proper processes, for example to make sure that any client software interacting with the backend is aware of the upcoming change. Same as with any security issue (e.g., the deprecation of SSLv3 ciphers in favour of newer TLS versions).
If a business can't handle a change like this, something is really broken in the development pipeline. Granted, that describes a lot of medium sized companies…
Take that one 'request' - possibly initiated by a jr-mid level developer - and stack it up against the 500 other todo items in the pipeline. You can make those arguments about "statistically increases the risk of people picking weaker passwords " - unless this increases a bottom line or comes out of someone else's budget, this sort of 'bug' is going to be really low down on the totem pole for all the reasons I mentioned, and a few others.
You can says "the process is broken" but it's also that same process that got people where they are, puts food on their table, pays for the lifestyle, and precious few people are willing to ever rock the boat at any company for anything.
Which gets back to some of the cargo cult thing too. It's not uncommon in a large enterprise to smack up against things like "Years ago we paid a highly trained Security Consultant a large amount of money to develop our Security Guidelines, who are you and where are your security credentials to tell us to do things differently?"
Even worse when that "Security Consultant" is still a retained coworker with a fiefdom to maintain by war at all costs, a wizened old greybeard whose seniority will always trump yours, and/or your boss.
There's a related issue that I've seen in some banks, which prevents you from using passwords LONGER than a certain number of characters (usually 8 or 10). The only reason I can think this is happening is because they don't hash them and they need to fit in their database column. I took my money out of that bank the next day.
Could just as likely be some process that still needs to pass the password around in a fixed-length file format to some ancient backend process written in COBOL living in the cthulian abyss deep in the bowels of the bank that no one dares rewrite from fear that it would destroy the bank from inside.
All of those checks can easily be done client-side, though. Of course this means you can't guarantee that none of these rules are violated, but I suspect that the users capable of bypassing this aren't the ones you're concerned about anyways.
The server does have to have access to the plain text password in memory, you know. I don't see why you think it's worth sacrificing guarantees of password strength to uphold some kind of taboo.
so you just replaced the password with another secret, the hashed password. An attacker would now not need to gain access to the original password, but needs the hashed password - which will be logged just like the PW.
The only security benefit is that it offers a bit of support for those that are reusing passwords since it doesn't expose the plain text.
Isn't that what SSL does? And if the SSL between you and the destination is compromised, you don't even know if the hashing algorithm you asked the client to use is actually the one they used.
Yep. And importantly, SSL is securing more than just the password - if you just salt + hash client side, then anyone watching gets to do a replay on that value instead of the original.
> You can perform the first pass of salted hashing on the client-side.
Not really. JavaScript crypto is fundamentally broken: an attacker, malicious server or disgruntled employee can replace server-side JavaScript and remove the client-side hashing at any time. This is, notably, why Firefox Accounts are completely and totally insecure (and hence why Sync is unsuitable for storing any private data at all).
Can you reassure me about the worst horror I've seen?
That is: a listed-and-enforced ban on obscenities in passwords. Is there any world in which that company isn't butchering their security in some serious way? (It's Time Warner, so not a small-time deal here).
Or, they submit the password over HTTPS, validate server side, then hash and store in the DB?
There is no reason to assume rejecting of a + means they don't hash, and browsers escape the password for you in POST/GET etc. The fact that + is used for space really isn't relevant.
(That said, I think it's pretty stupid to "validate" passwords beyond checking for "12345", "password", a min length, and other public info. e.g. your password should not be your name or date of birth. Anything any password generator spits out, so long as it meets the min length criteria, should be accepted).
> Anything any password generator spits out, so long as it meets the min length criteria, should be accepted
It annoys me to no end when I have to hand-craft a password to stay within the silly rules of some service.
Be vocal about this! Keep complaining to the services you use that don't accept such valid passwords. Banks for example are notorious in insisting on short passwords and arcane limits on which characters to use (ING in the Netherlands limits you to 20 characters, and nobody there seems to be able to explain why).
Société Générale uses the client ID and a 6 digit passnumber (sent via physical mail) with a digital numpad (you have to click) where each number is randomly placed.
So secure.
And they force a change every 3 months. As if a new 6 digit pass was more secure than the last 6 digit pass.
Sadly at some places it's intentional so your password matches existing PIN systems or is "easy to remember" so you're not as easily locked out of your account. I wish in those cases there was a check box that says, "I know what I'm doing, leave me alone."
Speaking of annoying validations, my name has a hyphen in it but you'd be surprised how often that's rejected with the familiar, "Please enter a valid last name." Sigh.
Because of code style (I do not mean spaces or tabs, it is beyond that). Different developers have different ideas of how a function should behave, where it literally has no effect on how the rest of the system works.
As much as I agree that most artificial limits on the passwords are silly, there's a reason for at least limiting to latin1 charset - or maybe even a bit stricter. How sure are you that strings are always unified the same way between the browser and hashing? Have you tested that you can login from a browser which prefers CJK encodings and each of the UTFs? From all browsers? Maybe it works, maybe your framework deals with this without issues. But if not, do you really want to chase those mysteries in the future?
Btw, almost nobody is actually storing hashes as binary blobs. Pretty much every framework I know stores either base64, or direct hex encoding of the value.
We reject Unicode control characters, surrogates, private use code points etc. Passwords are sent in a UTF-8 encoded JSON over TLS. They are normalized using NFC.
For more fun, try putting a "--" (two consecutive hyphens) in many fields in the AWS console. Not allowed, I guess because they're passing it to shell commands somewhere along the line?!
> Scarier still is when it's a server-side response that rejects my password for its contents...
Why? Assuming that you have established a secure connection with that server (i.e., HTTPS by means of TLS), then it is perfectly fine for the server to check, at the time you are setting the password, if your password confirms to the rules established. And when the password turns out to be suitable, it is okay for the password to be sent to the server as-is.
Now it goes without saying that as soon as you have picked a suitable new password, that the server will store only a (proper) hash — by using BCrypt for example. At no time is the plain text password stored anywhere, and any proper HTTP API will send the password via POST to prevent it from being logged in the server's access logs (which is where it could end up if sent as parameter via a GET request).
There are plenty of services that screw up and try to apply nonsensical rules (such as limiting the length of the password to anything less than, say, 256 characters), but in general this is done to prevent weak passwords. You can't exclusively do this client-side, because in security terms, the client cannot be trusted to actually apply the validation and to generate a proper hash; the client can be bypassed. Of course the client can and should validate any input before the server gets to it, so ideally the server need not even come into it during validation, but the server has the final word.
As user though, you have the responsibility of not reusing passwords anywhere (and you don't, because you sensibly use a password generator as you mention). Don't assume that any service handles security well.
And you should; you have no way of knowing if any service is practising proper security or not beyond what you can see (e.g., exclusive use of HTTPS with valid certificates), so assume the worst. It is best to see a password as a shared secret; something you share with a specific service. That means no reuse of passwords if you care about the consequences of that password getting out.
My bank limits passwords at 15 characters. The best thing? There is no verification, it just cuts off. Have fun figuring out why you can not login anymore.
Here is a fun one. My credit union limits passwords to 32 characters and since I use lastpass for my passwords I had it generate a 32 character password when registering at the website.
Next day I get around to downloading and setting up their mobile app. Login and get a prompt that since it is my first time using the mobile app. They have sent me an email with a 4 digit code I need to enter before proceeding in the app. The email with the code arrives quickly and I enter the code. Proceed to get an error message "Error 400: This service is not available at this time." Ok, they must be down I'll try again later. The code is good for 24 hours. Next morning find time to try again and get the same error message except this time it also informs me that my account has been locked. Call up customer service and with scarily little information get them to unlock my account. Explain what is happening with the app and the CS rep puts me on hold. Comes back and tells me the IT folks thought it must be a problem because of my email address. "Do you have a normal email address like from Gmail, Yahoo or hotmail?" I release a great sigh and give in to the stupidity I'm about to have to navigate. I provide a gmail address. They change the email associated with my account on their end and tell me I should try again in 24 hours. I do with the same results. Call back again and the CS person puts me on hold again while she reaches out to IT. This time whoever she talks to knows the issue right away.
Turns out that when you login to the mobile app for the first time and submit the code it is actually appended to the end of your password and submits it as your password. Which if you have a password with more than 28 characters means you are exceeding the 32 character password max which causes them to return the informative, "Error 400: This service it not available at this time.", message.
Fun fact PayPal (used to?) silently cut off passwords when signing up, but not when logging in. And the password rules are of course not shown on the login screen. So good luck remember what length they cut your password down to when your password manager has stored the longer version.
A previous place I worked did almost the same thing. There was an internal website they had built that everyone used for time entry. It authenticated via LDAP, so you didn't need a separate login for it. However, the password box on the page only permitted passwords of up to 10 characters, but it wouldn't notify you it would just truncate whatever you typed in. So if you had a Windows password longer than 10 characters, you couldn't enter your time.
Comcast has the same issue. I changed the password for a user on my account using a password manager, it accepted a 20 character password with no errors, but then I was unable to login with that account. Changing to a 12 character password finally worked.
Have you ever heard any reasoning behind why they do this? The "best" excuse I've heard is so that customers don't forget. As if they don't have a "Forgot password?" link right there.
When I forget my banks password, I have to go to the closest client centre. Ironically, the only time I've had to do that is when my password was truncated.
I remember Starbucks was doing it for their mobile app about two years ago, sacrificing security for a "better"/faster UX. It's one of those things that you think "No one would be dumb enough to do this", only to be surprised by the fact that a big player has been doing it for a while.
> Scarier still is when it's a server-side response that rejects my password for its contents...
A friend's project decided to disallow umlauts, combined characters like ´e (can't type the correct e with accent mark), the pipe symbol and a couple more in new passwords.
Not due to plaintext storage or so, but because of customer service issues - people were bugging support all the time because they were e.g. abroad and couldn't enter umlauts or on a Mac and couldn't find the pipe symbol (it isn't written on the keyboard!)... of course, a quick direction to the Zeichentabelle in Windows helped, but what for Mac users? At least it did take a huge load off the customer service guys.
I'm looking forward to the date when some clueless user will input a UTF8 emoji as a password, given that Android and iOS keyboards now include these on special keyboards...
I'm looking forward to the date when some clueless user will input a UTF8 emoji as a password, given that Android and iOS keyboards now include these on special keyboards...
Looking forward? That happened a year ago and caused problems because the OS X Yosemite login screen had no way to input emoji:
It's worth playing around with the option key when pressing letters on OS X. You're able to enter a ton of common (western) characters with that alone, without ever opening the "special characters" (ugh) palette.
Also the default configuration for a couple versions now (which I hate, but I know it benefits others) is if you hold a commonly (western) accented character, it'll suggest accents for it.
Even Linux tools like GNOME Archive Manager in Linux Mint 18 rejected my RAR password containing a $ as the incorrect password, even though it was the correct password for the RAR file I was trying to extract. I then used the command-line unrar utility with the exact same password, and it extracted successfully.
Now, why would you preemptively (and explicitly) throw out a candidate password string based on its characters?
At least make the validation consistent.
I also had trouble with connecting to a wifi network with a WPA password containing an ampersand in Arch Linux using the Arch Wiki's recommended command line network manager utility (netctl, if my memory serves me correctly), and I tried all sorts of ways of escaping it and quoting it in the configuration file.
The Arch devs on IRC just told me to change my AP settings so that the wifi network doesn't use a password containing an ampersand character. Well, that would have been a good idea, but it wasn't my network, so I didn't have the ability to change its WPA password.
I'm guessing both are because the are calling command line tools. One of my pet peeves with linux is that many of these tools are only callable via text and don't expose an API for other programs.
"callable via text"? What does that even mean? The command line is an API, isn't it? If people don't manage to pass an ampersand to another program via the command line, that's really no different than people failing to pass an ampersand as a URI parameter to an HTTP resource: Failure to encode properly. There is absolutely nothing that prevents you from passing an ampersand (or any other characters) to a program via its command line.
It means you have to execute commands through a shell. A real API would be something you could include in your program, execute a method against and get a list of objects back. Instead all these basic command are replicated in every framework.
As far as encoding properly, you're preaching to the choir, but out in the real world there is still injection attacks everywhere.
Word-splitting, escape character interpretation and so forth is done by the shell. The shell is only invoked if you use the system() call to run said tool.
Which you shouldn't. execve (and friends) will let you pass an explicit array of flags, and you should always use one of those functions if you're calling another program. No interpretation.
On a Unix system, a console application is effectively just a vararg function taking a bunch of const char* arguments. There's no limitation on what characters you can pass as those arguments, so that doesn't sound like a valid excuse.
(There are some characters that are treated specially by shells, and require escaping - but you don't normally spawn child processes via a shell.)
This very heavily implies that the database column which stores "passwords" is typed as "char(12)" and that the site is storing unhashed plaintext passwords in that column.
Why: Because if they were (at least) hashing the password, then the output of the hash would be a fixed size token unrelated to the length of the input plaintext password, and no such arbitrary short limit would be necessary on the plaintext password itself.
Ha! If only! Probably more than 50% of the sites I visit that have a maximum length that my password manager exceeds... give me an unrelated error message. Some times they tell me I haven't met the minimum length (100 chars, really?), sometimes they tell me that I've not met complexity requirements (I use upper/lower/numbers/special chars), etc.
It's as though the developer only ever thought of how people wouldn't meet their minimum requirements, wrapped it all in a case or switch and let the default be whatever they hadn't checked in previous conditions.... not thinking that anyone would exceed their requirements.
You default to 100 character passwords? Doesn't that make it extremely inconvenient on the rare occasions when you need to type a password out? I figure 14 characters is going to be effectively unbreakable, but still possible to manually copy in under a minute.
Yeah, basically this. In theory every time I need a password I can copy and paste.
In reality, sometimes I'm on someone else's computer or something else comes up that I need to open my database up on my phone and type it in by hand.
For instance, I can't imagine trying to enter a 100 character password with a PS3 controller to log myself back into Netflix...
If I'm already at "it's going to take 100 quintillion years to break this hash, even if they're only using MD5", then I really don't see any security benefit to using a longer password... But there's a definite loss of usability.
It depends on your needs. Clearly in my case: nope, not a problem. Otherwise I would have changed. And I think your use of the word "default" is appropriate; what I do most of the time is not what I do 100% of the time... there are exceptions.
In those cases where I might need to manually enter a passphrase and I can't rely on the password manager functionality, I use a pass-sentence that is both long and includes some random characters thrown in. That can still get me to 100 characters pretty easily. But those cases a fairly rare for me. I'm also only very, very rarely using a device that I don't own (ok, I do access systems I don't own frequently, but using my device... ssh, etc).
Most of the time my 100 character default doesn't give me any greater security than your 14 characters (assuming they're well constructed or random passwords) and things like multi-factor auth are often times much more important anyway. There are some times it can be useful; sometimes I can remember a 14 character password just by looking at it, even if it's random and not mine. 100 characters I can't. Since password managers can rely on things like cut and paste, rarely, but occasionally, I might accidentally paste a password somewhere I don't mean to: at 14 characters I might not notice, but at 100 I more likely will (or overrun the field and lose a piece of it anyway). At the end of the day, I have a password manager and I'm going to turn it to 11 and let it run. My personal practices are not necessarily my recommendations in this area, but they suit my needs.
Either way, my point still stands: if you build software that takes passwords and you limit the length (or can't take longer passwords), provide meaningful feedback if your requirements are exceeded... makes me wonder what else I might be able to do if you didn't expect me to exceed your password length and respond properly.
Depends on where you have to type it, but usually, if people have long "passwords", it's because they use passphrases, which tend to actually be easier to type on real keyboards, but which are long because of the low entropy per character.
My guess is that when you call them on the phone, they ask for your password to validate your identity. Which means it's stored in plain text in their database so that customer service can verify what you said is correct. Maybe they don't want their employees to have to be cursed at by customers.
I can't think of a good way for a business that has an online interface and frequently handles phone calls from customers to validate that they're talking to the correct person. Asking for other personal information can be used by an attacker to compromise multiple accounts via social engineering: http://www.wired.com/2012/08/apple-amazon-mat-honan-hacking/
Someone else in this thread mentioned a company that has customer service type in your password to open your account. So that would be a non-plaintext reason to insist on non-obscene passwords. But it's still terrible, because why the hell is customer service typing in your password.
Pretty much all organizations that allow phone authentication seem to be at risk of engineering attacks. The only ones that manage it send you something verifiable they can ask about like a credit card, and people who really care like the government just send an actual human to your house.
I'd prefer to have an obscenity in my password if a customer service representative is seeing it. That would help communicate my frustration with their system. Saves me from having to voice that same obscenity, most likely.
It doesn't have to be stored in plain text to validate it is correct. The phone operator could enter it into an authentication form to verify it is correct.
The concern isn't that there's no way to do it, it's that there's no reason to do it unless the plaintext is going to matter again in the future.
The best guess I've seen is that they might be hashing, but also having people read passwords to customer service reps. That would justify caring about rude plaintext, but it's also a terrible system.
Heh, to be fair, we are talking about a company with millions of customers that is consistently rated as "most hated" by consumers. I'd be surprised if a decent percentage of attempted passwords weren't "fuck time warner"...
Hah. I can see some poor support rep getting a dozen calls a day with insulting passwords, actually - maybe it's about reducing call center turnover...
An extremely large company I was involved in building a system for had a requirement that you be able to read your password over the phone to a call center agent, resulting in requirements like case insensitivity and character limits.
Terrible security, sure, but at least it came from the desire for usability, not just basic encryption idiocy.
Who said the text persists, you could take it off the phone and type it in to compare against any existing salted hash. Still not good security, but not necessarily stored in plaintext.
A former employer, that I will decline to mention by name, stored a hash of the password AND the plaintext in the database so it could be sent to people via email when they forgot it.
I tried to explain to my boss why this was such a terrible idea and he was not hearing any of it.
If I ever use the "forgot my password" functionality at a site and they mail out something that is probably my original password, I make a point of cancelling my account and sending them an e-mail explaining why I don't trust them any more.
They have two classes of site that they don’t distinguish to be fair. One is the awful “You’ve forgotten your password. It’s: xxxx” where they’re just storing the password in plaintext in the database.
The other is still not great but it’s less bad: “Thankyou for registering. Your registered password is: xxxx”. In this case they’ve just passed the user details to an emailer before (potentially) encrypting and storing them.
scarier still is when they let you set it but fail to let you log in (generally happens more with length). i don't know what you're doing, but i know it's not right and it scares the hell out of me
I had that happen because of length at an online service we needed to use in high school. I believe my password had 9 characters. After requesting a password reset and experimenting I found that the registration form would allow you to enter a password of any length, but the login form would only accept up to 8 characters. What really shocked me was that I was the only one of my classmates that used a password longer than 8 characters.
I see this all the time. They accept it and then when you go to log in, it fails. I usually try chopping it down to 32, 30, 24, 20, ... until I hit a match.
I've had that happen with Comcast. I think they fixed it but it was pretty annoying. The signup password input had no max length but the login form did.
At least you got a notification. One bank just stripped all of mine off and submitted like that. At least they were consistent about it, I didn't realize until I accidentally mistyped and still got in.
In disbelief I logged out, tried again but this time I intentionally didn't use any special characters. It worked....
Worse than that: I've encountered a few web sites which accept email addresses with '+' characters... and then tell me that my email address has a ' ' character in it. Every time I see this I think "there's got to be a multiple-form-decoding vulnerability here"...
I've got an account where they just plain stripped the + character. Since I happen to have used only alphanumeric charters after, I am now registered with an email address I can't actually receive mail on - it goes to somebody else's inbox.
Account synchronization was involved - IIRC the initial address confirmation message got thorough.
I thought that though the + is valid, nothing after the + is used to differentiate the email address? I use local+Organization when I sign up for an email list so that I can easily filter, plus I can see if that email address gets shared around. So on places that reject the + I just use everything before it as the local part. Maybe I'm missing something. Edit: I guess not all email providers do it this way but here's a link to more info: https://www.cs.rutgers.edu/~watrous/plus-signs-in-email-addr...
As far as email is concerned, "+" is just a character as any other. You might as well configure your mail server to ignore everything from the first "g" in the localpart when figuring out which mailbox to deliver to. Some mailservers happen to do this with a "+", if it happens to occur in a localpart, but that is just what they happen to be doing. There is nothing technically wrong with an email address like "++foo@example.net" or "x+y+z@example.net" or "+-@example.net", and "a+b@example.net" is a different address than "a+c@example.net", unless the operator of example.net explicitly specifies otherwise.
So this is true on the email provider side, sometimes. Gmail (and a lot of others) rout foo+bar@gmail to foo@gmail. It's not required, but it's common.
The problem here, though, is that the email sender just stripped the + and sent the content to foobar@gmail.com. That's just a totally different account, chosen without warning.
I recently registered [mylastname].email, thinking to switch over to firstname@lastname.email from my gmail address. Turns out a large percentage of forms don't accept new TLDs.
I co-founded the company that set up the ".name" TLD as part of the first batch of new TLDs back in 2001, and we had no ends of problems because of that.
A lot of people either had hardcoded lists, or they checked the length and refused TLDs longer than 3 characters. We kept e-mailing people about it, and kept getting messages back from people who had "fixed" it by adding just us (we'd generally have pointed people to articles listing other new TLDs too, to make it clear to them that this wasn't just one TLD), or increasing the limit to 4 characters... We quickly gave up on trying to get these people to stop having useless checks and settled for just getting them to accept ours.
It's quite shocking people still haven't learnt even after the number of expansions since.
It's because a lot of regex validations (including earlier versions of Angular) have a regex that ends in something like
\.[A-Za-z]{2,4}$ (this is the end of the regex in Angular 1.15, it also doesn't support brackets or quotes in the local part, poor showing from Google)
At the time when they were written, gTLDs didn't exist, so the longest a tld could be was 4 characters (.info, .mobi). It's pretty piss poor futureproofing to be honest.
Angular didn't even exist when the first expansion happened, so either they were flat out ignorant, or they copied an earlier regex without verifying it (.museum was part of the initial batch of new gTLDs in 2001)
Yes, the + is incredibly useful for tagging emails. When I register new web accounts, I always specify a new unique tag so that I can track down the source in case I receive spam. Furthermore, they help my mail server when filtering out junk mail.
These days I do this differently; I created a subdomain that forwards all email to my main account. hackernews@foo.example.com would forward to main@example.com, and I can just filter the prefixes. That way I can use the subdomain for my own unique addresses, without interfering or using up addresses on the parent domain.
I do as well. Keeps a nice track of which have been leaked (or guessed) by spammers. And easy mail filtering to relevant labels on my side.
But it does get awkward quite a few times when having to interact with a human (customer services, hotel bookings, etc) via email or phone when they get a bit confused why their company name is my email alias....
Spelling a long alias over the phone letter by letter is especially tedious...
Also responding to emails either means I have to configure yet another sender alias, or mostly just send from my normal alias, which sometimes gets rejected or confuse whomever I interact with.
Also hate unsubscribe links that insist on sending unsubscribe email from that alias(mailman etc).
Doesn't solve all of it, but that's why I use random localparts instead. Sometimes people wonder whether the address is correct, but if I confirm that it's ok that it looks weird, people aren't overly confused and just accept it. I then have it all integrated with Mutt so it tags emails with a human-readable label and automatically selects the correct source address when replying.
>But it does get awkward quite a few times when having to interact with a human (customer services, hotel bookings, etc) via email or phone when they get a bit confused why their company name is my email alias....
I had that problem so many times that I whipped up a quick Rails app that generates new email addresses. Type in the company name, hit submit. It uses a random project name generator gem to create something like SteelyFishSauce@<mydomain>.com, displays it on the screen, and emails "<Company name> has been associated with SteelyFishSauce@<mydomain>.com" to the spamcatcher address.
Same here, same (minor) problems. The human interaction part is pretty much covered when I used yourorg@mail.my-name.tld for registration, but the more provocative variation yourorg@spam.my-name.tld occasionally raises some eyebrows.
The nasty part is replying with the main address as the sender. I'd really love to have email clients with reasonable support for this usage pattern for the platforms I use, but maybe then the pattern might become popular enough to lose some of its advantages.
I do this too. Unfortunately some spammers decided that they'd use <random>@<mydomain>.com as their reply address and I have to filter out all of their bounces.
I do not use Gmail but a self-hosted Postfix instance. I have configured an alias for tagged use only and configured Postfix to reject all emails to this alias without a tag.
This means that currently any tag will be delivered but luckily, I am not receiving any spam to this address at all.
Originally, I wanted to create a Postfix filter based on an HMAC together with a browser extension which would simply let me generate new valid email addresses in the form of prefix.HMAC(secret, prefix)@example.com but I have never implemented it.
But I have an acquaintance who is constantly bickering and venting about this issue. If he strongly suspects that a signup form will reject his plus address, he will enter it even more fervently to prove they are idiots.
I'm firmly on the side of "Too bad. Maybe you could simply move on with your life? Just use a dash if you need sub-mailboxes"
Is a dash the same as a "+"? According to some quick googling it depends on your provider. If his provider relies on the "+" for sub-addressing what good will a dash do him?
Why are you so firm on this position when it seems to be a legitimate problem for him?
What is easier and better for your nerves? Moving to another mail provider or getting worked up for years?
Or if moving is not an option: how about stop wasting time and nerves on it? Just accept that your mail address doesn't work everywhere. There is no possible uture where you will force everyone on earth to stop mis-validating email addresses!
And I've never disputed that the problem is legitimate. I actually said in my comment!
I think best user experience in this case is to just prompt the user asking whether they have typed in correct email address, if they say yes just go ahead and submit the form.
I've got # in my address and some systems fail with it too. It's totally valid. As example outlook.com won't allow sending email to addresses with # . Also some services reject short addresses for no reason. As example m@em.mm or so, that could be totally valid. You don't know it before checking for MX info and asking from the SMTP server if the address is valid in rcpt to: .
Speaking as someone who uses + addresses to filter stuff from mostly well-known websites, I have never seen this. I have seen this a few times on old, crusty, finance websites etc. but I hardly ever need to use a + address with them anyway. (It does make me wonder about how good their internal security is, though.)
BestBuy allowed me to sign up with a trailing "+bestbuy@gmail.com", but their unsubscribe interface rejects it as invalid, so I can't unsubscribe from their promos. I just filter/mark as spam and move on.
I've seen this several times as well. If it's a referral link to unsubscribe and is meant to include your email, double check it's okay. If your plus is missing insert "%2B" where it should be.
It's less common, but some websites ask you to type your email address once they realise they can't find you on record. If the "+" doesn't work in that field (server side rejected), then you can try %2B there as well.
If they refuse it, I think it's generally because they _know_ you're filtering them, and don't want you to sign up with a single-use address. In which case I don't bother signing up because I know they'll be spamming.
As of a year or two ago, Garmin wouldn't accept email addresses with "+" signs in them when registering for some (but not all) of their services. I think they would let you register for the main Garmin Connect service, but not for their support forum or something. Leading to a nice situation where you couldn't get technical support via the forum for the same account you had created on Connect.
Interestingly, though, the support email address they had hanging around at the bottom of the login page actually routed to an engineer (somewhere), which was nice, although they did claim repeatedly that the "+" character wasn't valid in an email address, which is of course not true. I was floored just to get a response from my complaint, though, even though it didn't seem likely to lead to them fixing the problem.
The one thing I systematically do in term of email validation is catch the common typos of the main providers. So things like gmail.con, hotmai.com, gmall.com and so on.
In 99% of those cases, it prevents someone from entering a wrong email.
We do not do email activation by forcing people to click a link in their email to validate that they received it since that causes a drop in the funnel and reduces the amount of revenues (non technical users tend to not come back when you ask them to go to their emails to verify it). So, in this case, correcting the typical typoes is very important. In our case though the information is not extremely private so it's less of a problem to do this.
Having people type the email twice doesn't really prevent typoes, people copy and paste. And if you disable paste, then it becomes annoying to users and you don't want to annoy users during the signup process (plus I hate websites that mess up with paste so I won't be hypocritical and do it).
Lastly, I know that having an email with a local domain name with no TLD is valid but it'll never be valid in the context we are sending so supporting them just doesn't make sense.
> The one thing I systematically do in term of email validation is catch the common typos of the main providers. So things like gmail.con, hotmai.com, gmall.com and so on.
That can be helpful, but I assume you suggest the user that they may have made a typo and not flat out reject the input?
> […] having an email with a local domain name with no TLD is valid but it'll never be valid in the context […]
Well, depending on how the fairly recent brand generic top-level domains work out, I would not rule out someone actually using ceo@cocacola at some point (until the CEO of Coca-Cola realises that legacy email validation regexes mean a suspiciously quieter then usual inbox).
Our email provider flat out refuses to send to those domains actually so we actually reject the input for the domains that the email provider blocks.
> I would not rule out someone actually using ceo@cocacola at some point
Of course, this is subject to change if the practice changes. The aim is to make sure that we balance the percentage of emails that are correct versus the number of potential rejections. If 0.00001% of users use a top level domain and 99.9999% of emails without a tld are invalid, then it makes sense to reject it. We're not trying to find the most perfect solution just the most convenient solution for users.
How about rejecting all email addresses that contain the character sequence "jex"? 99.9999% of email addresses with "jex" in it are invalid, so it makes sense to reject it, doesn't it?
2. No, it still wouldn't make sense to reject existing email addresses if there is a method to figure out whether the email address you are being presented with actually exists instead of divining validity using some unreliable proxy. There is just absolutely no reason to ever reject an email address that you can successfully send emails to.
I mean, in this case there are two reasons. They're both bad reasons, but still.
1. "Our email provider won't send to them". That excuses OP's part in the thing, although now we need to ask why the email provider is being stupid.
2. "We don't do validation links, they cause too many lost users". I have serious problems with this, but from a pure-business standpoint they decided that rejecting valid emails loses fewer users than using account confirmation.
Number two is vaguely horrifying to me, but in terms of "new users gained" it probably works out.
Are they being stupid though? For example, hotnail.com is a parked domain. There's virtually no chance that a user actually has an email address there and sending an email to a wrong email address is bad for the email provider reputation...
I mean it's not like they block a huge amount of domain names but with their volumes, it makes sense to avoid sending emails that will never be received by their intended recipients anyway...
Could it inconvenience legitimate users? Yeah, there's a probably of that but it's negligible and so far we've never had a complaint about it... On the other hand, we've had users telling us that it was good that our system caught their typo.
This is a fair point. I guess it's the sort of thing that I'd prefer to see fixed with a double-check prompt instead of a rejection, but I doubt it's causing many problems.
I was primarily thinking of short domains like "gmail" and "aol". For those, I can see a company called, say, "Gail" getting blocked from legitimately using "Gail.com". "Hotnail" seems a lot less risky.
Not a big problem, I just have an aesthetic objection to very high-friction things like blocking possibly-legal domains. As a double-check option I wouldn't object, and in fairness it's possible no one has ever been inconvenienced.
> 1. "Our email provider won't send to them". That excuses OP's part in the thing, although now we need to ask why the email provider is being stupid.
Nope, actually, it doesn't. If your reaction to noticing that some service that you are using is incompetent is to adopt the same incompetence, that doesn't excuse anything.
> 2. "We don't do validation links, they cause too many lost users". I have serious problems with this, but from a pure-business standpoint they decided that rejecting valid emails loses fewer users than using account confirmation.
> Number two is vaguely horrifying to me, but in terms of "new users gained" it probably works out.
Well, sure, it's as much a reason as "I don't like your nose!"
If they don't do verification emails, they might as well just not ask for an email address in the first place (or make it optional). Misguided "validation" doesn't help with most mistyped addresses anyhow.
I oversold 'excuses'. Let's say "means the original error lies elsewhere". I also wonder who the hell they're using - who ever heard of an email service that bans domains for being likely misspellings?
The "it's as much a reason" I disagree with. Validating common typos will catch more errors than false positives, so you do get more users through your funnel than if you abandon it. "I don't like your nose" is a strict loss, this causes corrections to get real emails. So they'll still miss most typos, but it's a net gain compared to not doing it.
Of course, again, I don't endorse any of this. Decide if you're ok with bad emails, follow through on that decision, use verification, and get a not-incompetent email service.
I like them as well, but for a different reason - I have a way to find what email I used for particular website, if I used it, etc. Doesn't have to be the clicky linky mail, just a confirmation mail will do.
But if you include plain text password in it... ugh.
Oh, we do send a welcome email and actually include a link in the email to subscribe to the newsletter but we don't make it necessary to use the website.
In that case, I'd rather not have the site gather my email address at all. Just create an account with an arbitrary name (or no account at all) and let the user use the web site. As soon as an email address is connected to the account (or the user name itself is an email address), I'd rather have it verified.
But I can see how there may be other concerns (commercial or not) that interfere with this.
Users remember their emails, they don't always remember their nicknames especially if they had to select an alternative nickname because the one they usually chose was already used...
Honestly, it's ruthlessly pragmatic business reasons. Is it ideal? No but in the end, it's the most painless for most users...
And, also most users on that particular site have little computer experience, so it does affect how they react and how we work. If it were something targeting the HN crowd, I'd probably have enforced email validation and would see a much lower drop in conversion due to it because people on HN are used to that and do not have a problem with it.
So, validations and UI workflow have to be adapted to your audience and your business and that's the main point really.
In this particular case, we did experiments with enforcing email validation for a subset of new signups or even allowing a small number of signups to signup with a username instead of an email. So we do have data...
Most of the work I do for other customers doesn't touch those areas of their app, so I only have relevant on hand experience on that particular site but I think it's important to mitigate the recommendation of "Always validation emails" or "Always use regexp" and try to think of the best experience for the kind of users you're targeting.
Sometimes in HN, people talk in absolutes when things are instead very context-dependent.
> We do not do email activation by forcing people to click a link in their email to validate that they received it
Have you considered doing email validation without forcing users to comply? It's fairly common to require valid emails after X days, or in order to unlock N features. Gets the advantages of having valid emails without the disadvantages of a drop signups.
TL;DR the odds that the user entered an incorrect-but-valid address are way higher than that they entered one which will not actually be able to receive mail. Send a validation email.
Also relevant was, "If you have a well laid-out form with a label that says “email”, and the user enters an ‘@’ symbol somewhere, then it’s safe to say they understood that they were supposed to be entering an email address."
In other words, it does make sense to check that they entered an '@' symbol somewhere, since it shows that they understood it was an email field. Any 'validation' beyond that is useless.
It doesn't matter. If they have typed an @, they probably understand that it's an email field. Trying to validate to the spec beyond that is pointless, for reasons thoroughly covered in the article.
It actually used to (maybe still does) signal routing information[1]. An example from the RFC is "@ONE,@TWO:JOE@THREE"
[1]: https://tools.ietf.org/html/rfc821
For flip's sake, yes please! Close the loop, fer crying out loud.
I got a popular givenname.familyname@gmail.com address and I frequently get mail that's meant for other people who share my name. The vast majority of the time it's the individual themselves who sign up for a service or offering but there's rarely a validation upfront.
The best emails are the ones who extend full trust to the email recipient over some account during that first email. Facebook, shame on you.
Someone could have just created an account and accidentally put the wrong email address in. Therefore, putting a link that extends full trust in the welcome/confirmation email is a design error. The fix for this error is to require the newly created account holder to put in their password on this first login. While the account's in this state it shouldn't be possible to reset the password.
Address validation by sending an email should only be used if it is required for some reason to verify the user owns the email account. Otherwise, it's not a great UX.
I honestly can't think of a reason you'd ask a user for their email but not need to validate it.
For being able to do password resets later, permission to add to mailing list, avoiding sending private info to the wrong user, avoiding allowing someone to masquerading or impersonate someone they're not.. All should be validated.
If you're looking for a username as login identity and nothing more (and you don't have password reset functionality), then ask for a 'username', not email addresses.
"For being able to do password resets later" - don't need to validate.
Mailing list - reluctant OK
"avoiding sending private info to the wrong user" - how could this happen?
"avoiding allowing someone to masquerading or impersonate someone they're not" - how could this happen? Like if I signed up as tim@apple.com? What could realistically happen?
email addresses for usernames have the advantage of already being unique. None of this "gregmac already taken, try gregmac23, gregmc_595, or verbingnounXX instead?" nonsense
One argument against emails-as-names is that some people change email addresses frequently - unless you have some way of accounting for that (and you definitely wouldn't use the username as the contact field), the username will eventually go stale for some users.
You should allow changing the e-mail address. Of course, changing it requires the same validation, but I'm not really sure why there'd be an argument against it (though I do note many sites/applications don't have a mechanism for this).
At the same time, as a user, you need to consider your own e-mail setup. You should not use your ISP-provided e-mail address whatsoever, there is absolutely no reason to tie your contact to an account you may not keep.
Likewise, you should not be using your school or work-provided addresses for something that you expect to use beyond going to school or having that job.
If you're changing your e-mail address frequently for personal reasons (eg, someone harassing you?), then you should actually consider creating an account you use only for authentication, and a separate one that is your public-facing persona.
Having separate addresses for your contact and sign-ups is actually not a bad idea anyway. It actually makes it a bit harder for someone to break into your account somewhere (because they don't necessarily know what address you used), and you get the freedom to change/abandon your public e-mail address if necessary.
I mean unique as far as your end is concerned. Short of very invasive technology, there's no way to stop people from sharing a user token if they choose to do so.
Except, it's not a "user token", it's an address. It's as much a user token as a postal address or a telephone number is a "user token": not at all. It's a way to contact a person, not a way to identify a person. Just because you can enforce that only one person with a given postal address can create an account with your service, doesn't make it an inherent "user token".
Sure, but what if two humans share an email address, but both of them want to sign up for your game/service/whatever independently? This is a fairly common situation if you are (say) designing a game that might appeal to kids. Or old people.
If you block that, you're going to lose users. Either because they just can't sign up at all, or because there's too much unnecessary friction in requiring someone to go out and sign up for an email address just to register for your service.
Probably the most common example these days is spouses who don't email regularly.
Once upon a time, you only got one or two email addresses from your email provider (I'm looking at you, CompuServe and AOL), so at that time it was common for everyone in a family to have a single address. Eventually they added explicit multi-user features -- IIRC, AOL went as high as 5 emails per account before my family moved on.
I always assumed it was more a sanitization issue for security's sake. By allowing only a simple subset ("common") email address type, you can be ambivalent about what email server is running and how it reacts to the wide variety of specially crafted email addresses.
With no validation other than sending the email, you have to know, for example, what the server would do with an email address that claims to be @localhost. Now it becomes a problem- or at least a question and concern- for the backend system. Whether the backend interprets root@localhost as valid and does exactly what it's told or rejects it due to some configuration- it has become a backend complication and a DOS attack vector.
A simple policy of only handling a subset- the common class of email addresses- is one of the things that allows us to have a simple mental model of what the MTA is supposed to do. The fact that it sometimes caught a type-o, or not, is incidental. "Invalid email" wasn't meant to imply the email address doesn't fit the spec- it was meant to imply that a particular site or service has chosen not to accept email addresses like that.
> I always assumed it was more a sanitization issue for security's sake.
Sanitization is at best idiotic, at worst creates security problems. There is no such thing as "bad characters", there only is broken code that incorrectly encodes stuff. If you ever find yourself modifying user input "for security reasons" (or really, for any reason at all), you are doing it wrong. The only sane thing to do is to make sure that the semantics of everysinglecharacter of your user's input is preserved in whatever data format you need to represent it in.
An email address isn't a document though, it's a routing command. I don't mean sanitization in the sense of inserting backslashes. I mean sanitization in the sense of "we don't allow people to set their email address to a mailbox on localhost at our mail server."
1. Sanitization generally means changing information. As in, "removing bad characters", that kind of stuff. That's different from validation, which should result in rejection of bad input, and which can be perfectly fine. However, more often than not, validation is implemented badly and rejects perfectly fine input, which is why validation shouldn't be employed more than necessary either.
2. Rejecting @localhost addresses doesn't really make a whole lot of sense. People could just enter the public IP address or hostname of the server, or add a DNS A record under their own domain that points to 127.0.0.1, or an MX record that points to localhost, or any number of other weird stuff that you could not possibly validate anyway (if only because it could be changed at any point lateron). Just configure your mail server properly and then send the damn email, and if it does get sent to root@localhost, and possibly forwarded to the admin--so what? People obviously could just sign up using your admin's email address anyway, and that not only at your site, but at millions of sites out there, you won't be able to stop them. There is nothing particularly dangerous about receiving unsolicited signup emails or about sending emails to yourself.
> There is nothing particularly dangerous about receiving unsolicited signup emails or about sending emails to yourself.
Depends on what you do with them, in the latter case. There could be an amplification attack there.
Validating domain parts to a certain extent isn't a bad idea, at least as far as non-routable domain names and RFC1918 ranges go. I've seen this done (actually implemented some of it, in fact) at a past employer, who were basically looking to cover the 90% case in terms of not getting hosed by a trivial attack. It doesn't take much effort and it makes 4chan's life harder. What's not to like?
> Depends on what you do with them, in the latter case. There could be an amplification attack there.
Hu? How would that work?
> Validating domain parts to a certain extent isn't a bad idea, at least as far as non-routable domain names and RFC1918 ranges go.
What do you mean by "non-routable domain names" and what do you gain by checking for RFC1918 ranges?
> I've seen this done (actually implemented some of it, in fact) at a past employer, who were basically looking to cover the 90% case in terms of not getting hosed by a trivial attack.
Why did you prefer that approach over a robust solution?
My idea of a robust solution: Have one central outbound relay that's firewalled off from connecting to anywhere but the outside world, make all servers that need to send email use that relay as a smarthost (so they never connect to anything but that relay, regardless what the destination address is), use TLS and SMTP AUTH with credentials per client server to prevent abuse of the relay by third parties.
> What's not to like?
(a) that it's a lot easier to build a solution that's more robust, (b) it's extremely likely that your implementation is buggy, thus rejecting valid addresses, and (c) it's causing a maintenance burden (what happens when the first people drop IPv4 for their MXes? I'd pretty much bet that you don't check for AAAA records, so you'd probably suddenly start rejecting perfectly fine email addresses, thus making the transition to IPv6 unnecessarily harder, am I right?).
'Non-routable' as in a single label, or as in not resolvable. I don't think it is unreasonable to consider an address invalid when its domain part cannot be resolved. Checking for RFC1918 ranges means you don't try to send to another class of addresses that's never going to be received.
You would lose the bet. The product supported IPv6 from day one.
That is a robust, if somewhat complex, solution for a relatively small volume of mail. When you're sending ten million messages a day by the end of the first month, pushing everything into a single relay of any kind is asking for a lot of trouble.
> 'Non-routable' as in a single label, or as in not resolvable. I don't think it is unreasonable to consider an address invalid when its domain part cannot be resolved.
What exactly do you mean by "cannot be resolved"?
> Checking for RFC1918 ranges means you don't try to send to another class of addresses that's never going to be received.
But why check for it? Is that actually a common mistake people make? An attacker could change the address after you checked it, so it's not going to help against attackers, is it?
> You would lose the bet. The product supported IPv6 from day one.
Good for you! :-)
> That is a robust, if somewhat complex, solution for a relatively small volume of mail. When you're sending ten million messages a day by the end of the first month, pushing everything into a single relay of any kind is asking for a lot of trouble.
Complex? Certainly less so than implementing validation yourself.
As for scalability: Well, yeah, as described that's more the setup for a company that's operating various different services, none of which has a high volume of outbound email (which would be most, even the best startups don't have ten million signups per day and don't send much email otherwise, and even that should actually still be managable with a single server).
But that's trivial to adapt without changing the general approach. First of all, obviously, you could just add more relay servers and have client servers select one randomly, that scales linearly. But if you really need to move massive amounts of email for one service, so that adding an additional relay hop for each email you send actually adds up to noticable costs, you can still use the same approach: Just put the MTA onto the same machine(s) that the service is running on, into its own network namespace (assuming Linux, analogous technology exists on other platforms), and firewall it off there so it cannot connect to your internal network. Potentially you can even just add blackhole routes for your internal networks/RFC1918 ranges, so you would not even need a stateful packet filter (though currently you might still need it due to IPv4 address shortage).
Why assume email addresses only get checked in one place, and not all?
Ten million a day was a milestone. I left that company over a year ago; it would astonish me to find that figure now exceeded by less than a factor of twenty. Granted these are mostly not signups. They are outgoing emails nonetheless, which makes the case germane despite that superficial distinction.
Your proposed solution sounds pretty expensive in ops resource, to no obviously greater benefit than the rather simple (well under one dev-day) option we chose. You seem to feel yours is strongly preferable, but I still don't understand why.
NXDOMAIN can be a temporary error. The SMTP queuing protocol is designed to be resilient against DNS failures, internet outages, routing problems, and temporary mail delivery issues.
Unless some DNS server is broken, it actually cannot. NXDOMAIN is an authoritative answer that tells you that the domain positively does not exist. Not to be confused with SERVFAIL, which you should get if the DNS resolver ran into a timeout or got an unintelligible response or whatever, NXDOMAIN should only occur if the authoritative nameserver of a parent zone explicitly says "I don't know this zone either locally nor do I have a delegation for it".
OK, that at least shouldn't reject any valid addresses, so maybe ...
> Why assume email addresses only get checked in one place, and not all?
It's not an assumption, it's just a matter of simplicity and reliability.
> Ten million a day was a milestone. I left that company over a year ago; it would astonish me to find that figure now exceeded by less than a factor of twenty. Granted these are mostly not signups. They are outgoing emails nonetheless, which makes the case germane despite that superficial distinction.
Well, no clue how well common MTAs would cope with that, but 2500 mails per second could still be within the power of a single machine if it's designed for high performance. But regardless, I don't think that really matters: If you have a relatively low volume of emails, it's probably most efficient and secure to handle it all with one central outbound relay, if you need to send lots of emails, it obviously makes sense to distribute the load, but that shouldn't really otherwise change the strategy.
> Your proposed solution sounds pretty expensive in ops resource, to no obviously greater benefit than the rather simple (well under one dev-day) option we chose. You seem to feel yours is strongly preferable, but I still don't understand why.
What sounds expensive about it?
Whether there is any benefit to it: Well, depends on your goals and what exactly your solution actually does. I still don't understand why (or even how exactly) you do those RFC1918 checks, for example?! It seems like it's mostly a security measure? But then, it's not actually secure, it's essentially a race condition/TOCTTOU. Plus, it might even break valid email addresses.
Essentially, it's four factors why I think just delegating the validation of email addresses to the mail server is the best strategy:
1. Implementing your own checks risks introducing additional mistakes (which might lead to the rejection of valid addresses).
2. Implementing your own checks is additional work when you could just use the MTA which already knows how to do this (and which you have to install/configure/use anyway as soon as you want to actually use the email address), both for the initial implementation, and possibly for subsequent maintenance (if you just let your MTA do the work, only the MTA needs to be adapted to any changes in how emails get delivered, abstracting away the problem for any software that's supposed to be sending emails and isolating it from the lower layers).
3. My approach actually gives you the perfect result, in that it does not reject any valid addresses (assuming your MTA implements the RFCs correctly), and at the same time is perfectly secure against all possible abuses with weird addresses, which is impossible to achieve when you separate the check from the actual abuse scenario, and it's even kindof trivial to see that that is the case.
4. You have to have the infrastucture to deal with bounces anyhow, both because you ultimately cannot be sure an address actually exists unless you have successfully delivered an email to it, and because addresses that once existed might not exist anymore at a later time, so it's not like you can avoid that if only you validate your addresses better.
As for "ops resources": Assuming that any service that needs to send > 10 million mails per day will be deploying machines automatically anyway, what's so much more expensive about deploying the configuration of an additional network namespace? Writing that script certainly shouldn't be more than a day of work either, should it?
I may have erred in giving the impression that the application-level checks are the only line of defense here. They're not. The (bespoke) MTA underlying this product performs most if not all of these checks as well. I didn't really spend any time on that side of the business, so I might be wrong about that, but it would be something of a surprise. I do know our analytics needed to be able to cope usefully with an astonishing panoply of bogosity warnings that came back from the MTA, but I no longer recall exactly what they covered. And, in any case, it's nice when you can to tell the user "hey, this isn't deliverable" before it gets to the point of a bounce.
Checking whether an email address's domain-part is an RFC1918 IP is actually pretty easy. Split the address by '@'. The last piece is the domain part. Split it by '.'. If there are four pieces, all of which meaningfully cast to integers, treat it as an IP address. (Otherwise it's a domain name, which is fine as long as it has more than one part and isn't NXDOMAIN when the backend tries to resolve it.) If any part is negative or greater than 255, it's invalid. If it starts with [10] or [192, 168], or if it starts with [172] and the second part is between 16 and 31 inclusive, it's an RFC1918 address. Otherwise, it's fine.
Even with unit tests, that takes almost no time to write, and when your frontend and backend share a language as ours did, you can use the same logic both places. Node gives you a name resolver binding for free. I really can't imagine it being as quick to write, test, validate, and roll out a change to the MTA node manifest.
We were pretty sure it would already be impossible, or nearly so, for a malicious user to probe our infrastructure this way, but when it's so simple to be even more sure, why not?
Similarly, we'd already observed a low but nonzero rate of users inadvertently providing such addresses - not during signup or onboarding so much, but in recipient lists they submitted. Since we used the same recipient checking code everywhere, why not cut that back to zero, too?
Then reject email addressed to localhost. It shouldn't matter how the email got there. I'd suggest that especially given DNS trickery involving setting up a low TTL then redirecting to 127.0.0.1, you're probably not preventing this from happening or you'd have to invalidate any unrecognised domain. Better to solve that problem at a different layer -- validate the email by sending a validation link if you must...
And their point is, it's a backend issue, the backend being the mail client/server that already completely handling the sending/receiving of emails. Either the activation link gets clicked or it doesn't. The click is the only correct validation, and yes, the whole process happens on the "backend".
I mostly agree but there are cases where some definition of sanitization is the only appropriate thing. For example, if you allow users to create content with a lightweight subset of HTML for the sake of formatting control and want to render that html in your page. And in such cases, the correct way to sanitize it is not via regexps but via a DOM parser that takes user input and builds a DOM and then emits rendered html according to a whitelist of available tags/attributes. So you might argue DOM parsing isn't sanitization and so still matches your assertion, however, in general it's common and not really inaccurate to call this sanitization.
The important thing is to not change information. "Sanitization" as it is commonly used means doing something that (potentially) changes information. Which is in contrast to decoding/encoding/parsing/unparsing/translation/..., which, if done correctly, change representation, but not information.
So, to make it a useful distinction, I would call anything that potentially changes the semantics of the processed data "sanitization", and avoid using the term for anything else.
So, simply parsing a string with an HTML parser, possibly checking for acceptable elements, and then serializing back into some sort of canonical form that is semantically equivalent to the input, that's perfectly fine, and I wouldn't call that sanitization, but rather validation and canonicalization.
If you simply start dropping elements, though, that's probably a bad idea, just as simply dropping "<" characters is a bad idea, because those elements presumably bear some semantic meaning, just as a "<" in a message presumably bears some semantic meaning.
Now, it is not always obvious which level of abstraction to evaluate the semantics (and thus the preservation of semantics) at. So, it might be prefectly fine, for example, to remove or replace some elements where the semantics are known and you can show that, say, removing emphasis still generally preserves the meaning of a text.
But a whitelist approach where you simply remove everything that isn't on the whitelist usually is a bad idea. If you want to have a whitelist, use it for validation, and reject anything that's not acceptable, so the user can transform their input in such a way as to avoid any constructs you don't want, while still retaining the meaning of what they are trying to say.
I hear what you're saying and it represents an ideal. But there are circumstances where information really has to be removed. Perhaps because the user is no longer present and it was collected under circumstances that had more liberal validation. Or because you're handing information across a boundary of implementation ownership and can't trust the receiver to handle potentially dangerous information correctly. I agree that sanitization (in the sense of stripping bits out of a user data payload according to some security rules) shouldn't be the first tool in the toolbox, but I would really hesitate to say it's always the wrong thing to do.
Edit: here's a good example. I don't know if they still do this, but when I worked at Yahoo!, they used a modified version of PHP that applied a comprehensive sanitization process to all user inputs. As a frontend coder at Y!, all the information you pulled from request parameters, headers, etc, ran through this validation at the PHP level before your app code got to it. You can then literally splat this information into an html page raw, without any further treatment, and not expose the Y! property you work for to an XSS or other injection vectors. There were ways to obtain the raw input using explicit accessors when needed, and these workarounds were detectable by code monitoring tools and had to be reviewed and approved by security team(s). Overall this worked really quite well, in my opinion. Y! could hire junior frontend devs without deep knowledge of data encoding, security issues, etc etc and rest easy. I think the principle of safe-by-default, even if it means destruction of user input in some cases via aggressive sanitization, is a good principle to apply to a frontend framework.
No, that's just a terrible idea. It might work quite well in the sense that it prevents server security holes. But it makes for terrible usability, and potentially even security problems for the user. The user expects that their input is reproduced correctly, and if it isn't, that can potentially have catastrophic consequences because it might result in silent corruption.
If you think that it should be possible to have developers who don't understand this problem and its solution, you might as well argue that it should be possible to have developers who don't know any arithmetic or who are illiterate.
If you want to have some help from the computer in avoiding injection attacks, the solution is a type system that ensures that you cannot accidentally use user input as HTML or SQL, for example, or possibly automates coercion when you need to insert pieces of one language into another.
And that's a great philosophy until your email gets rejected by some service that picked a different "common class of email addresses" than you did. This is precisely why we have written standards.
Back when SMTP servers still had remnants of UUCP etc.- where the address could actually contain characters that specify intermediate servers to route to, I would have argued that front-end sanitization was important as, for example, html sanitization from end-users from a security perspective.
However, IETF made lots of progress simplifying things- to the point where, at the very least, the standard tells us specifically that we should leave it up to the destination host to interpret the local part of the email address-- that is, the thing to the right of the @ should be given the thing on the left unmolested ideally- even being ignored by intermediate relay servers. Since that's what most people complain about, any validation to the left of the @ should become extinct.
But off the top of my head that still leaves the thing on the right of the @ (such as localhost), buffer overflows by allowing longer strings than the standard allows (those limits do exist), and the problem with multiple @ which the MTA may or may not handle well... Since I'm not a security expert I'm going to go out on a limb and assume that I'm missing a bunch of other things.
My point is not that the article is wrong, though- my point is that if he wanted to convince me to only validate by sending the email on any string, he should convince me that those security concerns are not an issue- not that it's not good at catching type-os.
I think there is a reasonable middle-ground for validating the domain side of an email address. There are RFCs on all this stuff; it's not just a total free-for-all. The RFCs just aren't nearly as strict as a lot of badly-designed validation regexps are, presumably because most people are unaware of the diversity of acceptable email addresses.
There are some basic rules that you could safely apply to an address, which would prevent some attacks (buffer overflows, etc.) while also not blocking any legitimate addresses. E.g. limiting the overall length to 255 characters, for instance, could be defensible practice. There are also well-defined rules for validating the domain portion, since it has to be a routable address by definition.
What nobody ought to be doing is looking too hard at the string to the left of the @ symbol, because it's designed purely as instructions to the recipient server. Nobody else needs to care about it; only the receiving mailserver needs to actually parse that part of the address, in order to put the message into the right mailbox. From what I've seen, the vast majority of false-positive validation failures occur because people are looking at the mailbox portion of an email address when they have no business doing so.
Hmm, sorry but I don't buy that the "correct way to validate" is not to validate the input.
Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else.
Sure, the complete regex is complex, but it is defined and is hardly unconquerable. Look at Django's `EmailValidator` implementation for example [0] that is mature and well tested [1].
The author has not convinced me that ignoring validation is the right choice when options with a scope so thorough exist.
> Sure, the complete regex is complex, but it is defined and is hardly unconquerable.
If it is a regular expression, then it is not able to match all valid email addresses, because the grammar of email addresses is context-free, and regular expressions can only match regular grammars. It doesn't matter if it is defined or not: if it's a true regular expression, then it simply cannot validate email addresses.
(it may, of course, be a context-free expression masquerading as a regular expression)
I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)
I suspect that it will, but of course I could be wrong.
I don't think you're really correct about "email addresses" being context-free, or at least, citation, please?
When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`. I don't think most people are going to think they can enter that, nor would most programmers even think to implement it. And I certainly wouldn't want to try explaining it to a PM…
If you accept that assumption, what about `addr-spec` isn't regular?
Also, using that assumption, your "perfectly valud email addresses such as …" would appear to not be valid, as it has unbalanced quotes. (In fact, even under the grammar of `address`, I'm not sure it's valid; it feels like it should be invalid for the same reason, but I've not rigorously checked this.)
> I don't think you're really correct about "email addresses" being context-free, or at least, citation, please?
> When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`.
Well, sure. Let's look at what RFC 5322 defines as an addr-spec[1]:
Let's ignore quoted-string and obs-local-part for the moment. What is a dot-atom?
dot-atom = [CFWS] dot-atom-text [CFWS]
And what is CFWS?
CFWS = (1*([FWS] comment) [FWS]) / FWS
What's a comment?
comment = "(" *([FWS] ccontent) [FWS] ")"
So far, all of this has been matchable with a regular expression. But what's a ccontent?
ccontent = ctext / quoted-pair / comment
See that there? A comment is composed of a balanced pair of parentheses around, perhaps, another comment! Thus (this (is (a (heavily (commented (email \(address))))))foo@bar.example(some more (to prove (the point))) is a perfectly viable RFC5322 address!
Pair-balancing, of course, is impossible with regular expressions, since matching pairs requires push-down automata (which match CFGs) and cannot be done with finite-state machines, (which match regular expressions).
QED.
> Also, using that assumption, your "perfectly valud[sic] email addresses such as …" would appear to not be valid, as it has unbalanced quotes.
Nope, there are no unbalanced quotes in (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest): the first quote balances with the third, while the second quote is one of a quoted pair \" (which is allowed within a quoted-string, which is allowed within a local-part). It's all allowed per the spec.
I'll admit that it's a bit surprising, but it's true. One simply cannot match a valid RFC5322 addr-spec with a regular expression. One can, of course, match it with something which pretends to be regular but isn't really (as I noted).
Oh, shoot, I missed that. I saw "FWS" (meaning "folding white space"), and assumed that didn't include comments since they have nothing to do with folding whitespace.
I'm not sure if they are context-free, but talks about Parseable Expression Grammars, specifically Lua's LPeg, implies they might be. Conceptually, PEGs are to Context Free Grammars as RegEx is to Regular Expressions.
You can find multiple short email validation snippets using Lua LPeg pretty easily, but this is from the Lua creator's talk about LPeg which includes a part about proper RFC822 validation and how complicated it is for regex, but can be concisely done with PEGs.
Are those parens really permitted in the host name? I'm looking at RFC 3696.
> 3. Restrictions on email addresses
> .... The syntax of the domain part corresponds to that in the previous section.
and
> 2. Restrictions on domain (DNS) names
> Any characters, or combination of bits (as octets), are permitted in DNS names. However, there is a preferred form that is required by most applications. This preferred form has been the only one permitted in the names of top-level domains, or TLDs. .... The LDH rule, as updated, provides that the labels (words or strings separated by periods) that make up a domain name must consist of only the ASCII [ASCII] alphabetic and numeric characters, plus the hyphen. No other symbols or punctuation characters are permitted, nor is blank space.
> Are those parens really permitted in the host name? I'm looking at RFC 3696.
RFC5322 states that an addr-spec consists of a local-part, followed by @, followed by a domain. It states that a domain may be a domain-literal; a domain-literal may begin and end with commented folding whitespace (CFWS). It's all in https://tools.ietf.org/html/rfc5322#section-3.4.1
It's only the dtext portion of the domain-literal which must consist of printable ASCII characters, not including [, ] or \.
Yes, foo@bar!baz!quux is a valid RFC5322 email address (decimal 33, i.e. !) is allowed, per the dtext production of the spec) — and so is foo@(comment (nested comment (escaped \(comment)))bar!baz!quux(another (comment (to prove) the point))
Whether one should ever actually use such an address is, of course, another matter entirely.
The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.
> The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.
It is! Conceivably, it could be used to implement a new kind of domain-less mail system, e.g. foo@$megamail or something.
I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)
If it doesn't, remind me to patch it to not accept that.
I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.
> If it doesn't, remind me to patch it to not accept that.
You'd be wrong to do so. The whole point of RFCs and Standards is to take things out of the realm of personal preference.
Also, I suspect that a validator which allowed reasonable addresses like jim(somesite)@foo.invalid (which is both a good use of comments and what the + hack emulates) or "Ted Smith"@bar.invalid or "work@home"@jobs.invalid or "William \"Bill\" Jones"@baz.invalid but disallowed unreasonable ones would also be able to solve the halting problem. How many perfectly reasonable allowed features must one use before it's considered abuse?
> I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.
An email address is, simply, an address which can be given to an Internet mail server in order for it to properly route an email. All of the addresses I gave are perfectly valid ways to do so. An addresses is a property of the addressee, not of the addressor.
In other words: it may be your site, but it's my address.
There are already things that are technically allowed by RFCs for various types of input but should be disallowed because they cause problems. So something being in an RFC doesn't create any kind of binding requirement, and very often the things that get dropped are things that, on contact with the real world, turned out to be bad ideas.
Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that.
Also, no halting-problem issues at all; my preferred approach would be to disallow certain classes of characters.
> There are already things that are technically allowed by RFCs for various types of input but should be disallowed because they cause problems.
That may be true, but it's something else entirely for someone to drop them because the problems they cause … were enabled by him. An email address using characters you don't like is perfectly valid, and perfectly deliverable, right up until you refuse to deliver it. It doesn't cause any problems until you decide to make it cause a problem.
> Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that.
> Also, no halting-problem issues at all; my preferred approach would be to disallow certain class
I really don't see why you think it's a bad idea to disallow more than alphanumeric (and plus? and ampersand?) characters in the local-part. Cui malo?
I would prefer to allow plus, dot, hyphen, underscore, and alphanumeric characters in the local part and probably not much of anything else.
I prefer this because I work in a world where email addresses are provided to me, and possibly stored and then retrieved and used and even displayed, as text, and the RFCs allow some nightmarish things when you consider the interaction of the syntax the RFCs permit and the set of characters which are sensitive to one or more of the non-MTA components of that chain.
Your original example, for instance, contains characters that require escaping or at least careful handling for multiple situations, and even goes so far as to contain things that will be interpreted as escape sequences in some contexts. I'm sure that Robert(';DROP TABLE users;)(<script type="text/javascript">alert("Bobby Tables")</script>)@not.a.hacker(honest) will be terribly disappointed to know that he needs to use a different address to register on my site. I'll send him over to you instead.
It is true that regular expressions in the CS sense can't parse context-free grammars. However, PCRE, which is what most programmers are talking about when they say "regex", can do so. So you're both kinda right, I guess. But you're being a bit pedantic.
Interesting; I've never had to think about how regex interact with Unicode.
I guess what I really meant is that programmers are talking about PCRE in terms of power, not in terms of exact syntax. In particular, they have recursive patterns, which are sufficient to pull them up to context-free grammars.
By all means, validate. But err on the permissive side. The odds are pretty great that you're wrong, and you're preventing real email addresses (and almost certainly real users) by doing so.
I'd never looked closely at Django's email validation, but after looking at it I'm inclined to stop using it. Example reason:
> # max length for domain name labels is 63 characters per RFC 1034
There is a high likelihood that most of your users have shorter domain names, and also a high likelihood that most if not all registrars currently enforce this. But it also used to be highly likely that domains didn't begin with a numeral, and even though it was disallowed, 3m.com existed.
Another: neither the tests nor the patterns account for unicode, which is valid for domain names irrespective of the local part, and if not allowed for the local part by existing providers almost certainly will be eventually.
Also, a thing I noticed in the Django tests that is just upsetting:
- name: `TEST_DATA`
- values: `(validator, value, expected)` (a validator of the data is not the data)
- unstructured groups of such that should almost certainly be structured
The 63 character limit on domain name labels in RFC 1034/1035 isn't arbitrary, it's a limitation in the way DNS packets are encoded. The upper two bits of the label length are used for compression of labels in DNS packets.
Without protocol changes, DNS cannot support labels longer than 63 bytes.
However, I agree that validating it is unnecessary.
You can use a regex as a simple pre-check but you absolutely have to do more than that if you expect high-quality results.
Back in the 90s, we ran the customer rewards program mailing list for a mainstream business you've heard of. A [gnarly] regex took care of the gross failures but we still had double-digit percentage of invalid addresses and many spam reports because people mistyped their username, used their old ISP's email address which had been disconnected, etc.
The only approach which produced satisfactory results back then was to have a bit of (horrible) PHP code which did a full SMTP connection to deliver the welcome email. Even that wasn't enough to ensure you'd delivered it to the right person, however, so we had to track clicks on an activation link.
I would be surprised if the average internet user has gotten significantly more reliable over the last couple of decades and that's born out by the wide variety of misdirected legitimate email my shortname@gmail.com account receives.
Definitely, I think email validation links are important too. However, it's pretty senseless to let an obviously invalid email address pass all the way through to that layer (and potentially get billed for sending messages to invalid email addresses).
Obviously invalid can be a tough measure, though - many people who used domains other than .com/.net/.org have reported sites incorrectly denying their address. If I was still doing this, I'd start by doing some loose validation on the address and checking whether the domain resolved in DNS before returning a form validation error.
The other key step is having some contact form which doesn't require the same test so people can report bugs if you get the logic wrong.
The marketing or product team might prefer the "cost"(?) of sending a bad email here and there to the cost of losing a customer because the email is incorrectly rejected.
I am not sure what you are trying to say. Yes, of course, it costs money. Just like serving websites costs money. And just as checking the syntax of an email address costs money (CPUs cost money and need power to run!). Now, how is that an argument for checking the syntax of an email address to avoid sending a single email and seeing whether it bounces?
Sending a lot of bounced emails is going to get your mailservers reputation tanked, which is not an acceptable tradeoff for removing email validation.
Email validation is actually really important if you send even a medium amount of email. Testing by bouncing is not a very good idea, especially since there are a lot of ways to test the deliverability of the email before you try and send it.
That's not to say use a regex instead, you can do some simple checks on the domain etc or you can use one of the many services to check for you (for of a fee of course).
> Sending a lot of bounced emails is going to get your mailservers reputation tanked, which is not an acceptable tradeoff for removing email validation.
?!?
OK, let's break this down: What could be reasons for an email to be bounced due to an invalid address?
1. Because the address is syntactically invalid: Your own email server will notice and reject/bounce the email. Noone outside your own server will notice.
2. Because the domain doesn't exist or has no inbound mailserver: Your own email server will do the DNS lookup(s) and bounce the email. Noone outside your own server will make any note of that (and you can't find out without doing DNS lookups anyway).
3. Because the localpart doesn't exist: You can't find that out without trying to actually send an email (and consequently risking a bounce).
So, what exactly was the point of email validation "without bounces" again?
The only thing you can actually do to avoid unnecessary bounces is to make sure your emails have a valid return path and to make sure that you somehow process all the bounces that arrive there to make sure that you don't continue sending emails to addresses that start sending bounces (i.e., that don't exist (any more)). If you send lots of mails, it's probably best to employ VERP for that.
> Validating against the RFC is somewhat complex though.
Hence, not obvious :-)
If there is one thing we can learn from all the misguided attempts at validating email addresses that we have seen over the years, it is that it is obviously not obvious which email addresses are syntactically invalid.
With respect to (1), the cases I had in mind as "obviously invalid" is something that doesn't contain all of the required components of an email address. For example, if the user submits their email as "foo" or "foo@bar" or "foo@bar." I should have provided examples as the way I wrote it was definitely ambiguous.
Both "foo@bar" and "foot@bar." are valid email addresses. "bar" can be a host name or a top-level domain name. "bar." is a top-level domain name.
That said, "foot@bar." may be valid, but ICANN says it should not be functional. They prohibit dotless domain names (that is, top-level domain names may not have A, AAA, or MX records).
"foo", ok, I agree with the blogpost, if there isn't even an @ in there somewhere, maybe it's sensible to catch that and reject it outright. But that's probably about it.
Right. Wholeheartedly agree. Sanitize all input. If you _KNOW_ input you're receiving is invalid, throw it away.
Also whether you use RegExp on email or not totally depends on what you're doing with the address. Are you throwing that email into Salesforce? Great! Salesforce does RegExp validation on all email addresses and the field is required. You _HAVE_ to do it. Solution? Use the same validation that Salesforce does. That's what I do.
Also the OPs math about how common invalid email addresses are is wrong. Do you know what the most common way of screwing up an email address is? Hitting tilde when you're tabbing away from the field. And that error is in the tenths or hundreds of a percent range (which is still significant). I have a dataset of 5MM email addresses typed in by jaded call center agents to prove it.
> Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else.
Yes, they aren't special, and you shouldn't validate any of the other stuff either, unless you actually need to understand the semantics. So much breakage happens because people think that they need to validate all kinds of stuff, it's not funny. People have their legal real names rejected because they're "not a valid name", people have their correct postal address rejected because it's "not a valid postal address" ... just forget it, if someone tells you their name, just believe them it's their name, if someone tells you their postal address, just believe them it's their postal address, if someone tells you their email address ...
Oh, and don't ever think about "sanitizing" stuff. Just don't. If you think you need it, you are doing something wrong. The solution to SQL injection is not to disallow people using "'" characters in messages, the solution to cross site scripting is not to prevent people from using "<" characters in comments, ... that's what encoding/escaping is for.
I do a lot of optin email. Here are some examples of bounced emails that people use to sign up:
* somename@gmail.co
* anothername@yhoo.com
* myemail@hotmial.com
These are very common errors that occur nearly every day. A regex isn't going to help here. What does help, is a notification that asks people to verify what they typed –– if the email contains an obvious, common error, such as one listed above.
Asking to retype though being a simple solution IMHO is asking for a lot. Consider a user who uses mobile phone, even copy paste is annoying. Validating if the mail box exists and that it does not belong to a provider like mailinator and then sending a confirmation link to them works. While its not perfect, it does address lot of other concerns without sacrificing user experience.
Also, asking to retype makes sense for a password where you can't visually verify that you typed what you expected. For an email address, it's pointless and frustrating. Send an email. If it bounces or doesn't get verified in a timely fashion, it was a mistake and delete the account.
I don't think having users always retype the address is what the parent post suggested, but to look for common errors and if one is found prompt for verification.
I'm planning a follow-up post with all sorts of real validation just like what you mention here. Can't take too long to build an array of 'probably typos'. I wish had access to logs of failed emails.
There are other situations where you do have to guess whether a string of data is a "real" email address or if it's some garbage data that turns up in information systems. "Validating" email addresses (without emailing somebody) is a real problem. By the way, maximizing the accuracy of such a guess can require going against the standards.
One thing I've found that helps a lot is instant delivery notifications. When you try to register a new account, our "We've sent a confirmation email" screen will report within a few seconds if there was a mail delivery error and allow the user to correct their address. Common typo detection for popular email domains is also beneficial (https://github.com/mailcheck/mailcheck)
As an aside, I received an email from a government department recently that had an '&' character as part of the email address. I didn't think that was valid, but lo and behold, when I checked the specs, it IS indeed a valid character in an email address.
Just goes to show that assumptions are often wrong, and you have to crack open the spec document from time to time... [1]
I have a feeling somewhere on the Internet is the story of the sysadmin who got the request from that department for an email address with an ampersand. S/he probably stopped themselves from replying "uh, that doesn't work", tried it, and then learned it's valid.
I have a .link domain for my personal email and a lot of sites refuse to let me register because they don't recognize it as a valid TLD.
Then there's the textbook company that lets me register but refuses to let me reset my password claiming that I'm trying to enter an "invalid email address."
I've been using various emails on a personal domain for different sites. Some time this year Six Flags put in an email check and I can't register my season pass anymore. They give the following error:
We were unable to finish validating your connection with the "*.net" email server.
Yet I've been signed up for the pass holder announcements and get the email on a regular basis.
I had a .to domain for a while, and I had an email address "firstname@firstname.to" where firstname is my first name. I wanted to use it when I needed to give out emails causally, like if someone from Church or some other group wanted my email.
I gave up on it, not because of computer validation but because of stupid people! Nobody would "get" the .to domain and they'd always think there should be a .com or .gmail.com on the end of it. Especially older people.
As a counterpoint, I was surprised to see we've had very little trouble with our *.solutions domain, even from clients who are otherwise not very tech-savvy.
I sometimes have to repeat my @.ws email address more than once when reading it to someone, but I've never had a problem with anyone messing it up or thinking it's invalid.
Fully knowing where this was going to conclude, I still found myself reading through the "lets build a stats model" part. I just had to check my calendar to confirm it's not April 1st. :)
That is, at least one character for the inbox, at least one character for the domain, at least two for the TLD (we assume that TLD-less domains are undeliverable by us). This ensures we don't allow 'a@a' or 'a@a.a', but do allow 'a@a.io'.
Also, you don't know what new TLDs ICANN might come up with at some point. I am pretty sure there are service providers that do and did limit the TLD length and then TLDs like xn--vermgensberatung-pwb came up.
How about the common mistake of entering xyz@abc,com instead of .com. A lot of times I unintentionally make this mistake. If the system doesn't prompt me in this case then I would never know why I didn't receive any further communication from it. That's FAIL in my opinion.
they are up to no good and I don't want them as my customer.
Also, according to the standard email addresses supposed to be case sensitive,
since the username part refers to a unix user and unix is case sensitive.
I work with a lot of email address lists originally collected on paper and of course noone knows that. So as bad as it sounds, part of my sanitation process is to lowercase everything. Noone ever complained. What the standard says and what people actually do is very different.
Is it specifically the [, *, \, " symbols on the left that you're calling out, the presence of numbers on the right side, the presence of multiple periods on the right side, the lack of letters on the right side, or just the lack of non-numerics on the right side? I don't think most of those rules are needed technically; I wouldn't be surprised if at least one of those rules matches someone reading this comment thread. Any one of those individually seems like a strange minor reason to pre-emptively ban a customer if it doesn't necesarrily cause technical problems.
If it's only people hitting multiple of those conditions, then maybe you could use a scoring function that only rejects users that meet multiple conditions? If the reason for the validation is a character judgment instead of a technical limit, then you could add more conditions too. People with email addresses that contain accented characters probably don't even speak the same language as you, so you could save yourself some headaches by marking those invalid. If you don't like leetspeak, then I think banning the letter "x" could be a good move too on average. "+" is used by a lot of people who want to trace whoever you might be selling your email lists to, so you could ban them before they blow the whistle on anything. Email addresses containing references to political candidates probably aren't owned by savory characters, so you could mark those as invalid too... Now that I think about it, since it's not for technical reasons, then none of this is really specific to emails. You could run passwords and other user content through this validation too.
If the restrictions aren't for technical reasons, then they're just arbitrary tools for blocking users.
Sorry for ranting a bit. I'm just tired of how often I get mystery validation errors from services because something stupid like that it turned out they thought that english works that happen to be sql keywords like "insert" shouldn't be allowed in the subject field, or my friend with an apostrophe in his name repeatedly being turned away from software telling him his name is invalid, and the idea that these rules could be in place just to cause people like us this annoyance ticked me off.
Well, that one in particular, perhaps — but what about "sam & jill"@ourfamily.invalid? What about john(for sly010)@nowhere.invalid?
> Also, according to the standard email addresses supposed to be case sensitive, since the username part refers to a unix user and unix is case sensitive.
The local-part doesn't refer to a 'Unix user'; it is what it says on the tin: the local part. It's up to the receiving server to determine what it wants to do with case. It's the duty of other servers to preserve the email address as transmitted.
> I work with a lot of email address lists originally collected on paper and of course noone knows that. So as bad as it sounds, part of my sanitation process is to lowercase everything.
I might use the email address as given, and if that bounces try lowercasing. I don't know of anywhere that has case-sensitive names, but it's conceivable.
I am all for doing things by standards, but to me this is de facto.
No one uses case sensitive emails. If they did all sorts of weird things would happen: People couldn't log in, but other people could. There would be multiple accounts when there should be only one, etc. You would have to store every email in original format and in some canonical format for uniqueness, etc. It's a nightmare.
If I had to guess the main reason this apocalypse is not happening already is because by default mysql indexes are case insensitive, so dude@dude.com is the same as dude@DUDE.com,
so people who use mysql never realize this being an issue.
Yes, please do send activation emails (or perhaps a personal confirmation email if you are establishing contact with someone that wrote down an address for you).
Those of us with firstnamelastname@commonhost will appreciate not getting bills and job offers and such.
I've been getting monthly status reports from some guy's Hyundai for months... The unsubscribe link does nothing... I'm tempted to reset his password and change the email address.
Password resets are the only solution. I did that years ago when someone signed up for a Facebook account with my address and I kept getting friend notifications. I let it go for a few years and Facebook was happy to keep an unvalidated account active the whole time.
Send an email that they need to click on (or an email with a code they need to enter), OR ask the OAuth provider with authority for it to validate it (i.e. Google Oauth for Google addresses, Windows Live for Microsoft Accounts, etc).
The best way to identify someone with an email address:
Store a canonical version of their email address alongside the users email. Use the canonical version when signing-in/identifying and the raw version originally supplied to send email.
This is the only way to not have duplicate accounts for firstlast@googlemail.com vs first.last@googlemail.com vs first.last@gmail.com .
The canonical email is always lowercase, no dots, no + part, no prefix or suffix columns, known domain aliases are normalised to the most common alias (googlemail.com > gmail.com).
> This is the only way to not have duplicate accounts for firstlast@googlemail.com vs first.last@googlemail.com vs first.last@gmail.com .
Periods being optional in usernames is something that Gmail started, not a universal rule or part of the specification. Are you going to build into your system all the possible variations for every email provider?
I first encountered the idea of email validation in Agile Web Development with Ruby on Rails, which I read in middle school. In the book they give an example of validating email addresses to show how you could use regex to validate. I wonder if that contributed to the frustrating problem of developers trying to validate emails and not accepting valid inputs.
One more thing that irks me is that some websites capitalize or lowercase the part before the @. Email servers are allowed to treat that part case-sensitively, although most don't. (The part after the @ is indeed case-insensitive.)
In all my projects I use the same methods for validating an email address: 1) Does it contain a `@` 2) Split the string on the `@` and make sure that at least one character exists on both sides of the `@` 3) Send verification email.
This check is done server side, while on the client I just use an html5 input[type='email'] with a required attribute.
That's probably reasonable, but you might want to amend 2 to "split the string on the last '@'". There's no restriction to only '@' before the domain. It's exceptionally rare that a server even allows '@' in the mailbox name, fortunately for us.
In Dada Mail [0], there's quite a few steps to figuring out if the email address submitted for a mailing list subscription is valid, but most of it can really be organized under sanitizing the data you receive, which you should be doing anyways. Yes, we do validation for form of an address client side (which helps with hitting the server side so much), but we'll do it again server side. We also look at stats on how many times an address was submitted before, as well as per ip address over time across all fuctions of the app (as well as specifically for subscribing). Oh, and even if it's valid, and "real", sometimes we don't want to work with it either, ie: it shows up on something like StopFormSpam. It's actually a ton of work and much to orchestrate.
I always search out that a validation library hasn't already been done in my current language first before attempting validation with custom code - this leads to more easily maintainable code for future eyes.
For example in PHP I use filter_var function with the FILTER_VALIDATE_EMAIL [1] - while it's great to know why and how to do a particular thing programming, it's better to use a time tested library that is maintained by multiple eyes versus just your own.
Of course the only way to be 100% sure would be to actually test it out by sending an actual email isn't it. This is just so obvious. I think the point of email regexes out there is not to make up for the user's silly typing mistake. It's just more to reject nonsensical/malicious/blatantly false inputs etc. as the first layer of protection really, and nobody would really spend tons of time on crafting a "perfect" regex anyways I'm pretty sure, so that's never been a problem at all.
All of that statistical analysis was actually a bit silly, because I've never heard the "typo" argument as a reason for email grammar validation[1]. Sounds like a straw man. It didn't need to be disproven.
The conclusion is sound (although leaves out a discussion of the whether an email confirmation field is at least better than nothing).
[1]: (As a side note, I think the most common explanations for grammar validation are programmer perfectionism and proactively stopping user garbage, such as copy-paste errors or intentionally fluffed fields that will result in a bounced email anyway.)
I clicked to his podcast (where he reads Wikipedia pages) at the bottom and listened to this hilarious episode https://itunes.apple.com/us/podcast/david-reads-wikipedia/id.... His specific style of sarcastic humor (just like the article - is there a name for this style of humor btw?) is rare and hilarious.
1. International domain names... you cannot filter characters much because of this. If you want to filter characters, be damn sure it's careful and precise filtering.
2. For immediate feedback, you can reasonably check it's not at at a non-routable IP range, ie. address@127.x.x.x or address@192.168.x.x or similar. However, this check is best done on your email server (MTA policy). It's unlikely anyone would enter this without malicious intent so there's no need to optimize for their use case.
"user@127.0.0.1" isn't even valid. An IP address as a host requires square brackets a la user@[127.0.0.1] - but this is only by RFC, and no real site allows signing up with an IP address instead of a hostname.
Email validation is simple. You take the full RFC regex, and remove the notations permitting square brackets and comments. You wind up with a regex that accepts exactly 100% of real-world email addresses. If your regex contains "\.(com|org|...)", you're doing it wrong.
If you can't figure out how to do the full regex properly, then match against /^[^@]+@[^.]+\./ - or hell, just check for an '@' symbol - as a basic "hrm that kinda looks like an email address" and send the email. It's really not difficult. The only problem is naive English-speaking people who only deal with latin1 thinking that [a-z0-9]+ is somehow the only valid criteria for a username or hostname.
I can think of one reason to do RFC2822 validation: security. It reduces the chance that someone can give a bogus email address that makes some SMTP server on the route misbehave.
Correct way is run a regex for [wildcard @ wildcard . Wildcard] then send an opt-in email real-time as the user is typing additional info. If it bounces before the user finished the onboard form - alert them to the issue. If it gets validate, autologin the user. If bounces or there's zero response by the time the user completes the form, alert them, ask to type their email again without access to the reprior entry - then give them the optin option via text message.
Polling email on the server is awfully wasteful. I like the "request an invite" pattern. Get an email address and nothing more, send the "invite", clear the form, tell them if they don't get an email in 5 minutes, check spam, still nothing, reenter the email. Don't associate the link with any particular session, just the entered address so there is no risk of leaking any private info, if they type in an address that actually belongs to someone else the only options the other person would have is ignore the email or sign up themselves. When they finally get their email right they'll get a link to a place where they can fill in the rest of the info and you can be assured they gave you the email address they wanted you to have.
From the perspective of a user typing their own email address correctly, I use text substitution on OS X and iOS to never type my full email address. ex fg --> foo@gmail.com
I think the 'valid but wrong' email is a more common failure. Having just spent a few weeks with my wife trying to convince some poor person that they had mistyped their email address when they created their Amazon account. Reminded me of this xkcd (https://xkcd.com/1279/).
It does seem effective to have someone type the address twice as that can catch a typo fairly easily.
I hate it when sites make me type my password in twice like I'm some sort of idiot. What's really bad is when they disable pasting. This is so stupid because it makes sure I cannot copy and paste from my password DB. I will often not use a site for this reason alone.
I like the approach of accepting anything that is a reasonable length with an`@`, but suggestinging corrections for possibly misspelled common domains.
There is absolutely is a correct way to lexically validate an e-mail address: namely, implement a parser for the syntax specified in whatever RFC is the up-to-date successor of RFC 822.
There is such a thing as incorrect e-mail address syntax: namely, non-RFC-conforming syntax, whatever that is.
I agree, with two specific exclusions: notations for IP addresses in square brackets, and comments. Just removing those two never-encountered-in-real-life syntaxes reduces the RFC regex down to nothing. Comments are not a real thing, RFC be damned. And square bracketed IP addresses are never encountered in the real world, especially with the requirements for PTR records to pass all the major providers' spam filters.
Anyone trying to provide an email address with a square bracketed IP or a comment is specifically trying to find an excuse to cause drama when their email is rejected. Those exceptions aside, fuck anyone who validates email addresses that don't permit perfectly valid real-world characters like '+', or who whitelist specific TLDs (.com, .org, .net et al).
Email addresses are defined and have an interpretation not according to RFC 5322 (which defines how you can write them in a message), but rather RFC 5321 (which defines how to route email addresses). Note that the RFC 5321 definition does not permit CFWS--its existence is merely an artifact in RFC 5322 of being able to insert whitespace (and later comments) anywhere in the grammar. Any tool which accepts CFWS in an email address that is not reading an addr-spec field of an RFC 822 mail message is incorrect.
Beyond the issue of comments, I'd advocate rejecting the use of IP address literals and quoted localparts. Additionally, despite the injunctions on interpreting local-parts, in practice, it is best to treat email addresses as case-preserving: you'd consider a@example.com and A@example.com to be the same email address, but if the user typed in the latter, don't normalize it to the former.
Comments are and should be a real thing. They are, for example, how to use service-specific email addresses when using a mail provider who does not treat + as a folder extension (jack+suzy@provider.invalid is a perfectly valid email address for Jack & Suzy).
FWIW, I tested recently, and Google do exactly the Right Thing™ with comments in email addresses: they deliver the mail.
There seem to be a problem with the analysis. It seems to assume that everybody uses qwerty keyboards! What about qwertz, azerty, dvorak or even colemak... We need to recalculate with data aggregated and weighted by world-wide populations! :-D
Small anecdote: We recently had a visitor from the US (a physician) who didn't understand why we didn't need a .com at the end of an email address, she was exchanging email addresses with a french person who's address ended in .fr :)
I wish the author put the single sentence at the end to the up with a TL;DR; note: send verification emails, which I don't agree 100% while the article/author claims that's the 100% right way.
I respect the thought behind this article but if you don't want to build it yourself, we use the company .BriteVerify and it works pretty dam well for identifying invalid emails.
His number is probably wrong. I personally wrote/typed my email address wrong (when you have those two fields).
There are also common pitfalls people are doing like empty spaces at the beginning and at the end which should be cleaned up before trying to send the email to " whoever@gmail.com".
There is also an Email standard and when a normal library is able to validate it (and there are free good libs out there) than the effort to do so is similiar minimal but provides an additional support.
What you should do is also to make sure that you are not sending unlimited emails out there. Otherwise you might be missused as a mail relay / spammer.
That one fails " "@example.com and test@example, both of which are legal.
Also, since it relies on looking up MX records and then making an SMTP connection to check if the user exists, why not just send the confirmation email? You've already done all of the expensive stuff at that point.
The approach of attempting to deliver a test email synchronously and seeing whether that fails? That's fine in principle. After all, it's in principle just what the MTA would do once you actually sent the email, and what would cause the MTA to generate a bounce message in the case of failure.
The practical problem is essentially the same as with all the syntax validation out there: It's extremely likely that you are getting it wrong, thus, once again, rejecting valid addresses because your implementation is buggy and behaves differently than an actual MTA.
As for the how so: Because somebody obviously, as is so often the case, just made shit up and implemented that instead of reading the relevant standards and implementing what they say. That's how most of the interoperability problems out there start: People have some passing familiarity with some protocol or data format, and then they implement code handling that protocol or format based on their idea of how things work, instead of what the standards actually say. That's just how we end up with people rejecting email addresses that contain "+" characters in the localpart. I mean, RFCs are public, what more do you need? If you want to implement an actual email address validator, you don't have to pay anything, you don't need to sign up anywhere, you just have to read it, and you can actually build a reliable email address validator, and there is nothing wrong with doing so. Yes, that is more work than just implementing your fantasy protocol. But if you can't be bothered with that, then just forget it and use an existing implementation by people who did care--namely, use an MTA to send a verification email.
Even from http://emailregex.com, a popular regular expression summary for email, states it can only catch 99.99% of validate emails. However, I still like to include some basic form of email validation both on the client and server side, plus the activation email.
I can't tell if you're being sarcastic or not, because that's perfectly proving my point. That validation bans anyone with a number immediately before the @, which includes my email address.
The better question: In it's current form, is an email addresses really the best way to do what it is that's trying to be accomplished? (Hint: It's a fax machine.)
I mean, if I have have a phone number, why can't I have an email number? Okay, perhaps not the greatest example. But then again, if a phone number can be switched from one carrier to another, in the second decade of the 21st century shouldn't "email" get the same consideration?
Instead we're talking about regex or some other wonky validation? In 2016? That's just silly.
You can if you want (it is a valid email). But, if everyone had a numeric email it would be a lot harder to catch spam or know if you are sending to a bot. Sure, spammers can masquerade as a friendly name like yourmom313@domain.com now but it's easier to spot names you know than 471871731@domain.com.
> Why can't emails be carried over to other domains?
I like the idea in abstract. Practically speaking, though, this would not only require millions of emails to be purged from every ISP in the world it would also require some kind of creepy global email registry. That defies the decentralized spirit of the Web and would instantly become the Holy Grail of hacker targets.
No. Absolutely beginner level blog post. You would use a third party tool like http://www.datavalidation.com/ or mailgun's email validation service or BriteVerify etc etc. There are a ton of validation services now. We are living in a time when trillions of email addresses have been tried, entered and deployed to. So why reinvent the wheel on your website? 3rd party services are based not only on parsing the string but also on literally billions of emails actually deployed through various ESP's to tell you up front whether the email address entered by a user is correct. I could use a trashmail email address and it would validate by the OP's standards. First validate using a third party service and then send the double opt-in email to get a user click.
My philosophy is exactly the opposite. Allow anything as email address and make it as simple as possible for the user to sign up to use my application.
This means, no email validation, no address verification emails, heck, I don't even have a password confirmation field. One field for name, one for email and one for password and you are in.
If 10% of the users don't trust me with their real, or even throwaway email address, so be it. My goal is not to collect email addresses, but to have users use my application and add some value to their life.
If another 10% of them enter a mistyped email, they will figure it out eventually and change it if they care.
eh, password confirmation at least has a valid reason to exist - kinda sucks when someone's first hit on your site is having to go through a recovery, and if they didn't provide an email address, they're screwed.
Surprisingly, the validation is often done 100% client-side anyway, and simply modifying the incorrect regex lets my email address through... If I wrecked havoc on your back-end, then it's your fault for sucking ;)