
Outlook 2016’s New POP3 Bug Deletes Your Emails - luu
http://wp.josh.com/2016/02/24/outlook-2016s-new-pop3-bug-deletes-your-emails/
======
dmbaggett
From a client standpoint, POP is not actually trivial.

The main problem with POP is that unless you do something clever, determining
the changes in the mailbox from time _t0_ to time _t1_ is both conceptually
difficult and computationally expensive. This is because generic POP has no
concept of a message UID, so there's no principled way to diff between states.
In a very real sense, this means that POP is somewhat broken for the most
important use case for the client: syncing to server changes.

The UIDL extension adds a "UID" but it's just an MD5 hash of the contents --
meaning that multiple copies of a message appear to be the same message. And
you can ask for just the headers -- which means you can get the message-id
header -- but this is still very expensive to do repeatedly (say, every 5
minutes) on a 100,000 message POP store. And you can't ask for _just_ the
Message-Id header, which would fix the problem.

Even if you have valid UIDs -- which you won't -- you still have to run a diff
algorithm. Typical dynamic programming algorithms are O(N^2), which obviously
sucks big time for a 100,000 message POP store.

For Inky ([http://inky.com](http://inky.com)) we use a clever linear-time diff
algorithm based in part on [Meyers 86]: _An O(ND) Difference Algorithm and Its
Variations._ [Burns & Long 97] _A Linear Time, Constant Space Differencing
Algorithm_ is also a good treatment. But I know Outlook and Thunderbird both
use non-linear-time algorithms to diff, so "leave messages on server" gets
increasingly (non-linearly) expensive as the mailbox size grows on the server.

A few other points on comments made in this thread:

\- POP is still widely used. In the US, for example, Comcast has finally
migrated to IMAP, but Verizon is still POP only.

\- POP is, from the server standpoint, a very simple protocol, and it is
highly amenable to automated testing, as others here have pointed out. For our
own testing we generate both patterned and random mailbox modification
sequences, then have the test client cooperate with the test server to ensure
that the client has (independently) correctly determined what's happened to
the (test) mailbox. This is a perfect example of a situation where investing
significant effort into automated testing pays off -- and where a TDD approach
to development would also work well.

~~~
userbinator
_The UIDL extension adds a "UID" but it's just an MD5 hash of the contents --
meaning that multiple copies of a message appear to be the same message._

That sounds like a server problem; to quote the RFC,

 _The server should never reuse an unique-id in a given maildrop, for as long
as the entity using the unique-id exists._

Why would you hash the contents? The arrival time should be unique, assuming
no two messages could arrive at exactly the same time. That doesn't require
_any_ hashing.

I think POP could've been far better designed, without growing into the
complexity of IMAP, with just a few little changes like this.

~~~
dmbaggett
I know you know this, but RFC != what servers actually do. :)

------
username223
> UPDATE 02/26/2016: ... TL;DR: Disable automatic updates...

That's just good advice for dealing with modern software, for which fixes,
breakage, and feature churn are all mixed in a single awful stream. "Newer"
does not mean "better."

~~~
raverbashing
As someone who recently had Android 6.0 available for their phone and decided
to upgrade, exactly this

Some genius at Google decided to make the phone vibrate and beep every time
there is an open WiFi spot, or you need to sign up to a known one

Really

This kind of crap (not the only one) almost justifies the extra price for iOS

~~~
danieldk
I (Moto X 2014) got a different set of bugs on M. Mostly problems with basic
functionality:

\- When I get called, half of the time I don't see the name of the person when
they are in my address book.

\- Music controls (which were already very basic) usually stopped working a
certain amount of time after the last restart.

\- Music randomly pauses when browsing the web at the same time (as in, I have
to open up the music notification card and press 'Play' again).

\- E-Mail notifications have been flaky for some reason.

\- Active display music controls are worthless again. Google decided to add a
'favorite' button. Motorola just picks the first three buttons from the
notification card. Now you can't go to the next track anymore.

Security updates were also still at the 1 November 2015 level. After being
stuck for half a year on a buggy 5.0 release before getting 5.1, I know this
is going to take months to fix, if ever (Lenovo probably doesn't care about
the 2014 anymore).

I am now back to an iPhone after a Nexus 4, Moto X 2013, and Moto X 2014.

~~~
cft
Now that you need to bring your own phone to most carriers, there is no reason
not to stick with Nexus/pure Android. The updates work there for the most
part.

------
duncan_bayne
Many years ago I worked on the DPOP POP3 server and DList mailing list server,
and did a little work on our company's SMTP and IMAP servers too.

My experience was that most clients sucked.

For example, one (Eudora?) would move IMAP messages by copy and delete (which
may have been idiomatic, as I say it was a long time ago). But it wouldn't
check the success of the copy operation, so a failure to copy would result in
a delete, not a move.

Seems they still suck in 2016, for what seem like trivial reasons.

I mean, surely this sort of protocol interaction is very, very amenable to
automated testing of some sort. We had a bunch of automated regression tests
for our mail servers, written in C, back in the late nineties.

I'm genuinely uncertain whether to blame incompetence or another attempt at
the strategies spelled out in the Halloween Documents:
[http://www.catb.org/esr/halloween/faq.html](http://www.catb.org/esr/halloween/faq.html)

------
daemonhunter
"Inbox Zero Assistance Features"

------
singlow
I wasted a couple hours trying to help one of my clients who had this problem
last week. What a pain it was. He had 5 devices connected to his account, 2 on
pop and 3 on imap. It took 20 minutes just so listen to his explanation of
emails appearing and disappearing and re-appearing based on which account saw
them first. We spent the next hour turning on each device one at a time until
we determined that his outlook pop client was the one deleting them, even
though it was configured to leave them.

------
Aloha
I was actually more surprised someone still used POP3 for direct client access
- not a bad thing really - I just thought the world had migrated to IMAP.

------
makecheck
I think greater compartmentalization of software is long overdue.

For a program like this, there ought to be a Sacred Core that Does Very Little
and has implementations of key protocols that can’t be touched without the
blessing of about 5 senior engineers and the personal seal of the CEO or some
such.

In other words, it should be _unbelievably_ hard to screw with parts of the
program that are crucial, while making changes that shouldn’t have anything to
do with it. (Heck, for all we know, they were adding Windows 10 Tiles™ when
this screw-up occurred.)

~~~
fanf2
Microsoft keep rewriting their mail protocol implementations and fucking them
up in new ways.

------
walkingolof
Man, I waited for a feature like this for at least 15 years, how do I get
outlook in Linux ?!

------
Dolores12
Why cant they just improve what they have instead of re-writting it every X
years and introducing new bugs?

~~~
bcook
Because that method has it's own problems too.

[https://en.m.wikipedia.org/wiki/Software_rot](https://en.m.wikipedia.org/wiki/Software_rot)

~~~
astrange
Also

[https://en.m.wikipedia.org/wiki/The_Innovator%27s_Dilemma](https://en.m.wikipedia.org/wiki/The_Innovator%27s_Dilemma)

------
chris_wot
It's POP3. Seriously, one of the easiest to understand mailbox protocols ever.
HOW could Microsoft stuff up something like this so badly?

Look, I know that mistakes can be made. But in this case, I just can't think
of a single excuse that would be satisfactory.

~~~
frik
Look what Microsoft did with *.odt (OpenOffice/LibreOffice) compatibility. Now
they show bogus security and compatibility warning dialogs if you open or save
such a file.

Look what they do with IMAP support in Outlook 2016. Pre-loading just the
"Subject" (a feature supported since 1990s) got dropped, instead of the whole
email incl attachments is downloaded.

And now we learn POP3 got crippled too. The common Microsoft tactics to lock-
in end-user and enterprise customers to proprietary ever changing protocols
only fully supported by their most recent server software - their Exchange and
Outlook.com/Office365 cloud eco-system.

Microsoft wants you to upgrade to Windows 10 on PC, adopt it on smartphones
(even it has just 1.1% market share), on servers, SQL-Server 2016, Exchange
2016, Office365 (subscription based Office 2016 client, cloud based SharePoint
2016 = OneDrive for Business on Azure), Skype for Business (=rebranded Lync),
etc. Oh, and they ask you to install their telemetry services for Office on
your clients, to get a "full picture". The telemetry and other phone home
stuff cannot be turned off, except in the expensive LTSB license version. And
several IPs and URLs are white lists in the kernel mode network layer, in all
versions.

It's up to you, to help Microsoft to create another monopoly. Or you out smart
them and say no.

~~~
keithpeter
_" Look what Microsoft did with _.odt (OpenOffice/LibreOffice) compatibility.
Now they show bogus security and compatibility warning dialogs if you open or
save such a file."*

To be fair, oOo and LO _always_ warn you when you want to save a file in MS
Office format. Mind you, you can turn the notification off.

Food for thought: one of my employers now deploys OpenOffice as well as MS
Office after years of resisting that because of fears of user confusion. Guess
what? No confusion apparent.

~~~
xvilka
OpenOffice is dead, sadly refusing to accept it, so people still continuing to
install it - check the commits count for the 2015 year. While LibreOffice does
a very good job improving (including Microsoft formats support).

~~~
keithpeter
I tend to agree and I use LibreOffice myself. Employer is a Microsoft shop
(win10/7 clients, SharePoint, outlook/exchange, active directory &c). I
imagine someone just went to the oOo Web site and got the current download.

------
EugeneOZ
Update for those who dreamed about empty inbox.

------
nine_k
The widespread use of POP (well, _any_ significant use of POP) is what makes
me sad the most.

------
xpda
This is typical of software development in recent years. The emphasis on newer
platforms and technology (in the case iMap) results in new user interface
limitations, as well as bugs, for existing users of older technologies (i.e.
desktop, keyboard access, and Pop3).

I've found eM Client to be a good alternative to Outlook and WLM.

------
quattrofan
I loath outlook with the power of a 1000 suns. Give me gmail anyday.

------
slovette
Eh.. What's life without a little salsa.

------
ommunist
Does this mean testing software before release at MS is second to none and
relies on principle 'if users dont complain, we made it right'?

UPD: I was just about to persuade client to move to Outlook 2016 from 2011.
Glad I did not.

