During a rotational program out of college I was the product manager at Facebook working on maps as the team led up to the first ML contributions to OpenStreetMap (https://2016.stateofthemap.us/how-can-ai-help-us-make-maps/). There's an interesting story about a large mistake I made that is interesting to share:
I started working on this project because I believe that map data should be open as did everyone else on the team. There was (and still is to some extent) a lot of skepticism about whether large corporations have the best interest of the map at heart - and what tools need to be put in place to make sure the quality stays high. It's pretty clear to me that there can be an alignment in goals because whether it's a company, person, or nonprofit everyone prefers accurate map data. The questions are about margins of error & OSM rightfully leans toward only extremely high confidence.
One good thing about Facebook joining in on updating the open map is resources are not much of an issue. Our team was able to just call up and purchase the rights to the satellite images we needed instantly. For quite some time we had working models that generated edits for roads that were missing from the map. Leading up to the yearly conference there was a question of whether we should present all of our findings to the community and see what they thought before making any edits.
No one on the team was willing to push the button on submitting any of our edits because we weren't sure if they were good enough and it would feel shitty to be the one who broke the relationship between Facebook & OSM. Beyond the software engineers even the people who were paid hourly to train the models became invested in believing it was important that Facebook reach a way to channel its massive resources into the project.
Eventually we all gathered up and I just pushed the button to contribute a random 1km / 1km square in Egypt (we'd already computed edits across the country accounting for over 100% increase but things could always get better). Then we waited. No one ever reached out so we started contributing a little more at a steady rate to see if anyone else working in the country noticed (including improving the original 1km / 1km box as the models improved).
We were testing the ML in some of the harshest conditions we could find (low contrast in the desert) so the research team started working on another model for India (winding dirt roads, lots of rivers, tree cover). All of our edits went through street-by-street human approval which was working but slow going since the number of people on the project could fit in a Facebook friend recommendation unit. After not receiving much contact in Egypt (the map community there is small) we received initial positive feedback from local mappers in India and decided to ramp up the speed of contributing which meant getting a war room in Menlo Park and rounding up a larger group of people to review the machine suggested roads.
I ended up gathering a few too many people and myself along with the core team was too hasty in communicating the quality bar for submission. We planned to shadow edits as the week went on to make sure the new members were up to speed but things unfolded much more quickly. Within a few hours we accidentally submitted a few bad roads. A local mapper noticed as he'd just driven the area on his motorcycle the day before. I immediately left a meeting where we were negotiating buying more satellite imagery and jumped on a bike back to the war room. No harm done, we had scripts ready to undo edits.
When I got into the war room things were more problematic. Quite a few of the new folks had made similar mistakes so we paused everything. There was now a small group of local mappers in an IRC channel worried about large scale vandalism (though they quickly realized that wasn't the case). They noticed the breadth of the edits and tracked down the accounts of most everyone in the room. The map community in India is one of the better communities but still a room of this many people making edits at a such a scale was unlike anything you'd normally expect since the ML made editing 10-100x faster than hand tracing imagery.
During the time when the local mappers believed this might be intentional vandalism they escalated to the global community. So that's how a large chunk of the OSM leadership found out Facebook was doing work on the map - some of our worst edits. Shit! It ended up not being so bad however. The entirety of the bad edits were reverted in a few minutes with oversight from the head of the OSM data working group. IMO the situation accidentally showed some of the OSM corporate contribution skeptics that even the worst case wasn't so bad & Facebook was definitely wanting the community in control of any crises.
It wasn't all thought through but it's quite interesting in retrospect how the engagement strategy panned out.
Hi from the OSM data working group. Please don't do massive imports like this without discussing them first with the local community, and the global one if there's no significant local one.
The impact from the massive Facebook edit was more important, I think, and people are still very skeptical about whether massive ML mapping like this can provide value to OSM today. Announcing and discussing the edits first would have been better PR. Recovering goodwill will take a lot of time and effort.
Our guidelines prohibit undiscussed edits like these, and not only the ones that cause trouble. It is not respectful to the unpaid volunteers that cleaned up after this that LukeWalsh seems to think it was "not […] so bad".
I started working on this project because I believe that map data should be open as did everyone else on the team. There was (and still is to some extent) a lot of skepticism about whether large corporations have the best interest of the map at heart - and what tools need to be put in place to make sure the quality stays high. It's pretty clear to me that there can be an alignment in goals because whether it's a company, person, or nonprofit everyone prefers accurate map data. The questions are about margins of error & OSM rightfully leans toward only extremely high confidence.
One good thing about Facebook joining in on updating the open map is resources are not much of an issue. Our team was able to just call up and purchase the rights to the satellite images we needed instantly. For quite some time we had working models that generated edits for roads that were missing from the map. Leading up to the yearly conference there was a question of whether we should present all of our findings to the community and see what they thought before making any edits.
No one on the team was willing to push the button on submitting any of our edits because we weren't sure if they were good enough and it would feel shitty to be the one who broke the relationship between Facebook & OSM. Beyond the software engineers even the people who were paid hourly to train the models became invested in believing it was important that Facebook reach a way to channel its massive resources into the project.
Eventually we all gathered up and I just pushed the button to contribute a random 1km / 1km square in Egypt (we'd already computed edits across the country accounting for over 100% increase but things could always get better). Then we waited. No one ever reached out so we started contributing a little more at a steady rate to see if anyone else working in the country noticed (including improving the original 1km / 1km box as the models improved).
We were testing the ML in some of the harshest conditions we could find (low contrast in the desert) so the research team started working on another model for India (winding dirt roads, lots of rivers, tree cover). All of our edits went through street-by-street human approval which was working but slow going since the number of people on the project could fit in a Facebook friend recommendation unit. After not receiving much contact in Egypt (the map community there is small) we received initial positive feedback from local mappers in India and decided to ramp up the speed of contributing which meant getting a war room in Menlo Park and rounding up a larger group of people to review the machine suggested roads.
I ended up gathering a few too many people and myself along with the core team was too hasty in communicating the quality bar for submission. We planned to shadow edits as the week went on to make sure the new members were up to speed but things unfolded much more quickly. Within a few hours we accidentally submitted a few bad roads. A local mapper noticed as he'd just driven the area on his motorcycle the day before. I immediately left a meeting where we were negotiating buying more satellite imagery and jumped on a bike back to the war room. No harm done, we had scripts ready to undo edits.
When I got into the war room things were more problematic. Quite a few of the new folks had made similar mistakes so we paused everything. There was now a small group of local mappers in an IRC channel worried about large scale vandalism (though they quickly realized that wasn't the case). They noticed the breadth of the edits and tracked down the accounts of most everyone in the room. The map community in India is one of the better communities but still a room of this many people making edits at a such a scale was unlike anything you'd normally expect since the ML made editing 10-100x faster than hand tracing imagery.
During the time when the local mappers believed this might be intentional vandalism they escalated to the global community. So that's how a large chunk of the OSM leadership found out Facebook was doing work on the map - some of our worst edits. Shit! It ended up not being so bad however. The entirety of the bad edits were reverted in a few minutes with oversight from the head of the OSM data working group. IMO the situation accidentally showed some of the OSM corporate contribution skeptics that even the worst case wasn't so bad & Facebook was definitely wanting the community in control of any crises.
It wasn't all thought through but it's quite interesting in retrospect how the engagement strategy panned out.