

The Death of Upcoming.org - sp332
http://waxy.org/2013/04/the_death_of_upcomingorg/

======
jmathai
It's a little bit crazy that all the responsibility of preserving this data
resides with the Archive Team[1] and the Internet Archive[2].

[1] <http://archiveteam.org/index.php?title=Main_Page>

[2] <https://archive.org/web/web.php>

~~~
bjxrn
I just want to point out that anyone can help the Archive Team by running an
"Archive Team Warrior". It's a virtual machine which scrapes things that need
to be archived. Currently Posterous, Formspring, various URL shorteners and of
course upcoming.org (in testing).

It's really easy to set up and I would encourage anyone remotely interested to
give it a try. More info & download:
[http://www.archiveteam.org/index.php?title=ArchiveTeam_Warri...](http://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior)

~~~
jmathai
I ran one for Yahoo! Messages the other week and it was super simple. Really
great work on these.

------
nowarninglabel
I've been working on a replacement for Upcoming, called Daypace
<http://www.daypace.com> but I'm still a good ways from getting a product out
there. I really liked Upcoming before it went to hell with Yahoo and around
2010 got motivated to do something about it, but I wanted to also recommend
events to people and avoid the spam that slowly took Upcoming down, and it's
been a tough road getting that all to work and since I work at the most
amazing non-profit in the world, I don't get a lot of time to put into it.

Regardless, thanks to the people who built Upcoming and sorry that Yahoo
ruined it.

~~~
jmathai
Find a way to take advantage of the Upcoming shutdown.

This is a once in a lifetime opportunity for your project :)

At minimum, get email addresses.

~~~
cpeterso
An interesting way to bootstrap Daypace (and make conversion easier for people
coming from Upcoming) could be to import recent events from Upcoming.

------
bambax
> _But Yahoo's security makes scraping a challenge. Every time I've tried to
> back up pages, I can only grab a few files with curl or httrack before Yahoo
> starts serving blank responses. (...) If you have any idea how to scrape
> Upcoming's events_

A way to bypass security is to distribute scraping over many, many clients.

For example, one could write a browser extension and have many people use it
at once; this would have the added benefit of being able to scrape information
generated by JavaScript (just do the scraping after the page is completely
rendered).

If needed, I can help write such an extension for Chrome (I had written one
for del.icio.us some time ago that did exactly that, and another one for
IMDb).

(There would be a public server that would serve the ids of events yet to
scrape, and that would receive the data that has been scraped).

But for this strategy to work, you need a big community of people willing to
help (people who will download the extension and let it run).

Also, 11 days is a very short time...

~~~
sp332
ArchiveTeam already has a system for this. It's called "the warrior" and it's
a little VirtualBox image that checks in with a centralized tracker to get
jobs and upload rescued data. You can see the progress on the "upcoming"
project here <http://tracker.archiveteam.org/upcoming/>

~~~
bambax
Ok, I'm downloading this as I speak.

Still, it's much more complex than a simple webapp (so the audience willing to
do it will be smaller). If there are ~12 million events, we need 10 000 people
downloading 1200 pages...

(On the leaderboard, the number of "to do" items goes up instead of down: what
does it mean?)

~~~
creature
I don't know for sure, but I expect it's a standard thing from any spidering.
Let's say you start at one page and find 10 links; your tracker will then have
10 items. If each of those pages has 10 links, then you can then have up to
100 items to download. And so on.

Over time you're going to start seeing duplicates (ie. things you've already
archived/spidered), so the number will go down. But while you're still in the
'discovery' stage it's going to go up and up.

~~~
bambax
Oh, ok, if you're spidering.

But here's what the OP said: _"All of Upcoming's events and venues use
autoincremented ids, making it dead simple to generate a list of URLs to
scrape."_

So if you go from 1 to {current_id} you get them all, and you have the list at
the start.

------
mccolin
While this was a good bit too "name-droppy" for my taste, it's an interesting
story of decision-regret and a decent chronicle of a founder's view post
acquisition. At the time, the decision to sell seemed right, and it would've
been difficult to predict it would be so poorly managed by Yahoo! Along the
way.

~~~
205guy
Indeed. I have to think that it's hardly like there are (or were at the time)
no precedents. When you sell out^H^H^H your company, you give up ultimate
control.

However, I've thought about this space a bit (the idea-space, not the
solution-space), and I've always come to the conclusion it's too fragmented.
Too many things to too many different players.

But if the field is wide open again, better to invite someone into the
opportunity than dwell on the past.

------
npsimons
All due respect to the work that went into Upcoming, but _nothing_ else covers
the same things? I can't remember why or when I stopped using Upcoming.org,
but I was also using Eventful and Zvents at the same time, and I still use
them, for the exact same thing.

~~~
neilk
Disclosure: 1st non-founding employee at Upcoming.

Neither Eventful nor Zvents plugs into a social network to tell you stuff your
friends are going to, or thinking of going to. They are just sites that list
events.

Facebook's events app is 100% social - it tells you what your friends are
doing, and you can follow artists and venues. There are comments and such on
events. It is also good for non-public, informal events, something Upcoming
didn't do very well. But Facebook is useless for searching for events in your
area, or for following a particular genre of music or type of event.

Upcoming was sort of in the middle of all of this. Which was part of the
problem. It was hard to know what to focus on. I'm not sure the story is
entirely that Yahoo was solely to blame for fucking it up. We also made some
incorrect decisions. But unquestionably, Yahoo didn't give us the resources to
succeed, and prioritized things that had nothing to do with success.

~~~
npsimons
I'm sorry to hear about your loss. It always sucks to have something you
worked so hard on (your baby) canned or messed around with for non-technical
reasons.

Your elucidation of Upcoming's differences helps explain why I probably quit
using it, though. While I can see the value in having your social network
plugged into a facet of (if not your entire) calendar, I personally choose not
to participate in third party social networks. If Upcoming required a Facebook
(or just about any other social network) login, I would have stopped using it.
And yes, I realize I'm in the minority :)

~~~
neilk
Thanks but I don't feel any "loss". It's just a website. Even if it had been a
smashing success, by now it would either have evolved into something else or
died a natural death.

It was a loss of time for me, maybe, but I was well compensated. The real
losers were all the community members who built it into something.

------
mixmastamyk
I liked Upcoming a lot, and when I needed to learn Django I took a shot at
making a similar clone, (though focused more on today and this weekend,
something most sites don't feature).

It's a bit homely and needs some love but I'd like to upgrade and open-source
it. Anyone interested? It's called kpasa and is at <http://lax.kpasa.co/> .

------
asperous
I made <http://1fiesta.com>

It only works in Oregon, USA, but let me know if it it's interesting to you
guys and I'll add more places.

Also check out the list I made at
<http://dontstartaneventaggregator.1fiesta.com/>

------
funtober
This post-mortem from the Plancast founder which I just came across is also
worth reading for those interested in events:
<http://techcrunch.com/2012/01/22/post-mortem-for-plancast/>

------
chriscampbell
Curious who else on HN does large scale scraping? (ie can scrape Upcoming.com
entire site within a few days)

~~~
jmathai
Archive Team.

------
vsync
> they'd made a series of thoughtful hires, including PHP creator Rasmus
> Lerdorf

Uh.

~~~
jonursenbach
The creator of a programming language that's used on a large percentage of the
web. How dare they?

