
How MTA shut down my app for Penn Station commuters - zavulon
https://medium.com/@alexkharlamov/how-mta-shut-down-my-app-for-penn-station-commuters-39e1cf69395f
======
state_machine
I can see why the MTA wouldn't want you telling people to go to a track before
it is actually assigned -- if the train ends up on a different track, now you
need to get all those people back up, off that platform an on to the new one,
clogging stairs that that people who actually wanted that platform might be
trying to use too. Obviously it would be nice to assign tracks earlier, so
that people can head straight to the right platform, but sending people to
potentially the wrong platform seems even worse.

EDIT: Penn is extremely platform/track constrained -- NJ Transit, LIRR and
Amtrak are all sharing a fixed number of platforms, some of which are too
short.

To maximize platform utilization, they have to wait until the last minute to
finalize track assignments -- if you reserve one too early and the train ends
up late, you're wasting an empty platform. Once you send a horde of people to
a platform, moving them to a different one is a challenge (stairs/bottlenecks,
communication, etc).

~~~
spyspy
Have you ever taken a rush hour train at Penn Station? The alternative (i.e.
current system) is everyone congregates on the concourse staring at the
platform screens, and when a new one pops up hundreds of people flood in the
direction of that platform in a mad dash. It's a mess.

~~~
bogomipz
It is exactly this alternative(the current system) that the Amtrak/MTA/LIRR
prefer for safety reasons. Amtrak which also has the same procedure at Penn
states this here:

[http://www.slate.com/blogs/moneybox/2013/07/17/amtrak_s_unpe...](http://www.slate.com/blogs/moneybox/2013/07/17/amtrak_s_unpersuasive_response_on_boarding_procedures.html)

Although it is indeed inefficient and the MTA is completely inept, the
platforms for LIRR trains are very narrow, they are shared by two trains and
the escalators to get up to the main floor are also very narrow and crowded
already.

So although the "mad dash" is in fact a mess, its a safer mess than
overcrowding the platforms on the tracks.

~~~
eveningcoffee
I think this is also dehumanizing.

~~~
GauntletWizard
Mass transit is dehumanizing. Cities are dehumanizing. A significant amount of
the way that giant infrastructure works is making assumptions and rote;
Turning the human problem of "how do I get here" into a mechanical process of
"There are two ways to do so, X and Y, each of which follow these steps".

This is great for efficiency - that's why it's done. It's terrible for
humanity. It's stagnant, it kills creativity, it leads to these incredible
tragedies of the commons when so much relies upon the commons.

~~~
jogjayr
> It's terrible for humanity. It's stagnant, it kills creativity

Your other points may be valid but I'm not so sure about this one. Artists
tend to cluster in big cities, often with mass transit (and sometimes without,
like LA). I sincerely doubt there's any sort of correlation between mass
transit and creativity.

~~~
GauntletWizard
Mediocre artists cluster. Good artists hermit. Great artists do both - But
don't maintain city apartments.

~~~
CalRobert
Source?

------
bradleyjg
For about a year a decade and a half ago, I commuted into and out of Penn
Station with my father who had been doing so since the late 70s.

Not only did he know which track any of a half dozen trains that he might take
would come in on but he knew where the doors would open for each of the
tracks. And he wasn't the only one either. If you went down before they
announced the tracks you'd see little clusters of people waiting apart from
each other on an otherwise empty track. We'd usually go to the same car in
order to reduce the distance on the other side. Other commuters would do
likewise and so trains would have a contingent of regulars.

There's a fascinating kind of micro-expertise that develops when you do the
same thing over and over again.

~~~
mark212
same thing on the subway, with an extra dimension of starting on the local,
switching to the express, then transferring at another station. Pretty quickly
you get to know where to stand to get on the most efficient end of the train
to be closest to the stairway up to the transfer platform, etc. Makes an
otherwise tedious trip somewhat interesting as a puzzle.

~~~
cat199
And on some systems, one can usually tell if an inbound train is running
faster/slower than normal and where to adjust position accordingly to end up
at the right door/position/etc..

------
lukegb
As an interesting, although unrelated aside: platform data is available in the
UK's National Rail's feeds, but the terms and conditions[1] explicitly
prohibit displaying platform numbers early, as mentioned in their developer
guidelines[2].

I guess the wording doesn't technically ban you from displaying historical
platform information, but that would likely be a bad-faith use of the data
anyway...

[1]
[http://www.nationalrail.co.uk/static/documents/Terms_and_Con...](http://www.nationalrail.co.uk/static/documents/Terms_and_Conditions.pdf)

[2]
[http://www.nationalrail.co.uk/static/documents/Developer_Gui...](http://www.nationalrail.co.uk/static/documents/Developer_Guidelines.pdf)
"Occasionally Time-Bound Data will become available through NRE feeds before
it is ready to be published to the public. [...] One such example of Time-
Bound Data is platform numbers. Early display of platform numbers,
particularly at origin and destination stations, can lead to platform
overcrowding and/or staff not having sufficient time to prepare the train for
oncoming passengers. In some instances, platform numbers will be available in
Darwin before being displayed on screens in stations."

~~~
davidjbain
UK rail data is quite open in comparison to this story, for instance check out
these live track diagrams (of London Waterloo & many other areas) which show
real-time movements of trains. This is how I often find out the platform
numbers prior to travel and I also get information before staff during
disruption.

[http://www.opentraintimes.com/maps/signalling/wat#T_WAT](http://www.opentraintimes.com/maps/signalling/wat#T_WAT)

~~~
takno
There are maps of many more routes at traksy
[https://traksy.uk/live/M+2+CARLILE](https://traksy.uk/live/M+2+CARLILE)

~~~
davidjbain
Great to see another good implementation of rail maps, I do however prefer the
opentraintimes layout as its easier to scroll through while reflecting the
look of operational systems.

------
mjmasn
Would have been interesting to see if you could reliably crowd-source the data
from commuters themselves.

~~~
dmurray
Google almost certainly already could, without even asking the commuters,
since they have real time location for so many Android phones. But there's a
chicken and egg problem: it's hard to get the data before you're able to offer
anything in return.

~~~
mjmasn
Chicken and egg yes, but he had the recent historical data and a bunch of
paid-up users.

I'm just amazed this app was even necessary in a first-world transport system.
For all their faults, the UK rail network is pretty good at providing timely
information (and as some other comments have mentioned have solid APIs for
this stuff). Same with the LTA[1] in Singapore.

[1]
[https://www.mytransport.sg/content/mytransport/home/dataMall...](https://www.mytransport.sg/content/mytransport/home/dataMall.html)

------
jfries
For Penn Station in particular, one can beat the crowd by heading downstairs
and looking at the small CRT under the stairs. You will see the platform
assignment at the same time as the crowd, but you are closer to the platforms
and don't have to use the same stairs as everyone else.

------
harshaw
There is a similar issue with the MBTA commuter rail in Boston. At south
station, the signage in the main waiting area and at the platform only
announce the train 10 minutes before departure.

However, there are a couple of tools to get around this. For Starters, each
revenue service train has a number (like 508). In a push-pull system like the
MBTA the first coach in the consist is nearest to to the platform. This coach
has a number like 1827. Luckily, the MBTA publishes which trainsets will be
assigned to which departures. This makes it easy to know that 1827 is for
train 508. All you need to do is walk out to the platform and see if 1827 is
sitting there.. if so it's your train. You can get this mapping of trainset to
train via a bunch of apps.

This has been further expanded upon by micro-social apps like "MBTA Rail
Tracker" which has a comment section for every train. The whole thread is
basically "which platform" followed by a bunch of responses and then
snarkiness on why the trains are horribly late all the time.

------
DontSueMeBro
I've been collecting Penn Station NJT and LIRR track assignment data for
almost a year now. I've been intending to use ML to predict track assignment,
but haven't gotten there yet. I should post it to Kaggle or somewhere similar.

~~~
jrockway
Do you think the track assignment is something that's learnable? I feel like
there is some schedule where, under normal operations, the train will end up.
But then there are unpredictable events that cause this to change.

~~~
DontSueMeBro
I think it is. Like in the article, you can see that given certain parameters
like train number and time, you predictably end up with a certain track
assignment. If you can also consider track assignments leading up to this
time, you can discount those tracks. For example, the 8:02PM Long Beach train
always leaves on track 18, except when any 7:56 train left on track 18. As a
novice to ML it's currently beyond my ability to model, but my intuition is
that it should be doable.

------
daburninatorrr
Interestingly enough, I have had this same exact idea for an app, also being
one of those commuters who rely on LIRR trains, also being someone who noticed
that my peak train home arrives almost always on the same track, and also
being one of the (few) commuters who go down to the platform before the train
is called.

It is a shame that this had to happen to OP. I personally don't see the app as
an issue, because the amount of commuters that would actually use such a thing
is rather low (seriously, stand in the concourse and look around at people, a
large amount of them are not using their phones).

My original idea was to scrape the data from their webpage, or see if there
was a way to get the data from the Train Time app's Arrival Countdown page,
but according to the post, it has been removed from their website, so there
goes that idea.

The lesson to me is clear though: don't try to make an app that would make
commuter's lives easier. The MTA does not seem to want that, especially during
the track repairs. I find it interesting that it was shut down so close to the
start of the Penn Station track repairs...

------
midnitewarrior
Why didn't he switch to a crowd-sourced data model like GasBuddy?

~~~
sologoub
Was thinking the exact same thing. One possible reason would be wrong info,
but you could use some sort of weighted algorithm that would need more than
one submission to match and/or trust level per submitter based on historical
correctness.

~~~
tedmiston
Exactly. Like what Yelp / Foursquare / Google Places use.

------
dkrich
I wonder if there's any correlation or way to know based on where a train is
arriving from to know where it's headed. If that were the case, you could
potentially either look for that data, or if it isn't available, have a Waze-
like app where commuters can report which train they are on and which track
they are pulling into.

As an aside, I wonder whether the OP considered charging a hefty price (>$20
or so) for the app. This would lessen the number of users and page requests.

Having travelled from Penn many times and having had to deal with the massive
stampede that ensues when they suddenly reveal the track number, I would pay
dearly to have this info available to me. Even if I used it once every year,
not having to fight crowds or look forever to find an available seat, that
would be money well spent. Additionally, many of the peak travelers are
business travelers who probably have more money to spend on an app. I realize
this isn't the most democratic solution, but it could be a way to lessen the
crowds nonetheless.

------
x13
It's a great explanation of the need for this app, and how zavulon fixed a
problem.

But organizations change how they work with customers/the public all the time,
and I couldn't help but say Waaah.

Things change; tell us about the alternatives you tried to move around this
obstacle.

------
rishabhsagar
Ok, I understand that the Railway Authority don't want people to guess /
predict platform numbers because of various reasons (problems with platform
maintenance, safety and last min changes etc.

But what I don't understand is that why can't they change the seat allocation
process such that people can reserve remaining available seats on the trains
for free if they have arrived in station lobby? This can result is reduced
rush to grab seats and will help people to make a more informed decision
around whether to wait for the next train.

~~~
state_machine
it takes longer to load a train with assigned seating -- that's why say,
Amtrak might do it, but commuter rail trains, that at peak run minutes apart
and want to minimize dwell-times at highly contended platforms, optimize for
loading speed.

------
calvinbhai
Crowdsourcing the track data based on where commuters' phones travel,
something like waze for LIRR should help.

Instead of asking users to report the track info, it can be automated to a
great extent with the use of BTLE beacons on each platform (but that again
needs permission from MTA / Penn station authorities unless a long lasting BT
4.0 beacon can be sneaked in somewhere).

Sad that MTA is not cooperating. Can understand why the app had to be shut
down!

------
troisx
Why don't you sue them? It's a government agency, and I believe that the data
should be available under New York's public records laws (I only have limited
experience with NY's laws and IANAL). From personal experience, being nice to
a government agency never got me anywhere, but our lawyer has a 95% win rate
against them.

~~~
phonon
There is no law that NY government information has to be available through a
realtime API.

------
londons_explore
Why would you want to know the train more than 10 minutes before departure?
Surely you'd just get on the immediate next train?

Are tickets only for a specific time of train?

Or are trains less frequent than every 10 minutes?

~~~
dsr_
The next train for your destination may be 30 minutes later, or over an hour
away. LIRR has about 125 stations in 8 zones. It's not called Long Island
because it's small and round.

------
wfisher
www.njtwizard.com

We do this for NJ Transit with about 90% accuracy. We spent at least two
months figuring out how it all works, though. It's non-trivial to do right.

~~~
bayyar
How do you do it?

------
rocky1138
Whoever decided to close his access to the API should be fired.

~~~
danesparza
It sounds like The API changed. But same sentiment -- that was a shit design
decision.

~~~
strken
It sounds like locking down the API was an entirely deliberate decision made
at the same time as removing platform data from the website, which means
Chesterton's fence[0] applies - we should work out why that decision was made
before criticising it.

Discussion upthread is about commuters being pushed onto the tracks, which
sounds like a good reason not to display data that isn't 100% certain.

[0]
[https://en.m.wikipedia.org/wiki/Wikipedia:Chesterton's](https://en.m.wikipedia.org/wiki/Wikipedia:Chesterton's)
fence

~~~
camiller
how about
[https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton.27...](https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton.27s_fence)

------
drinchev
Why the OP did not continue with scraping the website then? I think there will
be no legal issue if he does that.

~~~
karrotwaltz
> They also stopped publishing the track number on the website — so going back
> to scraping was not an option either.

So the only way to gather the data would be manually from the station.

~~~
Sir_Cmpwn
Carefully placed raspberry pi + OCR?

~~~
vinnycoyne
I'd imagine that placing a "homemade" electronic device in a densely-
populated, public space is going to attract a lot of unwanted attention.

~~~
laylomo2
From all those military guards that parade up and down the station

~~~
InitialLastName
You joke, but Penn Station and Port Authority are effectively occupied
territory. There are at least four different law enforcement agencies present
and patrolling, stopping and frisking anybody who "looks out of place",
hassling homeless people, and generally making you feel "safer".

------
falcolas
Scraping the MTA site against the TOS, and intentionally obfuscating requests
to limit the chance of the MTA identifying and shutting it down? Yeah, no
surprises here.

EDIT: Oops, missed that he didn't end up going live with the scraping.

It would still be great for the MTA to publish this data intentionally; but
I'm sure the potential for trains to switch tracks and the associated backlash
when they do is what prevents them from doing it.

Perhaps they could adopt the airline model, where there's a best effort to
reach a particular pre-announced track, with notifications ringing out when
they can't.

~~~
Sir_Cmpwn
The author didn't do that, though. As far as MTA is concerned the app was
always using an API he had been given a legitimate key for.

As an aside, fuck websites that prohibit scraping. If you send me some bytes
I'm gonna do whatever I damn well please with them.

~~~
EGreg
In that case screw your API, if you allow DDOS I may as well do it for the
lulz.

Isn't it the same approach? If you can abuse a site you will, because they
opened themselves up to the public?

~~~
mustacheemperor
That's an illogical extreme. I don't think anyone thinks that, rather, the
question is whether this app was an abuse of MTA's system. They claim wide
distribution of the app is a hazard. Really, it's not like MTA is stopping
anyone with the inclination to do this themselves. If it's as reliable as the
author says, you could do it without any automation if you were patient
enough. Downvoting because that kind of exaggerated strawman reduces the
quality of discussion with no benefit.

~~~
EGreg
Thanks for explaining the downvote. But this is not exaggerated nor a
strawman. It mirrors the extreme "screw your website" reaction and
investigates whether the logic for such an extreme disregard holds.

So a website allows you to "download some bytes". Yes you can do anything you
want with THOSE bytes. Does this mean you should make an app to let people
scrape the website in an automated way and possibly overload it?

If airlines, restaurants give something free or discounted, does that mean you
can make an app to systematically let millions of people take advantage of the
arbitrage?

XKCD even has a comic for exactly this situation:
[https://xkcd.com/1499/](https://xkcd.com/1499/)

I wasn't the one who said "FUCK WEBSITES". I used a nicer word and questioned
the logic. Yet I get downvoted while the parent comment with expletives is
upvoted.

