
‘They Don’t Make These Anymore’: Maintaining the MBTA’s 100-Year-Old Signals - ilamont
http://www.wbur.org/2016/03/07/mbta-signal-maintenance
======
floatrock
The Atlantic did a great longform piece last year all about NYC's (also past-
useful-life) signaling infrastructure by asking a simple question: why is it
so hard to build a train arrival countdown clock?

They go into a lot of detail on how the past-useful-live infrastructure works,
modern replacement technologies, and even a race between infrastructure-giant
Siemans and a bunch of hackers kludging together a countdown clock:

[http://www.theatlantic.com/technology/archive/2015/11/why-
do...](http://www.theatlantic.com/technology/archive/2015/11/why-dont-we-know-
where-all-the-trains-are/415152/)

A more interesting read than this blog post.

\----------

On a different note, these "they don't make these anymore" problems make me
wonder if maybe we should be pushing government to look for open-source
hardware for these types of infrastructure work.

When you have equipment lifetimes of (officially) 35 years (realistically up
to 100 years), can you really rely on the same Sieman's division to still be
in business supporting it? In this Boston story, I wonder how much of their
repair workshop is reverse-engineering some hardware whos blueprints long ago
decayed in some rusting factory.

Given the right open-source hardware, could you make "they don't make these
anymore" obsolete because the technology has become accessible enough that you
could build it yourself? (And in the process employing "New Economy" type
high-tech jobs). More importantly, should a public entity push for making it's
infrastructure publicly accessible? (Pragmatically, could you find someone to
build it under those terms?)

~~~
rchowe
For most of the (new) equipment procurement the MBTA does now, they actually
own the specifications for the equipment, which means that if they want to
make more, they put out an RFP for companies to manufacture new equipment
based on their spec. The downside to this is that the T likes to over-
customize their orders, which drives costs up compared to buying more standard
"off-the-shelf" equipment, and if no responses to the RFP come back, they have
to relax some of the requirements and put it out again.

~~~
a2tech
While its true that they own the specification, in practice the large company
(i.e. Siemens in this example) is the only one capable of producing the
product at a non-prohibitive rate.

I suspect what they'll find when they finally rip out all this equipment is
that the new system will be unrepairable without huge long term contracts with
the original vendor.

These old mechanical systems have their own problems, but repairability is not
one of them. Almost any competent machining shop could duplicate one of the
switches in the article.

~~~
floatrock
Is owning the specification really the same thing as owning the hardware
source? Why is it that you need the long-term vendor contracts? If you go
beyond the spec and make all the hardware control boards, software, etc. open,
could you lower that cost?

Boston has one unique feature that makes public-ish ownership of
specifications and the hardware running more plausible than other cities... it
has a large number of technically-sophisticated potential stewards for such a
program who aren't going anywhere: all the colleges and universities.

MIT and Harvard are something like the largest land owners in cambridge and
boston... they're not leaving anytime soon, and ideally they have an interest
in maintaining relationships with their community. They're invested in the
area, and as academic institutions, inclined to support open source
initiatives like this (how many of you use the MIT keyserver for your PGP
keys?) Imagine a "civic service" initiative where the area universities are
stewards of the open source implementation for the city's infrastructure.

Not at all saying you want the T to run on overworked grad student code --
outsource that in a professional RFP -- but if the concern is that technical
stewardship might be out of the realm of a municipal infrastructure agency,
Boston's academic population could make for an interesting partnership.

Hairbrained stream-of-consciousness idea, but hey, that's what internet forums
are for ; )

~~~
Symbiote
As the first example of the money involved in railway signalling, in 2012
"Siemens has entered into an agreement to acquire Invensys Rail, the rail
automation business of Invensys for approximately €2.2 billion"

That's presumably not the kind of operation that can be run in someone's spare
time, and exactly the kind of work that graduates from these universities are
training towards, and would expect to be well-paid to do.

> if the concern is that technical stewardship might be out of the realm of a
> municipal infrastructure agency

...then they really must employ someone to do that work. Except perhaps for
the tunnels, the signalling system is their most important asset.

------
acomjean
live in MA and take the T (what we call the subway/bus service here). Public
transit is a major problem in this city and surrounding areas. It what make
this city OK and expensive vs being great and affordable.

While the transit system here needs upgrading badly, this piece is very
political. The quoting of Charles Chieppo is a give away.

"For many systems, Chieppo said, the focus is on improving stations or opening
new ones rather than fixing the core infrastructure." The T is working on the
first expansion of the subway rail system since the 1980s. Its going into the
neighboring city of Somerville and "surprise" its over budget. They might kill
the expansion with the focus on "fixing the existing" as kind of
justification, instead of doing both at once? They're also reducing late night
service for the extra couple hours they tacked on to friday and saturday,
while proposing a fair hike. Things cost money, but people don't like to pay
more to get less.

(I'm actually usually on one of the MASCO "private" buses that supplement the
T for its failure to provide adequate routing to the Longwood Medical area.)

~~~
roymurdock
Green line expansion (two stops) initially budgeted at $954M in June 2010.

5 years later, costs estimated at $3.09B and project is terminated.

    
    
      A Reliable Project Budget is Defined as being
      within +10% and -5% of what it would Cost
      to Design and Construct the Project
    

Costs went up over 300%. The level of incompetence on display here is
borderline criminal. Have a friend who works at one of the big 4 who is being
called in to audit this failed project. Should be interesting to get an
insider perspective on where taxpayer money was spent.

[http://www.mbta.com/uploadedfiles/About_the_T/Board_Meetings...](http://www.mbta.com/uploadedfiles/About_the_T/Board_Meetings/3-BRGLookback.pdf)

~~~
yummyfajitas
Wow, $3.09B for 4.3 miles of track and 4 stations? In possibly 7 years, if
everything goes according to plan?

Elsewhere in the world, the Delhi Metro Phase 2 cost $2.9B and took 3 years.
77 miles of track, 85 stations, and countdown clocks. The Bangalore Metro
(phase 1) is expected to be $2.5B (including a 25% cost overrun) for 26 miles
and 40 stations. It also includes countdown clocks. Barcelona spent $200M for
a 5.8km rail tunnel.

Where's your money going?

~~~
frandroid
Have you looked at the wages of Indian labourers versus American construction
workers lately?

~~~
yummyfajitas
In fact, most of the cost of train construction is _not labor_ (except in the
US). That's why the Egypt, China and India (low labor costs) aren't
significantly cheaper than Spain or Korea (high labor costs).

[https://pedestrianobservations.wordpress.com/2013/06/03/comp...](https://pedestrianobservations.wordpress.com/2013/06/03/comparative-
subway-construction-costs-revised/)

If only I thought of this - then I might have cited a _Spanish_ construction
project in the very comment you replied to, so that no one could cluelessly
bring up labor costs.

In any case, even if your claim were true, there is an easy solution. Import
Indians to get the job done. The goal is fixing our crumbling infrastructure
rather than union featherbedding, right?

------
codys
> It’s not as simple as just changing the signal. In order to change it out
> and modernize, you need to update the entire system.

Does anyone have any idea why this is? What prevents new signals from being
backwards compatible with the old system?

And if this were true, wouldn't all of the signals need to be the same hundred
year old type? How would line extensions ever be able to be done in that case?
Is the system segmented in some sense? If it is segmented, what prevents
further segmentation to allow incremental updates of the signals?

~~~
kesselvon
Because the old signals systems are analog or electromechanical, all modern
signals systems are computerized. You'd essentially have to build something
custom from scratch to do so, which would be way more expensive than simply
replacing everything. Since both options sound unappetizing, they end up doing
patch repairs like this to keep the system going, but that can only last so
long before it just breaks down.

~~~
Eric_WVGG
^ that

The Atlantic article linked above covers this in more detail, and is an
excellent read all-around.

------
giarc
I work in a 50 year old hospital. There was some work on the nurse call bell
system and the maintenance staff had to scour eBay for old parts to fix the
outdated system. I think procurement had a fit when they saw eBay receipts
show up.

~~~
acomjean
I worked on a radar project with a similar problem. The target system was a
computer called HP superdomes. HP discontinued those (the PA-RISC version
anyway) . I think they bought a couple spares. Ebay was mentioned as a source
for parts, which doesn't always give you a good feeling.

~~~
acveilleux
In a previous job at a University, we had a couple SGI big irons from the
1999-2002 era. A 64-way Origin 3800 and a 16-way Onyx 3800. Along with a
similar vintage SAN and some misc bits.

When I left in 2010, eBay was the source of something like one replacement HDD
per 10-14 days for the SAN (73GB dual-port fibre channel, full-height...) and
the two machines (made of modular "bricks") had been whittled down to a single
franken-SGI of something like 20 CPUs. Some of the older processing workload
still ran on there (i.e.: the student graduated ages ago...)

A few dual 5300-series (8 cores) Xeons had replaced the SGIs in most workloads
and left the old 500Mhz R14000 far in the dust. Lots more RAM too. Let's not
even get started on the advantage as far as HVAC and power went.

------
brk
FWIW, the same thing goes on in Miami. I've been in the electronics shop for
Miami-Dade Transit and there are always a bunch of electronic signs and
signals of different vintages under repair.

------
na85
A lesson in bad UX, or perhaps just bad editing: Nowhere in the article is the
abbreviation MBTA expanded. I shouldn't have to guess at or Google for your
article's main actor.

~~~
untog
WBUR is Boston's NPR radio station. It's the 21st Century equivalent of a
local news piece, while they could work harder to adapt it for a global
audience it isn't really in their remit to do so.

------
Animats
For a century, replacements were not much of a problem. Most of the components
were either from Union Switch and Signal, or General Railway Signal, both in
the US, and they still made them. Then US&S (which used to have www.switch.com
[1]) was sold to Ansaldo and then to Hitachi, and General Railway Signal was
acquired by Alstom. Alstom continues to make some GRS relays.[2] But not the
whole line of them. Ansaldo offers a remanufacturing service for the older
components.

The New York City system uses mostly General Railway Signal components, while
Boston uses US&S. NYC had to rebuild old relays after the big flood a few
years ago, because all available spare stocks, both theirs and the
manufacturers', had been exhausted. But they were able to get new ones from
Alstom over time. Boston seems to be having more of a parts problem.

These relay-based systems are designed to fail safe, in a very literal way.
"Vital" relays (the ones in safety-critical circuits) open by gravity, and the
open direction is always towards a red signal. Broken wires always force
things towards a safer condition. This is not a property of computer-based
systems. There are modern replacements, such as Ansaldo MicroLok II, which are
programmable logic controllers which emulate relay systems.[5] These,
unfortunately, can be Internet-connected.[6]

Here's a simulator for a GRS relay-based signaling system, down to the relay
level.[7]

[1]
[https://web.archive.org/web/20080103132002/http://www.switch...](https://web.archive.org/web/20080103132002/http://www.switch.com/)
[2]
[http://alstomsignalingsolutions.com/OurProducts/WaysideProdu...](http://alstomsignalingsolutions.com/OurProducts/WaysideProducts/Relays/BSeriesRelays)
[3]
[http://www.rrsignalpix.com/grs_relays.html](http://www.rrsignalpix.com/grs_relays.html)
[4] [http://www.ansaldo-sts.com/en/about-us/ansaldo-around-
world/...](http://www.ansaldo-sts.com/en/about-us/ansaldo-around-world/our-
companies/ansaldo-sts-usa) [5] [http://www.ansaldo-
sts.com/sites/ansaldosts.message-asp.com/...](http://www.ansaldo-
sts.com/sites/ansaldosts.message-asp.com/files/manuali-
ansaldo/ServiceManuals/SM-6800A.pdf) [6]
[http://www.westermo.com/web/web_en_idc_com.nsf/AllDocuments/...](http://www.westermo.com/web/web_en_idc_com.nsf/AllDocuments/56D68128C008FAD9C1257B90002FC54D)
[7]
[http://www.nycsubway.org/wiki/NXSYS,_Signalling_and_Interloc...](http://www.nycsubway.org/wiki/NXSYS,_Signalling_and_Interlocking_Simulator)

