Hacker News new | past | comments | ask | show | jobs | submit login
Big Oil’s Favorite Toy: Supercomputers (2018) (wsj.com)
39 points by ganeumann 23 days ago | hide | past | web | favorite | 36 comments

Shell Oil had a Cray 1 and an ncube when I worked there in 1989. Chevron had a Cray 1 and a raft of IBM mainframes in 1990. Chevron's DC was set up for visitors so every cluster had a sign describing the computing power for that system. Total compute power was very important because it meant they could do more analysis on seismic data to refine their oil and gas lease bids.

Chevron had so much seismic data they required (from memory) six robot tape libraries. They were ganged together so the picker robots could hand tapes between cabinets in case all drives in a given cabinet were in use. It was cool as hell to watch the camera mounted right above the picker flying around this dark wall of tapes to go grab one.

One of Shell's Quality training videos was about The Guy Who Lost The Tape; seems a seismic data tape was mislabeled and lost, causing Shell issues with bidding on a lease. Their Deming-style quality training was all about preventing that sort of thing happening again. I dare say their data was more valuable in total than the hardware.

If you knew the effort and cost that goes into large scale near and offshore seismic and non-seismic surveys then you wouldn't think twice about putting the value of the data well above the value of the hardware.

Offshore deepwater (~2500ft+) non-seismic surveys might cost 6-7 figures per day to operate, and might fill up a hard drive every 1-3 days.

Depending on how many drives fit on a tape, the raw data could get very expensive, very quickly, even before it's been processed, analyzed, etc.

Absolutely and of course the opportunity cost from missing out on a lucrative oil field could buy a room full of compute.

It's even worse than that.

I used to work with a guy who told me that the reason BP bought Amoco, and not the other way around, is that years before, the Amoco team misread the seismic map (not the chart), and bid on the wrong piece of land. BP got the other piece, and the difference was big enough that within a decade, one bought the other.

THAT is how much the data is worth.

I suspect the value of the seismic data keeps better over time than the value of the hardware, too.

Better than the hardware, certainly. But advances in surveying technology make new surveys worthwhile, occasionally, so the data does have a lifespan. Of course better analysis techniques or more powerful hardware for old data makes a big difference too.

Sure, adding more data is also good. Often new data can make the old data more valuable, too, I'd guess?

Yes, although it's not usually a case of doing the same thing with new tech and getting better results. You'd have to have some hint that oil might be there from the first survey.

But my experience is mostly not-seismic, so I'm not sure how the survey parameters would be adjusted for a revisit.

Do you have any good resources on that Deming-style quality training?

Deming refers to Edward Deming [1], and there is a W. Edward Deming Institute that still teaches his principles. [2] Since that was 30 years ago I doubt they still have the same training material today, but I wouldn't have the rights to share it even if it was the same, sorry.

[1] https://en.wikipedia.org/wiki/W._Edwards_Deming

[2] https://deming.org/

After research labs big oil has always been the 2nd largest client for supercomputers for decades. It's even mentioned in 90s films Hackers[0] as "The Gibson."

[0] https://en.wikipedia.org/wiki/Hackers_(film)

Aren't they Big Oil's old favorite toy? In the 90s, a local university near me had a Cray XMP that was a gift from some oil company after they'd decommissioned it.

My grandmother was an analyst for Gulf/Shell going back into the 60s. She was a resident FORTRAN/COBOL expert that was recalled in the 90s after retirement.

Data and systems analysis at scale is definitely not new to them.

I'm in software dev, my dad was in geophysics, he's always had a much better work computer than me.

Ok, given that and https://news.ycombinator.com/item?id=21531193, and frankly just about all the other comments here, we'll take newness out of the title above. Makes the story more interesting actually.

Exactly. I have been hearing about this stuff for ages. Oil companies optimized raping the land decades ago.

If only there were something we could do to stop the world’s oil dependency

When going to training classes on optimal I/O strategies for new supercomputers there is usually a bunch of people working for oil companies. The simulations I do for plasma physics are considered "easy" there because they need little input, generate only a few 10 terabyte output and are by-and-large compute bound.

I was working as a hired SGI/SUN admin via ABB for Statoil in 1997 in Stavanger. The HQ was packed with Ultra 60s with creator3d or elite3d cards. Some even had both. We had to keep a few hundred of these running and they were used to explore seismic data visually.

Those machines were pretty powerful for those days, almost at supercomputer level. One cost more than a years salary for me back then.

But Oil has always loved computers.

What kind of signal analysis do the oil companies do these days (like wavelet transforms)?

My (in my opinion failed) HonsBSc project was on signal analysis of GC-MS (Gas Chromatography coupled to Mass Spectrometry) signals.

If I had to do it again in 2019, then machine learning would be a much more pertinent focus. However, without the computing power of today and ubiquitous presence of programming libraries for that purpose, there are actually other ways of approaching such data (like wavelets).

Wavelet transforms were invented quite a while ago [1] but I think the seismic data analysts were some of the first to really investigate the applications of that field. The other application is for compression (and loading over an internet connection) [2].

[1] https://en.wikipedia.org/wiki/Haar_wavelet

[2] https://en.wikipedia.org/wiki/Lifting_scheme

Given a sparse data set recorded by methods that have known constraints and limitations, generate a physical model that could conceivably produce measurements that match the given data

Schlumberger was a major force in AI in the 1980s when I worked there. So was Texas Instruments, which started as an oil well instrumentation company. Computing has always been important to Big Oil.

Strangely enough, I had a couple of classes with Xukai.

Former energy VC with the Norwegian state energy co here. Two of the companies we invested into were HPC related—one of them developed node controller solutions for parallel processing, the other developed a platform to analyze massive amounts of subsurface data more quickly.

When the cost to drill a single exploratory well costs as much as some IPO’s, in some cases with <10% expected chance of success, there’s plenty of love for supercomputing in this industry.

Geotechnical users were always the second or third biggest users of supercomputers after national security users. Big oil always had the most astonishing data centers.

New big data jobs in oil exploration?

Yup: https://www.oilandgasjobsearch.com/Oil-and-Gas-Jobs/Search/B...

Nun new. Supercomputing has been used in Oil & Gas exploration for decades.

Indeed, but interestingly enough the book, The Prize: The Epic Quest for Oil, Money & Power, only has 9 mentions of the word computer and zero for supercomputer.

The Prize: The Epic Quest for Oil, Money & Power


From the authors Bio: "Daniel Yergin is the author of the bestseller The Quest: Energy, Security, and the Remaking of the Modern World which has been hailed as “a fascinating saga” about the “quest for sustainable resources of energy,” and “the book you must read to understand the future of our economy and our way of life,” ..."

So it's more a book to persuade (feelings) than inform (facts) - no wonder they aren't mentioned.

Know what you're reading.

I know I read a book that won the Pulitzer.


and I know you the passage you cite from his bio is not in related to the book I cited.

The book is loaded with facts. It just doesn't focus on the computational aspect (more focus on the geopolitical details). I strongly recommend this book- I understood a lot more about modern power after reading The Quest and The Prize.

The Prize is an excellent history, though one with some very curious omissions.

Actually, the news for me is that Oil & Gas exploration still has needs for computing that's considered significant today.

Basically, I can see that they needed super computers in the 80s. But I might have assumed that their computing needs wouldn't keep scaling with supply. So that by now, just renting some GPUs from Amazon might be enough.

A former employer of mine had lots of oil companies as clients for their CFD software...for most of the problem sets, the memory use was in the TBs, and it's tough to get a GPU with more than 16-24GB of onboard RAM.

can i hear an amen? and a huh? who did the research at wsj for this article?

Probably quite a few combinatorial problems not to ignore, https://phys.org/news/2018-09-quantum-mechanics-oil-industry...

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact