Hacker News new | past | comments | ask | show | jobs | submit | mkaufmann's comments login

Hyper, which was created by the same group, can now be used for free with the Tableau Hyper API https://help.tableau.com/current/api/hyper_api/en-us/index.h...

I especially like the super fast CSV scanning!


I've been following data Artisans for a while and love their tech! It seemed that being not from silicon valley made it more difficult for them to gain traction.

One part of the acquisition is the integration of Alibaba's in-house modifications called Blink. It will be interesting to see what is behind that. To me it looks like this could be a very health collaboration.


I think you are mixing the number up. I could not find a source that said anything close to the 1.7 million metric tons per day emitted by the leaking well you used for your calculation.

The leak is emitting 1247 metric tons of CO2e per hour that is 29928 metric tons of CO2e per day. So its 0.17% instead of 9% of nation-wide emissions.


I think I've corrected it, but we're still off from each other :) Can you take another look?

I'm not sure what you're using for US-wide methane and CO2e emissions figures.


Damn these numbers! :D

Regarding your correction. The percent value is now too low (.002%). I think you used the factor value of ~0.002 as percentage. So you probably meant 0.2% which would be close to my number.

Using your numbers

30000/18200000 = .00164835164835164835 = ~0.16% = ~0.2%


That was it - thanks!


On the topic of what went wrong I think this LA weekly[3] article is a much better source. The main problem why the well can't be shut down is that the security valve was removed about 40 years ago: "He pointed out that the valve was old at that time and leaking. It also was not easy to find a new part, so the company opted not to replace it.". Certainly a bad decision and it should be checked if regulations need to be changed to avoid similar problems in the future.

EDIT: The old version had used the wrong conversion factors, now corrected

The central number is the exhaust mass of 110,000 pounds per hour. So how much is this really? This is about 50 metric tons of methane per hour. To be able to compare it with other greenhouse emissions we can calculate the CO2 equivalent by multiplying with 0.01133[1] giving the rate of: 1247 metric tons CO2e per hour.

Using the EPA Online tool[2] we can relate this to the toal emissions in Calfornia or the US. The total emission of methan measured in CO2e for California in 2014 was: 9,546,270 metric tons CO2e. Converted to a rate per hour this gives: 1089 metric tons CO2e per hour.

So while the well is leaking it is releasing 114% of the normal methane emissions of California.

Compared to all greenhouse gas emissions the well is causing an increase of 10% in california and 0.3%at the US national level compared to the emissions from large facilities.

[1] http://www3.epa.gov/gasstar/tools/calculations.html

[2] http://ghgdata.epa.gov/ghgp/main.do

[3] http://www.laweekly.com/news/what-went-wrong-at-porter-ranch...


> we can calculate the CO2 equivalent by multiplying with 3.6

3.6x is how much CO2 is produced by burning methane (i.e. converting CH4 + 2 O2 => CO2 + 2 H2O).

Methane released into the atmosphere is 25x more potent as a greenhouse gas compared to CO2. See Global Warming Potential of methane in your [1] link.

So 50 metric tons of methane per hour is equivalent to to 1,250 tons of CO2e.


There's probably a really obvious answer to this really stupid question, but why don't they light the leak on fire until they stop it?


From what I understand, the gas is not flowing out of a pipe at ground level; the leak is deep underground and it is diffusing into rock and coming up over a wide area. It's not clear that combustion could be sustained in that configuration. I have as much oil/gas industry experience as rube goldberg, but I suppose if they put a huge upside-down funnel over the area maybe they could collect it and light the top, but I have no idea whether the radius of that funnel would have to be 10m or 10km.

I feel a good solution would be to implement a carbon tax, and an unburnt methane tax at the greenhouse equivalent, and start charging the company the estimated leak. I imagine their engineers would become more motivated.


This video shows the gas coming right out of the pipe:

http://arstechnica.com/science/2015/12/infrared-video-of-hug...


http://www.laweekly.com/news/what-went-wrong-at-porter-ranch... has a very clear diagram of what's happening. The inner casing of the well sprang a leak. The gas is blowing all the way back down the outer casing of the well, then exiting adjacent to the casing. So, while the leak is concentrated to an area directly adjacent to the well, it's not a matter of simply capping the pipe.


If you argue that the motivation of the engineers is at all an issue, I invite you to present yourself on the south field at dawn. Bring your second.


perhaps more acurately, it would motivate the budget holders... engineers work with the resources their company allocates. a large tax/fine would certainly help incentivise the right kind of response. although, the fine should be large enough to have prompted the correct behaviour in 1979!


That's how the Door to Hell[1] was made.

1: https://en.wikipedia.org/wiki/Door_to_Hell


I corrected the analysis, thanks for the clue, this paints a completely different picture. I corrected the analysis


But the atmospheric residence time of methane is short - a decade or something. Nothing to worry about in the grand scheme of things, once the leak is plugged.


The factor of 25 accounts for this - over a 100-year timespan, methane is 25x worse than the equivalent weight of CO2.

Over a shorter 20-year timespan, it is 86x worse.

https://en.wikipedia.org/wiki/Global-warming_potential#Value...


You're right, the LA Weekly article is superior. Its author has been reporting around LA for a long time.

Porter Ranch is an interesting place for this to happen. It is a cluster of gated communities with $1M+ homes. People generally move there to be left alone, and watch their property values increase -- this gas leak is deflating both of those dreams.


Sounds like a great time to buy one of these homes.


The figure used for converting metric tons CH4 to metric tons CO2e seems quite low, and I'm not sure what exactly it means in the context of the chart you cite. Indeed, if you plug "110,000" into the calculator [1] references you get 1,247 metric tons CO2e as a result.

That calculator uses a figure of 22.7 for conversion from metric tons CH4 to metric tons CO2e. Many other sources use the number 25.


25 is the correct number.

22.7 is used to convert short tons CH4 to metric tons CO2e. (A "short ton" is what we call a "ton" in the US)


Right. I still wasn't looking closely enough.


I corrected my calculation (using the correct 25 factor for metric tons). Thanks for the hint!


Ok, we changed the URL from http://motherboard.vice.com/read/why-we-cant-stop-the-enormo... to the more substantive LA Weekly article. Thanks.


I grew up in Southern California and the tone of the LA Weekly Article reflects the reputation of The Gas Company as we knew it, conservative, safety oriented, safe.

I guess a culture built around nothing going wrong doesn't always respond well when something does.


Another perspective, this accounts for 0.17 % of US total GHG emissions (for all sources excluding land use change and forestry; source: World Resources Institute, for 2010). One might argue that means it's "small" or insignificant, but I'm going to argue that the lesson should be how staggeringly large our emissions are. Change needed, and soon!


Thank you for the coherent and quantitative analysis. The original article is an example of the rampant & incoherent innumeracy that dominates reporting on quantitative issues.


It does not really compare, I guess thats also why its not mentioned.

The whole premise of the paper is that data can be analyzed in situ. That means loading from its original place in the original format without any previous transformations. This is in contrast to the traditional approach of database systems that the data has to be loaded first into a database.

This paper describes a way how unprocessed unindexed data can be efficiently used to answer queries using a database system. The novelty of this approach is mostly that they build a index on the fly that can be reused later and examining the idea of directly using the raw files. For efficient loading of CSV files which is also mentioned in the NoDB paper, I think those details are better described in a later paper from TU München[1] that examines this aspect in more detail.

HDT for RDF or Parquet[2] and ORCFiles[3] in the Big Data space, are (binary) formats where the data is already processed and stored in a more efficient format than plain old text CSV files. Creating these files can already be compared to just loading the data into a database. The only difference that format used for data storage is open and can be used by many systems. So its a completely different setting.

Still its an interesting thought to make databases aware of the indexed information in those file formats besides CSV so that these can also be directly used without loading.

[1] http://www.vldb.org/pvldb/vol6/p1702-muehlbauer.pdf

[2] https://github.com/Parquet/parquet-format

[3] https://cwiki.apache.org/confluence/display/Hive/LanguageMan...


The article lacks a description of which specific form of pollution is compared.

When talking about pollutants is important to keeps the following points in mind:

- Is the pollutant effecting health?

- Is the pollutant effecting climate change?

- Is the amount of pollution locally concentrated or very distributed?

Cargo ships typically have a very high emission of nitrogen Oxides (NOx) and sulphur dioxide. When emitted by cars / factories on the mainland these often strongly contribute to harmful smog especially in megacities or cities with poor ventilation. Also they can be generally bad to the ecosystem also on the water due to causing acid rain etc.

The main pollutants on cargo ships have a very strong short term effect but are often out of the air in a few weeks. Because of that they don't rise to the atmosphere and don't directly contribute to long term climate changes. Thus I think the comparison in the article is very dangerous. When considering pollutants that effect climate change cars are more dominant.

So depending on which effects are discussed reducing pollutions from cars can still be benefitial. As an additional thought, the polutions of the cargo ships are spread out over a very large geographich area while the exhausts of cars are much more concentrated around cities. So when considering ones own quality of living, cars have a much bigger impact.

I think an article that better manages to discuss the subject is this one from the guardian: http://www.theguardian.com/environment/2009/apr/09/shipping-...

Excerpts:

- Shipping is responsible for 18-30% of all the world's nitrogen oxide (NOx) pollution and 9% of the global sulphur oxide (SOx) pollution.

- Shipping is responsible for 3.5% to 4% of all climate change emissions


> Because of that they don't rise to the atmosphere

Funny choice of words. It reminded me of the classic sketch,

Interviewer: So what do you do to protect the environment in cases like this?

Bob Collins - Australian Senator: Well the ship was towed outside the environment.

Interviewer: Into another environment...?

Bob Collins - Australian Senator: No, no it's been towed beyond the environment, it's not in the environment.

Interviewer: No but from one environment to another environment...?

Bob Collins - Australian Senator: No it's been towed beyond the environment, it's not in an environment.


Here a link to the sketch if anyone's interested: https://www.youtube.com/watch?v=WcU4t6zRAKg


Official and better quality version:

https://www.youtube.com/watch?v=3m5qxZm_JqM


from the OP link

"the 15 largest ships in the world emit as much nitrogen oxide and sulphur oxide as the world’s 760 million cars"


With 760 million cars in the world and 7 billion people, that's one car per 921 people..

So if you have a car and your family is smaller than 921 children, you're ahead of the game, globally.


9.21 people, actually.


one car per 9.21 people, not 921


For x264 that is true, HEVC which is also mentioned is much slower. For a 4k source transcoding can take more than a second per frame. For a normal movie this can quickly result in encoding times of more than a day.

Another problem is that you have to encode the movie for each codec profile times the number of different bitrates per profile. The article mentions four profiles (VC1, H.264/AVC Baseline, H.264/AVC Main and HEVC) and bitrates ranging from 100 kbps to 16 Mbps. Assuming now there are 20 different bitrates per code you already get 4*20 => 80 encoded copies per source. But of course this can be solved by parallelism.


Are there any codecs that can output multiple versions of an input at the same time? Seems like a lot of the encoding process (like motion estimation) is the same every time, so why do it once for every output instead of reusing it?


That would be interesting to know. A lot of transcoders can make multiple passes over the source, so being able to reuse the meta data generated for subsequent passes at different output qualities might help speed up the process. I dunno, not my forte, just thinking out loud.


It's not worth it, because every single decision ends up depending on your output targets anyway.

(You can't afford accurate motion estimation at low bitrates because you can't fit the accurate info in your budget anyway. Except for when you can.)


I just did this page. The page is indexed. When I look for the search term "Update this was posted to Google on Friday the 17th of July, 2015. Monday, the 20th" the page is shown.

Trying to find any of the other search strings in the article for the different loading variants does not return any results. So no variant of javascript injected content is working on Bing currently.


I like to use -Wpedantic for my own projects to keep the code clean and it helped my well. Its not included in -Wall and -Wextra.


I would be interested why the go implementation is so much slower than the Java implementation. The algorithms behind both implementations seem to be exactly the same and both are garbage collection languages. There probably has to be a stupid performance mistake somewhere. I tested with the small dataset from the task page.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: