Hacker News new | comments | ask | show | jobs | submit login
Complete LiDAR Scan of England Publicly Available (environmentagency.blog.gov.uk)
303 points by alibarber on Oct 13, 2015 | hide | past | web | favorite | 67 comments



I took a look at this after it was mentioned on Hacker News two weeks ago and ended up building this map of all the DSM 1m data:

https://houseprices.io/lab/lidar/map

I've been quite fascinated to discover how many mysterious lumps and bumps are to be found all over the country, often with no apparent explanation in aerial photo maps, and to my great surprise I find myself cultivating an interest in armchair archaeology. I've stumbled across a few features which on further research have turned out to be sites of note, a couple of which were only discovered in recent years, which is quite exciting. Next mission is to discover something completely unknown. In fact I could do with some help interpreting some features if anyone here has any experience in this area.

Here's a couple of well known sites:

https://houseprices.io/lab/lidar/map?ref=SU1224642189 (Stonehenge)

https://houseprices.io/lab/lidar/map?ref=SU1025569962 (Avebury)

A few of my 'discoveries':

https://houseprices.io/lab/lidar/map?ref=ST5895844810 (Medieval and Iron Age/Roman field systems near Croscombe, Somerset)

https://houseprices.io/lab/lidar/map?ref=NY7217242430 (Potential henge near Alston, Cumbria)

https://houseprices.io/lab/lidar/map?ref=SX1025261066 (Roman Fort near Restormel Castle, Cornwall)

A couple of things I'm not sure about:

https://houseprices.io/lab/lidar/map?ref=ST4580543091

https://houseprices.io/lab/lidar/map?ref=TL2327777126


Nice job on the viewer. It would be cool to be able to toggle between (or have an overlay of) Ordnance Survey (or similar) data on known archeological features, and a google style aerial view.

Couple this with he ability to post and discuss interesting features, it could enable the "crowd-sourcing" of topological archeology.


Thank you, yes I should definitely look into adding both those facilities.

In the meantime, it is possible to compare other maps in a clunkier way by tapping to bring up a link to a variety of other sources via Wikipedia's GeoHack tool.


Unreal. A couple of years ago, I was smoking dope on my couch with my best friend and we came up with exactly this idea. It's so cool you're doing it! Something I've wondered about: How difficult is researching possible explanations? Scanning the data is easy, but (as you mention) those with "no apparent explanation" might be cataloged somewhere. Have you made any effort to track this data down? What are your results? Cool project!


The archaeological explanations I've found so far have typically been the result of googling the name of the nearest village combined with some keywords like "roman", "medieval", "archaeology" etc, and have appeared in disparate sources; council records, parish newsletters, academic publications and the like. It's a new field to me so I'm not sure whether there are any centralised catalogues which would be more useful.


You can make the further out zooms look better (not salt and pepper like) by using a sinc filter with a 5 or more lobes when to generate those layers. It looks like there is just some simple nearest neighbor or bilinear filtering happening, which is the source of the saltiness.


Oh nice tip, thank you very much. Yes, salt and pepper is a good description and they could definitely do with less of it. I have some reading up to do it seems.


Cool! How come my house (and much of central london) is missing?


Sorry to hear about your house, my entire principality is missing, as the Welsh data is now looked after by the devolved equivalent agency, Natural Resources Wales. Happily it appears that's due to be released on November 2nd (http://naturalresources.wales/lidar?lang=en) so hopefully I'll be able to incorporate that shortly after, depending on what it looks like.

The current map is built with all the available DSM (Digital Surface Model) 1m resolution data. It appears the coverage is focused on coast and rivers, which makes sense in light of the Environment Agency's flooding remit. There's more coverage available to download at 2m (covering approximately half of the 1m gaps at a guess) and some small patches available at 50cm and 25cm resolutions.


Looking forward to downloading this.

By the way: Wales is a country, not a principality. /pedant



Shwmae, (third?) Welshman checking in. The legal entity that was a Principality ended in 1542, but the P word has stuck over the centuries as a sort of colloquial reference, strongly enough that even ISO only got it right a few years ago. Some of us get offended by it, but mostly only when the Saxons next door use the term.


Wales was annexed to England and then formally incorporated by the Laws in Wales Act, viz (quoting Wikipedia, apparently quoting the LiW Act 1535):

>"That his said Country or Dominion of Wales shall be, stand and continue for ever from henceforth incorporated, united and annexed to and with this his Realm of England;" //

So, yes, Wales ceased to be recognised as a principality and instead the lands that formed the area at the time called Wales were, according to Henry VIII, to be referred to only as England. Wasn't this also when English was established as the language to be used in legal proceedings?

IIRC the Welsh Language Act 1993 was what changed this situation?

I've yet to find definitive details of when Wales officially became a country. It's always struck me that prior to annexation it was only a loose confederation of principalities and that not all of what we now call "Wales" was not a part of the possessions of the English crown. It seems Wales, perhaps first defined by the invading Romans, was then created as an entity only by the English crown's definition of the area in which "Welsh" courts could act.

Do you have any input on this?


Well, I suppose we need to define country, nation, and state first.

England annexed Wales, so there must have been a country to annex. Your quote lists Wales as a country. The nation, i.e. the peoples with a common language and culture and within some borders less than the Isle as a whole could I suppose be classified as forming a country post-Roman departure, until such a time as relatively stable borders could be recognised as the geographic borders of the country that nation inhabited, so 5th century?

However, your two replies speak more to statehood/governance than nation, to me you're asking the question "when did Wales become a sovereign state" which is a very different question - and as I'm guessing you well know the UK is comprised of four countries, the UK being the sovereign state - none of the four countries are sovereign states unto themselves.

> prior to annexation it was only a loose confederation of principalities

Which I would classify as a country. The borders were stable-ish, as much as any were for their time, and the people shared a common tongue and culture. Subdividing a country doesn't negate country-hood, does it? The United States has powerful member states, and has accreted states along the way, with borders changing up until what, 1950 something?

> I've yet to find definitive details of when Wales officially became a country.

Supposing we could find an agreeable definition of "official", I'd be surprised if more than a few relatively recently formed countries can be held to the same standard. When did England become a country? When did China? "Official" recognition creates the need for another entity to be the one recognising. If we go by inclusion in ISO 3166 then I suppose Wales became a country in 2011. if we go by nationality, then one could argue it was a country pre-Roman invasion, and it has progressively shrunk its borders ever since.

> It seems Wales, perhaps first defined by the invading Romans, was then created as an entity only by the English crown's definition of the area in which "Welsh" courts could act.

Wales, as in, a collection of people sharing a language and customs, was clearly in place for the Romans to come along and see it. Unless we are getting into some metaphysical quantum country state where a country can only be a country once observed, it seems odd to say that the Romans could call it Wales, but the Welsh couldn't do so prior to the Romans turning up and having at it. As for creation by the English, again, the English annexed the country of Wales. The country was there for them to annex. Wales was not in some cat-in-the-box superposition waiting to be seen by hungry Anglo-Saxons. It existed, thus it was annexed. Regardless of its unified authority or lack thereof, a people inhabiting a space called that place home.

Fomr your other reply: > When did it become a country? Or if you prefer more specificity: When did the region of land we today call "Wales" acquire any sort of significant unity of governance, as a totality, separate from England?

So, the question is in fact: Wales was once a country, got gobbled up and is slowly regaining some independence. When did it become a country _again_? Is that roughly what you're saying? If so, most people would likely point to 1998's creation of a National Welsh Assembly for that one.

The reason I think it is important to define the terms, is that while I - as a Welshman - wouldn't consider iron age Britain to be "Wales", I still recognise its ethnic importance on modern day Cymry. I'm no scholar, but given today's borders it's hard for me to not consider post-Roman Wales as the birth of the contemporary geographic and to some degree geopolitical country. So I suppose my answer is it became a country whenever people generally started calling it such, and it became a recognised state whenever some recognising entity recognised it as such.


>England annexed Wales, so there must have been a country to annex. //

I don't think that's true, the premise is false.

There wasn't a country to annex, there was a loose amalgamation of forces fighting under Gwynedd (from what I can tell), because Gwynedd had recently subdued those areas by force. Those areas hadn't been annexed in to a collective country with a single ruler, just like Gwynedd when it paid fealty to the English crown wasn't yet annexed. Alongside that some regions were already in effective possession of the English crown, and some few maybe were neither under Llywellyn's nor John's boot yet [I've not figured that out quite yet in my research].

[...cut for size...]

>So, the question is in fact: Wales was once a country, got gobbled up and is slowly regaining some independence. //

I think this is a nationalist misdirection. To recapitulate - Deheurbath and Gwynedd always seemed to be at odds, never united in to a common cause or stably under a common leader, nor common rule of law after the Romans, until they both came under the English Crown (just as it was in [the rest of] England). Wales was arguably a country - a single kingdom - for a very brief time but the people of the different kingdoms went their separate ways at the earliest opportunity. These separate kingdoms then got gobbled up, just like the separate Anglo-Saxon kingdoms were gobbled up by Wessex. Unless you consider Kent to have been a country in 1900, say, I can't see how you can consider Wales to have been one then either.

I think you're right that since the WAG was established Wales has had some claim to being a "country" but there seems a stronger claim that it's counties, like those of the rest of England are still a part of the country that is the dominion of the English Crown.

There has certainly been an effort in the last 80 years or so, since the Council for Wales was created, to make a new country: I'm certainly happy to go along with the WAG being a strong weight in favour of calling Wales a country.

It's strange to me that in the UK we had a great Welsh ruler of an historic Welsh line who ruled over all over England [aka "England and Wales"] as a unit, yet many in Wales instead want to look back to the time when the Wales was critically divided between two great powers and when it was most common for Cymry to be killing one another; a brief period that ended with Welsh killing their "own" king [Gruffydd ap Llewellyn].

It strikes me that had Henry VIII chosen a new name for the "land of the Angles", perhaps Cymru, then those who imagine they descend from the Celtic Cymry would have no complaint and would take full part in being citizens of that country. I wonder how getting access to genetic markers will, in the near future, change people's ideas of national identity - if people discover they descend from the invading Irish, Scots, Vikings, Romans, Angles, Jutes or some other group rather than the "native" Celts will they consider themselves less Welsh? More British?

Thanks for challenging my arguments and for walking through this with me, happy to get more responses to this if you have them.


When did it become a country?


The two are not mutually exclusive. "Countries" can be all sorts of things, including Principalities, Monarchies, Republics. Luxembourg, the country, is also a Grand Duchy. Sounds pretty spiffy. But if the Grand Duke stops being in charge, Luxembourg remains a country, but no longer a Grand Duchy. Country is geography, governance is political. Wales, the country, ceased to be governed by princes in 1542.


But Wales was just the region to the West that Rome conquered later because the 4 [or more] tribes there were protected by adverse geography [and because the region wasn't terribly useful]. Then some parts of what we call Wales were parts that the Vikings conquered, other parts stayed outside the English crown but continued to fight against one another, governed by various rulers (the principalities of Deheubarth, Gwynedd and such). The regions were subdued largely [but importantly not exclusively] by Llywelyn the Great, who then paid homage and fealty to - IIRC - King John.

There was much rebellion, both against the ruling Welsh lords, eg of Gwynedd, and against the local powerhouse of the English Crown. Edward I conquered Gwynedd by bringing Llywelyn ap Gruffudd ("the Last") under his boot and so imposed the power that Gwynedd previously held over large swathes of [what we now call] Wales.

Then the lands of "Wales" were annexed, this annexation taking various forms but certainly meaning that the lands of Great Britain to the West became a part of England (brought in to certainty by Henry VIII, whose ancestors were subordinates of the aforementioned rulers of Gwynedd).

So yes, a country can be many things, but it seems it needs something more than just geographic locality to be worthy of calling a country: "Wales" which we now know is an amalgamation of counties of the English crown won over a period of a couple hundred years from various rulers. It's a country in the same way that "Wessex, Sussex, Kent and Wight" is a country, an arbitrary grouping of a largely historic county regions based on locality - actually that's got a greater claim to be a country as it's the historic Kingdom of Caedwalla whilst "Wales" seemingly was only united as an administrative regional grouping of counties whose lords had been brought under fealty of the King of England. Long before that it had lost any unity of identity when the Romans, "Irish", Vikings and others came and pushed around the Celtic and Britthonic speaking people's who'd been there prior to the CE.

So I can accept Powys as a country, or Gwent, or Gwynedd, but Wales never really seems to have been a country until ... well I don't know, but I think since 1964?? The Acts of Union were between the countries of England and Scotland and Ireland. The Council of Wales, like the Council of the North, was abolished in the mid-17th Century in favour of direct government from London ... "the country of Wales" seems to be a pretty recent invention.

Looking at the more recent history of the 'electoral regions of England known as Wales': the Eisteddfod is seen as long standing historic event that epitomises the independence of Wales as a nation but it dates from the later half of the 19th Century and the druidic pomp of the Gorsedd seemingly has no historical basis other than being the invention of a London based academic in 1792; hardly the pre-Roman druid roots that people imagine the Eisteddfod as having when they see the gowns, the throne, and the sword (eg http://www.museumwales.ac.uk/articles/2010-07-25/Scrolls-swo..., http://www.bbc.co.uk/guides/ztbk87h).

Hence the question: When did it become a country? Or if you prefer more specificity: When did the region of land we today call "Wales" acquire any sort of significant unity of governance, as a totality, separate from England?

[It's amazing how tribal reach thousands of years ago, eg of the Silures, brings to bear on the nature of nationhood today. As you perhaps can tell this is a subject I find quite fascinating, encompassing as it does so much of British history and modern politics within such a small question.]


The Land Registry and OS maps have more data, and it's a shame they're aren't easily online.

It's interesting to compare across Google Maps.

Grid reference: SO9644600362) near Coates (Gloucestershire) is an old roman fort.

https://www.google.co.uk/maps/@51.7010323,-2.0505876,1017m/d...

EDIT: And the big lumps north of that are from the canal tunnel. These do not show on the Google map.


how did you find those features? was it just serendiptiy?


Just things that caught my eye while panning around, yes. Serendipity might be too strong a word for it, the country seems to be covered in such features.


People have been experimenting with using this to contribute to OpenStreetMap for a couple of weeks now. Here's one writeup: http://chris-osm.blogspot.co.uk/2015/09/extracting-building-...


There is a similar dataset available for The Netherlands. A potree point cloud visualisation can be seen at http://ahn2.pointclouds.nl/.


This is amazing. It took awhile to render at the max-zoom level but once the data loaded (and rendered), the level of detail is just astounding. You can see a construction crane and its counterweights imaged recognizably. Sagging power lines, just unbelievable.


LIDAR scans of Poland are available publicly from some time: http://geoportal.gov.pl/dane/numeryczne-modele-wysokosciowe


for a low low price of ~$1/km2 http://www.codgik.gov.pl/index.php/obsluga-klienta/oplaty-za...

There is some mention of free access, but its behind wall of legal paragraphs and applications


An almost complete LiDAR scan of Spain is also publicly available. I wrote about it here and included a few samples: http://wechoosethemoon.es/2015/09/05/lidar-espana-3D/

Sadly it is in Spanish but I hope available areas and pictures of expected results are clear enough! LiDAR data is provided as 2 Km x 2 Km squares of RGB-colored points in *.laz format. If someone is interested I can translate into English or point to the sources.


Hey Jorge, thanks for sharing! I would be interested in a translation, since I'm based in Barcelona and would love to do some experiments with the data, focused on Barcelona.


I will try to allocate time! Fortunately Barcelona is covered, you just need to head to this page: http://centrodedescargas.cnig.es/CentroDescargas/buscador.do (there's an English button on the upper right corner) and select "LiDAR" in the menu. You can then select by drawing a polygon or uploading a shapefile and click search. You will be taken to a download page with the files that intersect your selection =)

Edit: see for example this data in the Barceloneta area I just pulled: http://i.imgur.com/ayCK7MC.png


This is amazing. I love that they mention Minecraft as one of the use cases:

"LIDAR data – some surprising uses:"

"Computer games: Minecraft players have requested our LIDAR data to help them build virtual worlds: the data could be useful to anyone creating realistic 3D worlds."


If you're interested in LiDAR data from the United states, you should have a look at this wikipedia page and its corresponding links[0]. Most states have some kind of data freely available from the most recent survey, although it's neither uniform or always clear in terms of how to access it. The structure of the program allowed individual states to tackle their won territory differently in both surveying and data dissemination, so there's no easy and official central repo as far as I understand.

I was in grad school at Indiana and working at the geological survey while they were finalizing some of the state's pieces of this, and it was really fascinating to see some of the early products some of the people in the geography and geology departments were producing. I mucked around a bit with it myself, but never really produced anything useful. I can speak to finding the data fairly easy to acquire and quite comprehensive at the time, uncertain if that's changed, but might be a decent starting point[1].

[0] https://en.wikipedia.org/wiki/National_Lidar_Dataset_(United...

[1] http://gis.iu.edu/datasetInfo/statewide/in_2011.php


Can anyone recommend any 3d viewing programs for this data? This is all new to me but I'd love to try experimenting with it. The download zipfiles contain .asc files


I use the freely available FugroViewer, and manipulate the data with LAStools. If they are ASCII files you can use any program like python with matplotlib to make a 3D plot.


I was hoping to find some explanation of how they capture the data. I'm guessing it's from aircraft? It'd be interesting to read about how they stitch together and correct the data captured from a moving platform like that. And I wonder how long it takes to capture the whole country.


'All 11 terabytes of our LIDAR data (that’s roughly equivalent to 2,750,000 MP3 songs)' or a stack of paper 513 kilometers high :)


[pedantic]Using what encoding, font, font size, margins, etc? [/pedantic]


Paper's about 0.1mm thick. Therefore 513km of paper contains approximately 5.13 billion sheets.

Given that there's 11TB of data, that means that each sheet contains about 2360 bytes. (Printing on one side.) That's an 80x30 grid. A4 is 210x297mm, so (holding the paper landscape for convenience and assuming a 2cm margin) that makes for 3.5x6.3mm letters. Different encodings give you slightly different sized letters, but it's all roughly the same.

So I'd guess around 10 to 11 point.

Has this helped?


They aren't effectively using their paper then.


So... what is the most dense, reliably machine readable, format that can be printed on ordinary paper with a consumer inkjet?


Also what gsm is the paper


Also where would this stack of paper be placed? After all gravitational pull is not uniform around the planet.


Also, how embossed is the lettering?


Also, how resistant to compression and moisture is this paper? Because zombies might be trying to climb it to get to me, and they might have wet hands.


This is starting to look like reddit...


Any idea about the cost of doing a LIDAR scan for a region? Lets say you have 1200 square km (assume rectangular area). How much would that cost?


At one point in time[1], I hired a helicopter to act as a surrogate remote sensor doing data fusion with a ground robot. I flew as a passenger and told the pilot where to fly based on the ground robot's need for data.

Since we had to pay for the helicopter's time anyway, and the field trial was spread over two days, we left our equipment attached to the helicopter when it returned to its airfield that night. The next day we had a 78km long data set of LIDAR, GPS, visual imaging, and inertial measures, all from an altitude of about 25m giving us about +/- 2mm for the LIDAR's range data.

The sad end to this anecdote is that I have no idea what happened to that data. It's presumably sitting on a dusty server somewhere in academia.

[1] This point in time, as it happens: http://www.drtomallen.com/uploads/1/2/0/2/12026356/3361016_o...


Format of the data is listed as Arc/Info ASCII Grid (AAIGrid) which is an ASCII Esri grid https://en.wikipedia.org/wiki/Esri_grid.


Hell yes! This will be useful in my long term aim of procedurally generating rolling english hills for video game purposes...

EDIT: with the resolution of this thing, maybe I won't need to generate them, maybe I can just set the game in the real england...


You don't need this data to do that. OSM already releases everything you need including gridded height data and vector building maps.

I've used that data for this exact purpose before. Shoot me a message explaining what you're looking for.


Thanks. You don't have an email in your profile though. The one in the email field is just for HN itself to contact you, you have to put it in the `about` field for other people to see it.

EDIT: my email is in my profile, so hit me up if you dont want to expose yours.


Hey, I did, might be caught by spam given I have a terribly hacked up MX :D


Has to be very long-term then, given the fact it's 11 terabytes. Of course, you probably won't need all of that data / resolution and would have enough of 1/20th of that.


Exactly. Maybe I'd just need to extract some kind of statistical model.


Are you familiar with "Sir, You Are Being Hunted"? The game's developers have a great blog on their procedurally generated English countryside.


I had a look at that. They've done a great job of generating enclosed crop fields from voronoi diagrams, but that's not the kind of countryside I need, rolling hills and valleys. The south downs, basically. The topography is important for gameplay.

BTW for anyone interested they did a talk at some conf or other which goes into more detail than the blogpost.


I'm fascinated with the idea of simulating geological processes, particularly volcanism and glaciation - probably because I live in Edinburgh!


LIDAR is also being used at NASA to measure atmospheric CO2 concentrations.

http://decadal.gsfc.nasa.gov/ascends.html

[PDF]http://esto.nasa.gov/conferences/estf2011/presentations/Absh...


Anyone know how burial mounds/roman roads are found? Would love to see that reproduced for the non-academic world.


Basically you look for shapes. Usually you start from some vague idea, documents describing stuff that should be there, and then you look for shapes that look too peculiar for nature alone. See Portus for an example: natural ponds are not usually hexagonal...


This will be great as a base layer in OSM!


Do they have "first and last" LIDAR data, or just one value per point? It's common to capture the distance to both the first and the last reflection. This often indicates the top of vegetation and the ground level. With that, you can easily identify trees, brush, and crops.


Neat. I was wondering about that.


Awesome! Does anyone know if similar resources for the US are available?

Edit: possibly https://lta.cr.usgs.gov/LIDAR Still looking for more


This tutorial will direct you to download DEM data for the US:

http://www.gpsfiledepot.com/tutorials/how-to-create-garmin-t...

The map viewer you use:

http://viewer.nationalmap.gov/viewer/

Info about the elevation data you get:

http://nationalmap.gov/3DEP/3dep_prodserv.html

The data is kind of buried, but I went through this process for building maps and it appears to be very high quality.


http://opentopography.org/ is a great resource for US LiDAR data.


Could do some neat things with drone piloting with this.


Does anyone know of publicly available LIDAR data for the Czech Republic?




Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: