Unmapped Places of OpenStreetMap – 2016
by Pascal Neis - Published: June 19th, 2016
Back in 2010 & 2011 I conducted several studies to detect underrepresented regions a.k.a. “unmapped” places in OpenStreetMap (OSM). More than five years later, some people asked if I could rerun the analysis. Based on the latest OSM planet dump file and Taginfo, almost 1 million places have been tagged as villages. Furthermore, around 59 million streets have a residential, unclassified or service highway value. My algorithm to find unmapped places, works as follows:
- Use every place node of the OSM dataset which has a village-tag (place=village).
- Search in a radius of ca. 700 m for a street with one of the following highway-values: residential, unclassified or service.
- If no street can be found, mark the place as “unmapped”!
My results for the entire OSM planet can be found under the following webpage.
Overall we have more than 440,000 unmapped places in OSM. As you can see in the picture above, most of the places are around Central Africa, Saudi Arabia or China. However, I hope that this analysis helps to complete some of the missing areas or to revise some incorrect map data. Some remarks about “false=positives” or why your village is marked as unmapped? Some possible reasons: Is the used tag for your place correct? Compare the wiki page for further information. Sometimes “hamlet” could be the correct tag value. Are the nearby highways tagged correctly? (OSM wiki)
Amount of unmapped places for each continent:
- Africa 119,084
- Asia 241,833
- Australia 212
- Europe 44,819
- North America 16,464
- Oceania 837
- South America 15,576
Technical Stuff: The OSM data for the analysis is prepared by a custom OSM PBF reader. The webpage, which shows the results, is based on Leaflet 1.0.0-rc1 and the really fast PruneCluster plugin.
*Update*: You’ll find the date of the latest data update in the header -> “(Date: Apr. 9th, 2018)”
Thanks to maɪˈæmɪ Dennis.
Guess these will often be places imported with varying degrees of accuracy from third-party sources. Randomly zoomed in on three of your findings in Africa, and they were all from a 6-year-old GNS import (with some comment that since no population was given, they were “defaulting to village” – so they might not even be villages at all). Perhaps these places should really be un-mapped (removed) from OSM; if in six years there hasn’t been anyone willing or able to corroborate the existence of the place then it might just not be a place.
>if in six years there hasn’t been anyone willing or able to corroborate the existence of the place then it might just not be a place.
This sounds like a very narrow-minded, western-centric world view. In many places of the world, still few people have access to (fast) internet and a computer to edit large features, and then some of those fortunate enough to have would have to discover OSM. If you would take effort to map undermapped areas yourself, you’d notice that there are many towns, let alone villages, that are not on OSM (and often not on commercial maps either). I would encourage you to zoom in on a seeminly empty region in China for example, switch to satellite imagery and open your own eyes.
Ahem, somehow you seem to be implying that we should leave known utter rubbish in the database because it would be better for the narrow-minded, western-centric Frederiks of this world to not interfere with the Internet- and computer-less underprivledige not cleaning up the mess created by the former ……
Certainly the findings (Nottinghamshire UK) here aren’t “unmapped places” but are instead “things copied from an out-of-copyright map that aren’t really villages” such as http://www.openstreetmap.org/node/337676119 . It’s still a useful list though 🙂
@Felix if you were to zoom in on an empty region in China what value would you be able to add? Unnamed streets, a rough idea of landuse maybe, but presumably no points of interest. Realistically it’ll need the local community and actual on-the-ground visitors to be able to add those.
Is there any special reason you didn’t run this query with place=city/town? In many countries there are a lot of cities unmapped…
Hi wille, it’s mentioned in one of the older studies: “Why did I only use villages? Well for the case of Germany it can be assumed that places with “higher” place-type tags such as “town” or “city” have already been mapped.”
http://neis-one.org/2011/03/unmapped-places-1103/
I don’t understand why e.g. node
http://www.openstreetmap.org/node/2573424356 (see http://resultmaps.neis-one.org/unmapped#12/51.0042/4.4780) is marked as unmapped place. It’s in the middle of a triangle with residential roads. The same for the node of Walem. Plenty of streets are mapped around it.
Those streets have been there for a couple of years
Hi Marc, thanks for noticing. It seems that there is something wrong, in particular around Antwerpen. I will take a look at the issue.
There are several place nodes, which are not correct, e.g.:
http://www.openstreetmap.org/node/2575604140
http://www.openstreetmap.org/node/260758476
Two place nodes for Haasdonk? That is the issue. BTW, the same for Walem.
Thanks Pascal, I will take a look at that.
Already fixed those 🙂 , A few will remain now. I’ll try to fix the Belgian places as well.
There seems to be an issue with excluding nodes near to a service, wit: http://www.openstreetmap.org/node/301449092#map=18/46.41206/8.10143
But otherwise nice for finding not so obvious errors.
Pascal,
This looks pretty cool but I think the analysis for Africa is way off. Most villages in africa are near roads but a lot would be closer to “trunk, secondary, tertiary” roads than many other places. I would rerun the analysis with the same buffer but with “highway=*” that way you are grabbing a wider variety of potential settlement patterns. I also suspect that this will also show tons of past imports.
We did a similar analysis for our west africa mapping project where we looked at the presence of roads/buildings within a buffer (600 m) of a village node to determine if it needed to be mapped.
Dale, thanks for your comment. I will rerun the anaylsis in the coming week. But, do you think that the villages are tagged correctly? As mentioned above, “hamlet” could be the correct tag value for such places …
What’s up with this village?
http://www.openstreetmap.org/node/31530925
As mentioned above, IMHO hamlet should be the better place tag. See: http://wiki.openstreetmap.org/wiki/Tag:place%3Dhamlet
A village is a village, a hamlet is a hamlet, this is according to the official database of TERYT. Also it has population of 629, so really no hamlet to me.
There are tertiary and unclassified roads a few meters by the place node, but still it’s “unmapped”. Clearly your program is wrong, not OSM. That’s why I asked you 😉
Ok, but why isn’t the location of place node e.g. here?
http://osm.org/go/0PHpaadw1-?m=
there is plenty of false positives around here http://resultmaps.neis-one.org/unmapped#11/49.7671/22.2102 as well
Did you check the satellite imagery? There are a several streets which are not mapped, e.g. here:
http://www.openstreetmap.org/#map=17/49.75707/22.23932
http://www.openstreetmap.org/#map=17/49.72896/22.21876
I have now added more roads for that village, but, there seems to be local convention to tag most of them as “track”. Probably these tracks could be included in Pascal’s search radius.
false positive?
http://resultmaps.neis-one.org/unmapped#18/42.78234/-1.86532
Nope, IMHO the streets nearby should not be tagged as highway=living_street (http://wiki.openstreetmap.org/wiki/Tag:highway%3Dliving_street). Furthermore, such a place could be tagged as hamlet and not as a village.
This node is surrounded by ‘residential’ roads in a 50 meter radius. I think something isn’t analysed correctly.
http://resultmaps.neis-one.org/unmapped#17/51.03465/4.41442
Having a AHA-moment: There are 2 nodes in the same location, which is a validation error, 1 was in a relation, the other was not. I fixed this in OSM now. http://www.openstreetmap.org/node/332227169/history
Good Tool!
Checked in Hungary.
And I found some “secondary” tagging
http://wiki.openstreetmap.org/wiki/Tag:highway%3Dsecondary
====
>”The OSM data for the analysis is
> prepared by a custom OSM PBF reader.”
this is interesting for me. If you can give more info
Thanks for your comment.
Basically, you can do everything with Osmium or Osmosis:
http://wiki.openstreetmap.org/wiki/Osmium
http://wiki.openstreetmap.org/wiki/Osmosis
Personally, I use some Osmosis core classes. But you can also do some data manipulation tasks with the standard Osmosis implementation, as described here:
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.44#–node-key_.28–nk.29
As usual: Nice and helpful work! 😉 So thanks for your efforts.
Looking at two “unmapped” places in Germany, it seems to me that a search radius of 700m might be a bit too short.
The “unmapped” municipality Flörsbachtal consists of 4 existing villages (Flörsbach, Kempfenbrunn, Lohrhaupten, Mosborn) – which are up to 5 km away from each other.
Only 640 m away from the “virtual” point Flörsbachtal (see http://resultmaps.neis-one.org/unmapped#14/50.1211/9.4379) there are two residential roads mapped.
The “unmapped” municipality Jossgrund consists of 4 existing villages (Burgjoß, Oberndorf, Pfaffenhausen, Lettgenbrunn) – which are up to 6 km away from each other.
The nearest residential road is mapped 940 m away from the “virtual” point Jossgrund (see http://resultmaps.neis-one.org/unmapped#13/50.1857/9.4596). Several other residential roads are less than 1000m away from that point.
It might well be that re-thinking and re-adjusting the actual search radius would result in more precise results. Unfortunately I have no clue whether a stretched search radius would worsen results in other areas … 🙂
Is there areason why you did not incorprate highway primary, secondary and tertiary, they seem all to fall in your criteria. I found serveral places you detected as unmapped which are completly fine when you add these.
Eg: http://osm.org/node/1756313870
For the case of a place, it can be assumed that (at least for Germany or other European countries) a minimum of one highway with a “lower” value such as residential, unclassified or service, should be nearby. The mentioned values (primary, secondary and tertiary) are for “important roads in a country’s system” (see [1]).
However, of course false=positives can occur during during my analysis.
Are you sure, that the linked place is a village and not a hamlet? [2]
Did you check wikipedia for the amount of inhabitants? [3]
[1] http://wiki.openstreetmap.org/wiki/Key:highway
[2] http://wiki.openstreetmap.org/wiki/Tag:place%3Dhamlet
[3] https://de.wikipedia.org/wiki/Bisisthal
First data point I looked at for Waltham, IL, USA seems to be a false positive. There is a residential road passing right through a cluster of four or five houses and a church.
http://www.openstreetmap.org/node/153515752#map=17/41.43311/-89.00849&layers=D
Yes, you are right, there are some houses. Furthermore the satellite imagery shows some service roads. However, all mentioned objects aren’t in OSM, so I think, yes, this place isn’t mapped so far and this is not a false positive.
I have now cleaned up / added the details for the hamlet of Waltham. Wasn’t a false positive, this is exactly the kind of thing I like to find and work on 🙂
Hi Pascal, nice map as always 😉
I know that when mappers are identifying settlements from satellite imagery during remote mapping, they are often only identified by the boundary (landuse=residential), rather than with a place= point. I guess this means that they would all be excluded from this analysis? Maybe, we should start adding that place= point into the instructions for mappers….
Pete, thank you very much for your comment.
Right, I don’t use boundaries during my analysis.
I’ve done a lot of work on Uganda over the years and am noticing what I think might be false positives – bulk imported villages (possibly bad data) near minor/unclassified roads.
The example I’d give would be Bweyongede and around – should be within 500 feet of a road, I have no idea if a village even exists here.
Currently your map data is from June 8th, 2016. How often is planned to refresh data?
Just updated based on a fresh planet dataset. Next update maybe in 3 months?
In some areas, there does exist villages which are not connected to any road (notably on islands, or in the middle of rain forests or deserts, where it is not roads, but rivers (sometimes small airports) that connect them to the rest of the world.
In many desertic regions, there’s no roads but only unpaved tracks (not static and varying with winds or after occasional floodings by intermitent rivers, or because farmers want to move accesses to preserve cultures on the best lands, or want to rotate them)
Anyway you seem to be too strict on your definition of villages vs. hamlets. in many countries, places are still officially villages even if their population is below a threshold, for historic reasons (migrations, deaths dur to poverty or present/past conflicts) but that still exist administratively (expecting for a repopulation, or simply preserved as a culural heritage), or because these places have only a seasonal population, important them to qualify as villages even if this population is not permanent, or cannot be double-counted in multiple places in official population survey (but they are still counted in a separate figure allowing double-countings).
If a village has an administrative status (e.g. an admin_level 8), don’t insist into wanting them to be hamlets.
And anyway you should not map them as “unmapped” but only with warnings. And there’s not a single population threshold globally, the thresholds highly depend on how each country is administratively subdivided (in one country a place would be a municipality with a status of village, in another it could be a section of a municipality, e.g. a ward at level 9 or higher, and using other thresholds putting them as a quarter, or urban district, or suburb, or village…
If you already find existing nodes in the expected place, and with the correct expected name, these are NEVER “unmapped villages” but possibly may be tagged differently. Also don’t make absolute decisions based on what is tagged for the population: these tags may also be out of sync with official figures.
I found that ALL “unmapped villages” in france for examples are ALL false positive. But there are still missing villages (which are in fact smaller rural settlements within a “commune”/French municipality), and possibly historic communes, but in other places, villages have never been communes, or were split between two communes.
All current communes (that have existed at least some years after 2000) in France are mapped, with the exception of soem remaining “associated communes” (that exist in France since 1973 and are former communes, now merged into a larger commune but still with some very minimal delegations of powers and well definite borders). Even if these are “delegated communes” or “associated communes”, they remain as villages keeping their own name in the new commune, and will not turn into “hamlets”, unless they are fully merged.
As well you seem to assume that every village should be located near a crossroad connecting another smaller highway (such as a residential), but this is wrong in some cases, as there does exists villages that have only a single “street”, all houses being built along some important interconnection road linking two larger villages or towns. The other “highways” do not exist, or they are just private tracks, inaccessible due to closed fences, or water drains along the road, and only small private paths connecting to the road via a gate: almost all private homes are never tagged along with their private (with non normative geometry, not directly observable) access paths, and some houses in “no-car” areas (such as islands) have no access path at all, linked only through open fields or gardens, or by a small boat, or by walking along a beach where any construction or road/path is strictly forbidden by authorities or because these ways are regularly submersibles by river floods or maritime tide. Finally there are some rare places only accessible by helicopter or planes (e.g. in rainforests or some elevated islands), or by very sportive tracks in mountains, or only by cableways (no track at all). But there are really enough people living there to qualify the place as a village this is common in Borneo, New Guinea, western Brazil, eastern Peru/Bolivia/Colombia, Greenland, Saharian countries, Mongolia, and in many islands of the Pacific where nomadic or migrating people are frequent.