Category: Allgemein

Wait, someone did what?
Exploring Reverted Map Edits in OpenStreetMap

The OpenStreetMap (OSM) project has over 10 million registered members, with around 2 million user profiles having made at least one map contribution. However, a closer look reveals that there has been a slight decline in the number of active contributors over the last three years. Despite the extensive global mapping community, there are instances where individuals or automated bots disregard the consensus norms of the community when editing data. These situations arise due to disagreements regarding the appropriateness of certain tagging or features within the OSM database. To address these issues, a change rollback process, commonly referred to as reverting, is used to combat vandalism and correct ‘mistakes’ by restoring a previous version of the data.

Two years ago, I added additional statistics to the “How did you contribute to OSM?” page for quality assurance purposes. The numbers for each contributor profile were derived from an analysis of the full history OSM planet dump and changeset tags, including the specific editor used. While this pragmatic approach provides valuable insights, it’s important to acknowledge that the obtained numbers are estimations rather than exact figures. Furthermore, I received several inquiries regarding the implementation of the processing involved in identifying the displayed “reverted changes”.

Over the past few weeks, I have developed an advanced processing pipeline. This involved revisiting the comprehensive OSM planet dump and examining the evolution of each entity (node, way, relation) in relation to its previous states. Specifically, an entity with a higher version number was identified as a revert if it had the same latitude/longitude coordinates (for nodes), tags (key-value pairs), and/or members (for relations) as a previous version. In simpler terms, if a mapper changed “X” to “Y” and another mapper subsequently altered it back to “X”, it would be counted as a revert.

The following graph illustrates the amount of reverted map edits, changesets, and the contributors affected per month. This visualisation offers some initial insights into the scope and impact of reverted changes. It’s important to note, that the these numbers don’t include any actions related to reverted data imports.

 

It is also necessary to look more closely at the specific entities that are counted as “reverted”. Are they primarily nodes with a few tags, or are they ways and relations with extensive mapping histories in active areas? These specific aspects, among others, will be explored in an upcoming blog post or possibly published as part of a scientific research study.

What are your thoughts? I think many of you might be curious to discover whether your own map entities have been reverted. Please feel free to leave a comment on my OSM diary page, where I have cross-posted this article.

#100 – Thank you!

While I was working on my latest blog post, I realized that I had already written 100 posts over the past nine years. All posts have one thing in common: They are about the well-known and maybe never ending OpenStreetMap project. From time to time there are still emerging questions or issues which must be tackled by someone. This always fascinated me about OSM. However, this particular number 100 is not about a specific subject, it’s just a tiny post to say thank you! Thank you for your continuous interest in reading, commenting and of course sometimes criticizing my work. To me it’s still awesome to see that you, a few thousand people in total, use tools or services daily, that I implemented.

It’s still incredible that many people (not all) spent their spare time contributing to the project, not only as spatial data contributors but also as software engineers, system admins or coordinators of workshops, conferences or mapping events or by just validating or reviewing the latest map changes. Some of my webpages wouldn’t be as successful without your feedback. So, thanks again! Finally, I would like to thank all the people who I have met during the different meet ups, such as FOSSGIS, OSM hack weekends etc. over the past couple of years. There have always been friendly, respectful and useful chats: It’s always a pleasure.

Thanks to maɪˈæmɪ Dennis.

Processing compressed OpenStreetMap Data with Java

This blog post contains a summary on how you can write your own Java classes to process OpenStreetMap (OSM) pbf files. PBF is a compression format, which is nowadays more or less the standard utilized for reading and writing OSM data quickly. In the OSM world, many tools and programs implemented this file format (you can find additional information here). However, I think the following samples for reading and writing such compressed OSM data can be very helpful. In particular, if someone has to create some sort of test data or has to read some specific mapped objects of interest for her/his own project. The well-known Java Osmosis tool (command line application for processing OSM data), provides several libraries that are the basis for this brief tutorial.

Step 1: Maven is the key – If your Java project is already managed by Maven, you can just add the following lines to your pom.xml. It downloads and adds the required jar-files that are needed to process compressed OSM data to your project.

<dependency>
	<groupId>org.openstreetmap.osmosis</groupId>
	<artifactId>osmosis-pbf</artifactId>
	<version>0.46</version>
</dependency>

If you don’t use Maven, you can download the aforementioned dependencies here and add the jars as described here to your eclipse build path.

Step 2: Implementing the Sink interface for reading OSM data – Now, after you added the required libraries to your project, you can create your own class to read compressed OSM data. The following MyOSMReader class is an easy example for that scenario: It reads an OSM pbf files and prints all Ids of ‘ways’ with a ‘highway’ key.

package org.neis_one.osm.examples;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.util.Map;

import org.openstreetmap.osmosis.core.container.v0_6.EntityContainer;
import org.openstreetmap.osmosis.core.container.v0_6.NodeContainer;
import org.openstreetmap.osmosis.core.container.v0_6.RelationContainer;
import org.openstreetmap.osmosis.core.container.v0_6.WayContainer;
import org.openstreetmap.osmosis.core.domain.v0_6.Tag;
import org.openstreetmap.osmosis.core.domain.v0_6.Way;
import org.openstreetmap.osmosis.core.task.v0_6.Sink;

import crosby.binary.osmosis.OsmosisReader;

/**
 * Receives data from the Osmosis pipeline and prints ways which have the
 * 'highway key.
 * 
 * @author pa5cal
 */
public class MyOsmReader implements Sink {

	@Override
	public void initialize(Map<String, Object> arg0) {
	}

	@Override
	public void process(EntityContainer entityContainer) {
		if (entityContainer instanceof NodeContainer) {
			// Nothing to do here
		} else if (entityContainer instanceof WayContainer) {
			Way myWay = ((WayContainer) entityContainer).getEntity();
			for (Tag myTag : myWay.getTags()) {
				if ("highway".equalsIgnoreCase(myTag.getKey())) {
					System.out.println(" Woha, it's a highway: " + myWay.getId());
					break;
				}
			}
		} else if (entityContainer instanceof RelationContainer) {
			// Nothing to do here
		} else {
			System.out.println("Unknown Entity!");
		}
	}

	@Override
	public void complete() {
	}

	@Override
	public void close() {
	}

	public static void main(String[] args) throws FileNotFoundException {
		InputStream inputStream = new FileInputStream("/Path/To/Your/read.osm.pbf");
		OsmosisReader reader = new OsmosisReader(inputStream);
		reader.setSink(new MyOsmReader());
		reader.run();
	}
}

Step 3: Implementing the Source Interface for writing OSM data – Similarly to Step 1 to read an OSM file, you will have to implement again an interface to write data as well. This time the Osmosis core task Source interface is being utilized. The following example class writes 10 Nodes to a new OSM pbf file.

package org.neis_one.osm.examples;

import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.OutputStream;
import java.util.Date;

import org.openstreetmap.osmosis.core.container.v0_6.NodeContainer;
import org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData;
import org.openstreetmap.osmosis.core.domain.v0_6.Node;
import org.openstreetmap.osmosis.core.domain.v0_6.OsmUser;
import org.openstreetmap.osmosis.core.task.v0_6.Sink;
import org.openstreetmap.osmosis.core.task.v0_6.Source;
import org.openstreetmap.osmosis.osmbinary.file.BlockOutputStream;

import crosby.binary.osmosis.OsmosisSerializer;

/**
 * Writes OSM data to the output task.
 * 
 * @author pa5cal
 */
public class MyOsmWriter implements Source {

	private Sink sink;

	@Override
	public void setSink(Sink sink) {
		this.sink = sink;
	}

	public void write() {
		for (int idx = 1; idx <= 10; idx++) {
			sink.process(new NodeContainer(new Node(createEntity(idx), 0, 0)));
		}
	}

	public void complete() {
		sink.complete();
	}

	private CommonEntityData createEntity(int idx) {
		return new CommonEntityData(idx, 1, new Date(), new OsmUser(idx, "User"), idx);
	}

	public static void main(String[] args) throws FileNotFoundException {
		OutputStream outputStream = new FileOutputStream("/Path/To/Your/write.osm.pbf");
		MyOsmWriter writer = new MyOsmWriter();
		writer.setSink(new OsmosisSerializer(new BlockOutputStream(outputStream)));
		writer.write();
		writer.complete();
	}
}

Step 4: That’s it – You should now be able to write your own classes that allow you to process compressed OSM data.

One last thing: When writing your own Java classes to process OSM data, you should always keep in mind, that the PBF file has it’s own object ordering format of the OSM objects: Each file first contains Nodes, then Ways and at the end Relations. That means, a Way element only contains references to Node ids. It doesn’t contain the coordinates or any tags of the actual nodes that are used for the Way. Thus, if you want a complete Way element with its geometry, you have two options: Store all Nodes in some kind of map or read the entire OSM file twice. Furthermore, if you’re interested in additional compressing options during writing a compressed OSM file, you should have a look at this class.

Thanks to maɪˈæmɪ Dennis.

How to detect suspicious OpenStreetMap Changesets with incorrect edits?

Since its rise in popularity, the well-known online encyclopedia Wikipedia has been struggling with manipulation or, in the worst-case, vandalism attempts. Similarly, the OpenStreetMap (OSM) project suffered several times over the past few years of cases where incorrect map data edits were made. These erroneous edits can stem at times from (new) contributors or illegal data imports (or automated edits) which have not been discussed in advance with the community or the Data Working Group (DWG) and corrupted existing project data. The current OSM wiki page gives a great overview about general guidelines and e.g. types of vandalism. Another page in the wiki also mentions a prototype of a rule based system for the automatic detection of vandalism in OSM, which I developed in 2012. However, the system has never actually been implemented. Today, the contributors of OSM can use a variety of different tools to inspect an area or particular map changes. A few of them are listed below (complete list can be found here):

Based on the database which I use for multiple other services, I created an easy to use webpage to find suspicious OSM changesets with possibly incorrect map edits. The webpage offers some filter options such as the boundary of a country or the object change of interest. In contrast to the other aforementioned webpages you can also filter changesets based on the active “mapping days” of the contributor. A “mapping day” is a day on which the contributor created at least one changeset, independent from the registration date. I am also planning on adding additional user reputation information such as used editors or tagging behavior. And of course I am going to add some RSS feeds in the next version. The first version can be found here.

OSMSuspicious

What makes all of this different from other tools? Well, I think one of the major advantages is the simplicity of the webpage and that you can filter changesets based on the contributor activity and/or the changeset edits. In contrast to other tools, you can find changesets not only based on your area of interest, but also based on potential beginner mistakes and hopefully not vandalism attempts or fictional/ none existing map data.

Find Suspicious OSM Changesets here: http://resultmaps.neis-one.org/osm-suspicious

Thanks to maɪˈæmɪ Dennis.

OpenStreetMap Crowd Report – Season 2015

Almost one year has passed again. This means it’s time for the fourth OpenStreetMap (OSM) member activity analysis. The previous editions are online here: 2014, 2013 and 2012. Simon Poole already posted some interesting stats about the past few years. You can find all his results on the OSM wiki page. However, similar to last year, I try to dig a little deeper in some aspects.

Overall the OSM project has officially more than 2.2 million registered members (Aug, 9th 2015). For several of my OSM related webpages I create a personal OSM contributor database, based on the official OSM API v0.6. Anyway, when using this API, the final table will show a list with more than 3 million individual OSM accounts (Aug, 9th 2015). I’m not sure what the cause for this gap of almost 1 million members between the official number and the member number extracted with the API could be. Maybe some of you have a possible explanation? However, I think many accounts are created by spammers or bots.

The following chart shows a trend similar to the one of previous years: The project attracts a large number of newly registered members, but the sum of contributors that actively work on the project is fairly small. As mentioned in earlier posts, this phenomenon is nothing special for an online community project and has been analyzed for previous years already.

2015OSMMembers

Described in numbers (July 31st, 2015):

  • Registered OSM Members (OSM API): 3,032,954
  • Registered OSM Members (Official): 2,201,519
  • Members who created 1 Changeset: 562,670
  • Members who performed >= 10 Edits: 343,523
  • Members who created >=10 Changesets: 137,591

Personally, I really like the following diagram: It shows the increase in monthly contributor numbers over the past few years and their consistencies in collecting OSM data based on the first and latest contributed changeset of an OSM member. It’s great to see that at least some experienced mappers are still contributing to the project after more than five years.

2015OSMMembersSince

Some background information on how I created the stats: To retrieve the registration date of the members, I used the aforementioned OSM API. The other numbers are based on the OSM changeset dump, which is available for download here.

Next to the presented results above, you can find some daily updated statistics about the OSM project on OSMstats.

Thanks to maɪˈæmɪ Dennis.

Visualizing the #MissingMaps OpenStreetMap Contributions

The Missing Maps project is a collaboration between the Humanitarian OpenStreetMap Team (HOT) and various partner agencies, such as the American or the British Red Cross. One of their main objectives is “to map the most vulnerable places in the developing world, in order that international and local NGOs and individuals can use the maps and data to better respond to crises affecting the areas.” You can find additional information about the Missing Maps Project on the OpenStreetMap (OSM) wiki and their project page.

A year ago, I created a webpage where you can filter OSM changesets by a specific comment. Sadly the webpage provides only a search for the latest seven days. However, the Missing Maps project asked me, if it’s possible to “look over a longer time scale”? Here we go, based on a similar concept that I used for a webpage that I created for the HOT Ebola Response, I made a Map that displays all OSM changesets which have the hashtag #MissingMaps in the comment attribute and have been created since August 1st, 2014. It’s online here and being updated on an hourly basis: http://resultmaps.neis-one.org/osm-missingmaps

http://resultmaps.neis-one.org/osm-missingmaps

The webpage also contains overall information about the number of OSM Contributors and map changes. So far more than 800 contributors created more than 1.5 Mio map changes in almost 22,000 changesets. The volunteers contributed in more than 30 countries such as Congo-Kinshasa, Sudan, Central African Republic, Indonesia, Ethiopia, Chad, Zimbabwe or Rwanda. Additionally the number of created changesets and contributors of the last seven days are displayed in two charts at the left-hand side. The time slider at the bottom of the map can be used to show the changesets between two specific dates. Finally an overview page lists the names of all OSM contributors who used the hashtag #MissingMaps in their changesets comment. Same as last time: Thank you & keep up the good work!

Thanks to maɪˈæmɪ Dennis.

Your Explored OSM World

Gregory Marler had the great idea to implement an “explored” map, based on a concept that some of you might know as “fog of war” from strategy video games. So here you go: I extended my OSM Heat Map with the “Explored Map Style”. It essentially reveals the contribution areas of an OpenStreetMap member in a “fog of war” style. The following figure shows Gregory’s amazing “explored” OSM map.

ExploredMapStyleToner

The Heat & Explored Maps are available for almost all OSM members who contributed at least several changesets here: http://yosmhm.neis-one.org (The new “Explored Map Style” can be selected in the layer panel (upper right corner). Additionally, I added the awesome looking and well known Watercolor and Toner map styles from Stamen design)

Thanks to maɪˈæmɪ Dennis

The OSM Contributor Activity Report – Edition 2014

The OpenStreetMap (OSM) project celebrated its 10th anniversary in August 2014. For almost 10 years it has increased its number of registered members. Even though some contributors stopped their contributions to the project, each day new mappers start collecting features for the free wiki world map (aka database).

In my last contributor report in 2013, the OSM project had a total of 1.3 Mio registered members. For July 2014 this number has increased to almost 1.6 Mio registered members. Similarly to last year, I checked how many contributors created one or more than ten changesets or performed more than 10 map edits. This information can be retrieved from the changeset dump.

NewContributorsPerMonth.201408

The figure above reveals a similar trend to the ones we saw in the past few years: Less than 1/3 of the 1.6 Mio registered members actively contribute to the project (450,000 members). Furthermore, only a small group of 16% (270,000) or respectively 6% (100,000) of the contributors performed more than 10 edits or 10 changesets.

The long-term motivation of the contributors is quite important too. Therefore, similarly to the methods that we presented in our open access publication, I created a figure, which visualizes the increase in monthly volunteer numbers over the past few years and the consistencies in data contributions.

Activity.Stats.201408

As we already revealed in our study, only half of the monthly active members in OSM are also long-term contributors. Also, the previously discussed pattern which depicts a contributor loss of almost 70% over the years is again visible. However, it is good to see that at least some “senior” mappers still keep contributing to the OSM project.

Thanks to maɪˈæmɪ Dennis

Introducing OpenStreetMap Contributor Activity Areas

One month ago I wrote a blog post about a new website which allows you to see other OpenStreetMap contributors in your area. Overall the feedback was very positive, thank you very much for that! However, now it is time for a new extension to the “How did you contribute to OpenStreetMap?” (HDYC) webpage. As I mentioned in my last blog post, I used an algorithm (which is described in a paper that I wrote here) to compute and determine the activity area of a contributor based on her/his changeset centers. The following figure shows the new function that was added to the HDYC website visualizing the activity area of a contributor! Sorry Harry, as always you have to be our guinea pig, but you have a really awesome activity area 🙂

Next to the visualization of the overall activity area of a contributor, you can also click on a link at the bottom of the map to switch to the contributors’ activity area of the past six months. Furthermore, all maps on HDYC now use the great Leaflet map library instead of Openlayers. Also, your activity areas’ first and last Nodes have a direct link to the “Overview of OpenStreetMap Contributors aka Who’s around me?” webpage. This provides an easy way to locate other contributors in your area. I have to mention that not every contributor has an activity area for the past six months. It highly depends on the activity of the contributor within this time frame!

One more thing: The aforementioned “Who’s around me?” webpage has three new overlays. Two overlays show the contributors of the past six months with their first and last Nodes and one additional layer shows the activity areas also based on the past six months for each contributor. You can find all new layers in the upper right corner in the so-called “Layerswitcher”.

My HDYC database is updated more or less on a daily basis. The information about your changeset activities is updated once a week (based on the weekly changeset dumps from here). “The Created Nodes per Country”-section can only be updated when a new full history dump is available, but you can always find the latest date in the section-label. The “Who’s around me?” webpage uses almost the same database as HDYC, so the data up-to-dateness is similar.

Have fun with the new gadgets!

¡Muchas gracias maɪˈæmɪ Dennis

I Like OpenStreetMap (OpenLayers Plugin)

A few months ago, Frederik Ramm posted an idea on the German OpenStreetMap mailing list about a new (stochastic) approach to OSM data quality assurance. You can find his original German post here. His idea was to create a way to allow users to “like” or “dislike” a specific region on the OSM map, a function that other popular websites such as YouTube or Facebook implemented to allow users to provide feedback to videos or status updates. For OSM this particular function could give some indicators or trends about the OSM map data.

I really liked his idea and in collaboration with Frederik I created an Open Source OpenLayers plugin. For all new readers: OpenLayers is an Open Source library which can implement a dynamic (OSM) map into more or less any webpage. One of our goals was to make the integration of the ILikeOSM plugin as easy as adding a tile server to your OpenLayers map.

The following image shows the plugin in more detail, including the “like” and “dislike” buttons to provide feedback about the area on the map.

An additional feature of the plugin shows how many users have been viewing the same area of the map that the current user is taking a look at. More precisely: How many other users have been viewing a similar area of the map within the past two minutes with a zoom level of +-3 to yours. All components of the plugin are Open Source and available on github. The database which saves the likes and dislikes is running on a German OSM Dev server. A database dump file can be downloaded on a daily basis. It is important to note at this point that no private data is saved in the database when a user leaves his or her feedback. The plugin only saves an independent, randomly generated user ID, the feedback type i.e. thumbs up/down, the zoom level, the layer name and the bounding box of the map section. A map view is generally not saved to the database until the user accepts to do so via a pop up window.

Do you like this feature?
It is quite easy to integrate it into your own webpage. Here is how it works:
1. Add the following line below your OpenLayers script-tag:
<script src=”http://ilike.openstreetmap.de/ILikeOSM.min.js” type=”text/javascript”></script>
2. Then add the following lines to your OpenLayers Controls:
new OpenLayers.ILikeOSM()
3. Styling
<style type="text/css">
div.olILikeOSM { position: absolute; top: 15px; left: 50px; padding: 7px; color:white; border-radius: 10px; background: rgba(0, 0, 0, 0.6); }
div.olILikeOSM a { color: white; font-size:12px; text-decoration: underline; }
</style>
4. That’s it!

What is the benefit of this plugin or of the saved ILikeOSM data?
Based on the saved likes, dislikes and map views we can generate some statistics to provide you with information about the number of people who like or dislike your particular area of interest. Maybe we can even see some prove of Linu’s law “given enough eyeballs, all bugs are shallow”; meaning in this case, that a larger number of users that check a certain region of the map, results in “better” OSM data quality. As a first prototype, I generated a static webpage which shows an example result map.

Further ideas?
The plugin could potentially be expanded with an additional textbox in which a user could leave a comment why the area is not well represented in OSM. This information could then be saved e.g. in OpenStreetBugs. Anyway, we think that the current version of the plugin could provide some very useful information. You will find a webpage with all information, examples and downloads here: http://ilike.openstreetmap.de As a first step we integrated the plugin into the OpenStreetMap Germany webpage.

Frederik will give a short talk about the ILikeOSM plugin at the upcoming State of the Map 2012 in Tokyo. If our proposed session abstract about another topic for the State of the Map 2012 US gets accepted, Dennis will try to present it there too.

Thank you very much for your feedback: Frederik, Jonas, Dennis, Sven & Marc