Processing compressed OpenStreetMap Data with Java
This blog post contains a summary on how you can write your own Java classes to process OpenStreetMap (OSM) pbf files. PBF is a compression format, which is nowadays more or less the standard utilized for reading and writing OSM data quickly. In the OSM world, many tools and programs implemented this file format (you can find additional information here). However, I think the following samples for reading and writing such compressed OSM data can be very helpful. In particular, if someone has to create some sort of test data or has to read some specific mapped objects of interest for her/his own project. The well-known Java Osmosis tool (command line application for processing OSM data), provides several libraries that are the basis for this brief tutorial.
Step 1: Maven is the key – If your Java project is already managed by Maven, you can just add the following lines to your pom.xml. It downloads and adds the required jar-files that are needed to process compressed OSM data to your project.
<dependency> <groupId>org.openstreetmap.osmosis</groupId> <artifactId>osmosis-pbf</artifactId> <version>0.46</version> </dependency>
If you don’t use Maven, you can download the aforementioned dependencies here and add the jars as described here to your eclipse build path.
Step 2: Implementing the Sink interface for reading OSM data – Now, after you added the required libraries to your project, you can create your own class to read compressed OSM data. The following MyOSMReader class is an easy example for that scenario: It reads an OSM pbf files and prints all Ids of ‘ways’ with a ‘highway’ key.
package org.neis_one.osm.examples;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.util.Map;
import org.openstreetmap.osmosis.core.container.v0_6.EntityContainer;
import org.openstreetmap.osmosis.core.container.v0_6.NodeContainer;
import org.openstreetmap.osmosis.core.container.v0_6.RelationContainer;
import org.openstreetmap.osmosis.core.container.v0_6.WayContainer;
import org.openstreetmap.osmosis.core.domain.v0_6.Tag;
import org.openstreetmap.osmosis.core.domain.v0_6.Way;
import org.openstreetmap.osmosis.core.task.v0_6.Sink;
import crosby.binary.osmosis.OsmosisReader;
/**
* Receives data from the Osmosis pipeline and prints ways which have the
* 'highway key.
*
* @author pa5cal
*/
public class MyOsmReader implements Sink {
@Override
public void initialize(Map<String, Object> arg0) {
}
@Override
public void process(EntityContainer entityContainer) {
if (entityContainer instanceof NodeContainer) {
// Nothing to do here
} else if (entityContainer instanceof WayContainer) {
Way myWay = ((WayContainer) entityContainer).getEntity();
for (Tag myTag : myWay.getTags()) {
if ("highway".equalsIgnoreCase(myTag.getKey())) {
System.out.println(" Woha, it's a highway: " + myWay.getId());
break;
}
}
} else if (entityContainer instanceof RelationContainer) {
// Nothing to do here
} else {
System.out.println("Unknown Entity!");
}
}
@Override
public void complete() {
}
@Override
public void close() {
}
public static void main(String[] args) throws FileNotFoundException {
InputStream inputStream = new FileInputStream("/Path/To/Your/read.osm.pbf");
OsmosisReader reader = new OsmosisReader(inputStream);
reader.setSink(new MyOsmReader());
reader.run();
}
}
Step 3: Implementing the Source Interface for writing OSM data – Similarly to Step 1 to read an OSM file, you will have to implement again an interface to write data as well. This time the Osmosis core task Source interface is being utilized. The following example class writes 10 Nodes to a new OSM pbf file.
package org.neis_one.osm.examples;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.OutputStream;
import java.util.Date;
import org.openstreetmap.osmosis.core.container.v0_6.NodeContainer;
import org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData;
import org.openstreetmap.osmosis.core.domain.v0_6.Node;
import org.openstreetmap.osmosis.core.domain.v0_6.OsmUser;
import org.openstreetmap.osmosis.core.task.v0_6.Sink;
import org.openstreetmap.osmosis.core.task.v0_6.Source;
import org.openstreetmap.osmosis.osmbinary.file.BlockOutputStream;
import crosby.binary.osmosis.OsmosisSerializer;
/**
* Writes OSM data to the output task.
*
* @author pa5cal
*/
public class MyOsmWriter implements Source {
private Sink sink;
@Override
public void setSink(Sink sink) {
this.sink = sink;
}
public void write() {
for (int idx = 1; idx <= 10; idx++) {
sink.process(new NodeContainer(new Node(createEntity(idx), 0, 0)));
}
}
public void complete() {
sink.complete();
}
private CommonEntityData createEntity(int idx) {
return new CommonEntityData(idx, 1, new Date(), new OsmUser(idx, "User"), idx);
}
public static void main(String[] args) throws FileNotFoundException {
OutputStream outputStream = new FileOutputStream("/Path/To/Your/write.osm.pbf");
MyOsmWriter writer = new MyOsmWriter();
writer.setSink(new OsmosisSerializer(new BlockOutputStream(outputStream)));
writer.write();
writer.complete();
}
}
Step 4: That’s it – You should now be able to write your own classes that allow you to process compressed OSM data.
One last thing: When writing your own Java classes to process OSM data, you should always keep in mind, that the PBF file has it’s own object ordering format of the OSM objects: Each file first contains Nodes, then Ways and at the end Relations. That means, a Way element only contains references to Node ids. It doesn’t contain the coordinates or any tags of the actual nodes that are used for the Way. Thus, if you want a complete Way element with its geometry, you have two options: Store all Nodes in some kind of map or read the entire OSM file twice. Furthermore, if you’re interested in additional compressing options during writing a compressed OSM file, you should have a look at this class.
Thanks to maɪˈæmɪ Dennis.
