Friday, July 29, 2011

deegree Themes

After talking about layers I recently read the OGC WMTS specification. A new concept called themes was introduced there, and that lead to me being able to understand the problem I'm trying to solve here more thoroughly.

Layers in WMTS are no longer hierarchical. All that remains of the old WMS layer trees is a single list of layers. In order to create order in a potentially huge linear list of layers, the theme concept was introduced.

Themes in WMTS can be hierarchical, and each theme can reference any number of layers. Layers can even be referenced multiple times, and you can have multiple top-level themes.

That actually makes a beautiful distinction between structure and data. Speaking in deegree workspace language, it suddenly makes a lot of sense to have not only a bunch of layers configured in one place, but also have a bunch of themes, referencing layers, in another place. A WMS would then only reference themes.

Other ideas include simple layer collections (which aggregate layers similar to current logical layers within other layers).

I still have to figure out a lot of the details, but I can feel that this is going to make things simpler. When configuring a layer you only think about what data it's going to use, and when configuring a theme you only think about the bigger picture (where do my layers belong).

I'm always open for suggestions, so if you have an idea about how to make things good in the end, please speak up.

Monday, July 18, 2011

Cascading WMS with deegree

In this post I want to explain how cascading a WMS works in deegree. Cascading a WMS can be useful to 'repair' broken WMS, restructure layer hierarchies or to hide multiple WMS behind one endpoint.

So how does cascading work in deegree? First of all, there's an abstract concept of a remote OWS data source. Currently there is only a WMS implementation, but more will follow. That means a remote WMS can be a resource in the workspace. That resource can then be used in your WMS configuration as a data source for a layer.

One important consequence is that one remote WMS data source means one data source. Although a WMS might typically have multiple layers and the config allows you to select multiple layers for cascading, it's still one resource. Let's have a look at an example:

<RemoteWMSStore xmlns="http://www.deegree.org/datasource/remoteows/wms"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.deegree.org/datasource/remoteows/wms remotewms.xsd"
configVersion="3.1.0">
  <CapabilitiesDocumentLocation
location="http://deegree3-testing.deegree.org/deegree-utah-demo/services?request=capabilities&amp;service=WMS&amp;version=1.1.1"/>
  <RequestedLayer>
    <Name>ZipCodes</Name>
  </RequestedLayer>
</RemoteWMSStore>

It's always required to specify a capabilities document. This can also be a local file instead of a request. Then you can specify one or more requested layers. That's it, everything else (available crs, formats etc.) will be determined from the capabilities.

Now to actually make it available as WMS layer in deegree just add a layer with the datasource ID:

<wms:RequestableLayer>
  <wms:Name>zipcodes</wms:Name>
  <wms:Title>Utah ZipCodes</wms:Title>
  <wms:RemoteWMSStoreId>zipcodes</wms:RemoteWMSStoreId>
</wms:RequestableLayer>
 
And that's that, you're done.

But imagine the remote WMS has a slow png implementation, and you want to always request the map as jpeg. Also, transformation is broken and you want to let deegree always reproject the raster image. That's easy:

<RemoteWMSStore
xmlns="http://www.deegree.org/datasource/remoteows/wms"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.deegree.org/datasource/remoteows/wms remotewms.xsd"
configVersion="3.1.0">
  <CapabilitiesDocumentLocation
    location="http://deegree3-testing.deegree.org/deegree-utah-demo/services?request=capabilities&amp;service=WMS&amp;version=1.1.1" />
  <DefaultRequestOptions>
    <ImageFormat transparent="false">image/jpeg</ImageFormat>
<DefaultCRS useAlways="true">EPSG:4326</DefaultCRS>
  </DefaultRequestOptions>
  <RequestedLayer>
    <Name>ZipCodes</Name>
  </RequestedLayer>
</RemoteWMSStore>

Please note that deegree will automatically reproject anyway, if the requested crs is not available in the remote WMS. Setting useAlways to true as in the example will force reprojection even for crs that the remote WMS claims to know.

The configuration you've seen so far is a kind of configuration where you tell deegree what you want to achieve on a high level. That's pretty nice, but sometimes one wants control on a lower level. Imagine you want to send a vendor specific parameter for all requests.

Let's say you really like red backgrounds:

<DefaultRequestOptions>
    <ImageFormat transparent="false">image/jpeg</ImageFormat>
    <Parameter use="allowOverride" scope="GetMap" name="bgcolor">0xff0000</Parameter>
</DefaultRequestOptions>
 
In this case, you'll have a red background as default, and the user can override it using BGCOLOR in a GetMap request. If the use-attribute is set to fixed, the parameter's value will always be used. The scope-attribute can be set to GetMap, GetFeatureInfo and All.

The DefaultRequestOptions block can also occur within a RequestedLayer section (named RequestOptions). In that case, the store will not request the layers in a single request, but will do multiple requests and combine them into a single map. That enables you to combine layers with different options, such as when two layers should be requested in different crs.

Last but not least, some WMS need authentication. Currently, only HTTP Basic is supported:

<HTTPBasicAuthentication>
<Username>myuser</Username>
<Password>s3cret</Password>
</HTTPBasicAuthentication> 
 
Another note on why cascading might be useful is WMS implementations that don't support proper GML output for featureinfo. deegree will read in the broken format and serve proper GML. There are currently workarounds for ArcIMS and MyWMS, but others should be easy to add. Just drop me a note. 

I hope that provides some insight on how to use deegree for cascading WMS. For a couple of examples you can check out the deegree-wms-remoteows-test module.

Edit: sorry for the crappy XML formatting, I'm still trying to figure out how to use blogger...

Thursday, July 14, 2011

deegree Layers

So Bolsena is over, and it's time to get back to other issues. That is a little unfortunate, as the atmosphere was really nice and productive, but hey, the Code Sprint will return in 2012.

So what's next? In this post I want to talk about something that has been on my mind for quite some time: the layer configuration in deegree. This might not be so interesting for most people, but writing it down definitely helps to sort my own mind.

People familiar with deegree 3.1's workspace concept might know about our resource oriented approach. In essence, everything you can configure is a resource, and corresponds to a single configuration file. So a JDBC connection is a resource, a shape file configuration is a resource, and a SQL feature store configured with all INSPIRE Annex I themes is a resource. A service configuration for a WMS or WFS is also a resource.

For WFS this is all very well, the configuration is mainly concerns service specific issues, such as whether transactions are enabled, which output formats are available and what protocol versions are switched on.

For WMS this also makes sense, you configure the layer structure, and tell the WMS which data source and which style(s) to use. A disadvantage is that configuration files grow real big for many layers. Another disadvantage is that often a single feature store is used for all layers, using a different feature type for each layer. Switching the store requires to change the ID in all layers.

From a REST like web service perspective, it is very easy to add another feature store, a single file needs to be PUT. To add another layer requires fetching the current config, modifying it and then POST it again, which is obviously not very comfortable.

And last but not least, having this integrated in the WMS config means that only the WMS can access the layers. From a service point of view this might not be too bad (after all, only the WMS currently uses layers), but one can easily imagine other use cases where layers as a resource make sense. Think of offline-updating a cache, or writing a Java based client.

So I think that's reason enough to think about a different approach. A hierarchical layer structure is easy enough in a file system. A directory is an unrequestable group layer without, a .xml file is a layer with content. A directory called layer with a file layer.xml right next to it is a group layer possibly also with its own content.

A single layer resource might not just produce a single layer, but also a single root layer with many children. The default WMS config might use all available layers, if it's not configured to use a specific root layer. That would enable people to produce layer files which properly correspond to a single feature store, not a single feature type. The trees could even be generated from the feature type hierarchy, which might be a useful starting point for many applications.

Well, it's still a lot of work, and would require quite a bit of refactoring on the WMS side. But I believe it would make the workspace just that bit more consistent. Implemented via SPI (like the rest of the resources) extending deegree WMS with custom layers would become a walk in the park.

Wednesday, July 6, 2011

Setting up eclipse using maven

A popular topic for developers is always what development platform to use and how to set it up. Once a project becomes bigger and more people collaborate the project management often comes up with best practices, such as how code should be formatted and so on. In this post I'll try to describe how to set up a deegree development environment in eclipse quick and easy.

Here's a quick list on things you need to do:
  • download and install a recent eclipse
  • in eclipse, go to Window/Preferences/Java/Build Path/Classpath variables, and add M2_REPO, have it point to your local maven repository (usually $HOME/.m2/repository/). You can do a mvn -Declipse.workspace=/home/user/workspace eclipse:configure-workspace instead of configuring it manually if you want.
  • check out deegree trunk from https://svn.wald.intevation.org/svn/deegree/deegree3/trunk/. You can check it out into your eclipse workspace, we can import the projects properly later on.
  • run the maven eclipse plugin plus deegree maven plugin in the trunk folder using mvn -Declipse.workspace=/home/user/workspace -DdownloadSources=true -DdownloadJavadocs=true -Dwtpversion=2.0 eclipse:clean eclipse:eclipse deegree:create-links -Declipse.formatter=deegree (all in one line please)
  • use File -> Import -> General -> Existing Projects  into Workspace in eclipse to import all projects at once. Choose the directory with your deegree checkout as a starting point to scan for projects.
Now you should be good to go to hack deegree.

I like it when things just work, but in case they don't I usually want to know what exactly happens on my computer. So let me explain what happens when you run the maven plugins.

The Maven Eclipse plugin generates the .project, .settings and .classpath files/folders for you. It adds the correct source/resource folders in eclipse, output folder, project/library dependencies and even generated source folders (such as the ones generated by the jaxb plugin).

That's already neat. But we currently have 86 maven projects in a hierarchical project tree, and eclipse wants to have the projects flat in its workspace. That's where the deegree maven plugin can help. On Linux it automatically symlinks the projects to the eclipse workspace folder, although this is not strictly necessary any more (eclipse projects need not be directly in the workspace).

Now getting back to management telling you how to format your code. It's not an option to manually set the formatter for 86 projects. Other projects may require a different formatter, so just setting the default is also not an option. Version 1.4 of the deegree maven plugin let's you specify a formatter using -Declipse.formatter=xxx (as shown above). Currently deegree and eclipse120 are supported, with deegree being our custom code style and eclipse120 being the standard eclipse formatter modified to allow 120 characters per line (instead of the default 80).

So that's it. Please tell if something does not work out as expected.

By the way, the deegree maven plugin can do more stuff. It has some helpers for web service integration testing, deegree workspace management and other utilities. There is a wiki page describing some of it, but it's currently outdated, only some of the functionality is documented.

Edit: Added a few missing steps to get everything to compile properly (Lombok and M2_REPO). Thanks Martin!

Edit: Updated the post with a couple of new things we learned. Also Lombok is not needed any more.