Solutions for dealing with large numbers of features on a map

You can render points to the map, or use a mapserver or geoserver install which can do that, and then link to content items. However, to do this, you will need to save your points as a shapefile or import them into PostGIS. Obviously this is a tough solution. You can also create an OpenLayers layer with the Strategy.BBOX behavior and use the patched KML module with views arguments to only load points within the bounds of your map. Obviously this requires that the map should not display all points at the same time, so you may need to restrict zoom levels, which might impede usability of your map, depending.

So, the answer is yes, but not easily.

The WFS module may be of interest to you if you're still involved in this project.

I'm looking at much the same problem, but I'll have to resort to a few different methods of fixing this I think.

I already have about a thousand markers on the map, with the potential to add hundreds of markers in a zoom level below each one of those.

My plan is to create something like Google's Marker Manager, but for OpenLayers, which would allow me to do marker clustering in Javascript and hopefully improve performance. Using Google's marker manager, the map can comfortably handle thousands of markers assuming it's only actually showing around a hundred at a time. It filters by zoom level and bounds box so additional information would be needed by any displayed nodes for this to work in Drupal, but one stop at a time.

Additionally, I think I'd also need a server side clustering solution, or at least a zoom based filter. My idea is to have a marker which shows a location until you zoom right in, then it gets replaced by polygons to show detail, but that could lead to potentially hundreds of thousands of polygons over the whole map, so I'd need to only load them from the server for the current bounds at a certain zoom level.

Anyway, the reason I'm mentioning it here is to see if there is already a solution to this before I start working on it.

Cheers.

Martin...

As I was looking at that awesome site that you guys (@tmcw and development seed) built: http://haitiaidmap.org, I noticed just how much output gets put out in HTML.

So, do we want to support some sort of AJAX feature loading? Or, should we make it the responsibility of something like the WFS and KML module?

On a basic level, we could have a "use ajax" switch for features that calls the feature data from the server when loading. This will keep a lot of the output out of the HTML document and help with performance a little.

We could also take that idea a little farther and send through the bbox to only get the only the features that are needed. The OpenLayers bbox strategy doesn't seem like it could be all that efficient as it is doing all the processing in JS, but I don't have any research on this.

Overall, I understand that we don't want to create a huge AJAX system for features as there are plenty of more established methods, but I think there is an opportunity for a small amount of code to help performance.

Oh, and @darksnow, there are somethings for OpenLayers library that will help with this stuff. There is clustering, but this will not improve performance per-say as it still needs to load all the features to cluster. There is also the BBox strategy as well. Neither of these are actually implemented in our module at the moment, though it would be easy to create behaviors.

#805264: Add cluster behaviour

As I was looking at that awesome site that you guys (@tmcw and development seed) built: http://haitiaidmap.org, I noticed just how much output gets put out in HTML.

The site does have a pretty heavy load of markers. Moving them to the foot of the document reduces the perceived loading time of the page tremendously, though. One of the biggest performance improvements in 2.x, besides native json encoding.

So, do we want to support some sort of AJAX feature loading? Or, should we make it the responsibility of something like the WFS and KML module?

I think it's outside the sope of the module. Ideally the answer here is a module that tightly integrates with OpenLayers (simulates a layer) and provides BBOXed GeoJSON.

On a basic level, we could have a "use ajax" switch for features that calls the feature data from the server when loading. This will keep a lot of the output out of the HTML document and help with performance a little.

This isn't really doable - we don't provide a standard JSON format right now, just whatever's cooked up as the data scheme. An external module which provides GeoJSON would allow the OpenLayers parser to be used.

We could also take that idea a little farther and send through the bbox to only get the only the features that are needed. The OpenLayers bbox strategy doesn't seem like it could be all that efficient as it is doing all the processing in JS, but I don't have any research on this.

Huh? The BBOX strategy doesn't do processing in js, it adds parameters to the URL of the feature requests and thus requests a subset.

The KML module, combined with exposed arguments, can do this - and does it on FEBP (with OpenLayers 0.x). WFS will also have the capability soon.

Note that both of these methods require restrictions - a zoomed-out map with tons of pointers will still be slow. In that area, you'll need to do what Google does and rasterize points, something that the WFS module is working on.

Looking further at how things work for this module, I was thinking that adding a max and min zoom level to the layers created by the views module might help here.

If I have a map that's showing a hundred markers at a high zoom, but when you zoom in on each of those, there are another hundred under it, then for any given zoom level there will be a restricted set of markers actually on display.

Of course, if all layers that are in the map are loaded at once, then there is still a problem, but perhaps there's a way to remove markers from the map when they are out of the visible box. The details would still be in memory, but not in the map. I think this is how Google's marker manager works. It stores ALL markers and selectively adds and removes them from the map when you zoom and pan.

I also think an AJAX call would be a good idea, maybe supported by the bbox code mentioned above, which would allow a view, with exposed arguments, to selectively filter what it's getting when a user zooms and pans. I'm not familiar enough with OpenLayers, yet, to say whether this would be best implemented as a layer using the existing views system, which I like, it makes sense to me, or if there is something more complicated needed for WFS or KML.

My requirement is for vectors though, so I can get hover and click behaviours.

FYI, I am working on a solution with server-side clustering:

* source data (points) is in the form of Drupal nodes
* a new Drupal module maintains a DB table of multi-level geographic clusters, using hook_nodeapi() to track changes to source nodes and update the clusters table accordingly. Clusters table is accessible via a Views plugin
* WFS module accesses clusters view and serves out both clusters and source nodes, with clustering level and bbox as request parameters
* a new OpenLayers Strategy or layer type displays the map, selecting a suitable clustering level based on zoom and/or number of points in map view area

My scalability target for this is 1M points. Rasterising the markers using GeoServer would not be ideal for me because I want to have features data accessible in Javascript. WFS VirtualHover/VirtualClick alone will not suffice for me because when zoomed-out they would transfer all 1M points to browser.

Hopefully I can build this in a general way and contribute the modules to the public, if others find this approach useful.

The discussion in this issue more closely resembles a support request than a bug so I'm re-purposing it to avoid creating another issue on the same topic.

I am displaying lines on an openlayers view using the openlayers WKT field for data.
I also plan to pull feature information off the js map object for other use during run time with some external js (which may limit my options).

My options are:
1. Let all lines load at once at the start and take the loading time hit, which is not ideal.
2. Use the Strategy.BBOX behavior to only load what is required. I'm still not researched enough to fully understand how that process works in the background.
3. Use the WFS module to pull the data from my nodes (noting that the WFS module still needs work in regards to parsing WKT fields) and then display data from a geoserver.

In regards to option 2:

tmcw wrote in #1:

You can also create an OpenLayers layer with the Strategy.BBOX behavior and use the patched KML module with views arguments to only load points within the bounds of your map.

I'm not sure I understand the use of the KML module. Can anyone clarify the role of the KML module in this setup?

In regards to option 3:

I understand the data getting from drupal to geoserver but I'm not clear on displaying the data from geoserver on the map.
If it provides a layer that I can add to my openlayers map does that mean the layer contains all the data or is there some way can it be filtered using views exposed filters like I am doing now with my standard openlayers setup?

Would it still be possible in both these options to use js to pull feature data from the map object or would that possibility be lost?

Both KML and WFS are ways of dynamically fetch structured features data from server; there is no hard reason to use one over the other. In my solution at http://www.letsdoitworld.org/map/global-waste-map , I am actually using CSV instead of WFS/KML - the fetched data size (which is one of the main limiters if you are concerned about load time) is 20x smaller.

2. Strategy.BBOX does really work, I am using it on the abovementioned map. There are some downsides though:

• It does not help scalability at all if the user zooms out the map so that whole world is visible. Then Strategy.BBOX just fetches all features from server, which takes as much time as your initial page load hit when you are pre-loading everything. To solve that, I am combining Strategy.BBOX with the server-side clustering I described in post #8. This is cumbersome, but most other solutions would not work at the 100,000+ features scale I am designing for.
• You need to do some JavaScript and PHP programming to setup Strategy.BBOX (it is not doable by just clicking around in Drupal-OpenLayers admin interface)

3. I don't know geoserver, but I think this approach means that geoserver outputs rasterized features layer for your web display. For the user, this has the same scalability regardless of number of features. The downside is that the browser does not have structured data for features, which makes it difficult to do mouse pointer changes and other interactive elements when hovering over a feature. When the user clicks on the feature, you can do a server roundtrip to fetch structured data, but for hovering it is usually not practical.

Thanks for the quick reply.

2. Yeah, I have allowed for having to code the Strategy.BBOX integration. Due to my requirements I will have to do some coding regardless of the method I go with. I have done custom openlayers behaviours and layers before so this shouldn't be an issue.
Loading all feature when zoomed out should be acceptable too

3. The WFS module says that it provides
* VirtualVector layer type
* VirtualHover behavior
* VirtualClick behavior

So I assume (I know I shouldn't assume things but it's so easy :)) that will cover the issues you mentioned. The only thing I'm not sure about is what kind of data the virtual layer contains and if I can get to the individual feature data via javascript like I can normally with openlayers, it's quite possible that data will not be there. I can always just set it all up and try it out, it just means spending a bit more time.

I notice there is also a patch for the KML module to use WKT fields for data too, which would make things easier for me.

@ahtih:

The clustering module you mentioned in #8 sounds very interesting. Hopefully it turns out to be non-specific enough to be released.

What about OpenLayers.Strategy.Cluster?
Was that not a viable solution for you?
Is your server side clustering much faster?

My clustering module is at https://github.com/ahtih/Geoclustering , you can try using it, and give me feedback too. It should be ready to use for other people besides me, but it is beta quality and there are some loose ends/todos.

OpenLayers.Strategy.Cluster solves a completely different problem: it clusters features in browser (Javascript) to reduce bottleneck in displaying 500+ markers. So it still transfers all features/markers from server to browser, i.e. it is suitable only up to perhaps 5000 features, otherwise the transfer takes too long. Thus it does not solve my problem (having 100 000+ features) at all.

Closing as this issue is outside of this module, especially given that development capacity has diminished significantly on this module. This is a tough problem to solve and there are many different ways. But any solution could hopefully live outside of this module.

Closing ticket. Geoclustering looks interesting, though, I might have to look at something like that with Geofield.