High Level Technical Architecture

The story

Using an FTP program, Walter uploads a new expert range map for macaca_fuscata into the /ftp/new/animalia/species directory on the MOL backend server (a Linux node at Colorado running an Nginx web server and the Pylons web framework):

macaca_fuscata.dbf  
macaca_fuscata.sbn  
macaca_fuscata.shp      
macaca_fuscata.shx
macaca_fuscata.prj  
macaca_fuscata.sbx  
macaca_fuscata.shp.xml  
macaca_fuscata.xml

Meanwhile, the MOL frontend server (a node running Python in the Google App Engine cloud) is continually using the backend server Scan API REST implementation to scan for uploaded range maps. Since Walter just uploaded a range map, the scan gets a hit. The backend server creates a new task to process the range map and adds it to a FIFO queue. When the task runs, the following happens:

1. Verify range map data
2. Generate and store PNG range map tiles for first 5 zoom levels
3. Register range map metadata with frontend server using Layer API
4. Archive range map data
5. Remove range map data from the upload directory

The Layers API in step 3 allows the backend server to register metadata for Walter's new range map with the frontend server. These metadata are uniquely identified by a key named animalia/species/macaca_fuscata which happens to be the same key that uniquely identifies the corresponding Catalogue of Life species record stored by the frontend server. This is the essential link that allows us to search the Internal Taxonomy and then discover range maps associated with the search results.

The Layers API also allows the backend server to register any errors that occur while processing the range map. Fortunately though, Walter's new range map (and the backend server implementation) is perfect. The task completes successfully, the range map is registered with the frontend server, and map tiles are ready to serve.

Map tile architecture: present and planned

The current MOL prototype processes every SHP file to 5 zoom levels. Tiling is handled using Mapnik within the python app handled in the backend server. In the next phase of development, the window of tiling will be modified to reflect range sizes. For example, a very small range may be tiled for zoom levels 4,5,6,7,8 while a very large range will be tiled for zoom level 0,1,2,3,4.

When a user requests a map for an existing species, the javascript client goes through a 4 step process to load a layer,

1. Request layer metadata from the frontend server.
2. Use the known bounding box of the range to limit the tiles it requests.
3. Make tile requests to the frontend server
4. The frontend server forwards the client to the hard copy of the tile stored on the remote server

For the purposes of the prototype, we sacrificed some speed in tile requests and every tile request is forwarded directly to the backend server where tiles are permanently stored. We have planned a more distributed Tile Request Cascade for future development of MOL. The TRC will enable MOL to respond quickly to frequently requested tiles, while maximizing storage efficiency, and maintaining up to date tile sets in parallel with up to date SHP and other geometric formats.