Difference between revisions of "Overview of the system"

From CryoClim

(System architecture and interoperability)
(The portal)
Line 57: Line 57:
Here is an example of what it looks like in the Flash demonstrator of the user interface (from Phase 1 of the project):
Here is an example of what it looks like in the Flash demonstrator of the user interface (from Phase 1 of the project):  
[[Image:Flash-demo.jpg|center|700px|Screenshot from the Flash demonstrator of the user interface from Phase 1 of the project.]]<br>The user may be interested in doing a time series analysis of selected products. Thus, it must be easy to select a type of product and the first and last to define the time series to include.  
[[Image:data_portal.png|center|700px|Screenshot from the Flash demonstrator of the user interface from Phase 1 of the project.]]<br>The user may be interested in doing a time series analysis of selected products. Thus, it must be easy to select a type of product and the first and last to define the time series to include.  
<br>Metadata is recorded for each individual data product, and can be viewed. Each individual data product in the list or tree view can be clicked on, and the associated metadata is displayed.  
<br>Metadata is recorded for each individual data product, and can be viewed. Each individual data product in the list or tree view can be clicked on, and the associated metadata is displayed.  

Revision as of 17:11, 6 November 2013

System architecture and interoperability

The CryoClim system is a distributed system involving production chains located at several institutions. As such, the CryoClim system is a system of systems and the focal point of the system architecture has been to set up interoperability principles that support the distributed idea of CryoClim as well as the interoperability of the CryoClim system within a global environment as defined by, e.g., Global Earth Observation System of Systems (GEOSS), WMO Information System (WIS) and Infrastructure for Spatial Information in the European Community (INSPIRE) principles.

The architecture of the system (see figure) is fully decentralised and relies on Service Oriented Architecture (SOA) concepts and utilises web services to achieve the service orientation.


A basic design based upon SOA is in line with INSPIRE, WIS and GEOSS requirements and will ensure a potential for future development and addition of web-service interfaces as they become mature enough. Basically, SOA implies that the communication between the data user and the data provider is handled through services that may be used both interactively and as machine-readable interfaces.

The CryoClim production chains have implemented common standards in the following areas:

  1. Metadata
  2. Quality information
  3. File formats
  4. Map projections
  5. Data access interfaces

To ensure compatibility with upcoming systems/requirements (e.g., GEOSS, WIS and INSPIRE) standard interfaces have been utilised. This implies that each production chain within the system has published data and products using OpenGeoSpatial Consortium (OGC) interfaces. Metadata is published using Open Archives Initiative - Protocol for Metadata Harvesting (OAI-PMH). Internally a number of standards are used, externally metadata is exposed as Dublin Core, GCMD DIF or ISO19115. Data are available through a number of interfaces. Map oriented data can be viewed using OGC Web Map Service (WMS), access to data is provided using raw HTTP or Open-source Project for a Network Data Access Protocol (OpeNDAP).

Thematic Realtime Environmental Distributed Data Services (THREDDS) Data Server is used by some data providers to establish a number of interfaces to datasets served. THREDDS supports both OpeNDAP, Hypertext Transfer Protocol (HTTP), WMS and WCS access and provides a cost effective solution to establish interoperability at the dataset level.

The portal

The CryoClim products will likely be available through several portals, in particular the GEOSS portals. Also, there will be a particular portal hosted by the CryoClim consortium making CryoClim products available.

The purpose of the CryoClim portal is to allow an interactive user to do the following:

  • Log on to the service.
  • Find out what data products are available and view metadata.
  • View one or more data products.
  • Order data products.

The portal will have a number of modes. Before the user has logged on, the service is in the logon mode. When the user has logged on, the service will be in the search mode. The user may switch between this mode and the following other modes at will: metadata view, map preview, indicator graph, download, product information and general service information.

One section of the user interface will allow users to discover what data is available. The user enters the search criteria in a fixed format, and the individual data products matching the search criteria are listed.

Here is an example of what it looks like in the Flash demonstrator of the user interface (from Phase 1 of the project):

Screenshot from the Flash demonstrator of the user interface from Phase 1 of the project.

The user may be interested in doing a time series analysis of selected products. Thus, it must be easy to select a type of product and the first and last to define the time series to include.

Metadata is recorded for each individual data product, and can be viewed. Each individual data product in the list or tree view can be clicked on, and the associated metadata is displayed.

The ultimate goal for a user of the portal is to download data products for further analysis. In order to understand which data to actually download, it is practical to get a preview of data in a map viewer in the portal. All data can be viewed in any of the supported projections. The user may switch between different map projections.

The user may want to zoom in and out, and move around when zoomed in. Although the user may want to order many products, it may be impractical to view all at the same time. It must be possible to switch individual “layers” on and off, in order to see only one, a few, all, or none of the layers. This includes combining various product types, as viewing products for sea ice, snow, and glaciers at the same time. Also, additional background map layers may be switched on and off. Background layers typically include coastline, ocean area, land area, grid, lakes and rivers, national boundaries and administrative boundaries within a nation.

In the case that the user has selected a time series of data, it could be desirable to view the data as an animation. Some data products may include indicator variables, which may be viewed as a single measurement value as a function of time.
Different users may wish to download products in different ways, and the following methods will be supported:

  1. Some users prefer to download Network Common Data Form (NetCDF) files from an File Transfer Protocol (FTP) site, possibly with a temporary user name and password.
  2. Alternatively, the system may provide an HTTP link to a file with the download information, which can then be read by a small script to download the files.
  3. A subscription service to facilitate download of data at regular intervals as new data becomes available.

Preferably, all three modes could be supported by returning an Extensible Markup Language (XML) file from the server to the client, from which a manual or automatic download can be carried out.

Production chain for global sea ice products

The sea ice products in the CryoClim service, sea ice concentration (SIC) and sea ice edge (SIE), are based upon Ocean and Sea Ice Satellite Application Facility (OSI SAF) re-analysis. The Scanning Multichannel Microwave Radiometer (SMMR) and Special Sensor Microwave/Imager (SSM/I) brightness temperatures are corrected for contamination arising from atmospheric water vapour content and wind roughening of the open water. The correction is computed using a radiative transfer model and atmospheric input data from European Centre for Medium-Range Weather Forecasts (ECMWF) re-analysis.

Sample of average sea ice concentration (upper left), standard deviation in sea ice concentration (upper right), second minimum sea ice concentration (lower left) and second maximum sea ice concentration (lower right) - all shown in % - taken from the October 2000 monthly aggregated sea ice concentration file.

The OSI SAF ice concentration algorithm development is based on testing and evaluation of a number of established algorithms. Analysis of atmospheric sensitivity showed that the Bootstrap frequency mode algorithm (Andersen S., Evaluation of SSM/I sea ice algorithms for use in the SAF on Ocean and Sea Ice, DMI Scientific Report 00-10, Danish Meteorological Institute, Copenhagen, 2000) had the lowest sensitivity to atmospheric noise over open water. Conversely, comparison to high-resolution SAR imagery revealed that of the algorithms using the low-frequency channels (i.e., below 85 GHz), the Bristol algorithm (Andersen et al., Intercomparison of passive microwave sea ice concentration retrievals over the high concentration Arctic sea ice, JGR, vol. 112, 2007) gave the best agreement. Consequently a hybrid algorithm (Breivik et al., Sea Ice Products for EUMETSAT Satellite Application Facility, Canadian Journal of Remote Sensing, 27(5), 2001) has been established as a smooth combination of two of the tested algorithms, the Bristol algorithm and the Bootstrap frequency mode algorithm. To ensure an optimum performance over both marginal and consolidated ice, the Bristol algorithm is given little weight at low concentrations, while the opposite is the case over high ice concentrations. 

In order to achieve unambiguous estimates it is necessary to provide typical emissivities, commonly referred to as tie-points, of the pure type surfaces, i.e., first-year ice, multi-year ice and open water. To ensure stable performance and time/climate consistent results, a new method to estimate dynamic tie points has been developed and implemented. In this method new tie points are calculated based on the last two days of data. 

The added value of the CryoClim service to the OSI SAF products is the combination of products into aggregated climate products, adding standardised quality information to each product and providing sea ice products that are consistent with the other products delivered by CryoClim.

The sea ice production chain will run at the Norwegian Meteorological Institute (MET Norway). The production environment is based upon open standards and open source software. This concept has proven to be cost effective and reliable. Systems for data/product distribution and presentation have been and are being developed in various projects. These are now being harmonised to reduce costs of operation. Public services offered include OGC WMS for visual presentations, web services to the climate database and RESTful services to various data sets and products. No black box applications are allowed within the MET Norway production chain.

An important aspect for MET Norway when setting up various services is that new development should be available for use to fulfil the public commitment of the institute and it should not object to the open data policy of the institute.

Production chain for global snow products

The planned snow sub-service will provide various products for snow cover extent (SCE; binary snow/no-snow products) and snow cover fraction (SCF; percentage of snow cover). This sub-service will be developed by applying a similar production chain as for sea ice and by adapting algorithms and algorithm experience from other projects. The longest time series of historical snow products, all of 10 km spatial resolution, will be based on passive microwave data from SMMR (from 1978) and SSM/I (from 1987). An optical product (of 1 km spatial resolution) is also planned, based upon the NOAA AVHRR satellite/sensor series. A problem with this sensor is that there is no on-orbit calibration. An initiative is taken by the Committee of Earth Observation Satellites (CEOS) through Climate Action T-3 for calibration and cross-calibration of the whole global time series of AVHRR data. NOAA thinks that such a data time series (FCDR) will be available in 2-3 years. This would then be an extremely valuable time series of observations, from the beginning of the 1980s, for measuring high-resolution, accurate global, regional and local changes of the snow cover. Several algorithms are available for retrieval of the snow cover from optical data, and the Norwegian Computing Center (NR) has developed some of them and has experience with most of the algorithms.


The most advanced snow cover product envisioned for the snow sub-service is the multi-sensor time-series product. The idea here is to fuse passive microwave data with optical data through a model for retrieval of the snow cover variable at much higher quality and resolution than is possible with passive microwaves solely. NR has developed such a method combining optical and SAR data. A similar approach can be applied for optical and Passive Microwave Radiometer (PMR) data. Such an approach will, hopefully, solve the most significant deficits of snow retrieval algorithms for passive microwave data (often related to wet snow and mountainous terrain). The product will be a novel contribution to global snow monitoring as there is no such high-resolution product available today covering several decades, and no product at all fusing optical and PMR data in a sound way.

The snow production chain will run in the METNO production environment (see Production chain for sea ice products).

Production chain for glacier products mainland Norway

Glacier products from mainland Norway in the CryoClim service consist of Glacier Area Outline (GAO), Glacier Lake Outlines (GLO), Glacier Periodic Photo series (GPP) and Climate Indicator prodicts. The GAO and GLO products are based on Landsat TM/ETM+ imagery using image analysis and GIS techniques. The GPP products are time series of glacier photographs (terrestrial and airborne imagery without geo-referencing) and are used to illustrate glacier changes for selected glaciers .

In a pilot study conducted by Andreassen et al. in 2008 in the Jotunheimen region the applicability of standard glacier mapping methods were tested using segmentation of ratio images computed from the raw digital numbers for Landsat TM. The results confirmed that the applied method was robust and highly accurate for extracting glacier outlines in the test area. The methods have been used to map GAP for all of Norway and to create a new glacier inventory (Andreassen and Winsvold, 2012).

Glacier lake outlines (GLO)-dammed lakes were mapped by using normalised difference water index and by map overlay with lakes from topographic digital maps.
The algorithms required to effectively extract glacial variables from satellite images depend upon illumination condition, landscape types and scene conditions. Manual adaption is needed to check the suitability of algorithms for each region and scene.


The product processing chain for glacier products covering mainland Norway will be hosted by the Norwegian Water Resources and Energy Directorate (NVE) and consists of several steps, going from finding satellite imagery to product delivery. The first step in the processing chain is to find suitable satellite imagery for glacier mapping. It is important that the imagery are cloud free and acquired at the end of the ablation season when little seasonal snow is remaining. Raw imagery have to be ortho-rectified, and imagery that are already ortho-rectified (e.g., delivered by Norsk Satellittdataarkiv) has to be validated. Glacier products are derived from the imagery using the developed algorithms. Since the terrain and atmospheric conditions will vary in almost every image, the algorithms need manual adaption to insure good products. Some manual editing of the GAO product was needed in areas with debris cover, lake interfaces or area in cast shadow. In this step there was also an on-the-fly validation.

Production chain for glacier products Svalbard

At Svalbard about 1500 glaciers have been defined with a total area at about 36,000 km2. To monitor these glaciers efficiently there is a need for methods and algorithms which can be run more or less automatically based on remote sensing data. It is known that most of the glaciers at Svalbard are "surging glaciers". This causes very nonlinear changes in mass balance depending on whether a glacier is in a surging period or not. The project will mainly focus on establishing the necessary algorithms and the production chain which can provide climate-related variables.

The glacier products available for Svalbard in the CryoClim service will be Glacier Surface Type (GST) and Glacier Balance Area (GBA). These products are produced with the use of different image classification techniques on C-band SAR images. The Norwegian Polar Institute (NPI), who are responsible for the Svalbard glacier products, have historical satellite data available in C band is from ERS-1/2, Envisat ASAR and Radasat-1/2.

SAR data can be used to map glacier extent and the distribution of different surface types. The interaction between the microwave radiation and the surface layers of the glaciers is diverse, causing different characteristics of the backscatter depending on the surface.

An automatic method for classification of the glacier surface based on texture and the amount of backscatter in SAR images has been developed. This algorithm is also using information from different polarisations to increase the classification accuracy, if available. The work is an extension of the work done earlier at NPI in the EuroClim project. This algorithm is used as baseline for the classification of the facies, but with the possibility for an operator to do manual adjustments of the result. The results look promising for the glaciers where field data is available for validation. A correlation between k-means classification and mass balance for some glaciers in Kongsfjorden at Svalbard has been found in EuroClim. This algorithm will be used for the GBA product.

The glacier production chain will run in the NPI production environment. The production environment is based upon a combination of proprietary and open source software. The strategy is to use open standards and open source software whenever it is possible. Systems for distribution and presentation of data are now under development.

The products will be stored in the local database NPI is using for science data. The distribution will be based on:

  • RESTful (HTTP/GET) interfaces
  • OpeNDAP data server
  • OAI-PMH for metadata exchange

The OpeNDAP data server and the machine interface based on RESTful technology have to be developed. The OAI-PMH metadata exchange service has been developed during the DOKIPY project (Data handling and coordination service for Norwegian IPY projects), and only smaller adaptations will be needed for using it in CryoClim.