Overview of the system

From CryoClim

Revision as of 13:40, 20 December 2017 by Solveighw (talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

System architecture and interoperability

The CryoClim system is a distributed system involving production chains located at several institutions. As such, the CryoClim system is a system of systems and the focal point of the system architecture has been to set up interoperability principles that support the distributed idea of CryoClim as well as the interoperability of the CryoClim system within a global environment as defined by, e.g., Global Earth Observation System of Systems (GEOSS), WMO Information System (WIS) and Infrastructure for Spatial Information in the European Community (INSPIRE) principles.

The architecture of the system (see figure) is fully decentralised and relies on Service Oriented Architecture (SOA) concepts and utilises web services to achieve the service orientation.


A basic design based upon SOA is in line with INSPIRE, WIS and GEOSS requirements and will ensure a potential for future development and addition of web-service interfaces as they become mature enough. Basically, SOA implies that the communication between the data user and the data provider is handled through services that may be used both interactively and as machine-readable interfaces.

The CryoClim production chains have implemented common standards in the following areas:

  1. Metadata
  2. Quality information
  3. File formats
  4. Map projections
  5. Data access interfaces

To ensure compatibility with upcoming systems/requirements (e.g., GEOSS, WIS and INSPIRE) standard interfaces have been utilised. This implies that each production chain within the system has published data and products using OpenGeoSpatial Consortium (OGC) interfaces. Metadata is published using Open Archives Initiative - Protocol for Metadata Harvesting (OAI-PMH). Internally a number of standards are used, externally metadata is exposed as Dublin Core, GCMD DIF or ISO19115. Data are available through a number of interfaces. Map oriented data can be viewed using OGC Web Map Service (WMS), access to data is provided using raw HTTP or Open-source Project for a Network Data Access Protocol (OpeNDAP).

Thematic Realtime Environmental Distributed Data Services (THREDDS) Data Server is used by some data providers to establish a number of interfaces to datasets served. THREDDS supports both OpeNDAP, Hypertext Transfer Protocol (HTTP), WMS and WCS access and provides a cost effective solution to establish interoperability at the dataset level.

The portal

The CryoClim products will be available through several portals, in particular the GEOSS portals. Also, there is a particular portal hosted by the CryoClim consortium making CryoClim products available.

The purpose of the CryoClim portal is to allow an interactive user to do the following:

  • Find out what data products are available and view metadata.
  • View one or more data products.
  • Download data products.

Screenshot from the Flash demonstrator of the user interface from Phase 1 of the project.

The user may be interested in doing a time series analysis. Thus, it must be easy to select a type of product and the first and last to define the time series to include.

Metadata is recorded for each individual data product, and can be viewed. Each individual data product in the list or tree view can be clicked on, and the associated metadata is displayed.

The ultimate goal for a user of the portal is to download data products for further analysis. In order to understand which data to actually download, it is practical to get a preview of data in a map viewer in the portal. All data can be viewed in any of the supported projections. The user may switch between different map projections.

The user may want to zoom in and out, and move around when zoomed in. Although the user may want to order many products, it may be impractical to view all at the same time. It must be possible to switch individual “layers” on and off, in order to see only one, a few, all, or none of the layers. This includes combining various product types, as viewing products for sea ice, snow, and glaciers at the same time. Also, additional background map layers may be switched on and off. Background layers typically include coastline, ocean area, land area, grid, lakes and rivers, national boundaries and administrative boundaries within a nation.

In the case that the user has selected a time series of data, it could be desirable to view the data as an animation. Some data products may include indicator variables, which may be viewed as a single measurement value as a function of time.
Different users may wish to download products in different ways, and the following methods will be supported:

  1. Some users prefer to download Network Common Data Form (NetCDF) files from an File Transfer Protocol (FTP) site, possibly with a temporary user name and password.
  2. Alternatively, the system may provide an HTTP link to a file with the download information, which can then be read by a small script to download the files.
  3. A subscription service to facilitate download of data at regular intervals as new data becomes available.

Preferably, all three modes could be supported by returning an Extensible Markup Language (XML) file from the server to the client, from which a manual or automatic download can be carried out.

Production chain for global sea ice products

The sea ice products in the CryoClim service, sea ice concentration (SIC) and sea ice edge (SIE), are based upon Ocean and Sea Ice Satellite Application Facility
Sample of average sea ice concentration (upper left), standard deviation in sea ice concentration (upper right), second minimum sea ice concentration (lower left) and second maximum sea ice concentration (lower right) - all shown in % - taken from the October 2000 monthly aggregated sea ice concentration file.
(OSI SAF) re-analysis. The Scanning Multichannel Microwave Radiometer (SMMR) and Special Sensor Microwave/Imager (SSM/I) brightness temperatures are corrected for contamination arising from atmospheric water vapour content and wind roughening of the open water. The correction is computed using a radiative transfer model and atmospheric input data from European Centre for Medium-Range Weather Forecasts (ECMWF) re-analysis.

The OSI SAF ice concentration algorithm development is based on testing and evaluation of a number of established algorithms. Analysis of atmospheric sensitivity showed that the Bootstrap frequency mode algorithm (Andersen S., Evaluation of SSM/I sea ice algorithms for use in the SAF on Ocean and Sea Ice, DMI Scientific Report 00-10, Danish Meteorological Institute, Copenhagen, 2000) had the lowest sensitivity to atmospheric noise over open water. Conversely, comparison to high-resolution SAR imagery revealed that of the algorithms using the low-frequency channels (i.e., below 85 GHz), the Bristol algorithm (Andersen et al., Intercomparison of passive microwave sea ice concentration retrievals over the high concentration Arctic sea ice, JGR, vol. 112, 2007) gave the best agreement. Consequently a hybrid algorithm (Breivik et al., Sea Ice Products for EUMETSAT Satellite Application Facility, Canadian Journal of Remote Sensing, 27(5), 2001) has been established as a smooth combination of two of the tested algorithms, the Bristol algorithm and the Bootstrap frequency mode algorithm. To ensure an optimum performance over both marginal and consolidated ice, the Bristol algorithm is given little weight at low concentrations, while the opposite is the case over high ice concentrations. 

In order to achieve unambiguous estimates it is necessary to provide typical emissivities, commonly referred to as tie-points, of the pure type surfaces, i.e., first-year ice, multi-year ice and open water. To ensure stable performance and time/climate consistent results, a new method to estimate dynamic tie points has been developed and implemented. In this method new tie points are calculated based on the last two days of data. 

The added value of the CryoClim service to the OSI SAF products is the combination of products into aggregated climate products, adding standardised quality information to each product and providing sea ice products that are consistent with the other products delivered by CryoClim.

The sea ice production chain will run at the Norwegian Meteorological Institute (MET Norway). The production environment is based upon open standards and open source software. This concept has proven to be cost effective and reliable. Systems for data/product distribution and presentation have been and are being developed in various projects. These are now being harmonised to reduce costs of operation. Public services offered include OGC WMS for visual presentations, web services to the climate database and RESTful services to various data sets and products. No black box applications are allowed within the MET Norway production chain.

An important aspect for MET Norway when setting up various services is that new development should be available for use to fulfil the public commitment of the institute and it should not object to the open data policy of the institute.

Production chain for global snow products

The snow sub-service provides a snow cover extent (SCE; binary snow/no-snow) product based on a multi-sensor/multi-temporal algorithm. The algorithm performs a data fusion applying a Hidden Markov Model (HMM) combining data from optical and passive microwave radiometers (PMR). The optical component is based on NOAA AVHRR GAC starting from 1982. The PMR data are from the SMMR sensor from starting 1982 and then followed by SSM/I from 1987. The products are made available as netCDF CF files.

One of the ideas with the algorithm is to fuse PMR data with optical data through a model obtaining a product at much higher quality and resolution than is possible with PMR alone, and much better spatio-temporal coverage than with optical alone.


Production chain for glacier products mainland Norway

Glacier products from mainland Norway in the CryoClim service consist of Glacier Area Outline (GAO), Glacier Lake Outlines (GLO), Glacier Periodic Photo series (GPP) and Climate Indicator products. The GAO and GLO products are based on Sentinel-2 MSI and Landsat OLI/ETM+ imagery using image analysis and GIS techniques. The GPP products are time series of glacier photographs (terrestrial and airborne imagery without geo-referencing) and are used to illustrate glacier changes for selected glaciers.

In a pilot study conducted by Andreassen et al. in 2008 in the Jotunheimen region the applicability of standard glacier mapping methods were tested using segmentation of ratio images computed from the raw digital numbers for Landsat TM. The results confirmed that the applied method was robust and highly accurate for extracting glacier outlines in the test area. The methods have been used to map GAO for all of Norway and to create glacier inventories (Andreassen and Winsvold, 2012; Winsvold et al., 2014). Future work in the Copernicus glacier service (https://www.nve.no/hydrology/glaciers/copernicus-glacier-service/) includes mapping of GAO and GLO using both Landsat 8 OLI and Sentinel-2 MSI. Higher spatial resolution on Sentinel-2 images (10-20 m) results in a more accurate glacier mapping compared to earlier satellite images, but the mapping work can be more tedious for the interpreter (Paul et al., 2016).

Glacier lake outlines (GLO)-dammed lakes were mapped by using normalized difference water index and by map overlay with lakes from topographic digital maps.
The algorithms required to effectively extract glacial variables from satellite images depend upon illumination condition, landscape types and scene conditions. Manual adaption is needed to check the suitability of algorithms for each region and scene.


The product processing chain for glacier products covering mainland Norway will be hosted by the Norwegian Water Resources and Energy Directorate (NVE) and consists of several steps, going from finding satellite imagery to product delivery. The first step in the processing chain is to find suitable satellite imagery for glacier mapping. It is important that the imagery are cloud free and acquired at the end of the ablation season when little seasonal snow is remaining. Raw imagery have to be ortho-rectified, and imagery that are already ortho-rectified (e.g., delivered by Norsk Satellittdataarkiv) has to be validated. Glacier products are derived from the imagery using the developed algorithms. Since the terrain and atmospheric conditions will vary in almost every image, the algorithms need manual adaption to insure good products. Some manual editing of the GAO product was needed in areas with debris cover, lake interfaces or area in cast shadow. In this step there was also an on-the-fly validation.

Production chain for glacier products Svalbard

At Svalbard about 1500 glaciers have been defined with a total area at about 36,000 km2. To monitor these glaciers efficiently there is a need for methods and algorithms which can be run more or less automatically based on remote sensing data. It is known that most of the glaciers at Svalbard are "surging glaciers". This causes very nonlinear changes in mass balance depending on whether a glacier is in a surging period or not. The project will mainly focus on establishing the necessary algorithms and the production chain which can provide climate-related variables.

The glacier products available for Svalbard in the CryoClim service will be Glacier Surface Type (GST) and Glacier Balance Area (GBA). These products are produced with the use of different image classification techniques on C-band SAR images. The Norwegian Polar Institute (NPI), who are responsible for the Svalbard glacier products, have historical satellite data available in C band is from ERS-1/2, Envisat ASAR and Radasat-1/2.

SAR data can be used to map glacier extent and the distribution of different surface types. The interaction between the microwave radiation and the surface layers of the glaciers is diverse, causing different characteristics of the backscatter depending on the surface.

An automatic method for classification of the glacier surface based on texture and the amount of backscatter in SAR images has been developed. This algorithm is also using information from different polarisations to increase the classification accuracy, if available. The work is an extension of the work done earlier at NPI in the EuroClim project. This algorithm is used as baseline for the classification of the facies, but with the possibility for an operator to do manual adjustments of the result. The results look promising for the glaciers where field data is available for validation. A correlation between k-means classification and mass balance for some glaciers in Kongsfjorden at Svalbard has been found in EuroClim. This algorithm will be used for the GBA product.

The glacier production chain will run in the NPI production environment. The production environment is based upon a combination of proprietary and open source software. The strategy is to use open standards and open source software whenever it is possible. Systems for distribution and presentation of data are now under development.

The products will be stored in the local database NPI is using for science data. The distribution will be based on:

  • RESTful (HTTP/GET) interfaces
  • OpeNDAP data server
  • OAI-PMH for metadata exchange

The OpeNDAP data server and the machine interface based on RESTful technology have to be developed. The OAI-PMH metadata exchange service has been developed during the DOKIPY project (Data handling and coordination service for Norwegian IPY projects), and only smaller adaptations will be needed for using it in CryoClim.


Production chain for Greenland ice sheet products


The production chain for the Greenland ice sheet product is summarized in the figure to the right. Daily images during hours from 13:00 to 17:00 UTC (11:00 to 15:00 Nuuk local time) from April to September are downloaded. This typically results in four to eight scenes every day covering parts or the whole Greenlandic region. First, the geolocation product MOD03 is applied to each scene. The MOD35 L2 cloud mask is then applied and cloudy pixels within the study area are classified as “cloud data”. A geographical land/sea mask is used to make sure that the result only includes pixels on the GrIS to distinguish between ice sheet and bare soil in the marginal areas. All scenes for each day are combined into the daily product covering the entire Greenland area. Each pixel from the merged scenes are evaluated and represented by the most likely classification value. The monthly GST composite images are generated from the most frequently occurring ice or snow class for each pixel for each month. This allows exclusion of pixels with cloud cover and provides the best estimate for each pixel during that month. This estimate varies in quality, and for every daily mosaic in which a pixel is classified as a cloud pixel, the less reliable the monthly classification result becomes.


The figure to the left shows an example of a monthly Greenland surface type (GST) product. The number of class occurrences + standard deviation is provided for each pixel. The number of days when a pixel is cloud-free serves as a measure of the reliability of the classification for each month. The maximum and minimum classifications are also calulated. The maximum GST is calculated using every cloud-free pixel from a satellite image, and if one pixel during that month has experienced one day of melting, i.e. the classification was wet snow or ice, the pixel gets that classification. The maximum GST product thus contains information about brief melt events during the melting season. In the case of the minimum GST, a pixel is classified as dry snow if it receives that classification at least once during a month. This will therefore reveal areas experiencing consistent melting during the summer period. For the average GST, pixels are classified on the basis of the most frequently occurring cloud-free class during a month.