[CF-metadata] Multiple file datasets

From: John Caron <caron>
Date: Fri, 20 Nov 2009 07:08:06 -0700

Stephen Emsley wrote:
>>> Can anyone summarize what SAFE does?
>
> I will give it a shot as I brought it up in the first place!
>
> The Standard Archive Format for Europe (SAFE) was developed as a common format for archiving to ensure long-term preservation of EO data holdings, both historical and operational. The SAFE website [www.esa.int/safe] is the official ESA maintained site for the maintenance and distribution of the standard format, specification, XML-schemas and tools.
>
> SAFE is a specialisation of the XML Formatted Data Unit (XFDU), a CCSDS (Consultative Committee for Space Data Systems) recommended standard for the packaging of data and metadata to facilitate information transfer and archiving. Every SAFE product is an XFDU package. SAFE is a specialisation of XFDU, which defines a restriction of the generic XFDU package. SAFE inherits its main structure from XFDU packaging format and defines high level constraints and new rules for Earth Observation ground segment data products.
>
> A SAFE product wraps, or references, data and associates that data with metadata, both global and local. SAFE product metadata contains basic information, such as the acquisition period, platform and sensor identification and a processing history to ensure traceability. For each included, or external referenced, dataset another layer of associated metadata may be attached providing orbit and geo-location information, quality information and representational information.
>
> Basically a SAFE product is a directory. At the top level is a manifest file, written in XML, that provides both a map of the contained data sets, defines the relationships between these datasets, and contains global metadata (such as platform name, acquisition period etc.). There is a set of required metadata defined by the SAFE specialisation (e.g. there is an ENVISAT specialisation, further restricted to apply to, say, MERIS, and still further specialised to, say, Level 1 processed products).
>
> The contained datasets are collections of records. They are of three types:
>
> Measurement Data Sets: These are typically binary format files and, in our case, will be netCDF-CF files. As an example we will have 46 measurement data products and each will be stored at a netCDF file (data record) along with a data record containing associated quality information and another containing status flags.
>
> Annotation Data Sets: These contain metadata and common data. Although to be decided in the case of Sentinel 3 Level 2 we are considering storing a common set of coordinate data that is applicable to subsets of the measurement data. The manifest file will provide the association between specific measurement datasets and the associated coordinate data.
>
> Representation Data Sets: These are XML Schema descriptions of the measurement and annotation datasets. Firstly it is a key concept for OAIS digital preservation and secondarily third party applications may use these for displaying / accessing the corresponding measurement data sets. I appreciate that it might seem a little 'belt-and-braces' to have an XML schema for a netCDF file (which is by nature self-describing) but that is how the SAFE people have decided to include netCDF into the convention.
>
> There is a third type of data which can be considered as resources. These may be, for instance, data required for the generation of the end-user data products. For instance, for Level 2 data products they would include the Level 1 input products and possibly, for instance, ECMWF data required for processing (although the latter might equally be an annotation dataset). These resources are not packaged inside a SAFE container but are referenced (in the manifest file) using a URI.
>
> All of these taken together are a SAFE package.
>
> I hope that this provides a reasonably informative overview. The SAFE website is the place to go for more detailed info.
>
> Steve

Thanks, Steve for the summary.

A quick perusal of the SAFE spec for our purposes indicates that the referenced file is a full path HTTP URL:

  "The fileLocation element specifies an HTTP GET URL to request the latest version of data from an online registry/repository."

I suppose we are interested only in local netcdf files?
Received on Fri Nov 20 2009 - 07:08:06 GMT