[CF-metadata] CF and the CDM

From: John Caron <caron>
Date: Thu, 14 Sep 2006 13:13:05 -0600

Simon Wood wrote:
> Hi,
>
> As part of upgrading our satellite data reception systems we are
> planning to change how we store our remote sensing data. Currently an
> in-house binary file format is used to store remote sensing data at
> various levels of processing from raw swaths to composite gridded
> datasets. Some of it comes from locally received HRPT streams, whilst
> some of it comes from indirect sources (mostly in HDF-EOS formats). We'd
> like to store all these internally using a common data format. In some
> cases we'd like to be able to integrate with forecast model outputs too.
>
> Our current thinking tends towards standardising on netCDF (4) with
> something like CF for metadata, though I am aware that CF is based on
> the netCDF-3 data model and I don't see much discussion on how CF is
> expected to evolve to embrace the netCDF-4 data model.
>
> Reading up on netCDF-4 I see lots of references to the Common Data Model
> (CDM) and the prototype implementation in NJ22. This seems to offer all
> that CF does in the way of coordinate systems etc plus some higher level
> datatypes specifically for things like swaths and grids (I guess
> conceptually similar to the HDF-EOS types of the same name), though I
> note that most of these are either not implemented or tagged
> 'experimental'. I also note that NJ22 includes a CF reader to extract
> coordinate system data from CF files, but no other explicit references
> to CF metadata that I can see.
>
> And so to some specific questions:
>
> - Is there a plan for extending CF to exploit the netCDF-4 data model?
>
> - What advice do people have on the use of netCDF-4 with CF? (is the
> use of features from the netCDF-4 data model allowed in a CF compliant
> dataset? or should we stick to netCDF-4 *classic*?)
>
> - what exactly is the relationship between CF and the CDM? Where they
> cover the same functionality (eg coordinate systems) are they
> consistent? (If not then what happens when netCDF-4.1 includes these
> elements of the CDM as part of the netCDF standard?)
>
> - Is there any intention to include the higher level data types from the
> CDM as part of CF?
>
> Sorry for the long post and list of questions; any insight would be most
> gratefully received.
>
> kind regards
> Simon
>

The CDM is Unidata's attempt to create an abstract data model describing common semantics of scientific datasets, particularly coordinate systems. This allows as to create an API that is independent of a specific representation at the "data acccess" layer. The nj22 library maps the CF-1 representation into this model/API, as well as other NetCDF conventions like WRF and NUWG, plus non-NetCDF files like GRIB and BUFR as well.

Unidata is strongly recommending NetCDF file writers to use CF-1 when appropriate, and we are investing much time to continue CF's development. We expect the CDM to evolve along with CF, and will ensure inconsistencies dont occur. OTOH, The CDM is not just an implementation of CF, but must deal with many other important datasets, both currrent and archival, so there are places where the forces on the two efforts are different.

The netCDF-4 file format will be supported both by the netCDF-4 C library (built on top of HDF5 C library) and the pure Java nj22 implementation, although there may be some limits to the Java implemention. The netCDF-4 data model is the same as the CDM data access model. Some of the functionality of the CDM coordinate system layer is being coded into the netCDF-4 C library and an associated "libcf" library. The coordinate system layer is optional, users will always be able to get at the raw netcdf data in both the C and Java libraries.

CF is free to choose which features of the netCDF-4.1 data model to take advantage of, and I would guess that some CF features will eventually have "classic" and "netcdf-4" variants. I invite you to take an active part in the discussions around CF evolution, we hope to have some focussed subgroups starting soon that.

In short, its very unlikely that there will be a conflict between CF and CDM/netCDF-4.1, rather a co-evolution of the two.

Regards,

John Caron
Received on Thu Sep 14 2006 - 13:13:05 BST