[CF-metadata] New CF tools

From: David Hassell <d.c.hassell>
Date: Wed, 24 Apr 2013 14:54:58 +0100

Dear CF community,

I would like to advertise two new CF command line tools for inspecting
and combining datasets:

  cfdump - view CF fields
  cfa - create aggregated CF datasets

These utilities accept multiple input files (or URLs if DAP access is
enabled) and aggregate their contents according to the CF aggregation
rules (ticket #78) before either describing the combined fields
(cfdump) or writing them out (cfa). These utilities are useful even if
aggregation is not relevant. In that case they simply describe or
write out the data variables from the input file(s).

Aggregation means that all the input files are treated as though they
were a single file, and individual data variables in different files
are regarded as parts of a larger data variable if their coordinates
and other metadata indicate that this interpretation is possible. For
example, each file might contain data for surface temperature on the
same lat-lon grid but with a different range of dates. The aggregated
dataset then contains a single data variable covering all the
dates. The CF aggregation rules are fully general so, for example,
multidimensional aggregations are handled as are aggregations of
fields with ancillary variables, etc.

However, these utilities will not create the aggregated file unless
requested; the aggregation is done in memory, using only the metadata,
in the first place.

cfdump
------
The cfdump tool generates text representations on standard output of
the CF fields contained in the input files. It can describe fields in
various amounts of detail from one-line summaries, to medium-length
and complete dumps. It is complementary to other similar tools, such
as ncdump, but tells you about the data from a CF point of view
because it knows the CF conventions.

For example, a one-line summary of a complete dataset that has been
split across two files (file1.nc and file2.nc) might look like:

  $ cfdump -s file1.nc file2.nc
  <CF Field: air_temperature(time(1200), latitude(64), longitude(128)) K>

See http://www.met.reading.ac.uk/~david/cfdump.1 for more examples.

cfa

---
The cfa tool creates and writes to disk the aggregated CF fields
contained in the input files.
For example (using the same files as in the cfdump example):
  $ cfa -o new_file.nc file1.nc file2.nc
  $ cfdump -s new_file.nc
  <CF Field: air_temperature(time(1200), latitude(64), longitude(128)) K>
See http://www.met.reading.ac.uk/~david/cfa.1 for more examples.
CFA-netCDF format files
-----------------------
Both tools accept CFA-netCDF format files as input, and cfa also
outputs such files. A CFA-netCDF file is actually a netCDF file
following the CF convention with the addition of a private convention
(the CFA convention) to allow the data to be omitted, because it
resides in the individual files that are being aggregated. CFA-netCDF
files are consequently typically rather small. The suffix .nca is
suggested for CFA-netCDF files. The CFA convention has been designed
specifically to cope with the full generality of aggregations made
possible by the CF aggregation rules.
For example (using the same files as in the cfdump example):
  $ cfa -f CFA -o new_file.nca file[12].nc
  $ cfdump -s new_file.nca
  <CF Field: air_temperature(time(1200), latitude(64), longitude(128)) K>
See http://www.met.reading.ac.uk/~david/cfa/0.3/ for details.
Met Office (UK) PP format files
-------------------------------
Both utilities accept PP format files as input, but output from cfa is
only in CF-netCDF or CFA-netCDF format. Therefore cfa may be used as a
PP to CF converter.
cf-python
---------
Both utilities are built on the cf-python library, which also offers
an interactive programming environment in which you may read, write,
create and manipulate CF fields, including those which are larger than
the available machine memory.
How do you get these tools and cf-python?
-----------------------------------------
They are automatically installed with the open source cf-python
library version 0.9.7 (download from
http://cfpython.bitbucket.org/). Currently, this library only works on
Linux.
I hope that these utilities will be useful. I would welcome any
feedback, particularly if you find a dataset for which they don't work
as expected, and will be glad to make any improvements to cfdump, cfa
or the cf-python library.
All the best,
David
--
David Hassell
National Centre for Atmospheric Science (NCAS)
Department of Meteorology, University of Reading,
Earley Gate, PO Box 243,
Reading RG6 6BB, U.K.
Tel   : +44 118 3785613
E-mail: d.c.hassell at reading.ac.uk

Received on Wed Apr 24 2013 - 07:54:58 BST