Hi Mike - im forwarding this to the cf list, you posted to cf-pointobsconvention at unidata.ucar.edu which is no longer in use.
Its useful to separate these issues into 2 categories: 1) information content and 2) encoding.
So far CF has focussed on standardizing information content and encoding in netcdf-3 "classic" data model. Much of the information content standards are esp relevant to the "climate and forecast" community. As youve no doubt noticed, the proposed CF Observation conventions (
https://cf-pcmdi.llnl.gov/trac/ticket/37) deals mostly with encoding conventions for observational data and coordinates in the classic format. It sounds like you want to break some new ground for standardizing information content for metadata for observational data?
It certainly is possible to store structured metadata, either as attributes or variables in a netcdf-3 file. netcdf-4 adds groups, new data types, variable length arrays, etc. so it has richer constructs for these purposes, although CF conventions have not started to use them explicitly.
SensorML and SWE and other sensor infrastructures AFAIU have developed very detailed metadata models. We expect that the THREDDS Data Server will be developing capabilities in this area, esp for serving archived data. But exactly how and what the priority is for TDS development is TBD.
If theres enough interest in how this plays out in the context of CF, perhaps a subgroup could form to study these and make recommendations on how these could be integrated into CF conventions?
McCann, Mike wrote:
> Dear Point Obs and data model experts,
>
> I am following up on Nan's initial question - we were at the same meeting
> where this issue was identified by the oceanographic in situ instrument
> community. We are concerned about the proper archiving of climate quality
> measurements for the long term and recognize the need to store *very*
> complete provenance metadata *with* the instrument data.
>
> We are examining SensorML and ISO 19115 to meet this need, but haven't made
> any decisions yet and would entertain other options. We would very much
> prefer to embed this important metadata within the NetCDF file. The
> question is whether it is possible to store structured information in some
> area of the file - perhaps there is new capability in the NetCDF-4?
>
> A highly summarized example of the structured information that we'd like to
> store is:
>
> - (1) NetCDF file (containing global and variable metadata)
> |
> |- (2) Process that created this (1) file
> |
> |- (3a) Input data file to above process
> |
> |- (4) Process that created this (3a) file
> |
> |- (5) Input data stream
> |
> |- (6) Deployment of instrument
> |
> |- (7) Instrument details
> +
> |
> |- (8) Parent Deployment
> | ...
>
>
> |
> |- (3b) Input data file to above process
> | ...
>
>
> Where attributes of these elements would be like:
>
> 1) OPeNDAP URL to file, create time, etc.
> 2) Process name, process start and end time, reference to source code in
> version control, description of the process, host and user that executed the
> process, etc.
> 3) Same as 1
> 4) Same as 2, additionally with references to calibrations used if direct
> instrument datastream output was processed
> 5) Datastream access URL, create time, etc.
> 6) Deployment start and end time, references to parent deployment, if any
> (e.g. The instrument may be on an AUV, glider, mooring, ship, etc.), nominal
> latitude, longitude, depth, etc.
> 7) Instrument manufacturer, model, serial number and references to any
> supporting material: spec sheets, calibration documents, etc.
> 8) Same as 6
>
> An actual in situ data set would contain the descriptions of the processing
> and deployments of dozens of instruments and would have perhaps hundreds of
> elements with an indefinite number of processing steps and parent deployment
> levels.
>
> We are expecting many new types of more complex oceanographic in situ
> instrumentation that will measure essential climate variables such as pH,
> Nitrate, Carbon (in its various forms), particulate matter, and chlorophyll
> proxies. Having such detailed information about the instruments and the
> processing of their output is critical for understanding of the data. We
> would like to associate this metadata with the original files contributed to
> community archives and have it flow through to the THREDDS Data Servers
> where we expect the data will be accessed.
>
> Any advice or discussion on how to proceed would be appreciated.
>
> Cheers,
> Mike
>
Received on Thu Oct 01 2009 - 08:34:06 BST
