Hi all:
Lets see, I havent followed the entire conversation, but:
1) Andrew if you can send me a sample file (not just the CDL) I can
check if it works in the CDM with the new 1.6 conventions, and maybe
give you some advice from my POV.
2) Aggregation in the CDM comes in 2 flavors. 1) The original
implementation simply appends multidimensional arrays together, eg
"joinNew " and "joinExisting" NCML aggregation. I call it "syntactic
aggregation" because it doesnt know what its aggregating, and the
homogeneity requirements are strict. 2) "Feature Type collections" (aka
"semantic aggregation") are the more recent development. These
understand the coordinate information of the data, and so can handle
variations in the representation, eg ragged arrays, scalar coordinates,
etc. In this case, the CDM understands a dataset as a collection of
"features objects", which can be stored in a collection of files. The
interfaces to these collections is still under development. Most current
and future work in the CDM is in this category.
John
On 4/3/2012 12:48 PM, Upendra Dadi wrote:
> Hi,
> I think Jim is talking about NCML (virtual) aggregation. THREDDS
> can aggregate even when the variable is a scalar using "joinNew". But
> I think THREDDS can aggregate only over regular arrays. That is, all
> the dimensions other than over which a variable is aggregated should
> be of same size across all the files. This is possible, for example,
> only if all the profiles have same depth( or pressure) levels. Which
> in general is not true. NCML aggregation I guess is of limited use
> when dealing with jagged arrays. But I agree with Jim in that ability
> to aggregate should be an important consideration when coming up with
> a representation. Physical aggregation is still possible. But I prefer
> virtual aggregation, since letting data to be stored in individual
> files in often operationally convenient, but users benefit from
> aggregated views. I wonder if NCML could be extended to be able to
> aggregate jagged arrays into incomplete multidimensional array
> representations. I heard that ERDDAP has similar capability though I
> don't think it has anything like NCML where users can create views
> over remote data, not sure though.
>
> Upendra
>
>
>
> On 4/3/2012 1:27 PM, Jim Biard wrote:
>> Hi.
>>
>> A colleague of mine has told me that if you use size-1 array for a
>> variable, then data servers such as THREDDS can aggregate the
>> variable over multiple files and deliver a single file in which the
>> variables will be arrays of size > 1. He said that if a variable is
>> a scalar, THREDDS won't do this. (I don't mess with THREDDS, so I am
>> just parroting what he said.)
>>
>> If this is correct, then you might want to consider this point when
>> deciding which way to represent coordinates.
>>
>> Grace and peace,
>>
>> Jim
>>
>> On Tue, Apr 3, 2012 at 1:02 PM, Upendra Dadi <upendra.dadi at noaa.gov
>> <mailto:upendra.dadi at noaa.gov>> wrote:
>>
>> Andrew,
>>> However I have another small concern about all of this. Given
>>> there is more than
>>> 1 way to treat dimensions does the netCDF plotting and
>>> visualising sofware out
>>> there understand both approaches. That is can we expect
>>> something like
>>> ncBrowse, IDV or ODV? give me a plot of say salinity vs. pressure
>>> and work OK with BOTH approaches to coding the netCDF?
>> Software like ncBrowse do not understand CF conventions. They
>> can read netCDF files, but wouldn't know how to interpret the
>> attributes like "coordinates". So if you want to plot the profile
>> location on a map, for example, it wouldn't be able to do it by
>> itself. It would need to be told how to interpret the
>> coordinates. A CF based software wouldn't have to be supplied
>> with the additional semantics since it can read from the file by
>> itself. But if you want to plot salinity vs pressure, software
>> like ncBrowse can already do it since lot of these software make
>> it easy to make plots based on a shared dimension. And here
>> salinity and pressure share the same dimension. So I guess, the
>> correct answer to your question is - it depends on what kind of
>> plot or task you want to do and if the software can understand CF
>> conventions.
>>
>> Upendra
>>
>>
>>
>>> Thanks and Regards All,
>>> Andrew Walsh
>>>
>>> ----- Original Message -----
>>> *From:* Lowry, Roy K. <mailto:rkl at bodc.ac.uk>
>>> *To:* Upendra Dadi <mailto:upendra.dadi at noaa.gov> ; andrew
>>> walsh <mailto:awalsh at metoc.gov.au>
>>> *Cc:* Luke Callcut <mailto:luke at metoc.gov.au> ;
>>> cf-metadata at cgd.ucar.edu <mailto:cf-metadata at cgd.ucar.edu> ;
>>> sdn2-tech at seadatanet.org <mailto:sdn2-tech at seadatanet.org>
>>> *Sent:* Tuesday, April 03, 2012 9:31 AM
>>> *Subject:* RE: [CF-metadata] Ocean CTD data following CF
>>> Conventions v1.6
>>>
>>> Hi Upendra,
>>> I like the idea of a station dimension. It goes a long way
>>> to resolving the issue raised in my response to Jim which
>>> was based on the tunnel vision of having pressure/depth as a
>>> dimension. I have yet to look at the recently published
>>> NODC NetCDF templates. Is this CTD encoding included in
>>> them? If so, I'll bump up looking at them on my 'todo'
>>> list. I'd also recommend that Andrew and my colleagues in
>>> SeaDataNet take a look.
>>> Cheers, Roy.
>>> ------------------------------------------------------------------------
>>> *From:* cf-metadata-bounces at cgd.ucar.edu
>>> <mailto:cf-metadata-bounces at cgd.ucar.edu>
>>> [cf-metadata-bounces at cgd.ucar.edu
>>> <mailto:cf-metadata-bounces at cgd.ucar.edu>] On Behalf Of
>>> Upendra Dadi [upendra.dadi at noaa.gov
>>> <mailto:upendra.dadi at noaa.gov>]
>>> *Sent:* 02 April 2012 17:21
>>> *To:* andrew walsh
>>> *Cc:* Luke Callcut; cf-metadata at cgd.ucar.edu
>>> <mailto:cf-metadata at cgd.ucar.edu>
>>> *Subject:* Re: [CF-metadata] Ocean CTD data following CF
>>> Conventions v1.6
>>>
>>> Hi Andrew,
>>> Either way it should be okay as far as CF compliance is
>>> concerned. But the dimensions - latitude, longitude and time
>>> are not really required. If it is required to indicate that
>>> there is only one station(profile) in the file, there could
>>> be a dimension for number of stations instead, with a value
>>> of 1. Also, using a station dimension is the way to go if
>>> storing a collection of profiles in a single file. Here at
>>> NODC, we took the approach that we would use the same
>>> consistent representation whether there is a single instance
>>> or a collection in a file.
>>>
>>> Upendra
>>>
>>>
>>>
>>>
>>> On Mon, Apr 2, 2012 at 2:51 AM, andrew walsh
>>> <awalsh at metoc.gov.au <mailto:awalsh at metoc.gov.au>> wrote:
>>>
>>> Hi CF lis
>>> We are working on coding up some 1000's netCDF files off CTD
>>> instruments and want to make usre we are following the
>>> latest netCDF conventions (v1.6) OK. As background the
>>> CTD records
>>> a profile pressure, temperature and salinity.
>>>
>>> Here is a summarised CDL version (not all
>>> attributes+variables+qc flags there, just majors for now)
>>> of what we propose:
>>>
>>> dimensions:
>>> TIME=1
>>> PRESSURE=729
>>> LATITUDE=1
>>> LONGITUDE=1
>>>
>>> variables:
>>> double TIME(TIME) ;
>>> TIME:standard_name = "time" ;
>>> TIME:units = "days since 1950-01-01 00:00:00Z" ;
>>> TIME:axis = "T" ;
>>> TIME:valid_min = 0. ;
>>> TIME:valid_max = 999999. ;
>>>
>>> double LATITUDE(LATITUDE) ;
>>> LATITUDE:standard_name = "latitude" ;
>>> LATITUDE:units = "degrees_north" ;
>>> LATITUDE:axis = "Y" ;
>>> LATITUDE:valid_min = -90. ;
>>> LATITUDE:valid_max = 90. ;
>>>
>>> double LONGITUDE(LONGITUDE) ;
>>> LONGITUDE:standard_name = "longitude" ;
>>> LONGITUDE:units = "degrees_east" ;
>>> LONGITUDE:axis = "X" ;
>>> LONGITUDE:valid_min = -180. ;
>>> LONGITUDE:valid_max = 180. ;
>>>
>>> double PRESSURE(PRESSURE) ;
>>> PRESSURE:standard_name = "sea_water_pressure" ;
>>> PRESSURE:units = "decibars" ;
>>> PRESSURE:axis = "Z" ;
>>> PRESSURE:valid_min = 0. ;
>>> PRESSURE:valid_max = 12000. ;
>>> PRESSURE:positive = "down" ;
>>>
>>> double TEMPERATURE(PRESSURE) ;
>>> TEMPERATURE:standard_name = "sea_water_temperature" ;
>>> TEMPERATURE:units = "degrees_C" ;
>>> TEMPERATURE:_FillValue = -99.99 ;
>>> TEMPERATURE:valid_min = -2. ;
>>> TEMPERATURE:valid_max = 40. ;
>>> TEMPERATURE:coordinates="TIME LATITUDE LONGITUDE PRESSURE"
>>>
>>> double SALINITY(PRESSURE) ;
>>> SALINITY:standard_name = "sea_water_salinity" ;
>>> SALINITY:units = "psu" ;
>>> SALINITY:_FillValue = -99.99 ;
>>> SALINITY:valid_min = 0. ;
>>> SALINITY:valid_max = 40. ;
>>> SALINITY:coordinates="TIME LATITUDE LONGITUDE PRESSURE"
>>>
>>> // global attributes:
>>> :conventions = "CF-1.6" ;
>>> :featureType = "profile"
>>> :cdm_data_type = "profile"
>>> + several other attributes later for ISO19115 metadata
>>> generation
>>>
>>> I am not sure if I should have TEMPERATURE and SALINITY
>>> arrays with 4 dimensions
>>> like TEMPERATURE(TIME,LATITUDE,LONGITUDE,PRESSURE) or
>>> just 1 dimension
>>> like I have above i.e. TEMPERATURE(PRESSURE). ?
>>>
>>> Any feedback on the above is greatly appreciated.
>>>
>>> Andrew Walsh
>>> _______________________________________________
>>> CF-metadata mailing list
>>> CF-metadata at cgd.ucar.edu <mailto:CF-metadata at cgd.ucar.edu>
>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>>
>>>
>>>
>>> --
>>> This message (and any attachments) is for the recipient
>>> only. NERC
>>> is subject to the Freedom of Information Act 2000 and the
>>> contents
>>> of this email and any reply you make may be disclosed by
>>> NERC unless
>>> it is exempt from release under the Act. Any material
>>> supplied to
>>> NERC may be stored in an electronic records management system.
>>>
>>
>>
>> _______________________________________________
>> CF-metadata mailing list
>> CF-metadata at cgd.ucar.edu <mailto:CF-metadata at cgd.ucar.edu>
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>
>>
>>
>>
>> --
>> Jim Biard
>> Research Scholar
>> Cooperative Institute for Climate and Satellites
>> Remote Sensing and Applications Division
>> National Climatic Data Center
>> 151 Patton Ave, Asheville, NC 28801-5001
>>
>> jim.biard at noaa.gov <mailto:jim.biard at noaa.gov>
>> 828-271-4900
>>
>>
>>
>> _______________________________________________
>> CF-metadata mailing list
>> CF-metadata at cgd.ucar.edu
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
>
> _______________________________________________
> CF-metadata mailing list
> CF-metadata at cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20120403/d080c7d0/attachment-0001.html>
Received on Tue Apr 03 2012 - 13:55:29 BST
