Hi Bert,
This discussion is now turning towards an examination of alternative
approaches to meeting the needs that you have identified. The
discussion may lead back to changes in CF or it may not. But it an
important discussion to be having. A couple of days ago Rich Signell
presented the Wki page that Jonathan created of the three alternative CF
encodings as a question for some experts in the OGC community. He got
this response from Andrew Woolf:
"I think from an OGC perspective, there's not a lot to pick between them
- they all offer slightly different CF-mechanisms for specifying the
actual gridpoint positions under different CRS ["coordinate reference systems"].
Any OGC geometry is
associated with (and specified in terms of) just one CRS - through the
gml:srsName attribute of gml:AbstractGeometryType. Transformations
between CRS may in general be handled through algorithms (e.g. ref
proj4) using the geodetic parameters of the CRS."
So the OGC community has (at least so far) decided that explicit
translations between coordinate reference systems do not belong inside
of their data representations. CF has already gone one step beyond this
(and adds value in doing so). But I believe that keeping CF "lean and
mean" will pay off over the long term, so we should carefully consider
alternative solutions outside of CF before adding further complexity to
the standard. (Sorry if I'm sledge hammering this point.)
Bert Jagers wrote:
> Hi Steve,
>
> I'd like to open another outlook for this entire discussion -- an
> approach that might meet your goals _with no changes to CF at all_.
>
> In the detailed use cases that you presented yesterday you
> motivated the CF changes under discussion by pointing out that it
> would not be desirable to create multiple files to contain the
> same data, specifically "... one could of course generate
> multiple copies of the output file with one auxiliary coordinate
> system each, but I do not like this approach" What your use case
> has presented to us (I think) is a case of multiple "views" of the
> same data: The data variables and their coordinates in absolute
> lat/lon are immutable. But you also want to support two or three
> community-specific outlooks, each requiring specialized coordinate
> system information.
>
> This statement of the problem practically shouts out "service
> oriented" -- providing different flavors of the same data through
> multiple interfaces. I think we can achieve that today with
> existing tools -- OPeNDAP and TDS. Standard netCDF clients can
> "simply" (in quotes) be relinked to utilize the OPeNDAP service.
>
> Hm, a quite different approach indeed. I have no experience yet with
> OPeNDAP, my approach so far has been first to get netCDF CF support
> implemented correctly and take the next step to OPeNDAP. I associate
> OPeNDAP mostly with data access through the internet. An OPeNDAP
> compliant reader will allow you also to access local data files, but
> whether all features are available I am not sure. I know that it is
> possible in some way to serve multiple files as one big virtual file
> using OPeNDAP; however, I don't know whether that works also locally
> and what the requirements are for combining multiple files to one
> virtual file. I am looking for a local option because
>
> 1) model runs may be carried out by different project partners.
> 2) data files may be big and not easily transferred via internet.
>
This could get to be quite a long discussion -- the details of and
values from using TDS. Bullet form seems like it might get to the heart
of matters faster
* There is a sysadmin task to setting up the TDS server. For a
first-timer it is non-trivial, though I hesitate to make an
estimate. Say (??) 1/2 day to get the basics working and another
1/2 day to learn to configure this particular problem. That's
guessing. Say 1-2 hours from scratch for someone already familiar
with the software.
* Once the server is running if you are on the same machine as the
server the distinction between local and remote access is not
terribly important. The filename path becomes longer -- starts
with "http://...". NetCDF through OPeNDAP still looks just like
netCDF (there are a couple of glitches at the detail level -- more
issues for developers than users..)
* There will be a performance penalty, even on local access.
OPeNDAP performance is generally "good", but there are too many
unknowns to characterize it in greater detail. Hard to say
a-priori whether the penalty would be an irritating factor.
> One thing that I am not sure about is what you exactly propose. After
> some text regarding the immutabel part and special attributes and
> variables for the two separate communities, you propose:
>
> You can arrange for these views to show up neatly organized in a
> directory tree
>
> http://my_institution_server/
> Germany/
> ...files...nc
> Netherlands/
> ...files...nc
>
> Would all data be stored in both the Germany and the Netherlands
> directories (only different with respect to the local grids stored in
> the files)? If so, that approach would be quite redundant.
Using TDS there would be only single instances of the "immutable"
(geo-referenced) data files. There would be separate files that
contained only the coordinate representations needed by German, Dutch,
etc user groups. These would not change unless you changed the model
grid. And there would be an XML file (NcML) to tie them together. The
TDS software would act very much like the "small conversion programs"
you hypothesize in the next paragraph, except that it performs its work
on-the-fly. You might find that there were other useful tasks that you
could do with the TDS, too -- most obvious example being to aggregate
time series of files into a single (virtual) dataset.
> I don't need OPeNDAP for that and I can just generate both files and
> distribute them individually, or I could probably write a small
> conversion program to convert one file into the other using some
> separate grid files, or use separate grid files in addition to the
> data file. If the main data is stored in a central directory while
> community specific extensions are stored in the two subdirectories
> then I'm not sure how its going to work (only through internet
> access?, what about the variable specific grid_mapping attributes?).
If it is local files that your community really needs, and the
service-oriented aspects of TDS do not add any value, then (as you have
said) you don't need OPeNDAP. In fact, there is plenty of richness in
the netCDF utilities that already exist (e.g. nco), so that you can
create German and Dutch flavors of your files using nothing but simple
scripts to do the same things that are described from TDS above. No
changes to CF are required. And it would not be a particularly messy
data management task.
>
> P.S. (footnote) On the initial point about redundant "grid object"
> contents: I was referring to situations where variables like
> velocities (XYZ), share the same horizontal coordinates as
> variables like streamFunction (XY). The grid_mapping info for the
> horizontal has to be repeated in both grids ... or alternatively,
> complexity has to be introduced to represent parent (XY) and child
> (XYZ) grid objects.
>
> As long as the grid_mapping data sets do not refer to variables, there
> is no need to duplicate the grid_mapping data. I only see the need to
> specify for every variable in the file the grid_mapping attribute
> (current situation) or coordinate_system (one of the alternatives). So
> far, grid_mapping is a purely horizontal transformation, so, if the
> velocities are stored at the same grid locations as the streamfunction
> and the Z dimension is a separate dimension, there is no need to
> duplicate the horizontal grid variables either. If the velocities are
> stored at different X,Y locations than the streamFunction then we need
> of course two grid specifications.
In the footnote I was assuming that the theoretical discussion was of
grid objects that described both horizontal and vertical and files that
mixed 2D and 3D spatial grids. There are plenty of subtleties in the
vertical, too, particularly for hybrid coordinate models. Sorry for
opening up this distracting point.
Thanks for your patience with this process - Steve
>
> Best regards,
>
> Bert
> ------------------------------------------------------------------------
>
> _______________________________________________
> CF-metadata mailing list
> CF-metadata at cgd.ucar.edu
> http://www.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
--
--
Steve Hankin, NOAA/PMEL -- Steven.C.Hankin at noaa.gov
7600 Sand Point Way NE, Seattle, WA 98115-0070
ph. (206) 526-6080, FAX (206) 526-6744
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20061206/65864216/attachment-0002.html>
Received on Wed Dec 06 2006 - 10:07:58 GMT
