On 11/26/2011 9:14 AM, John Caron wrote:
> Im intending to incorporate the netcdf-4 C library into the netcdf-java
> library using JNI. Im hoping to have something working in the next few
> months, but we'll see. This will be an optional component, and will
> obviously make portability an issue.
Good idea, none the less.
> If you want to use Python, probably
> the one to use is Jeff Whittaker's at
> http://code.google.com/p/netcdf4-python/, which is also an interface to
> the netcdf-4 C library.
yes -- I think that's the best option for Python -- it's nicely done.
> I think its time to start using netcdf-4 for large collections of point
> data which need to be compressed. Instead of first making a standard, we
> need to try out the possibilities and see how it performs.
That may be true, time to move on eventually! However, you can use
netcdf4 for compression, but stille use a netcdf3 compatible data model,
so I'd like to see netcdf4-only features used only if they really are
necessary to get the data model we need.
> I think you
> want to use Structures, as well as multiple unlimited dimensions. With
> netcdf, we dont need the ragged array mecahnism - thats only needed to
> overcome the limitations of the classic model.
Can you tell us more? how do you express a "ragged_array" in netcdf 4?
Variable length user-defined types, maybe?
This is all a bit frustrating, as we've had a fair bit of discussion,
and I though had settled on ragged arrays, and I don't think anyone said
(this would all be so much easier in netcdf 4)
Ute: I'm a bit confused -- was your 11 GB file a result of using the
ragged array approach, or of using the rectangular array with LOTS of
empty values approach?
I don't think compression is the answer to the problem of how to store
what is naturally a ragged array -- partly because it simply doesn't
appeal to me aesthetically, but also because it hides, and moves the
problem -- the tools really should understand and be able to work with
the fact that the number oar particles is not the same at all times, and
we don't want to have the client apps to deal with a lot of empty arrays
either.
Note that it seems netcdf4 "groups" could be handy for dealing with
multiple "spills".
More soon on our current solution.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker at noaa.gov
Received on Mon Nov 28 2011 - 13:28:49 GMT
