On 11/25/2011 5:01 AM, Ute Br?nner wrote:
> Hi folks,
>
> I kind of lost track of our latest discussions and had the feeling
> that this was partly outside the mailing group;
yes, it was -- we had some discussion among a subset of teh CF list that
was interested in particle model output.
> so I will try to sum up what we were discussing.
IN our group, we've settled on format for the GNOME model (at least for
now, we needed to use something) based on the discussion -- I"ve been
remiss at posting about it to larger group -- I was waiting for the time
to write it up a bit more clearly. More on that soon...
> My latest try was to produce NetCDF for
> particle trajectory trying to write out the concentration grid which
> resulted in a 11GB netFCDF3 file :-(
when you say "grid" I'm wondering what you mean -- particle tracks don't
produce a grid of data -- maybe we're mixing issue here?
> So we have different motivations for discussion particle trajectory
> and netcdf4.
>
> First question: Does anybody know if and if yes, when writing netCDF4
> will be incorporated into the NetCDF Java library? Or will we use
> Python with the help of Jython etc.
> (http://www.slideshare.net/onyame/mixing-python-and-java) to write
> netCDF4?
I'm not sure mixin python and Java is going to help here -- the Python
libs use the C libs -- so mixing C and Java would probably be a better
bet, if you need Java. Jython isn't going to get you C-based Oython
packages. (JEPP might, as mentioned in that talk -- though if the goal
is functionality that really comes from C, straight JNI might make more
sense)
> Second question: Is there a de facto standard / proposal for writing
> Particle Trajectory Data which could be CF:featureType:<whatever we
> agree on>? The suggestion below is not suitable because: 1) we don't
> track a particle the whole time, it may disappear and show up again
> later, but if I have 1000 particles in time step 1 and 1000 in time
> step 2 we cannot be sure these 1000 are the same as before.
This was the whole point of the "ragged array" approach -- so that's
covered.
2) I cannot know the number of time steps in advance.
OK -- that is a challenge -- if we know neither the number of time
steps, nor the number of particles in advance, then we, by definition,
need two unspecified dimensions. I understand netcdf4 allows this -- may
be a good reason to go that route.
One question, though -- with the proposed ragged_array-specified format,
the time dimension is only used in one place - for the: int
rowSize(time) (or "particleCount", or whatever we want to call it) variable.
Is it possible, in netcdf3, to write the big array, with the UNLIMITED
dimension, then specify the time dimension and associated variable at
the end? Or does it need to all vbe defined at the start?
> and I might have int number_particles_per_timestep(time); :units =
> "1"; :long_name = "number particles per current timestep";
> :CF:ragged_row_count = "particle";
> That some of you need to know which spill a particle came from, may
> be solved with a 3rd dimension spill dimensions: spill = 3;
unless the spills all have the same number of particles at any given
time, that's not going to work.
Our solution is to have an "ID" variable to each particle, so they can
be isolated -- this can be used to track a given particle over time, and
also mapped to other data, like which spill it came from, etc.
// or how
> many one has particle = UNLIMITED; //because it may change each time
> step
actually ULIMITED does help if it's going to change each time step
(hence the ragged array solution) -- but it is required as we often
don't know how many particles are going to be used in the end.
> how would one write this? With coordinates or as hierarchical data
> structure? At least we need the ability to use several unlimited
> dimensions and the ragged-array feature.
apparently, yes.
> Third question: How can we compress big netCDF3 files? Or is it
> smarter to go for netCDF4 directly with hierarchical data.
I do think compression and hierarchical data structure are separate
issues. netcdf4 is certainly the easy way to get compression, IIUC, to
compress neetcdf3, you need to do it before/after file reading/writing
-- so helpful for storing and transmitting the data, but you still need
to deal with the big files at some stage.
(or has anyone adapted a netcdf lib to use on-the fly compression (like
with libz) -- that would be cool)
> Hoping to get up the discussion again and that we agree on a standard
> quite soon!
yes, thanks for reviving it!
-Chris
Have a nice weekend!
>
> Best, Ute
>
> -------- Original Message -------- Subject: [CF-metadata] Particle
> Track Feature Type (was: Re: point observation data in CF 1.4) Date:
> Fri, 19 Nov 2010 04:15:35 +0100 From: John
> Caron<caron at unidata.ucar.edu> To:
> cf-metadata at cgd.ucar.edu<cf-metadata at cgd.ucar.edu>
>
> Im thinking that we need a new feature type for this. Im calling it
> "particleTrack" but theres probably a better name.
>
> My reasoning is that the nested table representation of trajectories
> is:
>
> Table { traj_id; Table { time; lat, lon, z; data; } }
>
> but this case has the inner and outer table inverted:
>
> Table { time; Table { particle_id; lat, lon, z; data; data2; } }
>
> So, following that line of thought, the possibilities in CDL are:
>
> 1) If avg number of particles ~ max number of particles at any time
> step, then one could use multdimensional arrays:
>
> dimensions: maxParticles = 1000 ; time = 7777 ; // may be UNLIMITED
>
> variables:
>
> double time(time) ;
>
> int particle_id(time, maxParticles) ; float lon(time, maxParticles)
> ; float lat(time, maxParticles) ; float z(time, maxParticles) ; float
> data(time, maxParticles) ;
>
> attributes: :featureType = "particleTrack";
>
> note maxParticles is the max number of particles at any one time
> step, not total particle tracks. The particle trajectories have to be
> found by examining the values of particle_id(time, maxParticles).
>
> 2) The CDL of the ragged case would look like:
>
> dimensions: obs = 500000; // UNLIMITED time = 7777 ;
>
> variables: int time(time) ; int rowSize(time) ;
>
> int particle_id(obs) ; float lon(obs) ; float lat(obs) ; float z(obs)
> ; float data(obs) ;
>
> attributes: :featureType = "particleTrack";
>
> in this case, you dont have to know the max number of particles at
> any one time step, but you do need to know the number of time steps
> beforehand. The particle trajectories have to be found by examining
> the values of particle_id(obs). The particles at time step i are
> contained in the obs variables between start(i) to start(i) +
> rowSize(i).
>
> these layouts are optimized for processing all particles at a given
> time, and for sequentially processing time steps. If one wanted to
> process particle trajectories, that will be much slower. If you
> needed to do it a lot, you might want to rewrite the file. a more
> sophisticated application, possibly a server, could write an index to
> speed it up.
>
>
> _______________________________________________ CF-metadata mailing
> list CF-metadata at cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
> -----Original Message----- From: rsignell at gmail.com
> [mailto:rsignell at gmail.com] On Behalf Of Rich Signell Sent:
> Donnerstag, 18. August 2011 19:04 To: Christopher Barker Cc: Ute
> Br?nner; Ben Hetland; Mark Reed; Nils Rune Bodsberg; CJ
> Beegle-Krause; Caitlin O'Connor; Alex Hadjilambris; Rob Hetland
> Subject: Re: netcdf for particle trajectories
>
> Chris,
>
>
>>>> so I'll make part of my homework to deliver you a Python
>>>> script using Whitaker's NetCDF4 that writes a sample file.
>>
>> How did this go, Rich?
>
> Yes, I took Rob Hetland's Python short course, and yes, I wrote a
> small example showing how to take NetCDF3 particle tracking output
> and create a compressed NetCDF4 file with chunking. I just forgot to
> send it. ;-)
>
> Note: You can get a OpenDAP-enabled NetCDF4 Python module for both 32
> and 64 bit windows from: http://www.lfd.uci.edu/~gohlke/pythonlibs/
>
> -Rich
>>
>> We're getting closer to a prototype file (i.e. we've got GNOME
>> writing something, but it still needs some tweaking). I'll sent out
>> an example when I think we're close.
>>
>> One new issue:
>>
>> In GNOME, we have the concept of any number of "spills" -- each
>> spill is a set of particles that usually share some properties.
>>
>> So we're trying to figure out how to capture that. Two ideas:
>>
>> 1) each spill is a unique set of data -- but I think that it would
>> ony be possible to do this by using a convension on teh variable
>> names:
>>
>> data_1 particle_count_1 longitude_1 latitude_1 ...
>>
>> data_2 particle_count_2 longitude_2 latitude_2 ...
>>
>> That seems pretty ugly. Could netcdf4's "hierarchical data" help us
>> here? Maybe this provides the motivation to use it.
>>
>> Option two:
>>
>> put all the particles in one big array, but identify the different
>> "spills" by particle ID:
>>
>> ID_range_1 = 0-1000 ID_range_2 = 1000-2000 ...
>>
>> then they could get split up by the client software, if desired,
>> or the separate spills could be ignored, and it could all be
>> treated as one.
>>
>> -- thoughts?
>>
>>
>> -- Christopher Barker, Ph.D. Oceanographer
>>
>> Emergency Response Division NOAA/NOS/OR&R (206) 526-6959
>> voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA
>> 98115 (206) 526-6317 main reception
>>
>> Chris.Barker at noaa.gov
>>
>
>
>
> -- Dr. Richard P. Signell (508) 457-2229 USGS, 384 Woods Hole Rd.
> Woods Hole, MA 02543-1598
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker at noaa.gov
Received on Mon Nov 28 2011 - 12:41:13 GMT
