On 11/3/10 7:19 PM, John Caron wrote:
> 1) It seems clear that at each time step, you need to write out the data
> for whatever particles currently exist. I assume that if you wanted to
> break up the data for a very long run, you would partition by time, ie
> time steps 1-1000, 1001-2000, etc. would be in seperate files.
probably, yes.
> 2) Apparently the common read pattern is to retrieve the set of
> particles at a given time step. If so, that makes things easier.
yes -- or a sequence of time steps, but also easy.
> 3) I assume that you want to be able to figure out an individual
> particle's trajectory, even if that doesnt need to be optimized for speed.
It should be possible, yes, but correct, that is the less common use
case, and thus less important for good performance. And with particles
coming and going, a mess anyway.
> 1) is the avg number (Navg) of particles that exist much smaller, or
> approx the same as the max number (Nmax) that exist at one time step?
I think this is very use-case dependent -- how would that change what we
might want to do?
> 2) How much data is associated with each particle at a given time step
> (just an estimate needed here - 10 bytes? 1000 bytes?)
For us, it's currently about 40 bytes, but we'll be adding an unknown
amount in the future -- maybe up to a couple hundred.
> A more nasty example could be to represent an oil slick's shape and
> position with a polygon. The number of vertices of that polygon would be
> highly variable through time. (This is a typical GIS-like representation.)
I think that is an entirely different use-case -- and one probably bet
handles with GIS format -- although GIS does time (and 3d) really badly.
> How one decides to partition I think can depend a lot on the
> application. Sometimes splitting them on data type can be more
> appropriate. In a recent case I had, the data were to be transferred to
> a client computer over the Internet for viewing locally. In that case
> reducing the content of the file to the absolute minimum set of
> properties (that the client needed in order to visualize) became
> paramount. Even a fast Internet connection does have bandwidth
> limitations... :-)
I think that's a little different than partitioning -- it's more
subsetting the data, and yes, I think we would often want to only put
the relevant data for a given application in a given file.
My thought is that there would be very few required variables (but
hopefully some standard names for many more!)
>> 3) I assume that you want to be able to figure out an individual
>> particle's trajectory, even if that doesnt need to be optimized
>> for speed.
>
> Not my primary need, but if an object is "tracked" like that it would
> not be unlikely that the trajectory might need to be accessed
> "interactively", eg. while a user is viewing a visualization of the data
> directly on screen. Does that count as "optimized for speed"?
well, yes. IIUC, then if we group by time step, then you'd essentially
have to read all the data to follow a single particle through time. If
that was a common need, then the data should be arranged differently.
But then, the already-proposed standard for "trajectories" would work
well for that already.
> It also wouldn't be very atypical if
> this amount is then to be multiplied by say 20000 particles per time step.
or more -- we use 1000 typically for surface trajectories, and 10,000
for 3-d at a minimum -- if it's a long term spill, it could be a lot
more (i.e. the event in the Gulf this summer?)
I think we're converging on something here.
I just got some sample Sintef files sent to me, and I've been working on
my own sample -- I'll see if I can reconcile those, and post something here.
By the way -- it would be nice to have something that could accommodate
ASA's models, too -- is there anyone from ASA on this list and paying
attention?
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker at noaa.gov
Received on Thu Nov 04 2010 - 10:16:54 GMT
