Hi Russ,
Thanks for providing this reference, which I had previously come across
as part of some earlier investigative work around data packing. I may be
misunderstanding some of what's written there, but I wasn't able to see
anything in that particular section which described how to specify (or
derive) a fill value to use with the *unpacked* data array. But it's
quite likely I'm just missing some key point :-)
The CF spec currently states that "values that are identified as missing
should not be transformed". So if we have a packed data variable of type
short which uses a fill value of, say, -32768, all we can do is detect
those values, avoid applying the unpacking algorithm, and flag them as
missing in the in-memory data array.
Now, when we come to write out the unpacked data array to a new netCDF
file we need to choose a fill value appropriate to the type and range of
the unpacked data. We could use some semi-arbitrary value, such as
-1.0e20, or the netCDF default fill value. But as I hinted at in my
original email, both of those choices might be inappropriate, hence my
suggestion to explicitly define what the fill value should be via a
suitable new attribute.
Refining my earlier suggestion, that attribute could also be global, in
which case it would be used for all unpacked variables which didn't
define the new fill value via a variable-scope attribute of the same
name. All this would be optional of course.
Regards,
Phil
> -----Original Message-----
> From: Russ Rew [mailto:russ at unidata.ucar.edu]
> Sent: 26 September 2012 04:08
> To: Bentley, Philip
> Cc: cf-metadata at cgd.ucar.edu
> Subject: Re: [CF-metadata] Choice of fill value for unpacked data
>
> Hi Phil,
>
> > The final para of section 2.5.1 of the CF conventions document
> > describes the use of the _FillValue (or missing_value) attribute in
> > the case of data packed using the scale-and-offset method. What is
> > not clear - at least to me - is what the preferred application
> > behaviour should be in the case where the data is unpacked and then
> > written out to a new netCDF file. In particular, what fill value
> > should be used for the unpacked data variable?
> >
> > [snip]
>
> A more detailed recommendation for treating special values
> such as _FillValue or missing_value for packed data is
> described, with associated formulas, in the "Packed Data
> Values" section of a best-practices document that we wrote a
> few years ago:
>
>
> http://www.unidata.ucar.edu/netcdf/docs/BestPractices.html#Pac
> ked%20Data%20Values
>
> It provides a recommendation for ensuring the unpacked
> special value is not in the range of other unpacked data
> values. If that recommendation is followed, I think there is
> no need for an additional _FillValueUnpacked (or
> missing_value_unpacked) attribute.
>
> If you agree that is an acceptable approach, perhaps we
> should add it to CF ...
>
> --Russ
>
Received on Wed Sep 26 2012 - 08:59:53 BST
