[CF-metadata] Logarithmic scaling and CF-Convention

From: Mike Grant <mggr>
Date: Mon, 27 Sep 2010 18:54:08 +0100

On 27/09/10 16:45, Stephen Emsley wrote:
> (b) how to pack a float as a logarithm with scale_factor & add_offset to
> cast it to a byte.

Taking an alternative view, is there actually a real need to do this?
We currently store satellite chlorophyll as unpacked floats and rely on
netcdf (v4) internal compression to reduce the space consumed. In our
experience, the difference between packed floats and compressed floats
is fairly small overall.

(Perhaps this comes from the way the compression works. If there are
only 255 actual float values, as in satellite chlorophyll,
zip/Lempel-Ziv dictionary-type algorithms should just assign 255 symbols
and you'll get essentially the same space consumption as packing manually.)

It's also a lot easier to use unpacked floats as an end user. Libraries
that are made log-packing aware will help, but those don't exist at the
minute. Apart from historical ones, are there other reasons to continue
with log packing (for satellite chlorophyll)?

> To remain CF-convention compliant, the relevant CDL could be:
> byte chl(latitude, longitude) ;
> chl:long_name = "Logarithm (base 10) of chlorophyll-a concentration
> [mg/m3]" ;
> chl:scale_factor = "0.0150000" ;
> chl:add_offset = "-2.00000" ;
>
> Would this be the recommended approach?

I suspect existing code/libraries are likely to misinterpret this and
unpack the data as linear-scaled. If you're trying to stay CF
compliant, perhaps put the scaling information into a free text field
and let the user/developer unpack it manually.

If log packing does go into CF, I agree it would be very helpful to
include some sort of programmatically interpretable indication that the
packing type is logarithmic. I'd be inclined to suggest a new attribute
like "packing_type" rather than using a comment field, but others
probably have a better grasp on this than me.

Cheers,

Mike.

--------------------------------------------------------------------------------
Plymouth Marine Laboratory
 
Registered Office:
Prospect Place
The Hoe
Plymouth PL1 3DH
 
Website: www.pml.ac.uk
Registered Charity No. 1091222
PML is a company limited by guarantee
registered in England & Wales
company number 4178503

--------------------------------------------------------------------------------
This e-mail, its content and any file attachments are confidential.

If you have received this e-mail in error please do not copy, disclose it to any third party or use the contents or attachments in any way. Please notify the sender by replying to this e-mail or e-mail forinfo at pml.ac.uk and then delete the email without making any copies or using it in any other way.

The content of this message may contain personal views which are not the views of Plymouth Marine Laboratory unless specifically stated.

You are reminded that e-mail communications are not secure and may contain viruses. Plymouth Marine Laboratory accepts no liability for any loss or damage which may be caused by viruses.
--------------------------------------------------------------------------------
Received on Mon Sep 27 2010 - 11:54:08 BST