Dear all,
Note that if you use the new netCDF4 library, there is a compression
option that allows for efficient automatic compression of data in the
files containing lots of contiguous common numbers (like "missing" or
0.). If you stick to the so-called classical data model (and with CF
there is no reason not to), then most software can be easily built with
netCDF4, so reading these files is not impaired.
Best regards,
Karl
On 20-Jan-10 12:20 AM, andrea antonello wrote:
> Hi Janine,
> thanks for the reply.
>
>
>> I am very familiar with the problem you are having and have run into this
>> many times, and yes, you have to create timeSize*100, not timeSize*2. I
>> would use the _fillValue attribute to set a value to fill all the missing
>> temperatures. That way there will be no confusion that 0.0 is a measurement
>> value.
>>
>> If I understand correctly, the missing stations will be compressed in memory
>> to a value (0.0) and a repetition (100), so it doesn't take as much memory
>> as you think. But check the online documentation to confirm that.
>>
> That is one of the things I was guessing. In my understanding that
> helps to save disk space. Another issue would be heap memory. I assume
> that when the ArrayDoubles are read into memory, the are uncompressed.
> Is there any documentation that coveres that kind of doubts or is it
> something more to be looked in the javadoc? Any link would be
> appreciated.
>
> Thanks,
> Andrea
>
>
>
>
>> Janine
>>
>> andrea antonello wrote:
>>
>>> Dear Jonathan,
>>> thanks for your reply.
>>>
>>>
>>>
>>>> I don't know whether the problem is lack of memory. That would depend on
>>>> the software you are using, rather than on netCDF or the CF convention,
>>>> so
>>>> I'm not sure what to suggest really. You could try creating an empty file
>>>> of the right size by writing it in CDL and using ncgen, to test that it
>>>> can be done. Sorry not to be more helpful.
>>>>
>>>>
>>> At the moment I have no problem with the memory overload really. I
>>> just would like to understand if I am implementing the proposed CF way
>>> properly.
>>> Can you confirm me that in the above example with 2 stations of
>>> interest for the temperature variable (from 100) I am right in
>>> creating a variable with timeSize*100 instead of
>>> timeSize*2, which brings an 80% of 0.0 values of overload? Where
>>> overload here really means never used. And if yes, should those unused
>>> values stay 0.0 or be somehow flagged?
>>> I would love to do this the CF way and I am just wondering if someone
>>> stumbled into this doubt also.
>>>
>>> Best regards,
>>> Andrea
>>>
>>>
>>>
>>>> Best wishes
>>>>
>>>> Jonathan
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> CF-metadata mailing list
>>> CF-metadata at cgd.ucar.edu
>>> http://*mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>>
>>>
>>
>> --
>> Janine Aquino (Goldstein)
>>
>> "Life is a comedy for those who think... and a tragedy for
>> those who feel." - Horace Walpole
>>
>>
>>
> _______________________________________________
> CF-metadata mailing list
> CF-metadata at cgd.ucar.edu
> http://*mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
>
Received on Wed Jan 20 2010 - 06:29:45 GMT
