On 26/03/18 16:57, V. Balaji wrote:
> Groups can have quite significant performance issues when the datasets are large...
> something to keep in mind while designing standards or conventions that _require_ the
> use of netCDF4 groups. It's a last resort, in my opinion... to be used only if
> there's simply no other way to express what you want.
That is a bug in the netCDF4 library. When you use the HDF-5 library directly (and
still set the appropriate attributes), then groups have no performance impact at all.
On the other hand you'll still have a significant performance degradation if the
number of variables is large.
As far as I understand the issue is that a linear search is performed in the netCDF
library, where an appropriate algorithm (likely a hash) would have been more appropriate.
Kind regards,
Maarten
> Erik Quaeghebeur writes:
>
>> Dear list,
>>
>>
>> I have not come across any mention of netCDF4 groups (and other netCDF4 features)
>> in the CF conventions. I was wondering if there are nevertheless standard ways to
>> use groups.
>>
>> For example, I have repackaged some statistics data (csv files) from met masts in
>> netCDF4. I've been trying to apply the CF conventions, but bump into issues, e.g.,
>> of duplication.
>>
>> Currently, the structure I use is:
>>
>> root level:
>> * time dimension/variable
>> * per-instrument-type groups
>>
>> instrument level:
>> * height (of instrument) dimension/variable
>> * instrument metadata attributes (not covered by CF conventions)
>> * signal groups (some instruments measure more than one signal) signal level:
>> * signal specific metadata, e.g., units (covered by CF conventions, but for variables)
>> * signal statistics variables
>>
>> statistics variables:
>> * data
>> * some statistic-specific attributes like cell_methods
>> * should I duplicate the signal metadata such as unit here?
>>
>> The group structure is helpful, because its structure improves
>> self-descriptiveness, I feel. I guess the structure is also a form of metadata. But
>> perhaps it is considered out-of-scope for the CF conventions?
>>
>>
>> Best,
>>
>> Erik
>>
>>
>
Maarten Sneep
--
KNMI
T: 030 2206747
E: maarten.sneep at knmi.nl
R: A2.14
Received on Mon Mar 26 2018 - 09:33:46 BST
