Hi,
On 21/04/16 17:33, Pedro Vicente wrote:
[...]
> >>>Hmmm. Is there any big reason NOT to try to read a netCDF produced HDF5 file with
> the native HDF5 library if someone so chooses?
> It's possible to read a netCDF file using HDF5, yes.
> There are 2 things that you will miss doing this:
> 1) the ability to inquire about shared netCDF dimensions.
> 2) the ability to read remotely with openDAP.
> Reading with HDF5 also exposes metadata that is supposed to be private to netCDF. See
> below
I know that there are several groups (KNMI on L1B data for Sentinel 5 precursor, Jaxa
for the EarthCare mission) that use the plain HDF-5 library to *write* netCDF4 files,
including shared dimensions. This is done for performance reasons, including memory
management issues within the netCDF-4 library. The issues were reported, but a
solution was not forthcoming within the time frame required for the S5P schedule.
The L1B processor runs into these issues because of the large number of groups they
needed to support the on-ground calibration of the instrument (think several thousand
groups in a file, there is no interface for closing a group - or the memory
associated with a group was never actually released, either of these two). For level
2 processing we use the netCDF-4 library, as the number of groups is much smaller,
and the overall data volume is a lot smaller as well.
These L1B files can be read using the netCDF4 library, but not modified by this
library, opening in 'a' mode fails. The same happens if you use h5repack to compress
a netCDF-4 file or add a checksum filter, reading is fine, but modifying fails. For
the S5P project read-only compatibility is sufficient, but having a compression
utility for netCDF4 files that keeps them available for modification sure would be nice.
It would be a complete nuisance if the netCDF-4 library would refuse to open hdf-5
files that it considers incompatible based on this flag. Yes, I realise that
duplicating the effort of unicar for netCDF-4 and creating your own netCDF-4 write
tool is not efficient, but neither are memory leaks deep in netCDF.
Best,
Maarten Sneep
--
KNMI
T: 030 2206747
E: maarten.sneep at knmi.nl
R: A2.14
Received on Thu Apr 21 2016 - 09:55:03 BST
