[CF-metadata] Dealing with large numbers of flag values innetcdf cf

From: martin.juckes at stfc.ac.uk <martin.juckes>
Date: Tue, 27 Oct 2009 10:44:52 -0000

Thanks for the responses.

Seth is right, I was looking for an alternative to having a gigantic "flag_meanings" attribute.

I have tried Seth's approach, but can?t get it past the cf-checker. There appears to be a character set restriction: I can get the first 547 flag_meaning strings accepted if I remove all the "-", "." and "," characters. I'm reasonably sure there are no control characters in the following strings which could be responsible for the error message that I get (ERROR (3.5): Invalid syntax for 'flag_meanings' attribute) when I increase the number of flag_meanings to 550. Could it be something to do with the string length (16472 characters)?

I could try abbreviating the flag_meanings, but this would either be tedious work by hand or liable to create confusion if done automatically. If you could point me to some information about exactly what string format is acceptable, that would be very helpful.
 
To answer John's question: I'm currently using netcdf 3, in IDL -- I'm not sure what the options for upgrading to netcdf 4 are.

Cheers,
Matin

> -----Original Message-----
> From: Seth McGinnis [mailto:mcginnis at ucar.edu]
> Sent: 26 October 2009 23:13
> To: John Graybeal; Juckes, Martin (STFC,RAL,SSTD)
> Cc: cf-metadata at cgd.ucar.edu
> Subject: Re: [CF-metadata] Dealing with large numbers of flag values
> innetcdf cf
>
> On Mon, 26 Oct 2009 12:16:17 -0700
> John Graybeal <graybeal at marinemetadata.org> wrote:
> >Love the question! :->
> >
> >Are you saying every map location has one index value, which is one of
> the
> >numbers from 1 to 827? So these are mutually exclusive? Or are
> there 827
> >flag values assigned independently for each map location?
>
> It sounds to me like it's the former case, and the problem is that
> concatenating 827 flag values into a single string attribute makes
> something that is easy for a machine to interpret, but impenetrable to
> the human reader.
>
> As I read section 3.5, there are no other options for encoding the
> information according to spec; a gigantic flag_meanings attribute is
> the right answer.
>
> But if the issue is really usability, I think the answer is not to do
> it differently, but just to add an extra attribute that presents the
> same information in a more user-friendly format. You could add an
> attribute named something descriptive, like "category_legend", and
> pair the values & meanings up, one per line. Then you'd end up with
> something that looks like this:
>
>
> float landtype(yc, xc) ;
> landtype:long_name ="Land cover type" ;
> landtype:standard_name ="land_cover" ;
> landtype:flag_values = 1.f, 2.f, 3.f, 4.f, 5.f, 6.f, 7.f, 8.f,
> 9.f,
> 10.f, 11.f, 12.f, 13.f, 14.f, 15.f, 16.f, 17.f, 18.f, 19.f, 20.f, 21.f,
> 22.f,
> 23.f, 24 .f;
> landtype:flag_meanings ="Urban_and_Built-up_Land
> Dryland_Cropland_and_Pasture Irrigated_Cropland_and_Pasture
> Mixed_Dryland/Irrigated_Cropland_and_Pasture Cropland/Grassland_Mosaic
> Cropland/Woodland_Mosaic Grassland Shrubland Mixed_Shrubland/Grassland
> Savanna
> Deciduous_Broadleaf_Forest Deciduous_Needleleaf_Forest
> Evergreen_Broadleaf
> Evergreen_Needleleaf Mixed_Forest Water_Bodies Herbaceous_Wetland
> Wooden_Wetland Barren_or_Sparsely_Vegetated Herbaceous_Tundra
> Wooded_Tundra
> Mixed_Tundra Bare_Ground_Tundra Snow_or_Ice" ;
> landtype:category_legend ="\n",
> "1: Urban and Built-up Land \n",
> "2: Dryland Cropland and Pasture \n",
> "3: Irrigated Cropland and Pasture \n",
> "4: Mixed Dryland/Irrigated Cropland and Pasture \n",
> "5: Cropland/Grassland Mosaic \n",
> "6: Cropland/Woodland Mosaic \n",
> "7: Grassland \n",
> "8: Shrubland \n",
> "9: Mixed Shrubland/Grassland \n",
> "10: Savanna \n",
> "11: Deciduous Broadleaf Forest \n",
> "12: Deciduous Needleleaf Forest \n",
> "13: Evergreen Broadleaf \n",
> "14: Evergreen Needleleaf \n",
> "15: Mixed Forest \n",
> "16: Water Bodies \n",
> "17: Herbaceous Wetland \n",
> "18: Wooden Wetland \n",
> "19: Barren or Sparsely Vegetated \n",
> "20: Herbaceous Tundra \n",
> "21: Wooded Tundra \n",
> "22: Mixed Tundra \n",
> "23: Bare Ground Tundra \n",
> "24: Snow or Ice" ;
>
> (Sorry for the length, but it gives a real sense of what I mean.)
>
> The flag_values and flag_meanings attributes are still a mess to look
> at, but they're easy to deal with programmatically, while end-users
> can easily look at category_legend and figure out what cover type 22
> is if that's all they're after.
>
> Fundamentally, this is the same issue as the question of how to record
> the grid mapping, and the underlying problem is that while attributes
> allow you to associate key-value pairs with a variable, there's no way
> to nest another associative array within an attribute; the best you
> can do is reference a dummy container variable and look at *its*
> attributes.
>
> (Which is an approach that works well enough for the grid mapping
> problem. It may be worth formalizing / regularizing it and applying it
> to any attribute sets that are grouped together and have a tendency
> toward being composed of long lists of elements...)
>
> Cheers,
>
> --Seth
>
> ----
> Seth McGinnis
> mcginnis at ucar.edu
> NARCCAP Data Manager
> IMAGe / ISP / NCAR
> ----
>
> >On Oct 26, 2009, at 0710, <martin.juckes at stfc.ac.uk> wrote:
> >
> >> Hello,
> >>
> >> I wonder if anyone can help with this problem:
> >>
> >> I have a file with a map of index values, ranging from 1 to 827. The
> flag
> >meanings range from ?Admiralty Islands lowland rain forests? to
> ?Zambezian
> >halophytics?. I?m reluctant to combine all 827 flag meanings into a
> single
> >string for the ?flag_meanings? attribute ? is there a better way?
> >>
> >> Note that this is not my dataset, so that while suggestions for
> presenting
> >the data in a different way may be useful, they won?t solve my
> current
> >problem. The dataset is a eco-region analysis distributed by the WWF
> and has
> >wide usage in its present form.
> >>
> >> Sincerely,
> >> Martin Juckes
> >>
> >> --
> >> Scanned by iCritical.
Received on Tue Oct 27 2009 - 04:44:52 GMT