Time mean over area fractions which vary with time

[martin.juckes]

Following a discussion on the mailing list, I'd like to propose adding a new example to the CF Convention document to illustrate the use of cell_methods to specify different mean quantities when using a mask which is time varying (e.g. sea_ice). The qualifier where has been introduced into the cell_methods to specify masked spatial operations, e.g. area: mean where sea_ice to represent a spatial mean over sea ice. The current convention does not explicitly comment on whether the where construct can be used with other dimensions. For the CMIP6 data request there is a requirement to specify the temporal mean of quantities averaged over sea ice, and the spatial extent of the sea ice is generally varying in time.

The proposal is to make it clear that use of where for non-spatial dimensions is allowed by adding examples in section 7. It is also necessary to provide these examples to clarify the subtle differences implied by different formulations of the cell_methods statement.

1. Replace text of section 7.3.3 (first 3 paragraphs)

By default, the statistical method indicated by cell_methods is assumed to have been evaluated over the entire horizontal area of the cell. Sometimes, however, it is useful to limit consideration to only a portion of a cell (e.g. a mean over the sea-ice area). The portion concerned is constant in time in some cases, but it could be time-varying. Grid cell “portions” that can be considered are only those permitted to be associated with the standard_name of area_type. There are two options for indicating when a quantity represents a portion of a cell.

The first method can be used for the common case that the cell_method applies to a single area-type. In this case, the cell_methods attribute may include a string of the form name: method where type. Here name could, for example, be area and type may be any of the strings permitted for a variable with a standard_name of area_type. As an example, if the method is area: mean where sea_ice, then the data would represent a mean over only the sea ice portion of the grid cell. When this first option is adopted, none of the variables in the netCDF file should be given a name identical to the string that names the area_type. This restriction is imposed so that it will be clear that the metadata should not be interpreted following the second option (described in the next paragraph), which takes precedence.

The second method for indicating that a statistic applies to only a portion of a cell is more general because it can reference multiple area-types. This may be needed when a variable has a dimension that ranges across various area types. In this case, the cell_methods entry is of the form name: method where typevar. Here typevar is a string-valued auxiliary coordinate variable or string-valued scalar coordinate variable (see Section 6.1, "Labels") with a standard_name of area_type. The variable typevar contains the name(s) of the selected portion(s) of the grid cell to which the method is applied. This method provides a convenient way to store output from land surface models, for example, since they deal with many area types within each surface grid cell (e.g., vegetation, bare_ground, snow, etc.).

2. Caption of example 7.6

If the method is mean, various ways of calculating the mean can be distinguished in the cell_methods attribute with a string of the form mean where type1 [over type2]. Here, type1 can be any of the possibilities allowed for typevar or type (as specified in the two paragraphs preceding above Example). The same options apply to type2, except it is not allowed to be the name of an auxiliary coordinate variable with a dimension greater than one (ignoring the dimension accommodating the maximum string length).

A cell_methods attribute with a string of the form area: mean where type1 over type2 indicates the mean is calculated by integrating over the type1 portion of the cell and dividing by the area of the type2 portion. When over type2 is omitted, it is assumed to be the same as type1.

3. Addtional text for masks which vary over additional dimensions (e.g. time) .. form proposed by Karl

When the “where” construct is used, and when “area” is not the only “dimension” to which it applies, the interpretation more generally is that a “weighted” mean is reported. Specifically, the quantity of interest is integrated over the additional dimension(s) with weights proportional to the fraction of “type1” area_type that exists, and then this is divided by the integral over the same dimension(s) of the fraction of “type2” area_type that exists. [Note that certain variables might be undefined if the fraction of the area_type considered is 0; for example the temperature of sea ice is not defined if there is no sea ice. In this case, a time-mean value can still be computed for cells containing some sea ice during at least a portion of the averaging interval because no matter what the value assumed for temperature when sea ice is missing, those values are given zero weight in computing the time-mean.]

Note that "`all_area_types” is one of the valid strings permitted for a variable with the standard_name area_type, so a cell_methods string of the form “area: mean over type1 where all_area_types” indicates the mean is calculated by integrating over the type1 portion of the grid cell and dividing by the entire area of the grid cell.

The following three examples illustrate cases when one might want to use “where” or “where … over” in defining the cell_methods:

1. Suppose that in a grid cell the fractional sea ice varies over time, but there is interest in the time-mean surface temperature of the sea ice. The time-samples, each representing a spatially-averaged sea ice temperature can be summed and then divided by the number of samples to obtain an unweighted mean where sea ice exists. This would be indicated with:

cell_methods = “area: mean where sea_ice time: mean”

2. Suppose there is interest in recording the mean fractional area covered by sea ice and the mean sea ice thickness in such a way that their product would equal the time-mean volume of sea ice in each grid cell. In this case the sea ice area would be reported as an unweighted time-mean, while the mean sea ice thickness would be calculated with time samples weighted by the fractional area of sea ice. Thus, for sea ice thickness:

cell_methods = “area: time: mean where sea_ice”

3. Suppose the time-mean contributions to total heat flux from different portions of a grid cell (e.g., ice-free and ice-covered) are of interest, and there are reasons to report these in such a way that the total heat flux is the sum of the individual contributions. Then the cell_methods attribute would be defined:

cell_methods=”area: mean where sea_ice over all_area_types time: mean”

In some cases a variable referencing a specific area_type will actually be defined even in the absence of that area_type (i.e., over the entire grid cell). Consider the surface_snow_thickness, which could sensibly be considered to be 0 in the absence of snow. In this case one might in some instances want to report “area: time: mean where snow” (giving a measure of the typical snow depth when snow exists) and in other instances “area: time: mean where snow over all_area_types” (which in this case would be identical to “area: time: mean”) or “area: time: mean where snow over land”.

4. Clarification at start of section 7.3.3 (not needed if above is accepted)

Add a clarification after this sentence in the first paragraph of 7.3.3 "Sometimes, however, it is useful to limit consideration to only a portion of a cell (e.g. a mean over the sea-ice area)", to introduce the idea of time-varying area fractions:

The portion concerned is constant in time in many cases, but it could be time-varying.

5. New example for time-varying area fractions

The following new example and explanatry text should be added in section 7.3.3:

Example 7.8: Time mean over area fractions which vary with time

float simple_mean(lat,lon):
   simple_mean:cell_methods: area: mean where sea_ice time: mean

float weighted_mean(lat,lon):
   weighted_mean:cell_methods: area: time: mean where sea_ice

float partial_mean(lat,lon):
   partial_mean:cell_methods: area: mean where sea_ice over sea time: mean

When the area fraction is varying with time, there are several different ways in which a time mean can be formulated. Three of these are illustrated in this example. Suppose, for instance, we are averaging over three time steps and the data at one grid point is -10, -6, -2 with area fractions .75, .50, .25. The values of the simple_mean, weighted_mean and partial mean are, respectively, (-10 -6 -2)/3 = -6, (-10*.75 - 6*.5 -2*.25)/(.75+.5+.25) = -7.33 , and (-10*.75 - 6*.5 -2*.25)/3 = -3.667. The partial mean provides the contribution to the mean over the entire grid cell from a specified area type. The simple mean is weighting each time period equally, while the weighted mean provides equal weighting to each unit area of sea_ice.

In example 7.8, time could be replaced by any other coordinate over which an average is taken, such as an ensemble index.

[jonathan]

Dear Martin

Thank you for making this proposal, which I support. Is the text following the example to be included in the standard document as explanation? I suppose it should be.

I think that this proposal should also include some modification to the first paragraph of 7.3.3, which at the moment refers explicitly to "horizontal area". For instance, this could be changed to "extent in the dimension(s) concerned" or something, and in the parenthesis you could add "or over the time-interval when sea-ice is present". Possibly the next paragraph might need some minor tweaking for generality as well.

Best wishes

Jonathan

[martin.juckes]

Hello Jonathan,

The answer to the first question is yes, and I've modified the layout to make it a bit clearer.

On the 2nd point, I'm not sure. What I'm proposing is clarifying the usage with time-varying area fractions, or, in principle, area fractions which vary with any other coordinate dimension (e.g. an ensemble index). At the moment the construct is still dependent on the standard name area_fraction, so it can only apply to horizonal areas.

I've added a sentence to the proposed insert to clarify that time can be replaced by another dimension.

regards, Martin

[jonathan]

Dear Martin

Thanks for the first point.

CF 7.3.3 begins "By default, the statistical method indicated by cell_methods is assumed to have been evaluated over the entire horizontal area of the cell. Sometimes, however, it is useful to limit consideration to only a portion of a cell (e.g. a mean over the sea-ice area)." My concern is that this implies that the "where" syntax is concerned only with fractions of area, which is indeed what it was intended to imply. If we are now generalising it to fractions of a cell in any possible dimension, I think that we need to make this clear at the outset.

Best wishes

Jonathan

[martin.juckes]

Dear Jonathan,

Perhaps there has been a mis-understanding: what I am proposing is clarifying the cell_methods usage for time-varying area fractions. The dimensions over which the method (mean in the example) is applied will then be area and time. The area fraction is still, however, the fraction of the cell area identifying with the specified type at any time. I was not proposing to generalise to other kinds of cell fractions,

regards, Martin

[jonathan]

Dear Martin

You could then insert some text to acknowledge that the areas could be time-varying, which is not obvious currently. Alternatively you could take a broader view, which is how it seemed to me. Consider that you only had a dimension of time and were measuring sea-ice at a point. Would it not be natural still to allow a cell_method of "time: mean where sea_ice"?

Best wishes

Jonathan

[martin.juckes]

Dear Jonathan,

I'm not sure I understand the first point: the fact that the proposed example 7.8 is about time varying area fractions is in the example title.

On the 2nd point: yes, I can see that this is possible, but I don't think it would fit in 7.3.3, which is about portions of cells. If you want to generalise the idea of cell, I think that would raise quite a number of issues. As I don't have a use case for this, I'd like to leave it out of this ticket. Another approach might be to add a new sub-section, e.g. "7.3.5 Cell methods on a masked axis" (as you would essentially be using sea_ice as a mask). This looks like a sensible generalisation to me, but it is not directly relevant to my use case, so, again, I would like to leave it out of this ticket.

regards, Martin

[jonathan]

Dear Martin

I suggest inserting the a new third sentence of the first paragraph of 7.3.3: "The portion concerned is constant in time in many cases, but it could be time-varying." This will make the generality clear.

I think the generalisation to applying "where" in time when there is no space would be obvious, but we can leave it until it's explicitly requested, as you say.

Best wishes

Jonathan

[martin.juckes]

Dear Jonathan,

First point: Thanks, I see the point now. I've added your suggested sentence to the proposal.

Second point: OK

regards, Martin

[taylor13]

Dear Martin and Jonathan,

I think it is confusing (wrong?) to consider the construct “where <type1> over <type2>” as applying only to a horizontal dimension (even if “area_type” seems to imply this). In fact a time series taken at a single point location will have some sort of “area_type” defined at that point (and it won’t be fractional). If the area_type varies over space or over time, then we can define “cell_methods = <standard_name1: [<standard_name2]>: mean where <type1> over <type2>” to mean: “integrate quantity over the dimension(s) identified by their standard_names but conditional on the co-located existence of area-type1 and then divide by the integral of the unit scalar (i.e., 1) over that same dimension(s) conditional on the co-located existence of area-type2.” I think that just because we restrict the names allowed for type1 and type2 to be those names found in the area_type controlled vocabulary doesn’t mean that in the cell_methods attribute they *must* represent an area fraction. I think more generally they can be viewed as a binary indicator of the existence of the surface type at a particular point and a particular instant of time. (After temporal or spatial averaging, of course they would in fact generally represent a fraction.)

So, I think starting section 7.3.3 by referring to the “horizontal area of a cell” should be avoided. I suggest rewriting the beginning of section 7.3.3 as follows:

---

“By default, the statistical method indicated by cell_methods is assumed to have been evaluated over the entire domain of a cell (where a cell is not necessarily restricted to spatial dimensions). Sometimes, however, it is useful to limit consideration to only a subdomain of a cell (e.g. a mean over the sea-ice area, or a time mean considering only times when snow exists). The subdomains are restricted to be identified by one of the strings permitted for a variable with the standard_name area_type. There are two options for indicating when a quantity represents a subdomain of a cell.

“The first option is used for the common case that the quantity of interest has been recorded for a single subdomain. In this case, the cell_methods attribute may include a string of the form "name: method where type". As an example, if cell_methods is “area: mean where sea_ice”, then the data would represent a mean over only the sea ice portion of the grid cell. On the other hand for a point location, if the cell_methods is “time: mean where sea_ice”, then the data would represent the time-mean at that point based on samples obtained when sea ice existed there.

“When this first option (describe in the preceding paragraph) is adopted, none of the variables appearing in the netCDF file should be given a name identical to any string recording the area_type. This restriction is imposed so that it will be clear that the data writer has not elected to adopt the second option (described in the next paragraph).

"The second approach for indicating that a statistic applies to only a portion of a cell is more general because a single variable can contain statistics for multiple area-types. In this case, the cell_methods entry is of the form "name: method where typevar". Here typevar is a string-valued auxiliary coordinate variable or string-valued scalar coordinate variable (see Section 6.1, "Labels") with a standard_name of area_type. The variable typevar contains the name(s) of the selected portion(s) of the grid cell to which the method is applied. This convention can accommodate cases in which a method is applied to more than one area type and the result is stored in a single data variable (with a dimension which ranges across the various area types). It provides a convenient way to store output from land surface models, for example, since they deal with many area types within each surface gridbox (e.g., vegetation, bare_ground, snow, etc.)."

---

The discussion of the “where … over” option after example 7.6 should also I think be rewritten:

---

“If the method is mean, various ways of calculating the mean can be distinguished in the cell_methods attribute with a string of the form “mean where` type1 [over type2]". Here, type1 can be any of the possibilities allowed for typevar or type (as specified in the paragraphs preceding the above example). The same options apply to type2, except it is not allowed to be the name of an auxiliary coordinate variable with a dimension greater than one (ignoring the dimension accommodating the maximum string length).

"A cell_methods attribute with a string of the form "area: mean where type1 over type2" indicates the mean is calculated by integrating over the type1 portion of the cell and dividing by the area of the type2 portion. When “over type2” is omitted, it is assumed to be the same as type1.

"When “area” is not the only “dimension” for which the “where… over” construct is used in a cell_methods, the interpretation more generally is that a “weighted” mean is being reported. Specifically, the quantity of interest is integrated over the specified dimension(s) with weights proportional to the fraction of “type1” area_type that exists, and then this is divided by the integral of the fraction of “type2” area_type that exists.

"Note that “all_area_types” is one of the valid strings permitted for a variable with the standard_name area_type, so a cell_methods string of the form "area: mean where all_area_types over type2" indicates the mean is calculated by integrating over all types of area and dividing by the area of the type2 portion.

"The following three examples illustrate cases when one might want to use “where” or “where … over” in defining the cell_methods:

1. Suppose that in a grid cell the fractional sea ice varies over time, but there is interest in the time-mean surface temperature of the sea ice. The time-samples, each representing a spatially-averaged sea ice temperature can be summed and then divided by the number of samples to obtain an unweighted mean where sea ice exists. This would be indicated with:

cell_methods = “area: mean where sea_ice time: mean”

2. Suppose there is interest in recording the mean fractional area covered by sea ice and the mean sea ice thickness in such a way that their product would equal the time-mean volume of sea ice in each grid cell. In this case the sea ice area would be reported as an unweighted time-mean, while the mean sea ice thickness would be calculated with time samples weighted by the fractional area of sea ice. Thus, for sea ice thickness:

cell_methods = “area: time: mean where sea_ice”

3. Suppose the time-mean contributions to total heat flux from different portions of a grid cell (e.g., ice-free and ice-covered) are of interest, and there are reasons to report these in such a way that the total heat flux is the sum of the individual contributions. Then the cell_methods attribute would be defined:

cell_methods=”area: mean where sea_ice over sea time: mean”

---

best regards, Karl

[taylor13]

Hi all,

One further note:

There is a use-case that would be nice to fit into all of this, but I suspect it would be difficult:

Consider "sea ice velocity". If you wanted to report the time-mean velocity such that ice transport could be a correctly calculated as the product of time-mean velocity and time-mean sea ice mass, then each velocity time-sample would have to be weighted by the mass of sea ice (not its area) and then we would divide by the sum of the weights. The "where ... over" construct would not cover this case, so the mass-weighting scheme would have to appear as a "comment" in cell_methods.

Karl

[taylor13]

Hi all,

And one additional note:

The 3rd example assumes that the grid cell is occupied by open ocean plus sea ice (but no land or ice shelves). This should be made clear, or the cell_methods should be changed to:

cell_methods="area: mean where sea_ice over all_surface_types time: mean

Karl

[jonathan]

Dear Karl

Thanks for thinking about this. You've proposed to rewrite parts of text which seem clear enough already to me and don't need to be rewritten for the sake of this proposal. That might be an improvement, but it makes me uneasy because I'm not sure what's changed, and it would take more work to check it! Please could you propose text with the minimum necessary modification, alongside the existing text for comparison? That would be very helpful.

I agree that the case time: mean where sea_ice is an obvious extension, but it hasn't yet been requested (see comments 7 and 8 above) and thus following our normal rule we would not add it. The same applies to your comment 10.

Best wishes

Jonathan

[martin.juckes]

Dear Karl, Jonathan,

Karl's last comment is correct: there is an inconsistency between the 3rd example and the text. I think the best resolution is to make the example consistent with the text be replacing "over sea" with "over all_area_types". The example as it stands is a possible construction, but I think the mean over the whole grid cell, as stated in the text, is the more useful example to present.

For the 2nd comment: the statement that we cannot encode a the weighted mean of a velocity, with the current convention, in such a way that the time mean mass transport is correctly represented. The approach followed by SIMIP to deal with this in CMIP6 is to request mass transport variables (for which standard names exist, e.g. sea_ice_x_transport).

I agree that we should omit "horizontal area of the" from the sentence at the start of 7.3.3: "... is assumed to have been evaluated over the entire horizontal area of the cell." We may have, for instance, an array which represents sea ice properties at a certain point in an emsemble of simulations and as a fuction of time. In this case we would still be using an area_type, but using it to identify portions of an ensemble rather than portions of a horizontal area. The shorter text " .... is assumed to have been evaluated over the entire cell" conveys all the necessary information. I agree with Jonathan, however, that we do not need to move away from the restriction to the existing area_type usage at present. A more general ... where condition_xxx is conceivable, but I'm not sure we would want to fit it into the existing area_type framework.

Reviewing the ticket again I have, however, seen an additional problem with the first example: area: mean where sea_ice time: mean .. the problem is the area: mean where sea_ice is undefined where there is no sea ice, so how do we take the time mean? There are 3 options: (1) fill with zero where there is no sea ice, (2) report the time mean as undefined or (3) take the time average over those cells where there is sea ice. The last should be encoded area: mean where sea_ice time: mean where sea_ice. As far as the convention is concerned, we should try to avoid usage which is ambiguous. I think that (2) is the best interpretation of the first example as far as the convention goes, but this probably means it is not a very useful formulation from a scientific perspective and we should perhaps advise against its use. If people want a time mean over the whole time period they should use a formulation which gives a completely unambiguous definition at each time. Thus, I would change the wording to say that the first example is ambiguous for time varying area_type and should, in general be avoided in this case.

regards, Martin

[taylor13]

proposed text

[taylor13]

proposed text pdf

[taylor13]

proposed text (clean)

[taylor13]

I have attached a word document with "track changes" turned on showing the proposed changes in text to the conventions. I also have rendered it in pdf form (also attached). The 3rd attached document is a clean version of the proposed text without mark-up. Comments welcome.

Karl Taylor

From an offline discussion, I copy the following comment from Martin Juckes:

Dear Karl,

thanks, it is good that you have tackled the section on cell_methods for area_type arrays.

There is one place where I think there is an inconsistency between your text and the text in the ticket at present: you suggest that 'the interpretation more generally is that a “weighted” mean is reported.', and that this would also apply to "area: mean where sea_ice time: mean". I had interpreted this differently, that "time: mean" should mean an unweighted mean over time, and the way to report a weighted mean was to use "area: time: mean where sea_ice". The example in the current text reflects this, giving different results for the weighted and un-weighted time mean. This followed earlier discussion with Jonathan and Jim Baird, in which I learnt that the complete cell_methods string should be treated as a sequence of operations, carried out in the order presented from left to right. I feel that it is clearer, syntactically, to keep this distinction, but would be interested to hear the views of others,

regards, Martin

[jonathan]

Dear Martin and Karl

I suppose that the sentence "The interpretation more generally ..." in Karl's text applies to the particular cell_method concerned, not to the entire cell_methods attribute. if that is so, it doesn't contradict Martin's statement, which I agree with, about different methods for different dimensions. It would be good to clarify this by specifying this remark replies to an entry in the attribute. Also, I find the phrase "when area is not the only dimension to which is applies" rather confusing. I guess this phrase is there because if the cell method is about area it surely must be area-weighted, so it's redundant rather than wrong - is that the idea, Karl? If it is, I think we could omit it. I suggest running together this paragraph and the preceding one in Karl's new text, as follows:

An entry in the cell_methods attribute with the form "area: mean where type1 over type2" indicates the mean is calculated by integrating over the type1 portion of the cell and dividing by the area of the type2 portion. When "over type2" is omitted, it is assumed to be the same as type1. When the "where" construct is used, the interpretation is that a "weighted" mean is reported. Specifically, the quantity of interest is integrated over all dimensions associated with the method with weights proportional to the fraction of type1 area_type that exists, and then this is divided by the integral over the same dimension(s) of the fraction of type2 area_type that exists.

Best wishes

Jonathan

[martin.juckes]

Hello Karl, Jonathan,

I've copied Karl's proposed text into the ticket .. I thought it might make the discussion easier. I've also split it into 3 parts .. (1) modification of existing text, (2) modification of example 7.6 caption, (3) new text. The last is a variation on my original submission .. expanded.

I'm happy with with the first two of these parts. For the 3rd part, I see an ambiguity between the text saying "the interpretation more generally is that a “weighted” mean is reported" and the first example (area: mean where sea_ice time: mean) which I would not interpret as a weighted mean. This latter interpretation is supported by Jonathan's last comment, but it is not, I think, consistent with the bracketted part of Karl's text about dealing with time series when sea_ice is completely absent for part of the time period. In such cases, I would interpret area: mean where sea_ice as yielding a time series with some missing values, when sea_ice is completely absent, and time: mean would then also give a missing value. The construct area: mean where sea_ice time: mean where sea_ice would be needed to average over the portions of the time series for which sea ice is present,

Martin

[apamment]

Hi Martin, Jonathan and Karl,

On the mailing list I expressed support for Martin's suggestion to extend the use of 'where' in cell_methods to accommodate time means and I support the changes proposed in this ticket. I agree that we should confine ourselves to addressing Martin's current use cases and leave further extensions until they are needed.

The inclusion of the additional examples and detailed explanation is very useful - without them it would be difficult to understand the proposed changes. I can see that a lot of work has already gone into making the text as clear as possible. I am struck by the fact that this section of the conventions is getting steadily longer and more complicated so to help people navigate through the text it would be useful to signpost even more clearly which parts describe the use cases for constant grid cell portions and time varying portions, respectively.

I suggest re-ordering and extending the first proposed paragraph so that it serves as an introduction and overview for the whole of section 7.3.3:

'By default, the statistical method indicated by cell_methods is assumed to have been evaluated over the entire horizontal area of the cell. Sometimes, however, it is useful to limit consideration to only a portion of a cell (e.g. a mean over the sea-ice area). Grid cell “portions” that can be considered are only those permitted to be associated with the standard_name of area_type. There are two options for indicating when a quantity represents a portion of a cell and these are explained in the following two paragraphs. The portion of the grid cell to which the statistical method is applied may remain constant in time, for example, the portion occupied by land or sea; examples 7.6 and 7.7 illustrate this use case. The remainder of the section following example 7.7 describes instances when the grid cell portion varies in time, for example, when the area occupied by sea-ice changes during the sampling period.'

The first sentence of the second paragraph would then need to be reworded slightly: 'The first method for indicating that a statistic applies to only a portion of a cell can be used for the common case that the cell_method applies to a single area-type.'

The penultimate sentence in the paragraph following the proposed example 7.8 currently reads: 'The partial mean provides the contribution to the mean over the entire grid from a specified area type.' Shouldn't this say 'entire grid cell' rather than 'entire grid'?

Best wishes, Alison

[taylor13]

Responding to Alison's last paragraph, I have changed "entire grid" to read "entire grid cell". I also changed one instance of "gridbox" to read "grid cell". That change was made to the sentence just before section 2.