Karl,
I don't think it needs to be as detailed as what I wrote, but I think it
would be good to have this potential gotcha noted in whatever we end up
writing in the CF documentation.
Jim
On 7/20/15 12:51 PM, Karl Taylor wrote:
> Dear Jim,
>
> Yes, there are definitely cases like the one you mention where one
> shouldn't use the simple (approx.) gregorian calendar. Any
> measurements taken during the leap second would have to be either
> shifted in time or dropped. When that would be unacceptable one
> should use either greogrian_utc or gregorian_gps (and in converting
> from wall-clock time to elapsed time with the utc option, one would
> have to use a formula (and look-up table) to include all leap seconds).
>
> best regards,
> Karl
>
>
> On 7/20/15 6:25 AM, Jim Biard wrote:
>> Jonathan, Karl,
>>
>> I'm generally OK with what Karl has said. There is one more twist to
>> Karl's gregorian calendar. I think it is definitely an edge case, but
>> I feel it is worth noting.
>>
>> If you have a set of time measurements that start as UTC time stamps
>> and span a leap second event (from June 1, 2015 to July 31, 2015 for
>> example, since a leap second was added after 2015-06-30 23:59:59), a
>> discontinuous shift will be introduced into the time values. The
>> shift becomes more noticeable as the measurement precision and
>> frequency increase, and becomes most noticeable when the precision is
>> +-1 second or better and the frequency is 1/second or higher.
>>
>> The leap second added at the end of June produced a UTC timestamp of
>> 2015-06-31 23:59:60. An elapsed time calculated from that timestamp
>> with an algorithm that doesn't recognize leap seconds would have the
>> same value as an elapsed time calculated from 2015-07-01 00:00:00. If
>> your data had a measurement frequency of 1/second, you would have two
>> elapsed times with the same value. This would violate the requirement
>> that coordinate variables be monotonic. If your data had a
>> measurement frequency of 10/second, you would have 10 elapsed times
>> that repeated the previous 10.
>>
>> If you are measuring once per minute and your precision is +-1
>> second, you would likely be able to find an unhandled leap second
>> event through analysis, but it wouldn't likely affect calculations
>> being done with the elapsed times. If you are measuring 10 times per
>> second and your precision is +-1 microsecond (e.g. satellite data),
>> an unhandled leap second event would generate significant errors.
>>
>> Grace and peace,
>>
>> Jim
>>
>> On 7/17/15 9:53 AM, Jonathan Gregory wrote:
>>> Dear Karl
>>>
>>> I agree with your summary of the situation, thank you. This whole discussion
>>> began (as the title reminds us) because of a use-case of specifying that GPS
>>> time had been used, and not UTC time, so I expect there will be use of these
>>> new calendars, but that most data will continue be written with gregorian,
>>> which we will define more precisely when we make this change.
>>>
>>> Best wishes
>>>
>>> Jonathan
>>>
>>> ----- Forwarded message from Karl Taylor<taylor13 at llnl.gov> -----
>>>
>>>> Date: Wed, 15 Jul 2015 13:30:52 -0700
>>>> From: Karl Taylor<taylor13 at llnl.gov>
>>>> To:cf-metadata at cgd.ucar.edu
>>>> Subject: Re: [CF-metadata] How to define time coordinate in GPS?
>>>> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0)
>>>> Gecko/20100101 Thunderbird/31.7.0
>>>>
>>>> Hi Jonathan and Jim,
>>>>
>>>> Thanks for bearing with me as I educate myself and think about this
>>>> stuff. I rethought further about what I said yesterday and
>>>> realized that maybe we were still making this too complicated.
>>>> Jonathan's discussion helped me to come up with the following:
>>>>
>>>> Starting with the following claims:
>>>>
>>>> 1) All model data stored under CF has been generated by models with
>>>> strict 86400 second mean sidereal days (and no leap seconds are
>>>> necessary)
>>>>
>>>> 2) Up to now, the vast majority of all observational data stored
>>>> under CF has had reference times that can be associated with the UTC
>>>> calendar (or its predecessor GMT) and the elapsed times have been
>>>> recorded omitting leap seconds.
>>>>
>>>> I make the second claim because, for example, folks doing daily
>>>> measurements or even 3-hourly measurements collected for synoptic
>>>> weather prediction record that data at certain times of day as
>>>> dictated by clocks that follow UTC. For example a measurement might
>>>> be done every day at 0Z and the person performing the measurement is
>>>> probably not even aware that every once in a while he had to wait 1
>>>> second longer than usual because a leap second was inserted. When
>>>> these measurements are recorded as time series, I'd be surprised if
>>>> anyone has inserted a leap second in computing elapsed time.
>>>>
>>>> This "sloppiness" in recording elapsed time actually is a blessing
>>>> to those of us comparing climate model output to observations. If
>>>> we want to compare monthly means, for example, we can specify
>>>> exactly the same time bounds for both observed data and model output
>>>> and we will extract the data of interest. If instead leap seconds
>>>> had been accounted for in calculating elapsed time under the UTC
>>>> system, then the time bounds would be different from models,
>>>> certainly giving us headaches and likely leading to mistakes.
>>>>
>>>> I would therefore advocate to interpret "gregorian" as follows:
>>>>
>>>> This calendar is based on the assumption that the length of the mean
>>>> solar day is exactly 86400 seconds. For observations, the
>>>> assumption is not in fact exactly correct. To facilitate
>>>> comparisons between model results and climate and weather
>>>> observations, an adjustment in the true elapsed time is required
>>>> when calendar="gregorian". The reference time should be specified
>>>> according to the UTC time system, but leap seconds should be omitted
>>>> in converting from wall-clock time to elapsed time. This means that
>>>> elapsed time values will only be approximately correct, but this is
>>>> generally desirable because: 1) simple algorithms can be used to
>>>> convert from wall-clock time to elapsed time, 2) the mismatch
>>>> between the model's mean solar day and the actual mean solar day
>>>> (about .002 seconds) can be ignored in comparing models and
>>>> observations; observed elapsed times are effectively modified to
>>>> yield a time series that stays synced with the diurnal cycle we
>>>> would find if the earth's day were exactly 86400 seconds long. Note
>>>> that although elapsed times stored in the CF-conforming file will be
>>>> only approximate, they will differ by less than 17 seconds (as of
>>>> July 2015) from the true elapsed time.
>>>>
>>>> I think this definition means that there is no need for gregorian_utc_nls.
>>>>
>>>> Although I can see why the two new calendars (gregorian_utc and
>>>> gregorian_gps) could be used for special cases, I would hope most
>>>> providers of climate and weather data will continue to use a
>>>> "gregorian" calendar as defined above. This will make it much
>>>> easier for modelers to compare their output to observations.
>>>>
>>>> best regards,
>>>> Karl
>>>>
>>>>
>>>> On 7/15/15 1:34 AM, Jonathan Gregory wrote:
>>>>> PS. First, to repeat myself:
>>>>>
>>>>> So my conclusion (at present - but I suspect I haven't understood something
>>>>> you have said) is that gregorian is fine and sufficient (with the addition of
>>>>> gregorian_utc and gregorian_gps for the well-defined real-world systems) if
>>>>> we define it to mean that 86400-seconds are assumed to be used for conversion
>>>> >from clock time to elapsed time and elapsed time to clock time, that it is not
>>>>> defined whether the reference time and clock times are GPS and UTC, and that
>>>>> the elapsed times may not be exactly correct for the real world (due to the
>>>>> neglect of leap seconds).
>>>>>
>>>>> I also said, "This is then the same as gregorian_utc_nls, so we would not need
>>>>> that either, which was your final conclusion in your previous post." Actually
>>>>> it's not quite the same as gregorian_utc_nls, which asserts that clock times,
>>>>> if real-world, are UTC. Otherwise it's the same. If my conclusion above is
>>>>> correct, then I don't mind whether we introduce gregorian_utc_nls or not. No-
>>>>> one has definitely asked for it, as far as I remember; we discussed it because
>>>>> of anticipating the need, I believe. We could therefore follow the usual CF
>>>>> principle of omitting it until it's asked for.
>>>>>
>>>>> Cheers
>>>>>
>>>>> Jonathan
>>>>>
>>>>> ----- Forwarded message from Jonathan Gregory <j.m.gregory at reading.ac.uk-----
>>>>>
>>>>> Date: Wed, 15 Jul 2015 08:01:04 +0100
>>>>> From: Jonathan Gregory<j.m.gregory at reading.ac.uk>
>>>>> To:cf-metadata at cgd.ucar.edu
>>>>> Subject: [CF-metadata] How to define time coordinate in GPS?
>>>>> User-Agent: Mutt/1.5.21 (2010-09-15)
>>>>>
>>>>> Dear Karl
>>>>>
>>>>> I think the meaning of "gregorian" up to CF 1.6 is actually not clear, because
>>>>> we had not thought about these distinctions earlier. If we change the calendar
>>>>> definitions, it does not affect the interpretation of any existing data written
>>>>> with previous versions. However I agree that it would be inconvenient for data-
>>>>> users if they had to process "gregorian" times differently according to the
>>>>> setting of the Conventions version.
>>>>>
>>>>> I thought that the current proposal was that gregorian would be vague in
>>>>> future, like you wrote in your second option:
>>>>>
>>>>> 1) might or might not account for leap seconds, 2) might or might not assume
>>>>> the length of the solar day is exactly 86400 seconds long, and 3) might express
>>>>> the reference time according to either UTC or GPS
>>>>>
>>>>> and that is exactly the reason why, after agreeing with Jim last week, I
>>>>> changed my mind to argue that we needed gregorian_nls for model data, to
>>>>> preserve the exactness for times which really are always 86400-second days.
>>>>> You agree we would need gregorian_nls in that case. I agree that it means that
>>>>> models would use gregorian_nls in future, not gregorian.
>>>>>
>>>>> However in your first option you don't think this is necessary:
>>>>>
>>>>> Under the "gregorian" calendar the length of the solar day can be assumed to be
>>>>> exactly 86400 seconds long (i.e., there are no leap seconds). This means that
>>>>> for models where this assumption almost invariably is valid, conversion from
>>>>> elapsed time to clock time is straight-forward and exact, whereas for
>>>>> observations, conversion to clock time may introduce errors as large as 16
>>>>> seconds because it is unknown whether the UTC or GPS time system has been used
>>>>> in specifying the reference times (appearing in the time units attribute), and
>>>>> it is also unknown whether leap seconds have been properly accounted for in
>>>>> converting UTC clock times to elapsed time.
>>>>>
>>>>> I'm sorry, but I can't see what is the difference. In both cases you say it
>>>>> is unknown whether leap seconds have been included in converting from clock
>>>>> time to elapsed time. Is the crucial difference about the length of the day?
>>>>> Does "assume the solar day to be exactly 86400 seconds" mean "assume all days
>>>>> are 86400 seconds in converting elapsed time to clock time"?
>>>>>
>>>>> If that's what you mean, I think your first and preferred option is fine, and
>>>>> then I agree we do not need gregorian_nls. To spell it out, I think this is
>>>>> acceptable if "gregorian" means that you will recover the clock times (time-
>>>>> stamps) exactly by using an algorithm that assumes constant 86400-second days
>>>>> (i.e. with no leap seconds). If these clock times refer to the real world, it
>>>>> is undefined whether the reference time is GPS or UTC, and consequently it is
>>>>> undefined whether the decoded clock times are GPS or UTC.
>>>>>
>>>>> But my statement "you will recover the clock times (time-stamps) exactly by
>>>>> using an algorithm that assumes constant 86400-second days" is inconsistent
>>>>> with your statement "conversion to clock time may introduce errors ... because
>>>>> it is unknown whether ... leap seconds have been properly accounted for in
>>>>> converting UTC clock times to elapsed time". According to my statement,
>>>>> gregorian means that leap seconds were ignored when converting clock times to
>>>>> elapsed times i.e. 86400-second days were used. If that's not the case, you
>>>>> can't decode accurately. Thus the elapsed times may not be quite right for the
>>>>> real world. This is then the same as gregorian_utc_nls, so we would not need
>>>>> that either, which was your final conclusion in your previous post.
>>>>>
>>>>> So my conclusion (at present - but I suspect I haven't understood something
>>>>> you have said) is that gregorian is fine and sufficient (with the addition of
>>>>> gregorian_utc and gregorian_gps for the well-defined real-world systems) if
>>>>> we define it to mean that 86400-seconds are assumed to be used for conversion
>>>> >from clock time to elapsed time and elapsed time to clock time, that it is not
>>>>> defined whether the reference time and clock times are GPS and UTC, and that
>>>>> the elapsed times may not be exactly correct for the real world (due to the
>>>>> neglect of leap seconds).
>>>>>
>>>>> Best wishes
>>>>>
>>>>> Jonathan
>>>>> _______________________________________________
>>>>> CF-metadata mailing list
>>>>> CF-metadata at cgd.ucar.edu
>>>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>>> _______________________________________________
>>>> CF-metadata mailing list
>>>> CF-metadata at cgd.ucar.edu
>>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>> ----- End forwarded message -----
>>> _______________________________________________
>>> CF-metadata mailing list
>>> CF-metadata at cgd.ucar.edu
>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>
>> --
>> CICS-NC <http://www.cicsnc.org/> Visit us on
>> Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
>> *Research Scholar*
>> Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
>> North Carolina State University <http://ncsu.edu/>
>> NOAA National Centers for Environmental Information
>> <http://ncdc.noaa.gov/>
>> /formerly NOAA?s National Climatic Data Center/
>> 151 Patton Ave, Asheville, NC 28801
>> e: jbiard at cicsnc.org <mailto:jbiard at cicsnc.org>
>> o: +1 828 271 4900
>>
>> /Connect with us on Facebook for climate
>> <https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
>> <https://www.facebook.com/NOAANCEIoceangeo> information, and follow
>> us on Twitter at _at_NOAANCEIclimate
>> <https://twitter.com/NOAANCEIclimate> and _at_NOAANCEIocngeo
>> <https://twitter.com/NOAANCEIocngeo>. /
>>
>>
>>
>>
>> _______________________________________________
>> CF-metadata mailing list
>> CF-metadata at cgd.ucar.edu
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
>
> _______________________________________________
> CF-metadata mailing list
> CF-metadata at cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
CICS-NC <http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/>
/formerly NOAA?s National Climatic Data Center/
151 Patton Ave, Asheville, NC 28801
e: jbiard at cicsnc.org <mailto:jbiard at cicsnc.org>
o: +1 828 271 4900
/Connect with us on Facebook for climate
<https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
<https://www.facebook.com/NOAANCEIoceangeo> information, and follow us
on Twitter at _at_NOAANCEIclimate <https://twitter.com/NOAANCEIclimate> and
_at_NOAANCEIocngeo <https://twitter.com/NOAANCEIocngeo>. /
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20150720/f6b7f616/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 15784 bytes
Desc: not available
URL: <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20150720/f6b7f616/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CicsLogoTiny.png
Type: image/png
Size: 15784 bytes
Desc: not available
URL: <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20150720/f6b7f616/attachment-0003.png>
Received on Mon Jul 20 2015 - 12:34:47 BST
