I agree. We should be careful to provide guidance that is easy to follow.
thanks,
Karl
On 7/20/15 11:34 AM, Jim Biard wrote:
> Karl,
>
> I don't think it needs to be as detailed as what I wrote, but I think
> it would be good to have this potential gotcha noted in whatever we
> end up writing in the CF documentation.
>
> Jim
>
> On 7/20/15 12:51 PM, Karl Taylor wrote:
>> Dear Jim,
>>
>> Yes, there are definitely cases like the one you mention where one
>> shouldn't use the simple (approx.) gregorian calendar. Any
>> measurements taken during the leap second would have to be either
>> shifted in time or dropped. When that would be unacceptable one
>> should use either greogrian_utc or gregorian_gps (and in converting
>> from wall-clock time to elapsed time with the utc option, one would
>> have to use a formula (and look-up table) to include all leap seconds).
>>
>> best regards,
>> Karl
>>
>>
>> On 7/20/15 6:25 AM, Jim Biard wrote:
>>> Jonathan, Karl,
>>>
>>> I'm generally OK with what Karl has said. There is one more twist to
>>> Karl's gregorian calendar. I think it is definitely an edge case,
>>> but I feel it is worth noting.
>>>
>>> If you have a set of time measurements that start as UTC time stamps
>>> and span a leap second event (from June 1, 2015 to July 31, 2015 for
>>> example, since a leap second was added after 2015-06-30 23:59:59), a
>>> discontinuous shift will be introduced into the time values. The
>>> shift becomes more noticeable as the measurement precision and
>>> frequency increase, and becomes most noticeable when the precision
>>> is +-1 second or better and the frequency is 1/second or higher.
>>>
>>> The leap second added at the end of June produced a UTC timestamp of
>>> 2015-06-31 23:59:60. An elapsed time calculated from that timestamp
>>> with an algorithm that doesn't recognize leap seconds would have the
>>> same value as an elapsed time calculated from 2015-07-01 00:00:00.
>>> If your data had a measurement frequency of 1/second, you would have
>>> two elapsed times with the same value. This would violate the
>>> requirement that coordinate variables be monotonic. If your data had
>>> a measurement frequency of 10/second, you would have 10 elapsed
>>> times that repeated the previous 10.
>>>
>>> If you are measuring once per minute and your precision is +-1
>>> second, you would likely be able to find an unhandled leap second
>>> event through analysis, but it wouldn't likely affect calculations
>>> being done with the elapsed times. If you are measuring 10 times per
>>> second and your precision is +-1 microsecond (e.g. satellite data),
>>> an unhandled leap second event would generate significant errors.
>>>
>>> Grace and peace,
>>>
>>> Jim
>>>
>>> On 7/17/15 9:53 AM, Jonathan Gregory wrote:
>>>> Dear Karl
>>>>
>>>> I agree with your summary of the situation, thank you. This whole discussion
>>>> began (as the title reminds us) because of a use-case of specifying that GPS
>>>> time had been used, and not UTC time, so I expect there will be use of these
>>>> new calendars, but that most data will continue be written with gregorian,
>>>> which we will define more precisely when we make this change.
>>>>
>>>> Best wishes
>>>>
>>>> Jonathan
>>>>
>>>> ----- Forwarded message from Karl Taylor<taylor13 at llnl.gov> -----
>>>>
>>>>> Date: Wed, 15 Jul 2015 13:30:52 -0700
>>>>> From: Karl Taylor<taylor13 at llnl.gov>
>>>>> To:cf-metadata at cgd.ucar.edu
>>>>> Subject: Re: [CF-metadata] How to define time coordinate in GPS?
>>>>> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0)
>>>>> Gecko/20100101 Thunderbird/31.7.0
>>>>>
>>>>> Hi Jonathan and Jim,
>>>>>
>>>>> Thanks for bearing with me as I educate myself and think about this
>>>>> stuff. I rethought further about what I said yesterday and
>>>>> realized that maybe we were still making this too complicated.
>>>>> Jonathan's discussion helped me to come up with the following:
>>>>>
>>>>> Starting with the following claims:
>>>>>
>>>>> 1) All model data stored under CF has been generated by models with
>>>>> strict 86400 second mean sidereal days (and no leap seconds are
>>>>> necessary)
>>>>>
>>>>> 2) Up to now, the vast majority of all observational data stored
>>>>> under CF has had reference times that can be associated with the UTC
>>>>> calendar (or its predecessor GMT) and the elapsed times have been
>>>>> recorded omitting leap seconds.
>>>>>
>>>>> I make the second claim because, for example, folks doing daily
>>>>> measurements or even 3-hourly measurements collected for synoptic
>>>>> weather prediction record that data at certain times of day as
>>>>> dictated by clocks that follow UTC. For example a measurement might
>>>>> be done every day at 0Z and the person performing the measurement is
>>>>> probably not even aware that every once in a while he had to wait 1
>>>>> second longer than usual because a leap second was inserted. When
>>>>> these measurements are recorded as time series, I'd be surprised if
>>>>> anyone has inserted a leap second in computing elapsed time.
>>>>>
>>>>> This "sloppiness" in recording elapsed time actually is a blessing
>>>>> to those of us comparing climate model output to observations. If
>>>>> we want to compare monthly means, for example, we can specify
>>>>> exactly the same time bounds for both observed data and model output
>>>>> and we will extract the data of interest. If instead leap seconds
>>>>> had been accounted for in calculating elapsed time under the UTC
>>>>> system, then the time bounds would be different from models,
>>>>> certainly giving us headaches and likely leading to mistakes.
>>>>>
>>>>> I would therefore advocate to interpret "gregorian" as follows:
>>>>>
>>>>> This calendar is based on the assumption that the length of the mean
>>>>> solar day is exactly 86400 seconds. For observations, the
>>>>> assumption is not in fact exactly correct. To facilitate
>>>>> comparisons between model results and climate and weather
>>>>> observations, an adjustment in the true elapsed time is required
>>>>> when calendar="gregorian". The reference time should be specified
>>>>> according to the UTC time system, but leap seconds should be omitted
>>>>> in converting from wall-clock time to elapsed time. This means that
>>>>> elapsed time values will only be approximately correct, but this is
>>>>> generally desirable because: 1) simple algorithms can be used to
>>>>> convert from wall-clock time to elapsed time, 2) the mismatch
>>>>> between the model's mean solar day and the actual mean solar day
>>>>> (about .002 seconds) can be ignored in comparing models and
>>>>> observations; observed elapsed times are effectively modified to
>>>>> yield a time series that stays synced with the diurnal cycle we
>>>>> would find if the earth's day were exactly 86400 seconds long. Note
>>>>> that although elapsed times stored in the CF-conforming file will be
>>>>> only approximate, they will differ by less than 17 seconds (as of
>>>>> July 2015) from the true elapsed time.
>>>>>
>>>>> I think this definition means that there is no need for gregorian_utc_nls.
>>>>>
>>>>> Although I can see why the two new calendars (gregorian_utc and
>>>>> gregorian_gps) could be used for special cases, I would hope most
>>>>> providers of climate and weather data will continue to use a
>>>>> "gregorian" calendar as defined above. This will make it much
>>>>> easier for modelers to compare their output to observations.
>>>>>
>>>>> best regards,
>>>>> Karl
>>>>>
>>>>>
>>>>> On 7/15/15 1:34 AM, Jonathan Gregory wrote:
>>>>>> PS. First, to repeat myself:
>>>>>>
>>>>>> So my conclusion (at present - but I suspect I haven't understood something
>>>>>> you have said) is that gregorian is fine and sufficient (with the addition of
>>>>>> gregorian_utc and gregorian_gps for the well-defined real-world systems) if
>>>>>> we define it to mean that 86400-seconds are assumed to be used for conversion
>>>>> >from clock time to elapsed time and elapsed time to clock time, that it is not
>>>>>> defined whether the reference time and clock times are GPS and UTC, and that
>>>>>> the elapsed times may not be exactly correct for the real world (due to the
>>>>>> neglect of leap seconds).
>>>>>>
>>>>>> I also said, "This is then the same as gregorian_utc_nls, so we would not need
>>>>>> that either, which was your final conclusion in your previous post." Actually
>>>>>> it's not quite the same as gregorian_utc_nls, which asserts that clock times,
>>>>>> if real-world, are UTC. Otherwise it's the same. If my conclusion above is
>>>>>> correct, then I don't mind whether we introduce gregorian_utc_nls or not. No-
>>>>>> one has definitely asked for it, as far as I remember; we discussed it because
>>>>>> of anticipating the need, I believe. We could therefore follow the usual CF
>>>>>> principle of omitting it until it's asked for.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Jonathan
>>>>>>
>>>>>> ----- Forwarded message from Jonathan Gregory <j.m.gregory at reading.ac.uk-----
>>>>>>
>>>>>> Date: Wed, 15 Jul 2015 08:01:04 +0100
>>>>>> From: Jonathan Gregory<j.m.gregory at reading.ac.uk>
>>>>>> To:cf-metadata at cgd.ucar.edu
>>>>>> Subject: [CF-metadata] How to define time coordinate in GPS?
>>>>>> User-Agent: Mutt/1.5.21 (2010-09-15)
>>>>>>
>>>>>> Dear Karl
>>>>>>
>>>>>> I think the meaning of "gregorian" up to CF 1.6 is actually not clear, because
>>>>>> we had not thought about these distinctions earlier. If we change the calendar
>>>>>> definitions, it does not affect the interpretation of any existing data written
>>>>>> with previous versions. However I agree that it would be inconvenient for data-
>>>>>> users if they had to process "gregorian" times differently according to the
>>>>>> setting of the Conventions version.
>>>>>>
>>>>>> I thought that the current proposal was that gregorian would be vague in
>>>>>> future, like you wrote in your second option:
>>>>>>
>>>>>> 1) might or might not account for leap seconds, 2) might or might not assume
>>>>>> the length of the solar day is exactly 86400 seconds long, and 3) might express
>>>>>> the reference time according to either UTC or GPS
>>>>>>
>>>>>> and that is exactly the reason why, after agreeing with Jim last week, I
>>>>>> changed my mind to argue that we needed gregorian_nls for model data, to
>>>>>> preserve the exactness for times which really are always 86400-second days.
>>>>>> You agree we would need gregorian_nls in that case. I agree that it means that
>>>>>> models would use gregorian_nls in future, not gregorian.
>>>>>>
>>>>>> However in your first option you don't think this is necessary:
>>>>>>
>>>>>> Under the "gregorian" calendar the length of the solar day can be assumed to be
>>>>>> exactly 86400 seconds long (i.e., there are no leap seconds). This means that
>>>>>> for models where this assumption almost invariably is valid, conversion from
>>>>>> elapsed time to clock time is straight-forward and exact, whereas for
>>>>>> observations, conversion to clock time may introduce errors as large as 16
>>>>>> seconds because it is unknown whether the UTC or GPS time system has been used
>>>>>> in specifying the reference times (appearing in the time units attribute), and
>>>>>> it is also unknown whether leap seconds have been properly accounted for in
>>>>>> converting UTC clock times to elapsed time.
>>>>>>
>>>>>> I'm sorry, but I can't see what is the difference. In both cases you say it
>>>>>> is unknown whether leap seconds have been included in converting from clock
>>>>>> time to elapsed time. Is the crucial difference about the length of the day?
>>>>>> Does "assume the solar day to be exactly 86400 seconds" mean "assume all days
>>>>>> are 86400 seconds in converting elapsed time to clock time"?
>>>>>>
>>>>>> If that's what you mean, I think your first and preferred option is fine, and
>>>>>> then I agree we do not need gregorian_nls. To spell it out, I think this is
>>>>>> acceptable if "gregorian" means that you will recover the clock times (time-
>>>>>> stamps) exactly by using an algorithm that assumes constant 86400-second days
>>>>>> (i.e. with no leap seconds). If these clock times refer to the real world, it
>>>>>> is undefined whether the reference time is GPS or UTC, and consequently it is
>>>>>> undefined whether the decoded clock times are GPS or UTC.
>>>>>>
>>>>>> But my statement "you will recover the clock times (time-stamps) exactly by
>>>>>> using an algorithm that assumes constant 86400-second days" is inconsistent
>>>>>> with your statement "conversion to clock time may introduce errors ... because
>>>>>> it is unknown whether ... leap seconds have been properly accounted for in
>>>>>> converting UTC clock times to elapsed time". According to my statement,
>>>>>> gregorian means that leap seconds were ignored when converting clock times to
>>>>>> elapsed times i.e. 86400-second days were used. If that's not the case, you
>>>>>> can't decode accurately. Thus the elapsed times may not be quite right for the
>>>>>> real world. This is then the same as gregorian_utc_nls, so we would not need
>>>>>> that either, which was your final conclusion in your previous post.
>>>>>>
>>>>>> So my conclusion (at present - but I suspect I haven't understood something
>>>>>> you have said) is that gregorian is fine and sufficient (with the addition of
>>>>>> gregorian_utc and gregorian_gps for the well-defined real-world systems) if
>>>>>> we define it to mean that 86400-seconds are assumed to be used for conversion
>>>>> >from clock time to elapsed time and elapsed time to clock time, that it is not
>>>>>> defined whether the reference time and clock times are GPS and UTC, and that
>>>>>> the elapsed times may not be exactly correct for the real world (due to the
>>>>>> neglect of leap seconds).
>>>>>>
>>>>>> Best wishes
>>>>>>
>>>>>> Jonathan
>>>>>> _______________________________________________
>>>>>> CF-metadata mailing list
>>>>>> CF-metadata at cgd.ucar.edu
>>>>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>>>> _______________________________________________
>>>>> CF-metadata mailing list
>>>>> CF-metadata at cgd.ucar.edu
>>>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>>> ----- End forwarded message -----
>>>> _______________________________________________
>>>> CF-metadata mailing list
>>>> CF-metadata at cgd.ucar.edu
>>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>>
>>> --
>>> CICS-NC <http://www.cicsnc.org/> Visit us on
>>> Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
>>> *Research Scholar*
>>> Cooperative Institute for Climate and Satellites NC
>>> <http://cicsnc.org/>
>>> North Carolina State University <http://ncsu.edu/>
>>> NOAA National Centers for Environmental Information
>>> <http://ncdc.noaa.gov/>
>>> /formerly NOAA?s National Climatic Data Center/
>>> 151 Patton Ave, Asheville, NC 28801
>>> e: jbiard at cicsnc.org <mailto:jbiard at cicsnc.org>
>>> o: +1 828 271 4900
>>>
>>> /Connect with us on Facebook for climate
>>> <https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
>>> <https://www.facebook.com/NOAANCEIoceangeo> information, and follow
>>> us on Twitter at _at_NOAANCEIclimate
>>> <https://twitter.com/NOAANCEIclimate> and _at_NOAANCEIocngeo
>>> <https://twitter.com/NOAANCEIocngeo>. /
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> CF-metadata mailing list
>>> CF-metadata at cgd.ucar.edu
>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>
>>
>>
>> _______________________________________________
>> CF-metadata mailing list
>> CF-metadata at cgd.ucar.edu
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
> --
> CICS-NC <http://www.cicsnc.org/> Visit us on
> Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
> *Research Scholar*
> Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
> North Carolina State University <http://ncsu.edu/>
> NOAA National Centers for Environmental Information
> <http://ncdc.noaa.gov/>
> /formerly NOAA?s National Climatic Data Center/
> 151 Patton Ave, Asheville, NC 28801
> e: jbiard at cicsnc.org <mailto:jbiard at cicsnc.org>
> o: +1 828 271 4900
>
> /Connect with us on Facebook for climate
> <https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
> <https://www.facebook.com/NOAANCEIoceangeo> information, and follow us
> on Twitter at _at_NOAANCEIclimate <https://twitter.com/NOAANCEIclimate>
> and _at_NOAANCEIocngeo <https://twitter.com/NOAANCEIocngeo>. /
>
>
>
>
> _______________________________________________
> CF-metadata mailing list
> CF-metadata at cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20150720/3d915348/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 15784 bytes
Desc: not available
URL: <
http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20150720/3d915348/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 15784 bytes
Desc: not available
URL: <
http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20150720/3d915348/attachment-0003.png>
Received on Mon Jul 20 2015 - 13:15:29 BST
