Karl,
On 6/29/15 5:58 PM, Karl Taylor wrote:
> Hi all,
>
> After further reflection, I would like to clarify and slightly modify
> my proposal. I suggest:
>
> 1) CF not try to accommodate folks using "wrong" software.
>
I, in theory, agree with you, but it is actually not easy to get "right"
software for handling UTC time encoding (going from a timestamp to an
elapsed time) and decoding (going from an elapsed time to a timestamp).
Leap seconds (see detailed explanation below) don't follow a fixed
pattern like leap days do. The software for handling UTC must either
contact a site on the Internet that has the required information, or it
has some internal table or file containing the information that must be
kept up to date. (Note that the standard time functions for most
computers and languages don't handle leap seconds.) I don't think that
we should tell people that don't need the accuracy or precision that
they have to jump through such hoops. I think it's more important to
give data producers that have taken the trouble to make precise and
accurate elapsed times a way to indicate to data consumers that they
have done so.
> 2) we relax our requirement that udunits be able to handle the time
> coordinate because it won't recognize and interpret "UTC" and "GPS".
>
I like this idea. I had first proposed a time_system attribute to
contain this information, but that was not liked, and I then suggested
adding a separate word to the calendar attribute, such as 'gregorian
utc'. That was not liked either. Perhaps this idea will meet greater
acceptance. We could always petition UDUNITS to accept the presence of a
time system declaration, but they probably don't want to get involved
with doing any actual conversions based on time system differences.
> 3) If the units attribute indicates that "UTC" is used for reference
> time, this would imply that "UTC" should also be used to record the
> time coordinate values. If the units attribute indicates that "GPS" is
> used for reference time, this would imply that "GPS" should also be
> used to record the time coordinate values.
>
The identification of the time system used (UTC, GPS, etc) only applies
to the reference timestamp. I'm using timestamp as the name for a time
represented in a clock readout form such as yyyy-mm-dd hh:mm:ss. Elapsed
times are wholly independent of time system (as long as all systems
considered use the SI definition of 1 second). The elapsed times in time
variables may not have ever existed as timestamps. If you started with
GPS timestamps, then you surely should use "GPS" encoding functions
(ones that don't consider leap seconds) to produce your elapsed times,
and if you started with UTC timestamps you should use "UTC" encoding
functions (ones that do consider leap seconds). But your source data
could be a start timestamp in GPS and elapsed times associated with each
acquired measurement, and you might want to store the reference
timestamp as UTC. All you have to do is correctly convert your GPS
timestamp to UTC and you are done. There are too many different ways to
get to a reference timestamp and a set of elapsed times to make
statements like 3).
> 4) I originally proposed that there be no difference between "GPS"
> and the current units attribute. I now see that this won't allow us
> to handle all cases. So, I now propose that if "GPS" or "UTC" are not
> included as part of the units attribute (as is the case in all current
> CF-compliant files), then we should interpret this as meaning that the
> data provider has assumed that the calendar and earth orbital period
> neglect leap seconds entirely (and that leaving them out doesn't
> matter to them). The "no suffix" case would apply to all models I
> know of and observational data where differences in sampling time of
> up to 16 seconds (as of now) can be tolerated.
>
Yes. Whatever way we do this, there needs to be a way to accommodate
non-physical (model) time, and to provide a path for those who don't
need accuracy or precision great enough for all of this to become an issue.
> The above means that when comparing datasets where we want to consider
> the state of the system at a particular time of year (i.e., at the
> same orbital longitude), we should make sure to correctly compute the
> time-stamps from the elapsed time (which under CF is what is stored as
> a coordinate variable). Different algorithms would have to be used
> depending on whether or not the reference time stored as part of the
> units attribute includes the "GPS" suffix or the "UTC" suffix. The
> algorithm used to compute the "UTC" timestamp could also be used to
> generate the time-stamp for the normal model case which has no leap
> seconds (i.e. where the units attribute omits both "GPS" and "UTC").
> Once the time-stamps are known, then one could convert these back to
> elapsed time using a single approach (say, UTC) for all datasets
> (using a common reference time). We would then know how the sampling
> times across datasets were staggered, and the sample values could be
> interpolated in time to a common set of times. Going to this much
> trouble would, of course, be unwarranted in many cases. In fact I
> think only if one of the datasets is "GPS" would we ever have to do
> this. Even for GPS, if up to a 16 second errors in time could be
> tolerated, one could simply take the original time-values for each
> dataset and interpolate the data values to common times.
>
Note that leap seconds relate primarily to the Earth's rotational
motion, not orbital motion. Leap days relate primarily to orbital
motion. But anyway, you are right about the need to make sure you are
comparing different time variables correctly. The method you described
is overly complicated. If you have some time variables with GPS
reference timestamps and others with UTC reference timestamps, you only
need to convert the GPS reference timestamps to UTC (or UTC to GPS). You
can then calculate the offsets between the different reference
timestamps (using the proper GPS or UTC time functions) and apply them
to the elapsed times to get them all on a common footing. There is no
need to convert all elapsed times to timestamps and then convert them
back. But it that is easier for some reason, you certainly could do so.
There's more than one way to do the job.
As far as model time systems are concerned, they bear a closer
resemblance to the GPS time system than the UTC time system, since they
generally don't consider leap seconds. Model time systems have no direct
connection to the real Earth-Sun system (many also have 360, 365, or 366
day calendars), so comparisons between their time values and
observational data time values just about has to be done in a symbolic
fashion using timestamps.
Grace and peace,
Jim
> best regards,
> Karl
>
>
>
>
> On 6/29/15 11:36 AM, Jim Biard wrote:
>> Karl,
>>
>> On 6/29/15 2:00 PM, Karl Taylor wrote:
>>> Hi all,
>>>
>>> I haven't followed all this as closely as I would have liked, but
>>> will hazard some comments anyway:
>>>
>>> 1. I think we should require that elapsed time (recorded by the
>>> time-coordinate in CF files) be correct no matter what the
>>> calendar. So samples taken at a specific interval have identical
>>> time-differences (calculated from the difference between successive
>>> time coordinate values) whether or not the a leap second was
>>> introduced (or dropped?) or not.
>>>
>> I agree, in principle. The way leap seconds worm their way into
>> elapsed times is by using the "wrong" software when calculating
>> elapsed times from timestamps. POSIX (that is, Unix/linux) time
>> conversion functions do not normally account for leap seconds, so if
>> you use them with UTC times, you put yourself at risk a couple of
>> different ways.
>>> 2. I think we should allow for basetime (the reference time stamp)
>>> to be recorded either as UTC or GPS. Absence of either of these
>>> would imply "GPS", which will therefore apply to all data written
>>> until now under CF.
>>>
>> This is great for real-world acquired times, but model times are in a
>> non-real time system - neither GPS nor UTC. There is also the
>> possibility of reference times recorded in the TAI time system, but
>> we can handle that by adding yet another calendar (and so on for
>> sidereal time, etc, etc).
>>> 3. Folks *generating* CF files should use algorithms that correctly
>>> convert their timestamps to elapsed time (which is recorded in the
>>> files). Then users can regenerate the timestamps correctly by
>>> looking at whether UTC or GPS (or neither) appears.
>>>
>> They should, but many haven't in the past, and the added complexity
>> of getting UTC time conversion functions that understand leap seconds
>> (one is going to be added at midnight July 1!) is overkill. When your
>> time resolution is on the order of 1 hour, none of this matters.
>>> 4. Couldn't all of the above be simply accommodated with a single
>>> "gregorian" calendar, but with the basetime (reference time stamp in
>>> the units attribute) including either "UTC" or "GPS" at the end (or
>>> neither for compatibility with previous written CF data)? Examples:
>>> "days since 1990-1-1 0:0:0UTC" or "days since 1950-1-1 0:0:0 GPS"
>>> (which would be equivalent to "days since 1950-1-1 0:0:0"
>>>
>> This would be another way to tackle it. In the past, CF also allowed
>> for time zone offsets, so you would need to add that as well.
>>> 5. Data would not be considered CF-compliant if the elapsed time
>>> (recorded by the time coordinate) were incorrect because it had been
>>> incorrectly converted by the data provider. "UTC" would indicate
>>> that the basetime and conversion of elapsed time to timestamps
>>> should follow the rules of UTC and include leap seconds. "GPS" (or
>>> absence of "UTC" for backward compatibility) would indicate that
>>> thebasetime and conversion of elapsed time to timestamps should
>>> follow the rules of GPS and *not* include leap seconds.
>>>
>> Note that UTC and GPS in the reference times (or in the calendar)
>> don't say anything about how to create timestamps. It tells you how
>> to read the reference time. How you produce timestamps from the
>> elapsed times is up to the data consumer. It is entirely correct to
>> take a time variable with a reference time in GPS, correctly convert
>> the reference time to UTC, and then produce UTC timestamps. The
>> information in the time variable tells me what I have in hand (what
>> the data producer did), not what I'm supposed to do with it.
>>> I'm sure I have missed some important use case where the above
>>> simple scheme would be inadequate. (Or perhaps I'm just completely
>>> out to lunch, in which case please forgive me).
>>>
>> From my perspective there are two gaps. One is model time, which is
>> entirely non-physical (has no reference point that ties it into the
>> real world), and the other is the very real and likely continuing
>> case where the (in-)precision of the time measurements makes the
>> whole thing moot.
>>
>> I'm thinking that Nan's suggestion of specifying an uncertainty for
>> the elapsed time values (which when left off could imply that
>> "anything goes" as far as leap seconds is concerned) may be an
>> elegant way through this.
>>
>> We could (whether in the calendar or the units) have:
>>
>> * gregorian, which would not take an uncertainty attribute and
>> would indicate that the time base and precision are unknown (good
>> for models and for backward compatibility)
>> * gregorian + utc, which would take an uncertainty attribute and
>> would indicate that the reference time stamp is expressed in UTC,
>> and that elapsed times have no artifacts to the level of the
>> uncertainty (true UTC with "wrong" conversions but only accurate
>> to 1 minute would validly fit in this category)
>> * gregorian + gps, which would be like gregorian + utc, except that
>> the reference time stamp is expressed in GPS.
>>
>>
>> Grace and peace,
>>
>> Jim
>>> best regards,
>>> Karl
>>>
>>>
>>>
>>>
>>> On 6/29/15 9:21 AM, Jonathan Gregory wrote:
>>>> Dear Tim and Nan
>>>>
>>>> If I have understood correctly, I think your two emails suggest that we do need
>>>> a distinction of the precise and imprecise cases. As usual, I believe that CF
>>>> should not prescribe to users what they should do; its aim is to allow them to
>>>> describe what they have done. Different levels of precision are needed for
>>>> different datasets.
>>>>
>>>> Following the emails that Jim and I exchanged, we could distinguish:
>>>>
>>>> gregorian: Real world-times, but without specifying whether UTC or GPS
>>>> timestamps are intended, nor whether the encoding was done with or without leap
>>>> seconds. The decoded times could differ by several seconds from UTC. I think
>>>> this is Nan's use-case.
>>>>
>>>> gregorian_nls: UTC timestamps were encoded without leap seconds, with a
>>>> reference UTC timestamp. I think this is Tim's use-case. This is not accurate
>>>> according to UTC but it can be decoded precisely as intended. Jim points out
>>>> that it's not a real-world calendar, but it's not far off.
>>>>
>>>> Have I correctly described these as your cases?
>>>>
>>>> In addition, we propose two other calendars:
>>>>
>>>> gregorian_utc: The encoded and reference timestamps are UTC, and the encoding
>>>> is done with leap seconds allowed for. Hence the time coord is an accurate
>>>> elapsed time.
>>>>
>>>> gregorian_gps: The encoded and reference timestamps are GPS, and the encoding
>>>> is done without leap seconds. Again, the time coord is accurately elapsed time.
>>>> I think this is the use-case which originally started this thread!
>>>>
>>>> Best wishes
>>>>
>>>> Jonathan
>>>> _______________________________________________
>>>> CF-metadata mailing list
>>>> CF-metadata at cgd.ucar.edu
>>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>>
>>>
>>>
>>> _______________________________________________
>>> CF-metadata mailing list
>>> CF-metadata at cgd.ucar.edu
>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>
>> --
>> CICS-NC <http://www.cicsnc.org/> Visit us on
>> Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
>> *Research Scholar*
>> Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
>> North Carolina State University <http://ncsu.edu/>
>> NOAA National Centers for Environmental Information
>> <http://ncdc.noaa.gov/>
>> /formerly NOAA?s National Climatic Data Center/
>> 151 Patton Ave, Asheville, NC 28801
>> e: jbiard at cicsnc.org <mailto:jbiard at cicsnc.org>
>> o: +1 828 271 4900
>>
>> /Connect with us on Facebook for climate
>> <https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
>> <https://www.facebook.com/NOAANCEIoceangeo> information, and follow
>> us on Twitter at _at_NOAANCEIclimate
>> <https://twitter.com/NOAANCEIclimate> and _at_NOAANCEIocngeo
>> <https://twitter.com/NOAANCEIocngeo>. /
>>
>>
>>
>>
>> _______________________________________________
>> CF-metadata mailing list
>> CF-metadata at cgd.ucar.edu
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
>
> _______________________________________________
> CF-metadata mailing list
> CF-metadata at cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
CICS-NC <http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/>
/formerly NOAA?s National Climatic Data Center/
151 Patton Ave, Asheville, NC 28801
e: jbiard at cicsnc.org <mailto:jbiard at cicsnc.org>
o: +1 828 271 4900
/Connect with us on Facebook for climate
<https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
<https://www.facebook.com/NOAANCEIoceangeo> information, and follow us
on Twitter at _at_NOAANCEIclimate <https://twitter.com/NOAANCEIclimate> and
_at_NOAANCEIocngeo <https://twitter.com/NOAANCEIocngeo>. /
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20150630/e7600e57/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 15784 bytes
Desc: not available
URL: <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20150630/e7600e57/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CicsLogoTiny.png
Type: image/png
Size: 15784 bytes
Desc: not available
URL: <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20150630/e7600e57/attachment-0003.png>
Received on Tue Jun 30 2015 - 10:46:42 BST
