New electronics technicians setting up instruments, right?
I don't really think this is a CF problem. You apparently can guess the
date, it just
wasn't recorded by the instrument (otherwise, I'm guessing, you wouldn't be
thinking of using it in monthly stats).
My solution is to generate a date for each point, using whatever
external info you can -
sample scheme, start time, events that can be used as markers, and
whatever software
you normally use. Then document the process in an attribute, including
the fact that
the date is inferred, not recorded.
Oh, I see Dave A expressed a similar opinion, I should have read the
thread more carefully.
Cheers - Nan
On 11/1/16 2:17 PM, Seth McGinnis wrote:
> Hi Ajay,
>
> The problem is that the offset representation is the CF standard for how
> to record time, and that's what standards-aware software that displays
> it quickly and easily relies upon. If you don't know how much time has
> passed since the reference time, it's just plain not possible to record
> it that way.
>
> So sadly, I think you're in an unavoidable bind. You can keep the
> dataset in one piece, or you can represent it in the standard way, but
> you can't do both. Either option can be done in a CF-conformant way,
> but you'll have to choose one. (The third, simplest, and most
> conformant option is of course just to drop the missing-time data
> entirely. But that's outside the scope of the question you asked.)
>
> The objections that your team raised are sound, but I see no way to
> address both of them without dropping the missing-time data. Keeping
> the missing-time data, having a single dataset, or using the standard
> representation: pick any two.
>
> Cheers,
>
> --Seth
>
>
> On 11/1/2016 1:21 PM, Ajay Krishnan - NOAA Affiliate wrote:
>> Thanks a lot for the suggestions, Dave, Ed and Seth.
>>
>> Seth, here's the response from the team that's working on the data:
>>
>> The first solution is not realistic. There are too many
>> missing times - separating out into another data variable
>> would give the user a bifurcated data set.
>>
>> The second solution is doable, but still doesnt make much
>> sense. The power of using a convention is that the data
>> can be dumped, used, graphed, in software which follows
>> the conventions quickly and easily. What good is it to
>> graph against a sequential index? Date/time needs to
>> be a coordinate to interact seamlessly with existing
>> software.
>>
>> Ed,
>> I don't know to record the data as a time_offset in hours or seconds
>> when there is no information on the number of days that have passed
>> since the reference time.
>>
>> -Ajay
>>
>> On Wed, Oct 26, 2016 at 2:54 PM, <cf-metadata-request at cgd.ucar.edu
>> <mailto:cf-metadata-request at cgd.ucar.edu>> wrote:
>>
>> Send CF-metadata mailing list submissions to
>> cf-metadata at cgd.ucar.edu <mailto:cf-metadata at cgd.ucar.edu>
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>> <http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
>> or, via email, send a message with subject or body 'help' to
>> cf-metadata-request at cgd.ucar.edu
>> <mailto:cf-metadata-request at cgd.ucar.edu>
>>
>> You can reach the person managing the list at
>> cf-metadata-owner at cgd.ucar.edu
>> <mailto:cf-metadata-owner at cgd.ucar.edu>
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of CF-metadata digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: Handling time when date is "missing" (Seth McGinnis)
>> 2. Re: Handling time when date is "missing"
>> (Dave Allured - NOAA Affiliate)
>> 3. Re: Feedback requested on proposed CF Simple Geometries
>> (Jonathan Gregory)
>> 4. Re: Handling time when date is "missing"
>> (Armstrong, Edward M (398G))
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 25 Oct 2016 14:26:18 -0600
>> From: Seth McGinnis <mcginnis at ucar.edu <mailto:mcginnis at ucar.edu>>
>> To: "cf-metadata at cgd.ucar.edu <mailto:cf-metadata at cgd.ucar.edu>"
>> <cf-metadata at cgd.ucar.edu <mailto:cf-metadata at cgd.ucar.edu>>
>> Subject: Re: [CF-metadata] Handling time when date is "missing"
>> Message-ID: <de53af7d-c734-96f7-9631-ebf1e35c9cff at ucar.edu
>> <mailto:de53af7d-c734-96f7-9631-ebf1e35c9cff at ucar.edu>>
>> Content-Type: text/plain; charset=windows-1252
>>
>> But then the data is non-compliant, and it sounds like a valid CF
>> solution is needed.
>>
>> Two possible solutions come to my mind. The first way would be to store
>> the undated measurements separately. Record the normal measurements in
>> the normal way, and then record the undated measurements in a separate
>> data variable with an index coordinate instead of a time coordinate.
>>
>> The other way would be not to use time as a coordinate variable at all,
>> but only as a data variable. Record all the measurements with an index
>> coordinate instead of a time coordinate. Then define data variables for
>> year, month, day, and time of measurement, and just fill in what's known
>> for each one. (It sounds like the month and year are still known even
>> if the day is not.) This is very similar to the approach taken for
>> trajectories; see example H.12 in the spec.
>>
>> Cheers,
>>
>> --Seth
>>
>>
>> On 10/25/16 1:31 PM, Dave Allured - NOAA Affiliate wrote:
>> > Ajay,
>> >
>> > I think this is an exception to CF. I recommend using _FillValue or
>> > missing_value on the time coordinate. Document this in a comment
>> > attribute on the time coordinate variable.
>> >
>> > Also document this somehow in another global attribute that
>> explains you
>> > made this exception to the CF conventions. Follow CF conventions
>> in all
>> > other regards.
>> >
>> > Then, try to remember to warn people about this when you
>> distribute the
>> > data. CF compliant time coordinates are fundamental to many
>> application
>> > programs, and I expect they will choke or introduce subtle errors if
>> > missing values are in there. So users will need to provide special
>> > handling for such files. HTH.
>> >
>> > --Dave
>> > (Please reply to list only)
>> >
>> >
>> > On Tue, Oct 25, 2016 at 12:07 PM, Ajay Krishnan - NOAA Affiliate
>> > <ajay.krishnan at noaa.gov <mailto:ajay.krishnan at noaa.gov>
>> <mailto:ajay.krishnan at noaa.gov <mailto:ajay.krishnan at noaa.gov>>> wrote:
>> >
>> > Hi All,
>> >
>> > I have a user that's converting some IMMA format files to CF
>> > compliant NetCDF files.
>> >
>> > The problem is that, we've run into several measurements where
>> just
>> > the hour of measurement has been recorded without the
>> corresponding
>> > "date". We would prefer not to omit these data in the conversion,
>> > because they are considered valid measurements (and play a role in
>> > monthly summary statistics)
>> >
>> > How do we represent this in a valid CF NetCDF format since we
>> can't
>> > use _FillValues for 'time'? Any suggestions for handling such
>> > special cases?
>> >
>> > Thanks,
>> > Ajay
>> >
>> >
>> >
>> > _______________________________________________
>> > CF-metadata mailing list
>> > CF-metadata at cgd.ucar.edu <mailto:CF-metadata at cgd.ucar.edu>
>> > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>> <http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
>> >
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Tue, 25 Oct 2016 14:34:52 -0600
>> From: Dave Allured - NOAA Affiliate <dave.allured at noaa.gov
>> <mailto:dave.allured at noaa.gov>>
>> To: "cf-metadata at cgd.ucar.edu <mailto:cf-metadata at cgd.ucar.edu>"
>> <cf-metadata at cgd.ucar.edu <mailto:cf-metadata at cgd.ucar.edu>>
>> Subject: Re: [CF-metadata] Handling time when date is "missing"
>> Message-ID:
>>
>> <CALqwTFMhR_NDsLSeyuSePFdmDP=54N9vBGD90U3FpiTsLVLWXg at mail.gmail.com
>> <mailto:54N9vBGD90U3FpiTsLVLWXg at mail.gmail.com>>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Seth, Jim, et al:
>>
>> Thank you for the constructive alternatives.
>>
>> --Dave
>>
>>
>> On Tue, Oct 25, 2016 at 2:26 PM, Seth McGinnis <mcginnis at ucar.edu
>> <mailto:mcginnis at ucar.edu>> wrote:
>>
>> > But then the data is non-compliant, and it sounds like a valid CF
>> > solution is needed.
>> >
>> > Two possible solutions come to my mind. The first way would be to
>> store
>> > the undated measurements separately. Record the normal
>> measurements in
>> > the normal way, and then record the undated measurements in a separate
>> > data variable with an index coordinate instead of a time coordinate.
>> >
>> > The other way would be not to use time as a coordinate variable at
>> all,
>> > but only as a data variable. Record all the measurements with an
>> index
>> > coordinate instead of a time coordinate. Then define data
>> variables for
>> > year, month, day, and time of measurement, and just fill in what's
>> known
>> > for each one. (It sounds like the month and year are still known even
>> > if the day is not.) This is very similar to the approach taken for
>> > trajectories; see example H.12 in the spec.
>> >
>> > Cheers,
>> >
>> > --Seth
>> >
>> >
>> > On 10/25/16 1:31 PM, Dave Allured - NOAA Affiliate wrote:
>> > > Ajay,
>> > >
>> > > I think this is an exception to CF. I recommend using _FillValue or
>> > > missing_value on the time coordinate. Document this in a comment
>> > > attribute on the time coordinate variable.
>> > >
>> > > Also document this somehow in another global attribute that
>> explains you
>> > > made this exception to the CF conventions. Follow CF
>> conventions in all
>> > > other regards.
>> > >
>> > > Then, try to remember to warn people about this when you
>> distribute the
>> > > data. CF compliant time coordinates are fundamental to many
>> application
>> > > programs, and I expect they will choke or introduce subtle errors if
>> > > missing values are in there. So users will need to provide special
>> > > handling for such files. HTH.
>> > >
>> > > --Dave
>> > > (Please reply to list only)
>> > >
>> > >
>> > > On Tue, Oct 25, 2016 at 12:07 PM, Ajay Krishnan - NOAA Affiliate
>> > > <ajay.krishnan at noaa.gov <mailto:ajay.krishnan at noaa.gov>
>> <mailto:ajay.krishnan at noaa.gov <mailto:ajay.krishnan at noaa.gov>>> wrote:
>> > >
>> > > Hi All,
>> > >
>> > > I have a user that's converting some IMMA format files to CF
>> > > compliant NetCDF files.
>> > >
>> > > The problem is that, we've run into several measurements
>> where just
>> > > the hour of measurement has been recorded without the
>> corresponding
>> > > "date". We would prefer not to omit these data in the
>> conversion,
>> > > because they are considered valid measurements (and play a
>> role in
>> > > monthly summary statistics)
>> > >
>> > > How do we represent this in a valid CF NetCDF format since
>> we can't
>> > > use _FillValues for 'time'? Any suggestions for handling such
>> > > special cases?
>> > >
>> > > Thanks,
>> > > Ajay
>> > >
>> > >
>> > >
>> > > _______________________________________________
>> > > CF-metadata mailing list
>> > > CF-metadata at cgd.ucar.edu <mailto:CF-metadata at cgd.ucar.edu>
>> > > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>> <http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
>> > >
>> > _______________________________________________
>> > CF-metadata mailing list
>> > CF-metadata at cgd.ucar.edu <mailto:CF-metadata at cgd.ucar.edu>
>> > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>> <http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20161025/803102dd/attachment-0001.html
>> <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20161025/803102dd/attachment-0001.html>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Wed, 26 Oct 2016 14:33:37 +0100
>> From: Jonathan Gregory <j.m.gregory at reading.ac.uk
>> <mailto:j.m.gregory at reading.ac.uk>>
>> To: cf-metadata at cgd.ucar.edu <mailto:cf-metadata at cgd.ucar.edu>
>> Subject: Re: [CF-metadata] Feedback requested on proposed CF Simple
>> Geometries
>> Message-ID: <20161026133337.GA10147 at met.reading.ac.uk
>> <mailto:20161026133337.GA10147 at met.reading.ac.uk>>
>> Content-Type: text/plain; charset=utf-8
>>
>> Dear Ben and Bert
>>
>> Thanks for your emails, which help me to understand the simple geometry
>> proposals better. Just to be clear, I'd like to repeat my first
>> question.
>>
>> > You explain that the need is to specify spatial coordinates with a
>> simple
>> > geometry for a timeSeries variable. For example, this could be for the
>> > discharge as a function of time across some line in a river (your
>> example),
>> > or I suppose it could be an average temperature as a function of
>> time for
>> > the Atlantic Ocean, where you wanted to supply the polygon which
>> drew the
>> > outline of the basin. Have I got the idea?
>>
>> to which you replied
>>
>> > Yes, you have this mostly right. It?s common to have a collection
>> of points
>> > (weather stations), lines (stream reaches), or polygons (hydrologic
>> > catchments) with an associated time series
>>
>> I was asking whether this means that for each *collection* (of
>> points, lines or
>> polygons) there is a *single* timeseries. For instance, in your
>> example of a
>> single geometry composed of several polygons, there is a single
>> number for each
>> time. But that is not the case for weather stations; for each
>> weather station
>> there is a timeseries, and at each time there is a different number
>> (value of
>> temperature, precipitation or whatever) for each weather station.
>> You also
>> write, "The US National Weather Service?s National Water Model (NWM) ...
>> forecasts streamflow rates in about 2.7 million stream segments
>> averaging 2km."
>> The stream network is a MultiLineString geometry, but I don't think
>> there is
>> just one value of streamflow applying to the entire network at any
>> given time;
>> I guess there is a different timeseries for each stream segment. But
>> in my
>> example above, the Atlantic Ocean is a single polygon with a single
>> timeseries
>> for its average temperature, not a different timeseries for each
>> node. Thus I
>> am unclear about the dimensions of the data. In terms of your
>> original example,
>> does the data have dimensions (time,geometry, where geometry=1) or
>> (time,node)?
>>
>> This seems to me to be a crucial difference. In the former case the
>> simple
>> geometry can be regarded as a more complex alternative to cells
>> bounds - the
>> cell has a complicated geometry of nodes and lines, but it's still a
>> single
>> cell. In the latter case you're providing many timeseries in an
>> unstructured
>> geometry, which is what ugrid describes. Which do you have in mind?
>>
>> Nonetheless in both cases the geometries have to be described. I
>> think the
>> difference is how we attach this description to the data or
>> coordinates, rather
>> than how the description is constructed.
>>
>> You propose the index variable in order for the convention to be
>> like ugrid.
>> However this still seems to me to be an unnecessary complexity and
>> use of space
>> if you aren't going to have many shared nodes. I think the case for
>> having
>> another convention, distinct from ugrid, is stronger if it is
>> *unlike* ugrid
>> in this respect, and therefore simpler as well.
>>
>> I agree that repeating the inside/outside flag many times is
>> wasteful. That,
>> coupled with your clarification that you may have several
>> geometries, each
>> consisting of several elements (points, lines, polygons), means that
>> you need,
>> in effect, a ragged array of ragged arrays (geometry,element,node).
>> This is
>> more complicated than DSGs, but it seems to me it would be
>> reasonably easy to
>> understand if your multi-geometry example
>> https://github.com/bekozi/netCDF-CF-simple-geometry/wiki/VLEN-Arrays-in-NetCDF-3#multipolygon-example
>> <https://github.com/bekozi/netCDF-CF-simple-geometry/wiki/VLEN-Arrays-in-NetCDF-3#multipolygon-example>
>> was stored something like this:
>>
>> geom=3;
>> part=11;
>> node=36;
>> int number_of_parts(geom);
>> number_of_parts:parts="number_of_nodes";
>> int number_of_nodes(part);
>> number_of_nodes:inout="inout";
>> char inout(part);
>> float x(node);
>> float y(node);
>> number_of_parts=6, 3, 2;
>> number_of_nodes=4, 3, 3, 3, 3, 3, 3, 5, 3, 3, 3;
>> inout="OIIIOOOIO";
>> x=0, 20, 20, 0, 1, 10, 19, 5, 7, 9, 11, 13, 15, 5, 9, 7, 11, 15,
>> 13, -40,
>> -20, -45, -20, -10, -10, -30, -45, -30, -20, -20, 30, 45, 10, 25,
>> 50, 30;
>> y = 0, 0, 20, 20, 1, 5, 1, 15, 19, 15, 15, 19, 15, 25, 25, 29, 25,
>> 25, 29,
>> -40, -45, -30, -35, -30, -10, -5, -20, -20, -15, -25, 20, 40, 40,
>> 5, 10, 15;
>>
>> where I assume that all polygons are closed.
>>
>> What do you think?
>>
>> Best wishes
>>
>> Jonathan
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Wed, 26 Oct 2016 18:54:18 +0000
>> From: "Armstrong, Edward M (398G)" <Edward.M.Armstrong at jpl.nasa.gov
>> <mailto:Edward.M.Armstrong at jpl.nasa.gov>>
>> To: Ajay Krishnan - NOAA Affiliate <ajay.krishnan at noaa.gov
>> <mailto:ajay.krishnan at noaa.gov>>,
>> "cf-metadata at cgd.ucar.edu <mailto:cf-metadata at cgd.ucar.edu>"
>> <cf-metadata at cgd.ucar.edu <mailto:cf-metadata at cgd.ucar.edu>>
>> Subject: Re: [CF-metadata] Handling time when date is "missing"
>> Message-ID: <D43648EB.1A0EA%edward.m.armstrong at jpl.nasa.gov
>> <mailto:D43648EB.1A0EA%25edward.m.armstrong at jpl.nasa.gov>>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Jay,
>>
>> You could use the variable time as a single value to establish the
>> time (in complete CF date format) of the first observation
>>
>> Another multi dimension array can then be used to store the time
>> offset (in hours or seconds etc.) of each measurement from variable
>> time
>>
>> Or else convert the hourly measurements into a proper CF date format
>> to store in variable time
>>
>> From: CF-metadata <cf-metadata-bounces at cgd.ucar.edu
>> <mailto:cf-metadata-bounces at cgd.ucar.edu><mailto:cf-metadata-bounces at cgd.ucar.edu
>> <mailto:cf-metadata-bounces at cgd.ucar.edu>>> on behalf of Ajay
>> Krishnan - NOAA Affiliate <ajay.krishnan at noaa.gov
>> <mailto:ajay.krishnan at noaa.gov><mailto:ajay.krishnan at noaa.gov
>> <mailto:ajay.krishnan at noaa.gov>>>
>> Date: Tuesday, October 25, 2016 at 11:07 AM
>> To: "cf-metadata at cgd.ucar.edu
>> <mailto:cf-metadata at cgd.ucar.edu><mailto:cf-metadata at cgd.ucar.edu
>> <mailto:cf-metadata at cgd.ucar.edu>>" <cf-metadata at cgd.ucar.edu
>> <mailto:cf-metadata at cgd.ucar.edu><mailto:cf-metadata at cgd.ucar.edu
>> <mailto:cf-metadata at cgd.ucar.edu>>>
>> Subject: [CF-metadata] Handling time when date is "missing"
>>
>> Hi All,
>>
>> I have a user that's converting some IMMA format files to CF
>> compliant NetCDF files.
>>
>> The problem is that, we've run into several measurements where just
>> the hour of measurement has been recorded without the corresponding
>> "date". We would prefer not to omit these data in the conversion,
>> because they are considered valid measurements (and play a role in
>> monthly summary statistics)
>>
>> How do we represent this in a valid CF NetCDF format since we can't
>> use _FillValues for 'time'? Any suggestions for handling such
>> special cases?
>>
>> Thanks,
>> Ajay
>>
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20161026/e7b7ecfa/attachment.html
>> <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/attachments/20161026/e7b7ecfa/attachment.html>>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> CF-metadata mailing list
>> CF-metadata at cgd.ucar.edu <mailto:CF-metadata at cgd.ucar.edu>
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>> <http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
>>
>>
>> ------------------------------
>>
>> End of CF-metadata Digest, Vol 162, Issue 13
>> ********************************************
>>
>>
>>
>>
>> _______________________________________________
>> CF-metadata mailing list
>> CF-metadata at cgd.ucar.edu
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>
> _______________________________________________
> CF-metadata mailing list
> CF-metadata at cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
Received on Wed Nov 02 2016 - 10:17:54 GMT
