[Linuxptp-users] UTC offset change when loosing grand master

[Boris Boucher]

Hi all,

I have a system with a grand master clock that is GPS synchronized (secure
sync),
a SERVER machine running Linux/ptp4l acting as boundary clock for set of
CLIENTS systems.

The SERVER and the CLIENTS behind it requires precise time synchronization,
hence
the usage of PTP.

The grand master is aware that the current UTC offset is 37s today.

My SERVER on the other hand, is built with a code base from 2016 and think
the UTC offset is 36s. But as long as the grand master is present, the
boundary clock
distribute time with the correct offset.

The problem is, if the grand master is lost (power failure, network
issue....), my boundary
clock promotes itself as grand master, and uses its own knowledge of the
UTC offset instead
of the last known offset from the secure sync, and this cause a 1s step on
the UTC time !

This change propagate to the CLIENTS and also to the system clock of the
SERVER via phc2sys.
And during 1 or 2 minutes, the system clock of the SERVER and the CLIENTS
are not fully synchronized,
causing the system to fail to provide its service.

Looking at the documentation, I can't find any way of updating the UTC
offset of the SERVER in a simple
manner.

Rebuilding the SERVER with a new code base each time a new leap second is
inserted is not an option (the system
once deployed can not be updated easily).

What are my options ?

Thank you in advance for your advice !

Boris Boucher

[Richard Cochran]

Post by Boris Boucher
The problem is, if the grand master is lost (power failure, network
issue....), my boundary
clock promotes itself as grand master, and uses its own knowledge of the
UTC offset instead
of the last known offset from the secure sync, and this cause a 1s step on
the UTC time !

Your are correct. After the GM goes away, the ptp4l program does not
remember the UTC offset learned as a slave. It makes sense that we
should store the learned value in this case.

Post by Boris Boucher
Looking at the documentation, I can't find any way of updating the UTC
offset of the SERVER in a simple
manner.

You can set the UTC offset at run time using the
GRANDMASTER_SETTINGS_NP management message.

Post by Boris Boucher
What are my options ?

The best solution is to patch the program to store the UTC offset from
the GM, provided that the currentUtcOffsetValid is set.

One work around that you could use without patching is to monitor the
status (by scripting the pmc program, for example) using GET
GRANDMASTER_SETTINGS_NP. Then, when currentUtcOffsetValid = 1, write
the value of currentUtcOffset back into ptp4l using SET
GRANDMASTER_SETTINGS_NP.

Thanks,
Richard

[Boris Boucher]

Thank you for your responses.

I have tested the GRANDMASTER_SETTINGS_NP solution, it may work but I dont
think the solution is very elegant.

I also swear that I may also have a problem during system startup, if the
serveur is started before the master clock, it may
start to broadcast a time with the wrong UTC offset until the GM is ready.

Richard, I understand that the daemon should write the received offset on a
persistant storage on use it on startup, regardless
of the actual time zone or hardcoded info ? Is it a difficult modification
to implement in the daemon ?

Regards,

Boris.

[Richard Cochran]

Post by Boris Boucher
I also swear that I may also have a problem during system startup, if the
serveur is started before the master clock, it may
start to broadcast a time with the wrong UTC offset until the GM is ready.

Be sure to check the value of currentUtcOffsetValid. If zero, then
the offset should be ignored.

Post by Boris Boucher
Richard, I understand that the daemon should write the received offset on a
persistant storage on use it on startup, regardless
of the actual time zone or hardcoded info ? Is it a difficult modification
to implement in the daemon ?

No, it doesn't make sense to automatically store the offset. Why?
Because when the program starts again, there is no guarantee that one
or more new leap seconds have occurred, and there is no way to know,
except via the network.

Having said that, if you can verify the offset before starting ptp4l,
then you can provide it via the utc_offset configuration file option
or the --utc_offset=x command line switch.

Thanks,
Richard

[Boris Boucher]

Post by Richard Cochran

Post by Boris Boucher
I also swear that I may also have a problem during system startup, if the
serveur is started before the master clock, it may
start to broadcast a time with the wrong UTC offset until the GM is ready.

Be sure to check the value of currentUtcOffsetValid. If zero, then
the offset should be ignored.

Post by Boris Boucher
Richard, I understand that the daemon should write the received offset on a
persistant storage on use it on startup, regardless
of the actual time zone or hardcoded info ? Is it a difficult modification
to implement in the daemon ?

No, it doesn't make sense to automatically store the offset. Why?
Because when the program starts again, there is no guarantee that one
or more new leap seconds have occurred, and there is no way to know,
except via the network.

I agree that on startup, we have no guarantee that no leap second was
added or removed
since the last shutdown.
But, in a global system that is powered up, the GPS helped master
clock will take some some
time to start, eventually more time than my SERVER machine. In between
the SERVER will
assume grand master role and start broadcast the wrong time (because
its utc offset knowledge
is old).
So, on one hand, we face the risk of having an offset error each time
the system is started, on the
other hand, we have e very little change of starting up the system
just after a leap second
update and have the wrong time until the GPS helped master clock is available.
From my point of view, having a risk of temporary false time after
each startup is more problematic.

Post by Richard Cochran
Having said that, if you can verify the offset before starting ptp4l,
then you can provide it via the utc_offset configuration file option
or the --utc_offset=x command line switch.

Happy to know about this option. This may be the solution in my case
to start the daemon
with the correct offset each time. Much better than having to update
the offset afterward with
the pmc command while the daemon is already broadcasting a bad offset.
I while try this ASAP.

By the way, this configuration option seams not to be documented
(never seen it before). I just
looked in the source code to retrieve it. But I didn't see it as a
command line switch in the code.
I will check that at work tomorow.

Thank you very much,

Boris.

Post by Richard Cochran
Thanks,
Richard

[Richard Cochran]

Post by Boris Boucher
But, in a global system that is powered up, the GPS helped master
clock will take some some time to start, eventually more time than
my SERVER machine. In between the SERVER will assume grand master
role and start broadcast the wrong time (because its utc offset
knowledge is old).

As long as currentUtcOffsetValid=0, then the behavior of the SERVER is
correct according to the standard.

Still, that is little consolation to you, when your clients ignore the
currentUtcOffsetValid flag.

Post by Boris Boucher
So, on one hand, we face the risk of having an offset error each
time the system is started, on the other hand, we have e very little
change of starting up the system just after a leap second update and
have the wrong time until the GPS helped master clock is available.
From my point of view, having a risk of temporary false time after
each startup is more problematic.

I disagree. If I were to implement this "magic" saving of the offset,
then it would appear to work, even though it is in fact broken. Then
users would rely on this, until the day comes when someone hits our
latent leap second bug.

I prefer to force users to come to grips with their configuration WRT
leap second handling early on, rather than waiting until the rare
event causes very unpleasant surprises.

Post by Boris Boucher
By the way, this configuration option seams not to be documented
(never seen it before). I just looked in the source code to retrieve
it. But I didn't see it as a command line switch in the code.

This is a new option, post version 1.8 but in the latest git head.

From 33e62f992542ac5ce6bdbb8ae6c34dec7011b543 Mon Sep 17 00:00:00 2001
From: Viliam Lejcik <***@kistler.com>
Date: Tue, 17 Jan 2017 18:25:26 +0100
Subject: ptp4l: Make UTC offset configurable.

Currently UTC offset is defined as a constant - CURRENT_UTC_OFFSET, and if
a leap second is added, that constant is no longer valid. Ptp4l was
updated to read the UTC offset from configuration instead.

Signed-off-by: Viliam Lejcik <***@kistler.com>

Thanks,
Richard

[Scott Silverman]

I don't know if it's the recommended solution, but I encountered a similar
problem a while back. My solution was to use pmc as follows:

# pmc -u -b 0 'set GRANDMASTER_SETTINGS_NP clockClass 248 clockAccuracy
0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37 leap61 0 leap59 0
currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 0 frequencyTraceable 0
timeSource 0xa0'

Hopefully someone more knowledgeable will have something better to suggest,
but this does work.





Thanks,

Scott Silverman | IT | Simplex | 312-360-2444
230 S. LaSalle St., Suite 4-100, Chicago, IL 60604

Post by Boris Boucher
Hi all,
I have a system with a grand master clock that is GPS synchronized (secure
sync),
a SERVER machine running Linux/ptp4l acting as boundary clock for set of
CLIENTS systems.
The SERVER and the CLIENTS behind it requires precise time
synchronization, hence
the usage of PTP.
The grand master is aware that the current UTC offset is 37s today.
My SERVER on the other hand, is built with a code base from 2016 and think
the UTC offset is 36s. But as long as the grand master is present, the
boundary clock
distribute time with the correct offset.
The problem is, if the grand master is lost (power failure, network
issue....), my boundary
clock promotes itself as grand master, and uses its own knowledge of the
UTC offset instead
of the last known offset from the secure sync, and this cause a 1s step on
the UTC time !
This change propagate to the CLIENTS and also to the system clock of the
SERVER via phc2sys.
And during 1 or 2 minutes, the system clock of the SERVER and the CLIENTS
are not fully synchronized,
causing the system to fail to provide its service.
Looking at the documentation, I can't find any way of updating the UTC
offset of the SERVER in a simple
manner.
Rebuilding the SERVER with a new code base each time a new leap second is
inserted is not an option (the system
once deployed can not be updated easily).
What are my options ?
Thank you in advance for your advice !
Boris Boucher
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Linuxptp-users mailing list
https://lists.sourceforge.net/lists/listinfo/linuxptp-users

--
DISCLAIMER: NOTICE REGARDING PRIVACY AND CONFIDENTIALITY

The information contained in and/or accompanying this communication is
intended only for use by the addressee(s) named herein and may contain
legally privileged and/or confidential information. If you are not the
intended recipient of this e-mail, you are hereby notified that any
dissemination, distribution or copying of this information, and any
attachments thereto, is strictly prohibited. If you have received this
e-mail in error, please immediately notify the sender and permanently
delete the original and any copy of any e-mail and any printout thereof.
Electronic transmissions cannot be guaranteed to be secure or error-free.
The sender therefore does not accept liability for any errors or omissions
in the contents of this message which arise as a result of e-mail
transmission. Simplex Trading, LLC and its affiliates reserves the right to
intercept, monitor, and retain electronic communications to and from its
system as permitted by law. Simplex Trading, LLC is a registered Broker
Dealer with CBOE and a Member of SIPC.

[Richard Cochran]

Post by Boris Boucher
This change propagate to the CLIENTS and also to the system clock of the
SERVER via phc2sys.
And during 1 or 2 minutes, the system clock of the SERVER and the CLIENTS
are not fully synchronized,
causing the system to fail to provide its service.

Are the CLIENTS also running ptp4l?

Thanks,
Richard

[Boris Boucher]

Post by Richard Cochran

Post by Boris Boucher
This change propagate to the CLIENTS and also to the system clock of the
SERVER via phc2sys.
And during 1 or 2 minutes, the system clock of the SERVER and the CLIENTS
are not fully synchronized,
causing the system to fail to provide its service.

Are the CLIENTS also running ptp4l?

No, the CLIENTS are running ptpd.

Post by Richard Cochran
Thanks,
Richard

[Richard Cochran]

Post by Boris Boucher
No, the CLIENTS are running ptpd.

Ok, then ptpd has a bug. It ignores the currentUtcOffsetValid member
of the TIME_PROPERTIES data set. You will need to fix that in order
to solve your start up problem, or you could instead work around it by
setting the offset in the ptp4l configuration or in its command line.

Looking at ptp4l, sending an incorrect currentUtcOffset with
currentUtcOffsetValid=0 is technically not wrong, but I agree that we
should retain the correct offset when provided by a GM, even after
that GM disappears.

I will review the ptp4l client behavior when using SW time stamps.
Maybe there is a bug there as well.

Thinking out loud, the rational for the current behavior is the common
use case of an isolated LAN without any global time source. In this
case, users don't care what the offset is. They simply want the nodes
in the system to be synchronized to each other.

Thanks,
Richard

[Richard Cochran]

Post by Richard Cochran
Looking at ptp4l, sending an incorrect currentUtcOffset with
currentUtcOffsetValid=0 is technically not wrong, but I agree that we
should retain the correct offset when provided by a GM, even after
that GM disappears.

Over on the linuxptp-devel list, I posted a series that should fix
this issue. Please see if it fixes the problem for you.

In order to test it, you can force the TAI-UTC offset to an out of
date value (like 35) using the configuration file or the command line.

Thanks,
Richard