Discussion:
[Linuxptp-users] ptp inducing as many hops as possible
Mace, Kyle P.
2017-04-20 19:08:52 UTC
Permalink
I am in need to test ptp with as many 'hops' as possible. I have multiple systems with dual Ethernet ports. I would like to have a setup something like this: grandmaster -> BC 1 port 1 -> BC 1 port 2 -> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3 -> etc.. Anyone have any ideas if this is possible?
Richard Cochran
2017-04-20 21:26:58 UTC
Permalink
Post by Mace, Kyle P.
I am in need to test ptp with as many 'hops' as possible. I have
multiple systems with dual Ethernet ports. I would like to have a
setup something like this: grandmaster -> BC 1 port 1 -> BC 1 port 2
-> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3 ->
etc.. Anyone have any ideas if this is possible?
The linuxptp stack supports this.

Remember that all the ports of the BC must share the same clock (that
is, the same PHC).

Also, long chains of BC are prone to "gain peaking".

HTH,
Richard
Mace, Kyle P.
2017-04-24 11:58:59 UTC
Permalink
That is very good news. What do you mean by "gain peaking"?

Thank you,
Kyle Mace

-----Original Message-----
From: Richard Cochran [mailto:***@gmail.com]
Sent: Thursday, April 20, 2017 4:27 PM
To: Mace, Kyle P.
Cc: linuxptp-***@lists.sourceforge.net
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible
Post by Mace, Kyle P.
I am in need to test ptp with as many 'hops' as possible. I have
multiple systems with dual Ethernet ports. I would like to have a
setup something like this: grandmaster -> BC 1 port 1 -> BC 1 port 2
-> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3 ->
etc.. Anyone have any ideas if this is possible?
The linuxptp stack supports this.

Remember that all the ports of the BC must share the same clock (that is, the same PHC).

Also, long chains of BC are prone to "gain peaking".

HTH,
Richard
Richard Cochran
2017-04-24 17:08:22 UTC
Permalink
Post by Mace, Kyle P.
That is very good news. What do you mean by "gain peaking"?
@book{gardner2005phaselock,
title={Phaselock Techniques},
author={Gardner, F.M.},
isbn=9780471732686,
url={https://books.google.at/books?id=r5yjPuWQde0C},
year=2005,
publisher={Wiley}
}

See page 19.

HTH,
Richard
Mace, Kyle P.
2017-04-25 13:19:47 UTC
Permalink
Alright I am having troubles forcing the ports to use the same PHC. When I specify which PHC to use I get an error about PHC device mismatch. Is there a way to unattach the hardware clock from a specific port?

Output of error
sudo ptp4l -i eno1 -p /dev/ptp1 -m -s
ptp4l[61532.475]: selected /dev/ptp1 as PTP clock
ptp4l[61532.475]: port 1: PHC device mismatch
ptp4l[61532.476]: port 1: /dev/ptp1 requested, but /dev/ptp0 attached
ptp4l[61532.476]: failed to open port eno1

Or would I be able to enable the boundary_clock_jbod option in a config file and then sync the clocks off the system clock using phc2sys?

-----Original Message-----
From: Richard Cochran [mailto:***@gmail.com]
Sent: Thursday, April 20, 2017 4:27 PM
To: Mace, Kyle P.
Cc: linuxptp-***@lists.sourceforge.net
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible
Post by Mace, Kyle P.
I am in need to test ptp with as many 'hops' as possible. I have
multiple systems with dual Ethernet ports. I would like to have a
setup something like this: grandmaster -> BC 1 port 1 -> BC 1 port 2
-> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3 ->
etc.. Anyone have any ideas if this is possible?
The linuxptp stack supports this.

Remember that all the ports of the BC must share the same clock (that is, the same PHC).

Also, long chains of BC are prone to "gain peaking".

HTH,
Richard
Luke Bigum
2017-04-25 13:39:59 UTC
Permalink
Kyle,

I think you've misunderstood what Richard was talking about. You can't just arbitrarily set any network interface to use any PHC - the oscillator under the PHC device is physically attached to the NIC. Unless you have a multi-port network card that shares one oscillator (eg: SolarFlare 7x22), most NICs will probably have their own PHC. That's why you're getting the device mismatch error.

I've used multiple phc2sys processes in the past to sync different NICs, but this is not a true Boundary Clock. You'll be copying time up and down the Linux stack and through the system clock, so you are subject to all sorts of kernel scheduling jitter and PCI bus latency. If you just want some sort of PTP signal and don't care so much about accuracy though, you can do this.

Also regarding your question earlier about Gain Peaking, Richard was warning you about cumulative error. So if you are proposing to go:

grandmaster -> BC 1 port 1 -> BC 1 port 2 -> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3

And it costs, say, +/- 10us of jitter to go between ports, by the time you get to BC3 port 3 you'll be at +/- 60us.
--
Luke Bigum
Lead Engineer

Information Systems

----- Original Message -----
From: "Mace, Kyle P." <***@gd-ms.com>
To: "Richard Cochran" <***@gmail.com>
Cc: "linuxptp-users" <linuxptp-***@lists.sourceforge.net>
Sent: Tuesday, 25 April, 2017 14:19:47
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible

Alright I am having troubles forcing the ports to use the same PHC. When I specify which PHC to use I get an error about PHC device mismatch. Is there a way to unattach the hardware clock from a specific port?

Output of error
sudo ptp4l -i eno1 -p /dev/ptp1 -m -s
ptp4l[61532.475]: selected /dev/ptp1 as PTP clock
ptp4l[61532.475]: port 1: PHC device mismatch
ptp4l[61532.476]: port 1: /dev/ptp1 requested, but /dev/ptp0 attached
ptp4l[61532.476]: failed to open port eno1

Or would I be able to enable the boundary_clock_jbod option in a config file and then sync the clocks off the system clock using phc2sys?

-----Original Message-----
From: Richard Cochran [mailto:***@gmail.com]
Sent: Thursday, April 20, 2017 4:27 PM
To: Mace, Kyle P.
Cc: linuxptp-***@lists.sourceforge.net
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible
Post by Mace, Kyle P.
I am in need to test ptp with as many 'hops' as possible. I have
multiple systems with dual Ethernet ports. I would like to have a
setup something like this: grandmaster -> BC 1 port 1 -> BC 1 port 2
-> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3 ->
etc.. Anyone have any ideas if this is possible?
The linuxptp stack supports this.

Remember that all the ports of the BC must share the same clock (that is, the same PHC).

Also, long chains of BC are prone to "gain peaking".

HTH,
Richard

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Linuxptp-users mailing list
Linuxptp-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linuxptp-users
---

LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN
http://www.LMAX.com/

Recognised by the most prestigious technology and business awards

Financial technology awards
2017 Best FX trading venue, Fund Technology & WSL Awards
2016 Best trading & execution venue, HFM US Technology Awards

FX industry awards
2016, 2015, 2014, 2013 Winner, Profit & Loss Readers' Choice Awards
2016, 2015, 2014, 2013 Winner, WSL Institutional Trading Awards

Business growth awards
2016, 2015 Winner, Deloitte UK Technology Fast 50
2015, 2014, 2013, Winner, The Sunday Times Tech Track 100
2016, 2015 Winner, Deloitte EMEA Technology Fast 500
2015 Winner, Tech City UK Future 50

---

FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved.

This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). This message is not intended for any recipient(s) who based on their nationality, place of business, domicile or for any other reason, is/are subject to local laws or regulations which prohibit the provision of such products and services. This message is subject to the following terms (http://lmax.com/pdf/general-disclaimers.pdf), if you cannot access these, please notify us by replying to this email and we will send you the terms. If you are not the intended recipient, please notify the sender immediately and delete any copies of this message.

LMAX Exchange is the trading name of LMAX Limited. LMAX Limited operates a multilateral trading facility. LMAX Limited is authorised and regulated by the Financial Conduct Authority (firm registration number 509778) and is a company registered in England and Wales (number 6505809).

LMAX Hong Kong Limited is a wholly-owned subsidiary of LMAX Limited. LMAX Hong Kong is licensed by the Securities and Futures Commission in Hong Kong to conduct Type 3 (leveraged foreign exchange trading) regulated activity with CE Number BDV088.
Mace, Kyle P.
2017-04-25 14:11:28 UTC
Permalink
Ahhhhh I understand the PHC thing now. Is there any way to induce as many "hops" as possible then? I set a test earlier where I just looped through a switch a whole bunch of times using VLANS, but I wasn't sure if that truly counted as a "hop" I would need to check if the switch supports PTP or not.

If push comes to shove the phc2sys might work for me.

Thank you so much for the help.
-----Original Message-----
From: Luke Bigum [mailto:***@lmax.com]
Sent: Tuesday, April 25, 2017 8:40 AM
To: linuxptp-users
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible

Kyle,

I think you've misunderstood what Richard was talking about. You can't just arbitrarily set any network interface to use any PHC - the oscillator under the PHC device is physically attached to the NIC. Unless you have a multi-port network card that shares one oscillator (eg: SolarFlare 7x22), most NICs will probably have their own PHC. That's why you're getting the device mismatch error.

I've used multiple phc2sys processes in the past to sync different NICs, but this is not a true Boundary Clock. You'll be copying time up and down the Linux stack and through the system clock, so you are subject to all sorts of kernel scheduling jitter and PCI bus latency. If you just want some sort of PTP signal and don't care so much about accuracy though, you can do this.

Also regarding your question earlier about Gain Peaking, Richard was warning you about cumulative error. So if you are proposing to go:

grandmaster -> BC 1 port 1 -> BC 1 port 2 -> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3

And it costs, say, +/- 10us of jitter to go between ports, by the time you get to BC3 port 3 you'll be at +/- 60us.

--
Luke Bigum
Lead Engineer

Information Systems

----- Original Message -----
From: "Mace, Kyle P." <***@gd-ms.com>
To: "Richard Cochran" <***@gmail.com>
Cc: "linuxptp-users" <linuxptp-***@lists.sourceforge.net>
Sent: Tuesday, 25 April, 2017 14:19:47
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible

Alright I am having troubles forcing the ports to use the same PHC. When I specify which PHC to use I get an error about PHC device mismatch. Is there a way to unattach the hardware clock from a specific port?

Output of error
sudo ptp4l -i eno1 -p /dev/ptp1 -m -s
ptp4l[61532.475]: selected /dev/ptp1 as PTP clock
ptp4l[61532.475]: port 1: PHC device mismatch
ptp4l[61532.476]: port 1: /dev/ptp1 requested, but /dev/ptp0 attached
ptp4l[61532.476]: failed to open port eno1

Or would I be able to enable the boundary_clock_jbod option in a config file and then sync the clocks off the system clock using phc2sys?

-----Original Message-----
From: Richard Cochran [mailto:***@gmail.com]
Sent: Thursday, April 20, 2017 4:27 PM
To: Mace, Kyle P.
Cc: linuxptp-***@lists.sourceforge.net
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible
Post by Mace, Kyle P.
I am in need to test ptp with as many 'hops' as possible. I have
multiple systems with dual Ethernet ports. I would like to have a
setup something like this: grandmaster -> BC 1 port 1 -> BC 1 port 2
-> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3 ->
etc.. Anyone have any ideas if this is possible?
The linuxptp stack supports this.

Remember that all the ports of the BC must share the same clock (that is, the same PHC).

Also, long chains of BC are prone to "gain peaking".

HTH,
Richard

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Linuxptp-users mailing list
Linuxptp-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linuxptp-users
---

LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN http://www.LMAX.com/

Recognised by the most prestigious technology and business awards

Financial technology awards
2017 Best FX trading venue, Fund Technology & WSL Awards
2016 Best trading & execution venue, HFM US Technology Awards

FX industry awards
2016, 2015, 2014, 2013 Winner, Profit & Loss Readers' Choice Awards 2016, 2015, 2014, 2013 Winner, WSL Institutional Trading Awards

Business growth awards
2016, 2015 Winner, Deloitte UK Technology Fast 50 2015, 2014, 2013, Winner, The Sunday Times Tech Track 100 2016, 2015 Winner, Deloitte EMEA Technology Fast 500
2015 Winner, Tech City UK Future 50

---

FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved.

This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). This message is not intended for any recipient(s) who based on their nationality, place of business, domicile or for any other reason, is/are subject to local laws or regulations which prohibit the provision of such products and services. This message is subject to the following terms (http://lmax.com/pdf/general-disclaimers.pdf), if you cannot access these, please notify us by replying to this email and we will send you the terms. If you are not the intended recipient, please notify the sender immediately and delete any copies of this message.

LMAX Exchange is the trading name of LMAX Limited. LMAX Limited operates a multilateral trading facility. LMAX Limited is authorised and regulated by the Financial Conduct Authority (firm registration number 509778) and is a company registered in England and Wales (number 6505809).

LMAX Hong Kong Limited is a wholly-owned subsidiary of LMAX Limited. LMAX Hong Kong is licensed by the Securities and Futures Commission in Hong Kong to conduct Type 3 (leveraged foreign exchange trading) regulated activity with CE Number BDV088.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Linuxptp-users mailing list
Linuxptp-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Mace, Kyle P.
2017-04-25 15:41:12 UTC
Permalink
Why would you need multiple phc2sys processes? Shouldn't you only need one? Shouldn't the command phc2sys -s /dev/ptp0 -c /dev/ptp1 -w sync the two clocks? And would this still go through the system clock?

Thank you
Kyle

-----Original Message-----
From: Luke Bigum [mailto:***@lmax.com]
Sent: Tuesday, April 25, 2017 8:40 AM
To: linuxptp-users
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible

Kyle,

I think you've misunderstood what Richard was talking about. You can't just arbitrarily set any network interface to use any PHC - the oscillator under the PHC device is physically attached to the NIC. Unless you have a multi-port network card that shares one oscillator (eg: SolarFlare 7x22), most NICs will probably have their own PHC. That's why you're getting the device mismatch error.

I've used multiple phc2sys processes in the past to sync different NICs, but this is not a true Boundary Clock. You'll be copying time up and down the Linux stack and through the system clock, so you are subject to all sorts of kernel scheduling jitter and PCI bus latency. If you just want some sort of PTP signal and don't care so much about accuracy though, you can do this.

Also regarding your question earlier about Gain Peaking, Richard was warning you about cumulative error. So if you are proposing to go:

grandmaster -> BC 1 port 1 -> BC 1 port 2 -> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3

And it costs, say, +/- 10us of jitter to go between ports, by the time you get to BC3 port 3 you'll be at +/- 60us.

--
Luke Bigum
Lead Engineer

Information Systems

----- Original Message -----
From: "Mace, Kyle P." <***@gd-ms.com>
To: "Richard Cochran" <***@gmail.com>
Cc: "linuxptp-users" <linuxptp-***@lists.sourceforge.net>
Sent: Tuesday, 25 April, 2017 14:19:47
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible

Alright I am having troubles forcing the ports to use the same PHC. When I specify which PHC to use I get an error about PHC device mismatch. Is there a way to unattach the hardware clock from a specific port?

Output of error
sudo ptp4l -i eno1 -p /dev/ptp1 -m -s
ptp4l[61532.475]: selected /dev/ptp1 as PTP clock
ptp4l[61532.475]: port 1: PHC device mismatch
ptp4l[61532.476]: port 1: /dev/ptp1 requested, but /dev/ptp0 attached
ptp4l[61532.476]: failed to open port eno1

Or would I be able to enable the boundary_clock_jbod option in a config file and then sync the clocks off the system clock using phc2sys?

-----Original Message-----
From: Richard Cochran [mailto:***@gmail.com]
Sent: Thursday, April 20, 2017 4:27 PM
To: Mace, Kyle P.
Cc: linuxptp-***@lists.sourceforge.net
Subject: Re: [Linuxptp-users] ptp inducing as many hops as possible
Post by Mace, Kyle P.
I am in need to test ptp with as many 'hops' as possible. I have
multiple systems with dual Ethernet ports. I would like to have a
setup something like this: grandmaster -> BC 1 port 1 -> BC 1 port 2
-> BC 2 port 1 -> BC 2 port 2 -> BC 3 port 1 -> BC 3 port 3 ->
etc.. Anyone have any ideas if this is possible?
The linuxptp stack supports this.

Remember that all the ports of the BC must share the same clock (that is, the same PHC).

Also, long chains of BC are prone to "gain peaking".

HTH,
Richard

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Linuxptp-users mailing list
Linuxptp-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linuxptp-users
---

LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN http://www.LMAX.com/

Recognised by the most prestigious technology and business awards

Financial technology awards
2017 Best FX trading venue, Fund Technology & WSL Awards
2016 Best trading & execution venue, HFM US Technology Awards

FX industry awards
2016, 2015, 2014, 2013 Winner, Profit & Loss Readers' Choice Awards 2016, 2015, 2014, 2013 Winner, WSL Institutional Trading Awards

Business growth awards
2016, 2015 Winner, Deloitte UK Technology Fast 50 2015, 2014, 2013, Winner, The Sunday Times Tech Track 100 2016, 2015 Winner, Deloitte EMEA Technology Fast 500
2015 Winner, Tech City UK Future 50

---

FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved.

This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). This message is not intended for any recipient(s) who based on their nationality, place of business, domicile or for any other reason, is/are subject to local laws or regulations which prohibit the provision of such products and services. This message is subject to the following terms (http://lmax.com/pdf/general-disclaimers.pdf), if you cannot access these, please notify us by replying to this email and we will send you the terms. If you are not the intended recipient, please notify the sender immediately and delete any copies of this message.

LMAX Exchange is the trading name of LMAX Limited. LMAX Limited operates a multilateral trading facility. LMAX Limited is authorised and regulated by the Financial Conduct Authority (firm registration number 509778) and is a company registered in England and Wales (number 6505809).

LMAX Hong Kong Limited is a wholly-owned subsidiary of LMAX Limited. LMAX Hong Kong is licensed by the Securities and Futures Commission in Hong Kong to conduct Type 3 (leveraged foreign exchange trading) regulated activity with CE Number BDV088.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Linuxptp-users mailing list
Linuxptp-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linuxptp-users
Richard Cochran
2017-04-25 16:53:32 UTC
Permalink
Post by Mace, Kyle P.
Why would you need multiple phc2sys processes? Shouldn't you only
need one? Shouldn't the command phc2sys -s /dev/ptp0 -c /dev/ptp1 -w
sync the two clocks? And would this still go through the system
clock?
With 'phc2sys -a' and boundary_clock_jbod you don't involve the system
clock at all, but everything else Luke said is right. You *will* have
"true" BC WRT the protocol, but this BC will introduce relatively
large errors, like a dozen usec or so.

However, if you can connect your JBODs together with a hardware
signal, like the SDPs on the Intel i210, then you can reduce the error
to 100 nanoseconds or so, but you will have to write the program to
synchronize the group of cards yourself. Still, that is not really
the kind of performance you would expect from a proper BC.

If you really need long chains, then you should use TCs configured
like the Time Aware Bridge (TAB) of 802.1AS. In order to avoid the
ill effects of a chain of servos, they let the clock of the TAB run
free and calculate the master/local rate ratio in order to correct the
residence times.

HTH,
Richard

Loading...