Stumped by NTP. Yes really!

I didn’t think that in 2022 I would have to make a write up like this but here goes…

I have a client’s Linux PBX server that gives time information to all the IP phones in the company.
Thing is, NTP isn’t working.

From the server I can ping any NTP pool like:

linux-server:/var/lib/ntpdate# ping pool.ntp.org
PING pool.ntp.org (193.136.152.71) 56(84) bytes of data.
64 bytes from ntp1.tecnico.ulisboa.pt (193.136.152.71): icmp_req=1 ttl=54 time=10.3 ms
64 bytes from ntp1.tecnico.ulisboa.pt (193.136.152.71): icmp_req=2 ttl=54 time=10.2 ms
64 bytes from ntp1.tecnico.ulisboa.pt (193.136.152.71): icmp_req=3 ttl=54 time=9.91 ms
64 bytes from ntp1.tecnico.ulisboa.pt (193.136.152.71): icmp_req=4 ttl=54 time=9.62 ms

I can also use nmap to confirm port 123 works:

linux-server:/var/lib/ntpdate# nmap -p 123 -sU pool.ntp.org

Starting Nmap 6.00 ( http://nmap.org ) at 2022-05-18 16:33 WEST
Nmap scan report for pool.ntp.org (193.136.152.71)
Host is up (0.00014s latency).
Other addresses for pool.ntp.org (not scanned): 193.136.152.72 51.77.89.237 135.125.157.35
rDNS record for 193.136.152.71: ntp1.tecnico.ulisboa.pt
PORT    STATE         SERVICE
123/udp open|filtered ntp

Nmap done: 1 IP address (1 host up) scanned in 0.35 seconds

I can confirm NTP is running with ps aux, which gives me this:

ntp       3776  0.0  0.0  33636  4252 ?        Ss   May17   0:10 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 106:111

Despite all this, the ntp server is stuck at INIT:

linux-server:/var/lib/ntpdate# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ntp21.kashra-se .INIT.          16 u    - 1024    0    0.000    0.000   0.000

And whenever I try syncronize time manually through ntpdate I get this:

linux-server:/var/lib/ntpdate# ntpdate pool.ntp.org
18 May 16:36:46 ntpdate[27945]: no server suitable for synchronization found

And if I add the -q flag which is just to query the pool, I can see that ntpdate can reach the servers but every single one returns with every parameter set to 0:

linux-server:/var/lib/ntpdate# ntpdate -q pool.ntp.org
server 51.89.13.34, stratum 0, offset 0.000000, delay 0.00000
server 178.33.203.115, stratum 0, offset 0.000000, delay 0.00000
server 195.22.17.7, stratum 0, offset 0.000000, delay 0.00000
server 51.77.89.236, stratum 0, offset 0.000000, delay 0.00000
18 May 16:38:02 ntpdate[28030]: no server suitable for synchronization found

I get the exact same result if I use the -u flag to force the communication through a non privileged port like 123.

Turning off the firewall does nothing.

Any ideias before I hurl this machine into the closest river?

Can you post the output for ntpdate -dv pool.ntp.org?

2 Likes

Sure, here it is:

linux-server:/var/lib/ntpdate# ntpdate -dv pool.ntp.org
18 May 16:52:48 ntpdate[29922]: ntpdate [email protected] Fri Jul 22 17:24:22 UTC 2016 (1)
transmit(135.125.157.35)
transmit(51.77.89.236)
transmit(51.77.89.237)
transmit(51.77.89.238)
transmit(135.125.157.35)
transmit(51.77.89.236)
transmit(51.77.89.237)
transmit(51.77.89.238)
transmit(135.125.157.35)
transmit(51.77.89.236)
transmit(51.77.89.237)
transmit(51.77.89.238)
transmit(135.125.157.35)
transmit(51.77.89.236)
transmit(51.77.89.237)
transmit(51.77.89.238)
transmit(135.125.157.35)
transmit(51.77.89.236)
transmit(51.77.89.237)
transmit(51.77.89.238)
135.125.157.35: Server dropped: no data
51.77.89.236: Server dropped: no data
51.77.89.237: Server dropped: no data
51.77.89.238: Server dropped: no data
server 135.125.157.35, port 123
stratum 0, precision 0, leap 00, trust 000
refid [135.125.157.35], delay 0.00000, dispersion 64.00000
transmitted 4, in filter 4
reference time:    00000000.00000000  Sun, Dec 31 1899 23:23:28.000
originate timestamp: 00000000.00000000  Sun, Dec 31 1899 23:23:28.000
transmit timestamp:  e62f94d6.9fa1c503  Wed, May 18 2022 16:52:54.623
filter delay:  0.00000  0.00000  0.00000  0.00000 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 0.000000 0.000000 0.000000 0.000000
         0.000000 0.000000 0.000000 0.000000
delay 0.00000, dispersion 64.00000
offset 0.000000

server 51.77.89.236, port 123
stratum 0, precision 0, leap 00, trust 000
refid [51.77.89.236], delay 0.00000, dispersion 64.00000
transmitted 4, in filter 4
reference time:    00000000.00000000  Sun, Dec 31 1899 23:23:28.000
originate timestamp: 00000000.00000000  Sun, Dec 31 1899 23:23:28.000
transmit timestamp:  e62f94d6.d2d4cafb  Wed, May 18 2022 16:52:54.823
filter delay:  0.00000  0.00000  0.00000  0.00000 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 0.000000 0.000000 0.000000 0.000000
         0.000000 0.000000 0.000000 0.000000
delay 0.00000, dispersion 64.00000
offset 0.000000

server 51.77.89.237, port 123
stratum 0, precision 0, leap 00, trust 000
refid [51.77.89.237], delay 0.00000, dispersion 64.00000
transmitted 4, in filter 4
reference time:    00000000.00000000  Sun, Dec 31 1899 23:23:28.000
originate timestamp: 00000000.00000000  Sun, Dec 31 1899 23:23:28.000
transmit timestamp:  e62f94d7.0607a8d6  Wed, May 18 2022 16:52:55.023
filter delay:  0.00000  0.00000  0.00000  0.00000 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 0.000000 0.000000 0.000000 0.000000
         0.000000 0.000000 0.000000 0.000000
delay 0.00000, dispersion 64.00000
offset 0.000000

server 51.77.89.238, port 123
stratum 0, precision 0, leap 00, trust 000
refid [51.77.89.238], delay 0.00000, dispersion 64.00000
transmitted 4, in filter 4
reference time:    00000000.00000000  Sun, Dec 31 1899 23:23:28.000
originate timestamp: 00000000.00000000  Sun, Dec 31 1899 23:23:28.000
transmit timestamp:  e62f94d7.393bc530  Wed, May 18 2022 16:52:55.223
filter delay:  0.00000  0.00000  0.00000  0.00000 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 0.000000 0.000000 0.000000 0.000000
         0.000000 0.000000 0.000000 0.000000
delay 0.00000, dispersion 64.00000
offset 0.000000

18 May 16:52:57 ntpdate[29922]: no server suitable for synchronization found

Looks like NTP is being blocked. Likely going to be the perimeter firewall but I suppose ISP could also be filtering.

1 Like

The only firewall in the client’s scenario is this machine before the traffic goes to the internet.

So ISP…
I couldn’t think of much else to be honest…

Thank’s!

I know you said you disabled the firewall but I’d still run a capture just to verify the host firewall isn’t blocking traffic before yelling at the ISP.

sudo tcpdump 'port 123'

1 Like

I’m really sorry I didn’t reply until now, I didn’t have notifications enabled.

Anyway I did what you suggested, and I see the IP phones sending traffic to the server, and the server replying.

Endind the command I get something like:

230 packets captured
230 packets received by filter
0 packets dropped by kernel

I’m an idiot that’s not what matters

I did another capture, this time on the interface where the server goes to the internet, and after restarting NTP i see that the server sends traffic to the ntp pool but it does not get an answer back.

This pretty much confirms that it’s the ISP blocking traffic correct?

If none of the packets were dropped, then yes it has to be something upstream blocking it or a broken package on your server. If you have access to another machine onsite I’d run the same tests to confirm it’s upstream.

Here’s what your tcpdump should look like with ntpdate:

Dump
13:46:38.601549 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > 38.229.54.9.ntp: NTPv4, Client, length 48
13:46:38.609665 IP 38.229.54.9.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:38.801545 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > time.nullroutenetworks.com.ntp: NTPv4, Client, length 48
13:46:38.861352 IP time.nullroutenetworks.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:39.001482 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > t1.time.gq1.yahoo.com.ntp: NTPv4, Client, length 48
13:46:39.066733 IP t1.time.gq1.yahoo.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:39.201484 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > time.cloudflare.com.ntp: NTPv4, Client, length 48
13:46:39.213484 IP time.cloudflare.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:40.601583 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > 38.229.54.9.ntp: NTPv4, Client, length 48
13:46:40.609700 IP 38.229.54.9.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:40.801529 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > time.nullroutenetworks.com.ntp: NTPv4, Client, length 48
13:46:40.861235 IP time.nullroutenetworks.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:41.001478 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > t1.time.gq1.yahoo.com.ntp: NTPv4, Client, length 48
13:46:41.066711 IP t1.time.gq1.yahoo.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:41.201517 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > time.cloudflare.com.ntp: NTPv4, Client, length 48
13:46:41.213491 IP time.cloudflare.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:42.601550 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > 38.229.54.9.ntp: NTPv4, Client, length 48
13:46:42.609736 IP 38.229.54.9.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:42.801557 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > time.nullroutenetworks.com.ntp: NTPv4, Client, length 48
13:46:42.861244 IP time.nullroutenetworks.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:43.001504 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > t1.time.gq1.yahoo.com.ntp: NTPv4, Client, length 48
13:46:43.066829 IP t1.time.gq1.yahoo.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:43.201476 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > time.cloudflare.com.ntp: NTPv4, Client, length 48
13:46:43.213459 IP time.cloudflare.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:44.601556 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > 38.229.54.9.ntp: NTPv4, Client, length 48
13:46:44.609731 IP 38.229.54.9.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:44.801506 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > time.nullroutenetworks.com.ntp: NTPv4, Client, length 48
13:46:44.861222 IP time.nullroutenetworks.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:45.001295 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > t1.time.gq1.yahoo.com.ntp: NTPv4, Client, length 48
13:46:45.066551 IP t1.time.gq1.yahoo.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48
13:46:45.201314 IP ub03.subnet03230935.vcn03230935.oraclevcn.com.41687 > time.cloudflare.com.ntp: NTPv4, Client, length 48
13:46:45.213385 IP time.cloudflare.com.ntp > ub03.subnet03230935.vcn03230935.oraclevcn.com.41687: NTPv4, Server, length 48

Edit: The likelihood of an ISP intentionally blocking NTP seems quite remote to me. Maybe a misconfiguration but surely others would had complained by now.

1 Like

the ntp server conf could have restricted what ip/networks can query it.

I tried from another server on the same network, this is what I got:

10:22:21.309141 IP my-server.ntp > pool.ntp.org.ntp: NTPv4, Client, length 48
10:22:21.509134 IP my-server.ntp > ntp21.kashra-server.com.ntp: NTPv4, Client, length 48
10:22:21.709132 IP my-server.ntp > ntp.rnl.tecnico.ulisboa.pt.ntp: NTPv4, Client, length 48
10:22:21.909168 IP my-server.ntp > 0.ubuntu.pool.ntp.org.ntp: NTPv4, Client, length 48
10:22:23.309133 IP my-server.ntp > pool.ntp.org.ntp: NTPv4, Client, length 48
10:22:23.509128 IP my-server.ntp > ntp21.kashra-server.com.ntp: NTPv4, Client, length 48
10:22:23.709084 IP my-server.ntp > ntp.rnl.tecnico.ulisboa.pt.ntp: NTPv4, Client, length 48
10:22:23.909100 IP my-server.ntp > 0.ubuntu.pool.ntp.org.ntp: NTPv4, Client, length 48
10:22:25.309134 IP my-server.ntp > pool.ntp.org.ntp: NTPv4, Client, length 48
10:22:25.509137 IP my-server.ntp > ntp21.kashra-server.com.ntp: NTPv4, Client, length 48
10:22:25.709126 IP my-server.ntp > ntp.rnl.tecnico.ulisboa.pt.ntp: NTPv4, Client, length 48
10:22:25.909133 IP my-server.ntp > 0.ubuntu.pool.ntp.org.ntp: NTPv4, Client, length 48
10:22:27.309076 IP my-server.ntp > pool.ntp.org.ntp: NTPv4, Client, length 48
10:22:27.509156 IP my-server.ntp > ntp21.kashra-server.com.ntp: NTPv4, Client, length 48
10:22:27.709145 IP my-server.ntp > ntp.rnl.tecnico.ulisboa.pt.ntp: NTPv4, Client, length 48
10:22:27.909075 IP my-server.ntp > 0.ubuntu.pool.ntp.org.ntp: NTPv4, Client, length 48
26 May 10:22:29 ntpdate[4830]: no server suitable for synchronization found

NTP is a common vector for DDoS amplification attacks using botnets , so I could easily see an ISP blocking it for residual connections. You can try calling the support line and tasking for port blocks to be removed.

Can you try “ntpdate -u pool.ntp.org”? This sets an arbitrary source port instead of “123”, which may not trigger ISP blocking of requests/replies.

To rule out if your ISP is blocking it I’d suggest running tcpdump on a VPS with a public IP, and pointing your NTP client at that. Then you can see if your requests are reaching the server.

2 Likes

Hi @cowphrase, executing ntpdate with -u yields the same results.

I didn’t quite understand your suggestion.

You’re saying I should set up a VPS with a public IP and point this server to that for NTP queries?

That traffic would still have to go through the ISP, how would this test rule out a problem with them?

It’s the opposite, it confirms that there is an issue with them. If the request gets to the VPS then it means the reply is getting blocked somewhere, which could be on your own firewall.

Also to raise this with the ISP its helpful to know exactly what’s being blocked (request or reply), and it’s hard to verify that unless you have a second node on the other side of the ISP.

Either way worth calling up your ISP and just asking if they block it. If you’re on a business plan the phone support is usually a little better.