Need help troubleshooting mail server

Dexter_Kane · June 6, 2021, 2:52am

I’m having some troubles with my mail server, starting maybe a month or so ago I’ve noticed that e-mails from some senders have stopped arriving. All other e-mail works no problem.

the log entries for the problem e-mails look like this (postfix 3.4.13 on ubuntu server 20.04) :

Jun  6 10:27:33 HERMES postfix/smtpd[7621]: Anonymous TLS connection established from mx4.slc.paypal.com[173.0.84.229]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Jun  6 10:27:39 HERMES policyd-spf[7627]: prepend Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=173.0.84.229; helo=mx4.slc.paypal.com; [email protected]; receiver=<UNKNOWN> 
Jun  6 10:27:39 HERMES postfix/smtpd[7621]: 2296741618: client=mx4.slc.paypal.com[173.0.84.229]
Jun  6 10:27:41 HERMES postfix/cleanup[7630]: 2296741618: message-id=<[email protected]>
Jun  6 10:32:42 HERMES postfix/smtpd[7621]: timeout after DATA (49136 bytes) from mx4.slc.paypal.com[173.0.84.229]
Jun  6 10:32:42 HERMES postfix/smtpd[7621]: disconnect from mx4.slc.paypal.com[173.0.84.229] ehlo=2 starttls=1 mail=1 rcpt=1 data=0/1 commands=5/6

I’ve also seen entries with connection lost after DATA and maybe something I’ve done while trying to figure it out has changed it from connection lost to timeout, I’m not sure yet. Anyway it’s vague and doesn’t really explain what the problem is. When using verbose logging it appears that the e-mail is passing all the checks, moves on to sending data and then fails. But I did try disabling everything and removing the restrictions and it’s the same result.

I’ve tried lowering the MTU and disabling TCP window scaling but it hasn’t made a difference. I’m guessing it’s a network problem but I don’t know how to track it down, there’s nothing obvious like the the firewall blocking something.

Unfortunately the only reliable way I have of testing it on demand is to buy something on ebay as I won’t get the order confirmed e-mail, but will receive order update e-mails (what the difference is there I don’t know) so I just have to wait for something to attempt to resend which is making it a slow troubleshooting process.

Anyway, I really don’t know how to go about tracking down exactly what’s going wrong so any help would be fantastic .

level1 · June 6, 2021, 3:13am

On your mail server’s network card as well as the switch/router port that it connects to? Both have to have the same value.

Dexter_Kane · June 6, 2021, 3:19am

I’ve only changed it on the server but I’ll try that now. Do you know a way to test that MTU is set correctly?

level1 · June 6, 2021, 3:32am

What do you mean by “set correctly”? As long as both ports on either end of your cable have the same MTU, there shouldn’t be any MTU-related weirdness. There is no ‘one special number’ that works. Weirdness happens simply because the numbers on either side of the cable are different.

How you set the MTUs depends on your OS and switch/router. Pretty-much all tools/commands that are used to set MTUs can also be used to read/display MTUs. You should be able to trust what they report.

Once you’ve got both MTUs synchronised you might want to reboot both devices and check to make sure the MTU changes have persisted.

Dexter_Kane · June 6, 2021, 3:35am

I mean a way of checking if changing the MTU is having an effect other than just waiting to see if it starts working

level1 · June 6, 2021, 3:59am

Dissimilar MTUs are one possible cause of the problem you reported. Is it the cause? There’s no way to tell right now because there are too many moving parts to the system you are dealing with.

If the MTUs weren’t synchronised, but now are, then all you can do is wait and see what happens. If the timeouts stop, then it’s likely that was your problem.

If the timeouts continue, then it’s unlikely dissimilar local MTUs were the cause, and you have to look at something else. In that event, at least you’ve eliminated one possible cause and can focus your attention elsewhere.

I’m not sure how viable sending yourself lots of small Paypal payments is, or if that would result in the same sort of messages coming through, but you could look into that if you want to do some 1st-person testing.

The firewall for mx4.slc.paypal.com is operating in stealth mode, so the usual approach (of pinging the destination with various sized packets until you force a split and thus discover the largest possible packet size) is not available, unfortunately.

Dexter_Kane · June 6, 2021, 4:07am

I’m still getting e-mails from paypal except for whatever this one is, so I don’t know how to trigger it. the only one I’m certain about is order confirmed e-mails from ebay. I’m not sure if it’s just coming from a different server or if it has something to do with the content of the e-mail itself but there’s a handful of things no getting through while everything else still works.

I’m also not sure what it was that has changed it from connection lost to timeout

nx2l · June 6, 2021, 4:12am

are you running AV?

Dexter_Kane · June 6, 2021, 4:15am

Yes it’s got clamav but as far as I can tell it’s not getting to that stage. But there’s no real time AV running if that’s what you mean.

level1 · June 6, 2021, 4:26am

If you trawl through your existing logs and see that only a single server (or a small number of servers) are involved in the error, then you could enable debug mode explicity for connections from those servers.

/etc/postfix/main.cf:
  debug_peer_level = 3
  debug_peer_list = 173.0.84.229

(Where 173.0.84.229 is the address of mx4.slc.paypal.com, for example.)

The extra debug information might shed more light on what is going on? (Debug level 2 is default, but I think 3 is the max.)

See:
http://www.postfix.org/postconf.5.html#debug_peer_list

Dexter_Kane · June 6, 2021, 4:34am

I had verbose logging on before but it didn’t really add anything useful. It showed that all the checks were passed and my server sent the message to say to start sending data, but then it loses the connection or times out while the data is being sent (it always shows some number of bytes). So it’s something going wrong while the data is being sent, but increasing the log level doesn’t add anything to that message.

I’m currently running tcpdump to see if that will show anything but I haven’t received any e-mails so far.

level1 · June 6, 2021, 5:15am

While you are waiting for your next mail, I’ll just leave this here for some casual reading. Translated from the original German (Postfix: Timeout after DATA / lost connection - Heinlein Support - Unser Linux-Blog für AdminsHeinlein Support – Unser Linux-Blog für Admins):

If outgoing e-mails are not sent, error messages such as “Timeout after DATA” can be found in the log file and it may also be necessary to note that mainly larger e-mails are affected, then the diagnosis is quickly clear: The admins of the firewall overshot the target.
ICMP in firewalls must not be banned across the board

In almost all of these cases the cause can be traced back to the following problem: The firewall blocks all incoming ICMP packets, although some of them are indispensable and not bad either :-). However, this also blocks the meaningful and desired (and necessary) ICMP packets “fragmentation needed”. With these ICMP palettes, routers can inform the sending host in the meantime that it has chosen a packet size (MTU) that is too large and that it must send smaller packets. If these ICMP messages are lost, the connection “freezes” from the point of view of the sending server: It has sent out IP packets and (apparently) suddenly no longer receives any response. Thanks to his firewall, he does not find out that his IP packets are lost and he has to send smaller packets … Postfix runs into a timeout and the whole thing ends in the log file with:

Sep 26 14:45:30 mail postfix / smtpd [12106]: timeout after DATA from
mail.example.com [xxx.xxx.xxx.xxx]

The scenario at a glance

The sending server has a normal MTU packet size of 1500. A smaller MTU is required on the way to the target system: for example 1,492 or something even smaller. This typically occurs when the target system is connected via business SDSL, cable modem or similar. In this respect, the phenomenon described here typically only occurs with certain remote stations, while the connections otherwise mostly function normally. The ICMP packets “fragmentation needed” that have now been sent back are lost - and the connection freezes.
Quick remedy as a workaround / test

As a quick hack and workaround, you can reduce the MTU packet size on the Postfx mail server. Just as a quick test, for something absurdly small such as 1,250.

ifconfig eth0 mtu 1250

The problems described should then suddenly stop and the messages in the log should not appear. This would already provide the proof for the problem described here (“q.e.d.” :-)). Such a low MTU is of course not sensible and desirable - and it does not really solve the problem, especially since other services such as web servers also have the same problems in a completely analogous manner, but the operator rarely notices it there. For a real solution, the firewall admins have to go and check which incoming ICMP rules they have set.
Allow ICMP!

The following ICMP packets should be allowed on firewalls:
ICMP Unreachable
ICMP Unreachable, Fragmentation Needed
ICMP Time Exceeded in Transit
Depending on which side the connection freezes on, the message “Timeout after DATA” is in our Postfix from smtpd (incoming e-mails, connection froze - have we blocked outgoing ICMPs?) Or from smtp (outgoing mails - we have incoming ICMPs blocked?).

Haven’t been ‘tightening up’ security on your firewall lately, have you?

Dexter_Kane · June 6, 2021, 5:38am

I’ve tried really low MTU values and haven’t seen a change, I’m not really convinced it is an MTU problem but that’s what’s worked for everyone else.

It’s possible that weeks ago I changed something on the firewall that broke this and I don’t remember what it is, but ICMP is definitely not blocked on my end.

Dexter_Kane · June 6, 2021, 7:42am

Here’s another failed e-mail

from mail.log

Jun  6 15:16:19 HERMES postfix/smtpd[2164]: initializing the server-side TLS engine
Jun  6 15:16:23 HERMES postfix/smtpd[2164]: connect from o1131.abmail.my.zip.co[167.89.95.108]
Jun  6 15:16:24 HERMES postfix/smtpd[2164]: setting up TLS connection from o1131.abmail.my.zip.co[167.89.95.108]
Jun  6 15:16:24 HERMES postfix/smtpd[2164]: o1131.abmail.my.zip.co[167.89.95.108]: TLS cipher list "aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH"
Jun  6 15:16:24 HERMES postfix/smtpd[2164]: SSL_accept:before SSL initialization
Jun  6 15:16:24 HERMES postfix/smtpd[2164]: SSL_accept:before SSL initialization
Jun  6 15:16:24 HERMES postfix/smtpd[2164]: SSL_accept:SSLv3/TLS read client hello
Jun  6 15:16:24 HERMES postfix/smtpd[2164]: SSL_accept:SSLv3/TLS write server hello
Jun  6 15:16:24 HERMES postfix/smtpd[2164]: SSL_accept:SSLv3/TLS write change cipher spec
Jun  6 15:16:24 HERMES postfix/smtpd[2164]: SSL_accept:TLSv1.3 early data
Jun  6 15:16:25 HERMES postfix/smtpd[2164]: SSL_accept:TLSv1.3 early data
Jun  6 15:16:25 HERMES postfix/smtpd[2164]: SSL_accept:SSLv3/TLS read client hello
Jun  6 15:16:25 HERMES postfix/smtpd[2164]: SSL_accept:SSLv3/TLS write server hello
Jun  6 15:16:25 HERMES postfix/smtpd[2164]: SSL_accept:TLSv1.3 write encrypted extensions
Jun  6 15:16:25 HERMES postfix/smtpd[2164]: SSL_accept:SSLv3/TLS write certificate
Jun  6 15:16:25 HERMES postfix/smtpd[2164]: SSL_accept:TLSv1.3 write server certificate verify
Jun  6 15:16:25 HERMES postfix/smtpd[2164]: SSL_accept:SSLv3/TLS write finished
Jun  6 15:16:25 HERMES postfix/smtpd[2164]: SSL_accept:TLSv1.3 early data
Jun  6 15:16:26 HERMES postfix/smtpd[2164]: SSL_accept:TLSv1.3 early data
Jun  6 15:16:26 HERMES postfix/smtpd[2164]: SSL_accept:SSLv3/TLS read finished
Jun  6 15:16:26 HERMES postfix/smtpd[2164]: o1131.abmail.my.zip.co[167.89.95.108]: Issuing session ticket, key expiration: 1622965585
Jun  6 15:16:26 HERMES postfix/smtpd[2164]: SSL_accept:SSLv3/TLS write session ticket
Jun  6 15:16:26 HERMES postfix/smtpd[2164]: Anonymous TLS connection established from o1131.abmail.my.zip.co[167.89.95.108]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (2048 bits) server-digest SHA256
Jun  6 15:16:32 HERMES policyd-spf[2170]: prepend Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=167.89.95.108; helo=o1131.abmail.my.zip.co; envelope-from=bounces+7397607-124a-paypal=aquinasnet.com@abmail.my.zip.co; receiver=<UNKNOWN> 
Jun  6 15:16:32 HERMES postfix/smtpd[2164]: 78CE841618: client=o1131.abmail.my.zip.co[167.89.95.108]
Jun  6 15:16:33 HERMES postfix/cleanup[2172]: 78CE841618: message-id=<[email protected]>
Jun  6 15:21:34 HERMES postfix/smtpd[2164]: timeout after DATA (61398 bytes) from o1131.abmail.my.zip.co[167.89.95.108]
Jun  6 15:21:34 HERMES postfix/smtpd[2164]: disconnect from o1131.abmail.my.zip.co[167.89.95.108] ehlo=2 starttls=1 mail=1 rcpt=1 data=0/1 commands=5/6
Jun  6 15:24:54 HERMES postfix/anvil[2165]: statistics: max connection rate 1/60s for (smtp:167.89.95.108) at Jun  6 15:16:23
Jun  6 15:24:54 HERMES postfix/anvil[2165]: statistics: max connection count 1 for (smtp:167.89.95.108) at Jun  6 15:16:23
Jun  6 15:24:54 HERMES postfix/anvil[2165]: statistics: max cache size 1 at Jun  6 15:16:23

and the corresponding tcpdump

15:16:18.956736 IP (tos 0x0, ttl 51, id 51059, offset 0, flags [none], proto TCP (6), length 60)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [S], cksum 0x14e2 (correct), seq 1424647911, win 29200, options [mss 1288,sackOK,TS val 3593911713 ecr 0,nop,wscale 9], length 0
15:16:18.956856 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 56)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [S.], cksum 0x1410 (incorrect -> 0x92b3), seq 3543317145, ack 1424647912, win 64440, options [mss 1444,nop,nop,TS val 943868123 ecr 3593911713], length 0
15:16:19.332871 IP (tos 0x0, ttl 51, id 12478, offset 0, flags [none], proto TCP (6), length 52)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x32a0 (correct), ack 1, win 29200, options [nop,nop,TS val 3593912074 ecr 943868123], length 0
15:16:23.295120 IP (tos 0x0, ttl 64, id 65121, offset 0, flags [DF], proto TCP (6), length 91)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x1433 (incorrect -> 0xb1b6), seq 1:40, ack 1, win 64440, options [nop,nop,TS val 943872462 ecr 3593912074], length 39: SMTP, length: 39
	220 mail.aquinasnet.com ESMTP Postfix
15:16:23.696576 IP (tos 0x0, ttl 51, id 13421, offset 0, flags [none], proto TCP (6), length 52)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x106b (correct), ack 40, win 29200, options [nop,nop,TS val 3593916453 ecr 943872462], length 0
15:16:23.715310 IP (tos 0x0, ttl 51, id 24514, offset 0, flags [none], proto TCP (6), length 81)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [P.], cksum 0xee8f (correct), seq 1:30, ack 40, win 29200, options [nop,nop,TS val 3593916471 ecr 943872462], length 29: SMTP, length: 29
	EHLO o1131.abmail.my.zip.co
15:16:23.724739 IP (tos 0x0, ttl 64, id 65122, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x8503), ack 30, win 64411, options [nop,nop,TS val 943872891 ecr 3593916471], length 0
15:16:23.725479 IP (tos 0x0, ttl 64, id 65123, offset 0, flags [DF], proto TCP (6), length 198)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x149e (incorrect -> 0x9885), seq 40:186, ack 30, win 64411, options [nop,nop,TS val 943872892 ecr 3593916471], length 146: SMTP, length: 146
	250-mail.aquinasnet.com
	250-PIPELINING
	250-SIZE 10240000
	250-ETRN
	250-STARTTLS
	250-ENHANCEDSTATUSCODES
	250-8BITMIME
	250-DSN
	250 CHUNKING
15:16:24.313579 IP (tos 0x0, ttl 51, id 59011, offset 0, flags [none], proto TCP (6), length 62)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [P.], cksum 0xc60a (correct), seq 30:40, ack 186, win 30016, options [nop,nop,TS val 3593917070 ecr 943872892], length 10: SMTP, length: 10
	STARTTLS
15:16:24.313646 IP (tos 0x0, ttl 64, id 65124, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x7fcd), ack 40, win 64401, options [nop,nop,TS val 943873480 ecr 3593917070], length 0
15:16:24.313913 IP (tos 0x0, ttl 64, id 65125, offset 0, flags [DF], proto TCP (6), length 82)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x142a (incorrect -> 0x6883), seq 186:216, ack 40, win 64401, options [nop,nop,TS val 943873481 ecr 3593917070], length 30: SMTP, length: 30
	220 2.0.0 Ready to start TLS
15:16:24.704713 IP (tos 0x0, ttl 51, id 39327, offset 0, flags [none], proto TCP (6), length 323)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [P.], cksum 0x0ac1 (correct), seq 40:311, ack 216, win 30016, options [nop,nop,TS val 3593917460 ecr 943873481], length 271: SMTP, length: 271
15:16:24.704832 IP (tos 0x0, ttl 64, id 65126, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x7ca2), ack 311, win 64130, options [nop,nop,TS val 943873871 ecr 3593917460], length 0
15:16:24.705311 IP (tos 0x0, ttl 64, id 65127, offset 0, flags [DF], proto TCP (6), length 151)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x146f (incorrect -> 0xe5a5), seq 216:315, ack 311, win 64130, options [nop,nop,TS val 943873872 ecr 3593917460], length 99: SMTP, length: 99
15:16:25.098485 IP (tos 0x0, ttl 51, id 63988, offset 0, flags [none], proto TCP (6), length 58)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [P.], cksum 0xe7e2 (correct), seq 311:317, ack 315, win 30016, options [nop,nop,TS val 3593917856 ecr 943873872], length 6: SMTP, length: 6
15:16:25.112468 IP (tos 0x0, ttl 51, id 7183, offset 0, flags [none], proto TCP (6), length 388)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [P.], cksum 0x3386 (correct), seq 317:653, ack 315, win 30016, options [nop,nop,TS val 3593917869 ecr 943873872], length 336: SMTP, length: 336
15:16:25.112851 IP (tos 0x0, ttl 64, id 65128, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x791b), ack 653, win 63788, options [nop,nop,TS val 943874279 ecr 3593917856], length 0
15:16:25.122102 IP (tos 0x0, ttl 64, id 65129, offset 0, flags [DF], proto TCP (6), length 2604)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x1e04 (incorrect -> 0xc778), seq 315:2867, ack 653, win 63788, options [nop,nop,TS val 943874289 ecr 3593917856], length 2552: SMTP, length: 2552
15:16:25.122139 IP (tos 0x0, ttl 64, id 65131, offset 0, flags [DF], proto TCP (6), length 1596)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x1a14 (incorrect -> 0xea03), seq 2867:4411, ack 653, win 63788, options [nop,nop,TS val 943874289 ecr 3593917856], length 1544: SMTP, length: 1544
15:16:25.677054 IP (tos 0x0, ttl 51, id 21100, offset 0, flags [none], proto TCP (6), length 52)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xda74 (correct), ack 2867, win 35728, options [nop,nop,TS val 3593918433 ecr 943874289], length 0
15:16:25.684569 IP (tos 0x0, ttl 51, id 24010, offset 0, flags [none], proto TCP (6), length 52)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xc076 (correct), ack 4411, win 40832, options [nop,nop,TS val 3593918439 ecr 943874289], length 0
15:16:25.684616 IP (tos 0x0, ttl 64, id 65133, offset 0, flags [DF], proto TCP (6), length 697)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x1691 (incorrect -> 0x86dc), seq 4411:5056, ack 653, win 63788, options [nop,nop,TS val 943874851 ecr 3593918439], length 645: SMTP, length: 645
15:16:26.080165 IP (tos 0x0, ttl 51, id 5917, offset 0, flags [none], proto TCP (6), length 177)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [P.], cksum 0xbb51 (correct), seq 653:778, ack 5056, win 43384, options [nop,nop,TS val 3593918837 ecr 943874851], length 125: SMTP, length: 125
15:16:26.081422 IP (tos 0x0, ttl 64, id 65134, offset 0, flags [DF], proto TCP (6), length 307)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x150b (incorrect -> 0xbeb5), seq 5056:5311, ack 778, win 63788, options [nop,nop,TS val 943875248 ecr 3593918837], length 255: SMTP, length: 255
15:16:26.491914 IP (tos 0x0, ttl 51, id 17696, offset 0, flags [none], proto TCP (6), length 52)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xa19c (correct), ack 5311, win 45936, options [nop,nop,TS val 3593919249 ecr 943875248], length 0
15:16:26.491972 IP (tos 0x0, ttl 64, id 65135, offset 0, flags [DF], proto TCP (6), length 206)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x14a6 (incorrect -> 0xbc6d), seq 5311:5465, ack 778, win 63788, options [nop,nop,TS val 943875659 ecr 3593919249], length 154: SMTP, length: 154
15:16:26.862012 IP (tos 0x0, ttl 51, id 65232, offset 0, flags [none], proto TCP (6), length 52)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x93fe (correct), ack 5465, win 48488, options [nop,nop,TS val 3593919618 ecr 943875659], length 0
15:16:26.891719 IP (tos 0x0, ttl 51, id 37543, offset 0, flags [none], proto TCP (6), length 161)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [P.], cksum 0x55f1 (correct), seq 778:887, ack 5465, win 48488, options [nop,nop,TS val 3593919647 ecr 943875659], length 109: SMTP, length: 109
15:16:26.909824 IP (tos 0x0, ttl 64, id 65136, offset 0, flags [DF], proto TCP (6), length 88)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x1430 (incorrect -> 0x5a38), seq 5465:5501, ack 887, win 63788, options [nop,nop,TS val 943876076 ecr 3593919647], length 36: SMTP, length: 36
15:16:27.403809 IP (tos 0x0, ttl 51, id 31545, offset 0, flags [none], proto TCP (6), length 107)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [P.], cksum 0xd6ff (correct), seq 887:942, ack 5501, win 48488, options [nop,nop,TS val 3593920161 ecr 943876076], length 55: SMTP, length: 55
15:16:27.447104 IP (tos 0x0, ttl 64, id 65137, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x5198), ack 942, win 63788, options [nop,nop,TS val 943876614 ecr 3593920161], length 0
15:16:32.502943 IP (tos 0x0, ttl 64, id 65138, offset 0, flags [DF], proto TCP (6), length 88)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x1430 (incorrect -> 0x48a6), seq 5501:5537, ack 942, win 63788, options [nop,nop,TS val 943881670 ecr 3593920161], length 36: SMTP, length: 36
15:16:32.903579 IP (tos 0x0, ttl 51, id 40909, offset 0, flags [none], proto TCP (6), length 80)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [P.], cksum 0x9d5a (correct), seq 942:970, ack 5537, win 48488, options [nop,nop,TS val 3593925661 ecr 943881670], length 28: SMTP, length: 28
15:16:32.903645 IP (tos 0x0, ttl 64, id 65139, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x268c), ack 970, win 63788, options [nop,nop,TS val 943882070 ecr 3593925661], length 0
15:16:32.904965 IP (tos 0x0, ttl 64, id 65140, offset 0, flags [DF], proto TCP (6), length 111)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x1447 (incorrect -> 0xb999), seq 5537:5596, ack 970, win 63788, options [nop,nop,TS val 943882072 ecr 3593925661], length 59: SMTP, length: 59
15:16:33.295179 IP (tos 0x0, ttl 51, id 18043, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x1cd1 (correct), seq 970:2246, ack 5596, win 48488, options [nop,nop,TS val 3593926050 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.295180 IP (tos 0x0, ttl 51, id 10416, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xbfd3 (correct), seq 2246:3522, ack 5596, win 48488, options [nop,nop,TS val 3593926050 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.295303 IP (tos 0x0, ttl 64, id 65141, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x1e48), ack 2246, win 63788, options [nop,nop,TS val 943882462 ecr 3593926050], length 0
15:16:33.295360 IP (tos 0x0, ttl 64, id 65142, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x1e3c), ack 3522, win 62524, options [nop,nop,TS val 943882462 ecr 3593926050], length 0
15:16:33.295411 IP (tos 0x0, ttl 51, id 14412, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xba39 (correct), seq 3522:4798, ack 5596, win 48488, options [nop,nop,TS val 3593926050 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.295433 IP (tos 0x0, ttl 64, id 65143, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x1e3c), ack 4798, win 61248, options [nop,nop,TS val 943882462 ecr 3593926050], length 0
15:16:33.296904 IP (tos 0x0, ttl 51, id 35623, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x283a (correct), seq 4798:6074, ack 5596, win 48488, options [nop,nop,TS val 3593926050 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.296961 IP (tos 0x0, ttl 64, id 65144, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x0f46), ack 6074, win 63800, options [nop,nop,TS val 943882464 ecr 3593926050], length 0
15:16:33.297021 IP (tos 0x0, ttl 51, id 30916, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xa71c (correct), seq 6074:7350, ack 5596, win 48488, options [nop,nop,TS val 3593926050 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.297032 IP (tos 0x0, ttl 64, id 65145, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x0f46), ack 7350, win 62524, options [nop,nop,TS val 943882464 ecr 3593926050], length 0
15:16:33.297195 IP (tos 0x0, ttl 51, id 18343, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xac8f (correct), seq 7350:8626, ack 5596, win 48488, options [nop,nop,TS val 3593926050 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.297220 IP (tos 0x0, ttl 64, id 65146, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x054e), ack 8626, win 63800, options [nop,nop,TS val 943882464 ecr 3593926050], length 0
15:16:33.297409 IP (tos 0x0, ttl 51, id 41091, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xcb4a (correct), seq 8626:9902, ack 5596, win 48488, options [nop,nop,TS val 3593926051 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.297421 IP (tos 0x0, ttl 64, id 65147, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x0051), ack 9902, win 63800, options [nop,nop,TS val 943882464 ecr 3593926051], length 0
15:16:33.297565 IP (tos 0x0, ttl 51, id 45828, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x505b (correct), seq 9902:11178, ack 5596, win 48488, options [nop,nop,TS val 3593926051 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.297579 IP (tos 0x0, ttl 64, id 65148, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x0051), ack 11178, win 62524, options [nop,nop,TS val 943882464 ecr 3593926051], length 0
15:16:33.297757 IP (tos 0x0, ttl 51, id 9750, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xc618 (correct), seq 11178:12454, ack 5596, win 48488, options [nop,nop,TS val 3593926051 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.297757 IP (tos 0x0, ttl 51, id 4897, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x8255 (correct), seq 12454:13730, ack 5596, win 48488, options [nop,nop,TS val 3593926051 ecr 943882072], length 1276: SMTP, length: 1276
15:16:33.297789 IP (tos 0x0, ttl 64, id 65149, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x0051), ack 12454, win 61248, options [nop,nop,TS val 943882464 ecr 3593926051], length 0
15:16:33.297816 IP (tos 0x0, ttl 64, id 65150, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x0051), ack 13730, win 59972, options [nop,nop,TS val 943882464 ecr 3593926051], length 0
15:16:33.672867 IP (tos 0x0, ttl 51, id 54936, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xb1c0 (correct), seq 13730:15006, ack 5596, win 48488, options [nop,nop,TS val 3593926428 ecr 943882462], length 1276: SMTP, length: 1276
15:16:33.672938 IP (tos 0x0, ttl 64, id 65151, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0xe96f), ack 15006, win 63800, options [nop,nop,TS val 943882840 ecr 3593926428], length 0
15:16:33.672993 IP (tos 0x0, ttl 51, id 45826, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x241c (correct), seq 15006:16282, ack 5596, win 48488, options [nop,nop,TS val 3593926428 ecr 943882462], length 1276: SMTP, length: 1276
15:16:33.672993 IP (tos 0x0, ttl 51, id 58624, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x04dd (correct), seq 16282:17558, ack 5596, win 48488, options [nop,nop,TS val 3593926428 ecr 943882462], length 1276: SMTP, length: 1276
15:16:33.673006 IP (tos 0x0, ttl 64, id 65152, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0xe96f), ack 16282, win 62524, options [nop,nop,TS val 943882840 ecr 3593926428], length 0
15:16:33.673077 IP (tos 0x0, ttl 64, id 65153, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0xe96f), ack 17558, win 61248, options [nop,nop,TS val 943882840 ecr 3593926428], length 0
15:16:33.673218 IP (tos 0x0, ttl 51, id 39399, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x67c3 (correct), seq 17558:18834, ack 5596, win 48488, options [nop,nop,TS val 3593926428 ecr 943882462], length 1276: SMTP, length: 1276
15:16:33.673228 IP (tos 0x0, ttl 64, id 65154, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0xda7b), ack 18834, win 63800, options [nop,nop,TS val 943882840 ecr 3593926428], length 0
15:16:33.673345 IP (tos 0x0, ttl 51, id 59095, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x6a63 (correct), seq 18834:20110, ack 5596, win 48488, options [nop,nop,TS val 3593926429 ecr 943882462], length 1276: SMTP, length: 1276
15:16:33.673345 IP (tos 0x0, ttl 51, id 5781, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x3056 (correct), seq 20110:21386, ack 5596, win 48488, options [nop,nop,TS val 3593926429 ecr 943882462], length 1276: SMTP, length: 1276
15:16:33.673536 IP (tos 0x0, ttl 51, id 5292, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xd5ac (correct), seq 21386:22662, ack 5596, win 48488, options [nop,nop,TS val 3593926429 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.673537 IP (tos 0x0, ttl 51, id 11596, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xfa9e (correct), seq 22662:23938, ack 5596, win 48488, options [nop,nop,TS val 3593926429 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.673545 IP (tos 0x0, ttl 64, id 65155, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0xd082), ack 21386, win 63800, options [nop,nop,TS val 943882840 ecr 3593926429], length 0
15:16:33.673597 IP (tos 0x0, ttl 64, id 65156, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0xcb86), ack 23938, win 62524, options [nop,nop,TS val 943882840 ecr 3593926429], length 0
15:16:33.676623 IP (tos 0x0, ttl 51, id 39487, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xf2af (correct), seq 23938:25214, ack 5596, win 48488, options [nop,nop,TS val 3593926433 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.676707 IP (tos 0x0, ttl 51, id 54158, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xb5af (correct), seq 25214:26490, ack 5596, win 48488, options [nop,nop,TS val 3593926433 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.676723 IP (tos 0x0, ttl 64, id 65157, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0xc187), ack 26490, win 62524, options [nop,nop,TS val 943882843 ecr 3593926433], length 0
15:16:33.677729 IP (tos 0x0, ttl 51, id 37736, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x41c9 (correct), seq 26490:27766, ack 5596, win 48488, options [nop,nop,TS val 3593926433 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.677914 IP (tos 0x0, ttl 51, id 30936, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x0050 (correct), seq 27766:29042, ack 5596, win 48488, options [nop,nop,TS val 3593926433 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.677925 IP (tos 0x0, ttl 64, id 65158, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0xb78d), ack 29042, win 62524, options [nop,nop,TS val 943882845 ecr 3593926433], length 0
15:16:33.678086 IP (tos 0x0, ttl 51, id 42649, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x0066 (correct), seq 29042:30318, ack 5596, win 48488, options [nop,nop,TS val 3593926434 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.678263 IP (tos 0x0, ttl 51, id 22345, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x5282 (correct), seq 30318:31594, ack 5596, win 48488, options [nop,nop,TS val 3593926434 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.678274 IP (tos 0x0, ttl 64, id 65159, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0xa898), ack 31594, win 63800, options [nop,nop,TS val 943882845 ecr 3593926434], length 0
15:16:33.678548 IP (tos 0x0, ttl 51, id 47490, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x0cc9 (correct), seq 31594:32870, ack 5596, win 48488, options [nop,nop,TS val 3593926434 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.678740 IP (tos 0x0, ttl 51, id 14613, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x960b (correct), seq 32870:34146, ack 5596, win 48488, options [nop,nop,TS val 3593926434 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.678768 IP (tos 0x0, ttl 64, id 65160, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x9ea0), ack 34146, win 63800, options [nop,nop,TS val 943882845 ecr 3593926434], length 0
15:16:33.678903 IP (tos 0x0, ttl 51, id 1453, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x2c49 (correct), seq 34146:35422, ack 5596, win 48488, options [nop,nop,TS val 3593926434 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.679088 IP (tos 0x0, ttl 51, id 30885, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x564d (correct), seq 35422:36698, ack 5596, win 48488, options [nop,nop,TS val 3593926434 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.679088 IP (tos 0x0, ttl 51, id 46125, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x7162 (correct), seq 36698:37974, ack 5596, win 48488, options [nop,nop,TS val 3593926434 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.679101 IP (tos 0x0, ttl 64, id 65161, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x94a7), ack 36698, win 63800, options [nop,nop,TS val 943882846 ecr 3593926434], length 0
15:16:33.679242 IP (tos 0x0, ttl 51, id 65172, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x75c1 (correct), seq 37974:39250, ack 5596, win 48488, options [nop,nop,TS val 3593926434 ecr 943882464], length 1276: SMTP, length: 1276
15:16:33.679263 IP (tos 0x0, ttl 64, id 65162, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x8aaf), ack 39250, win 63800, options [nop,nop,TS val 943882846 ecr 3593926434], length 0
15:16:34.123014 IP (tos 0x0, ttl 51, id 58471, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x34b3 (correct), seq 39250:40526, ack 5596, win 48488, options [nop,nop,TS val 3593926878 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.123084 IP (tos 0x0, ttl 51, id 58107, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x38bf (correct), seq 40526:41802, ack 5596, win 48488, options [nop,nop,TS val 3593926878 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.123242 IP (tos 0x0, ttl 51, id 8045, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xe40b (correct), seq 41802:43078, ack 5596, win 48488, options [nop,nop,TS val 3593926878 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.123247 IP (tos 0x0, ttl 64, id 65163, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x7d3f), ack 41802, win 63800, options [nop,nop,TS val 943883290 ecr 3593926878], length 0
15:16:34.123278 IP (tos 0x0, ttl 64, id 65164, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x717c), ack 43078, win 65535, options [nop,nop,TS val 943883290 ecr 3593926878], length 0
15:16:34.123511 IP (tos 0x0, ttl 51, id 8006, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x5aa2 (correct), seq 43078:44354, ack 5596, win 48488, options [nop,nop,TS val 3593926878 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.123511 IP (tos 0x0, ttl 51, id 52047, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x0aea (correct), seq 44354:45630, ack 5596, win 48488, options [nop,nop,TS val 3593926878 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.123540 IP (tos 0x0, ttl 64, id 65165, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x6c80), ack 44354, win 65535, options [nop,nop,TS val 943883290 ecr 3593926878], length 0
15:16:34.123591 IP (tos 0x0, ttl 64, id 65166, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x6784), ack 45630, win 65535, options [nop,nop,TS val 943883290 ecr 3593926878], length 0
15:16:34.123818 IP (tos 0x0, ttl 51, id 8463, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xe777 (correct), seq 45630:46906, ack 5596, win 48488, options [nop,nop,TS val 3593926878 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.123818 IP (tos 0x0, ttl 51, id 30847, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x93ea (correct), seq 46906:48182, ack 5596, win 48488, options [nop,nop,TS val 3593926878 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.123818 IP (tos 0x0, ttl 51, id 21268, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x28c7 (correct), seq 48182:49458, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.123852 IP (tos 0x0, ttl 64, id 65167, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x6288), ack 46906, win 65535, options [nop,nop,TS val 943883290 ecr 3593926878], length 0
15:16:34.123882 IP (tos 0x0, ttl 64, id 65168, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x5d8c), ack 48182, win 65535, options [nop,nop,TS val 943883290 ecr 3593926878], length 0
15:16:34.123907 IP (tos 0x0, ttl 64, id 65169, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x588e), ack 49458, win 65535, options [nop,nop,TS val 943883291 ecr 3593926879], length 0
15:16:34.124385 IP (tos 0x0, ttl 51, id 8616, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x9039 (correct), seq 49458:50734, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.124415 IP (tos 0x0, ttl 64, id 65170, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x5392), ack 50734, win 65535, options [nop,nop,TS val 943883291 ecr 3593926879], length 0
15:16:34.124907 IP (tos 0x0, ttl 51, id 6589, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x5254 (correct), seq 50734:52010, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.124939 IP (tos 0x0, ttl 64, id 65171, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x4e95), ack 52010, win 65535, options [nop,nop,TS val 943883292 ecr 3593926879], length 0
15:16:34.125073 IP (tos 0x0, ttl 51, id 57326, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x7bc9 (correct), seq 52010:53286, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.125073 IP (tos 0x0, ttl 51, id 47867, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x9db6 (correct), seq 53286:54562, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.125073 IP (tos 0x0, ttl 51, id 20667, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x29c1 (correct), seq 54562:55838, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882840], length 1276: SMTP, length: 1276
15:16:34.125104 IP (tos 0x0, ttl 64, id 65172, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x4999), ack 53286, win 65535, options [nop,nop,TS val 943883292 ecr 3593926879], length 0
15:16:34.125131 IP (tos 0x0, ttl 64, id 65173, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x449d), ack 54562, win 65535, options [nop,nop,TS val 943883292 ecr 3593926879], length 0
15:16:34.125161 IP (tos 0x0, ttl 64, id 65174, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x3fa1), ack 55838, win 65535, options [nop,nop,TS val 943883292 ecr 3593926879], length 0
15:16:34.125259 IP (tos 0x0, ttl 51, id 13663, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xa72c (correct), seq 55838:57114, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882843], length 1276: SMTP, length: 1276
15:16:34.125259 IP (tos 0x0, ttl 51, id 56416, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x4a79 (correct), seq 57114:58390, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882843], length 1276: SMTP, length: 1276
15:16:34.125259 IP (tos 0x0, ttl 51, id 44848, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x5228 (correct), seq 58390:59666, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882843], length 1276: SMTP, length: 1276
15:16:34.125283 IP (tos 0x0, ttl 64, id 65175, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x3aa5), ack 57114, win 65535, options [nop,nop,TS val 943883292 ecr 3593926879], length 0
15:16:34.125317 IP (tos 0x0, ttl 64, id 65176, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x35a9), ack 58390, win 65535, options [nop,nop,TS val 943883292 ecr 3593926879], length 0
15:16:34.125332 IP (tos 0x0, ttl 64, id 65177, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x30ad), ack 59666, win 65535, options [nop,nop,TS val 943883292 ecr 3593926879], length 0
15:16:34.126791 IP (tos 0x0, ttl 51, id 20830, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xb9c8 (correct), seq 59666:60942, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882843], length 1276: SMTP, length: 1276
15:16:34.126826 IP (tos 0x0, ttl 64, id 65178, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x2bb0), ack 60942, win 65535, options [nop,nop,TS val 943883293 ecr 3593926879], length 0
15:16:34.127016 IP (tos 0x0, ttl 51, id 6828, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x12ab (correct), seq 60942:62218, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882843], length 1276: SMTP, length: 1276
15:16:34.127033 IP (tos 0x0, ttl 64, id 65179, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x26b3), ack 62218, win 65535, options [nop,nop,TS val 943883294 ecr 3593926879], length 0
15:16:34.127183 IP (tos 0x0, ttl 51, id 41736, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x66e0 (correct), seq 62218:63494, ack 5596, win 48488, options [nop,nop,TS val 3593926879 ecr 943882843], length 1276: SMTP, length: 1276
15:16:34.127183 IP (tos 0x0, ttl 51, id 63330, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0x7560 (correct), seq 63494:64770, ack 5596, win 48488, options [nop,nop,TS val 3593926883 ecr 943882845], length 1276: SMTP, length: 1276
15:16:34.127196 IP (tos 0x0, ttl 64, id 65180, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x21b7), ack 63494, win 65535, options [nop,nop,TS val 943883294 ecr 3593926879], length 0
15:16:34.127219 IP (tos 0x0, ttl 64, id 65181, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x1cb7), ack 64770, win 65535, options [nop,nop,TS val 943883294 ecr 3593926883], length 0
15:16:34.127948 IP (tos 0x0, ttl 51, id 52916, offset 0, flags [none], proto TCP (6), length 1328)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [.], cksum 0xfe32 (correct), seq 64770:66046, ack 5596, win 48488, options [nop,nop,TS val 3593926883 ecr 943882845], length 1276: SMTP, length: 1276
15:16:34.128010 IP (tos 0x0, ttl 64, id 65182, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [.], cksum 0x140c (incorrect -> 0x17ba), ack 66046, win 65535, options [nop,nop,TS val 943883295 ecr 3593926883], length 0
15:21:34.211408 IP (tos 0x0, ttl 64, id 65183, offset 0, flags [DF], proto TCP (6), length 129)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [P.], cksum 0x1459 (incorrect -> 0x6fe6), seq 5596:5673, ack 66046, win 65535, options [nop,nop,TS val 944183378 ecr 3593926883], length 77: SMTP, length: 77
15:21:34.212405 IP (tos 0x0, ttl 64, id 65184, offset 0, flags [DF], proto TCP (6), length 76)
    10.1.3.31.25 > 167.89.95.108.24079: Flags [FP.], cksum 0x1424 (incorrect -> 0x2c96), seq 5673:5697, ack 66046, win 65535, options [nop,nop,TS val 944183379 ecr 3593926883], length 24: SMTP, length: 24
15:21:34.586082 IP (tos 0x0, ttl 51, id 3157, offset 0, flags [none], proto TCP (6), length 40)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [R], cksum 0x8002 (correct), seq 1424713957, win 0, length 0
15:21:34.593224 IP (tos 0x0, ttl 51, id 15635, offset 0, flags [none], proto TCP (6), length 40)
    167.89.95.108.24079 > 10.1.3.31.25: Flags [R], cksum 0x8002 (correct), seq 1424713957, win 0, length 0

There’s nothing that really stands out to me, but I don’t really know what I’m looking for.

level1 · June 6, 2021, 4:05pm

All of your TCP headers seem to be failing their checksums (e.g. cksum 0x140c (incorrect → 0x0051)). On most systems, checksum offloading to the NIC causes this error. Basically your monitoring program (tcpdump, wireshark, whatever) gets the packets from the OS before they are handed over to the NIC and the actual checksums generated and inserted. The tool compares both and because they are different (the packet from the OS has the checksum field set to zero) the check fails. These errors are, therefore, usually red herrings. You can hide them with tcpdump --dont-verify-checksums to avoid (likely) distraction.

Computers send outbound IP packets that contain the DF (Don’t Fragment) flag. That’s part of an automated process known as Path MTU Discovery (PMTUD). Basically your network interface is trying to negotiate the largest MTU possible with all devices on the path to, and including, the remote mail server’s network interface. If any device along the path, or the destination, can’t handle the size of the packet being sent, they drop it and respond with a ICMP Fragmentation Needed packet. If that packet doesn’t get back to the sender then the source network interface will continue to send oversized packets, that keep getting dropped, until the 5-minute timeout is reached and the connection is aborted/closed by the source server.

In a nutshell: If a firewall somewhere along the path blocks ICMP then these Fragmentation Needed packets don’t make it through, PMTUD fails, and the transfer times out. You don’t have control over the majority of these machines, so in many cases this problem is not rectifiable. If you are the destination NIC, then failure is silent and invisible to you. You can make sure that you aren’t causing a problem, but that’s about it (unless you fancy counting packets and sequence numbers looking for ‘missing’ packets).

Long shot, and probably something you’ve already done: Can you check to make sure that you haven’t blocked outbound traffic on port 25 either on your server or your firewall. Also check that your ISP isn’t blocking outbound traffic on port 25 (they sometimes do this automatically if they believe your IP is an open relay).

Dexter_Kane · June 7, 2021, 12:53am

Port 25 is working.

ICMP I think is working (I ran these tests and they appeared to work http://icmpcheck.popcount.org/)

I have found that it isn’t the mail servers which are the problem but the size of the e-mail. I’ve been sending myself an e-mail with a 75kb attachment and that e-mail fails with the same problem.

With some more testing I can send a 43kb attachment but not a 57kb attachment. But I have no idea why.

I can send the attachment from one local account to another on the local network but when doing so externally it fails. So it’s definitely a network problem. The way this is set up is with the mailserver running on a server on my local network, I have an openvpn connection to a VPS which is configured to forward traffic to the server. So the problem could be with the openvpn connection or more likely with the firewall on the VPS.

level1 · June 7, 2021, 3:12am

That tool only verifies that ICMP works from its own server to your server. It doesn’t (and can’t) say anything about the path from Paypal’s server to your server, unless the two have one or more devices in common. Since the end point is under your control, and common, it at least means that is working — which is good news.

Yep, it’s an IP/TCP problem, not specific to mail. You should be able to replicate it in a variety of ways — as long as you can push the data in from the outside world. Not sure if you have an external/3rd-party server you can use, or whether you have a command line option available on your phone (or an otherwise useful app installed) that you could leverage for this.

A long time ago there used to be a case where data streams would fail to transmit if double newlines (CRLF) fell on a segmentation boundary. So it’s not only size, but content as well, that can make weird things happen.

At least you can test at zero cost now — that’s great! If the problem is actually within your domain of control, it’s just a matter of time now before you track it down.

That means the mail server NIC and your router’s LAN NIC are no longer suspects (on the hardware failure front, at least).

Is OpenVPN embedded into your router, or are you running software OpenVPN on your mail server?

Are you able to easily (temporarily) disable OpenVPN, or open a port in your firewall (use port forwarding), to give your VPS direct access to HERMES? Alternatively, can you spin up a VM on your LAN, set up a basic mail server on that, and put it in the DMZ with direct access, then route specific mail traffic to that instead of HERMES? Either would allow you to tell if OpenVPN is causing the problem.

Dexter_Kane · June 7, 2021, 3:19am

I checked the firewall logs on the VPS and there’s nothing unusual there, which is to be expected as I haven’t changed the configuration on the VPS in a very long time.

I will try testing a direct connection to the mail server but for now I’m updating the VPS to see if it’s just an old version of openvpn not being completely compatible with the version in pfsense. I know that pfsense updated earlier this year so possibly that’s part of the problem.

I have other services running through this VPN and none of them have had any issues that I’ve noticed, but they’re mostly sending data not receiving it.

Dexter_Kane · June 7, 2021, 3:47am

Still updating but I did set up port forwards on my local network to test if mail works when connecting directly and it does. So that confirms the problem is with the VPS or VPN. I’ll see what happens when I finish updating.

level1 · June 7, 2021, 4:55am

That test means the problem is now clearly within your realm of control, and thus can certainly be fixed. There’s light at the end of the tunnel, and it isn’t a train. In a few more hours you’ll have this licked.