Return to Level1Techs.com

I've Broken My Postfix Server

I had a perfectly working Postfix server for a long time and made the mistake of changing something without backing up my main.cf first. I’ve occasionally received emails that spoofed my own domain, so I attempted to follow the instructions here to prevent it.

Because I had followed the guide on flurdy.com, SASL authenticated submission on 587 was already enabled, so I made no changes to master.cf.

After setting up check_sender_access as described above, I could no longer receive emails from outside domains. I attempted to undo everything, but it’s still broken, and I’ve almost certainly made things worse trying to fix it. I even went as far as starting fresh on another machine and going through the guide again, but eventually reverted back to my old server.

smtp/smtpd connections to/from external servers still show up in my log, but I just get timeouts and SSL/TLS errors.

Oct 6 11:28:23 ns1 postfix/smtp[8191]: SSL_connect error to mta6.am0.yahoodns.net[98.136.96.91]:25: Connection timed out Oct 6 11:28:23 ns1 postfix/smtp[8191]: 30E9C65FD2: Cannot start TLS: handshake failure

Oct 6 11:38:46 ns1 postfix/smtpd[8214]: SSL_accept error from mail-lj1-f173.google.com[209.85.208.173]: Connection timed out Oct 6 11:38:46 ns1 postfix/smtpd[8214]: lost connection after STARTTLS from mail-lj1-f173.google.com[209.85.208.173]

I am, however, still able to send/receive email between my two domains hosted on this server. I’m at my wit’s end. Please help!

edit - configs removed

What happens if you use Python to manually connect to your sever using the smtp library.

Or, can you try to manually connect using openssl?

In other words if you manually connect to the smtp server and issue the starttls command do you have issues?

If you need help doing this let me know I can provide some links for how to do this.

Also, did you make any firewall changes or modify the cert file in anyway?

Two last things: the link doesn’t seems to show configs. Please help me find those and it looks like 465 is open on your machine. Does that port work still?

I’m able to send and receive between my own domains, so STARTTLS is working fine there. Below is testing using openssl from another machine. My main.cf and master.cf are in the first post. Which other configs did you need?

edit - log removed

Shouldn’t the aliases database be aliases.db file?

Aliases were configured per the this guide, and have worked fine.

https://flurdy.com/docs/postfix/index.html

I wonder if the last rule (defer_unauth_destination) may be getting hit here.

Have you defined anywhere:

$mydestination, $inet_interfaces, $proxy_interfaces, $virtual_alias_domains, or $virtual_mailbox_domains,

Here is a link to postfix documentation for that section:
http://www.postfix.org/postconf.5.html#reject_unauth_destination

I tried temporarily removing defer_unauth_destination just to see if it made a difference. Sent myself another test message from gmail and it still showed as connecting and then timed out.

mydestination is blank in main.cf because I’m using virtual domains.
inet_interfaces = all to accept connections from anywhere
proxy_interfaces hasn’t ever been in my config and the default is empty.
virtual_alias_domains is defined within virtual_alias_maps

(“The default value is $virtual_alias_maps so that you can keep all information about virtual alias domains in one place.”)

virtual_alias_maps = mysql:/etc/postfix/mysql_alias.cf

user=mail
password=xxxxxxxxxxxxx
dbname=maildb
table=aliases
select_field=destination
where_field=mail
hosts=127.0.0.1
additional_conditions = and enabled = 1

virtual_mailbox_domains = mysql:/etc/postfix/mysql_domains.cf

user=mail
password=xxxxxxxxxxxxx
dbname=maildb
table=domains
select_field=domain
where_field=domain
hosts=127.0.0.1
additional_conditions = and enabled = 1

It appears that mail destined for external domains is being routed locally. Also, it appears that my DKIM configuration is invalid.

edit: log removed

This is going to sound weird, but maybe google can’t use high ciphers?

Might be helpful: https://serverfault.com/questions/927877/postfix-2-6-6-with-tls-unable-to-receive-emails-from-gmail-and-a-couple-of-ot

I turned up smtpd_tls_loglevel to 2 and noted the following:

Oct  7 22:22:06 ns1 postfix/smtpd[5401]: setting up TLS connection from mail-vs1-f74.google.com[209.85.217.74]
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: mail-vs1-f74.google.com[209.85.217.74]: TLS cipher list "aNULL:-aNULL:HIGH:@STRENGTH"
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:before SSL initialization
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:before SSL initialization
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:SSLv3/TLS read client hello
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:SSLv3/TLS write server hello
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:SSLv3/TLS write change cipher spec
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:TLSv1.3 write encrypted extensions
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:SSLv3/TLS write certificate
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:TLSv1.3 write server certificate verify
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:SSLv3/TLS write finished
Oct  7 22:22:06 ns1 postfix/smtpd[5401]: SSL_accept:TLSv1.3 early data
...
Oct  7 22:27:06 ns1 postfix/smtpd[5401]: SSL_accept error from mail-vs1-f74.google.com[209.85.217.74]: Connection timed out
Oct  7 22:27:06 ns1 postfix/smtpd[5401]: lost connection after STARTTLS from mail-vs1-f74.google.com[209.85.217.74]
Oct  7 22:27:06 ns1 postfix/smtpd[5401]: disconnect from mail-vs1-f74.google.com[209.85.217.74] ehlo=1 starttls=0/1 commands=1/2
...
Oct  7 22:46:46 ns1 postfix/smtp[5785]: initializing the client-side TLS engine
Oct  7 22:47:16 ns1 postfix/smtp[5785]: connect to mta7.am0.yahoodns.net[67.195.228.110]:25: Connection timed out
Oct  7 22:47:46 ns1 postfix/smtp[5785]: connect to mta5.am0.yahoodns.net[98.136.96.91]:25: Connection timed out
Oct  7 22:48:16 ns1 postfix/smtp[5785]: connect to mta6.am0.yahoodns.net[67.195.204.79]:25: Connection timed out
Oct  7 22:53:17 ns1 postfix/smtp[5785]: A50486279F: conversation with mta5.am0.yahoodns.net[67.195.228.111] timed out while receiving the STARTTLS response
Oct  7 22:53:17 ns1 postfix/smtp[5785]: setting up TLS connection to mta7.am0.yahoodns.net[98.136.96.76]:25
Oct  7 22:53:17 ns1 postfix/smtp[5785]: mta7.am0.yahoodns.net[98.136.96.76]:25: TLS cipher list "aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH"
Oct  7 22:53:17 ns1 postfix/smtp[5785]: SSL_connect:before SSL initialization
Oct  7 22:53:17 ns1 postfix/smtp[5785]: SSL_connect:SSLv3/TLS write client hello

See end of this post about specifying tls_ciphers. https://lists.andrew.cmu.edu/pipermail/info-cyrus/2016-November/039274.html

Not saying to copy/paste that, but it looks like not specifying those may cause this type of behavior.

Explicitly defined tls_high_cipherlist. Same timeouts.

Oct  9 18:00:57 ns1 postfix/smtp[45379]: setting up TLS connection to alt1.gmail-smtp-in.l.google.com[209.85.145.26]:25
Oct  9 18:00:57 ns1 postfix/smtp[45379]: alt1.gmail-smtp-in.l.google.com[209.85.145.26]:25: TLS cipher list "aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH"
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:before SSL initialization
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSLv3/TLS write client hello
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSLv3/TLS write client hello
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSLv3/TLS read server hello
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:TLSv1.3 read encrypted extensions
Oct  9 18:00:57 ns1 postfix/smtp[45379]: alt1.gmail-smtp-in.l.google.com[209.85.145.26]:25: depth=2 verify=1 subject=/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign
Oct  9 18:00:57 ns1 postfix/smtp[45379]: alt1.gmail-smtp-in.l.google.com[209.85.145.26]:25: depth=1 verify=1 subject=/C=US/O=Google Trust Services/CN=GTS CA 1O1
Oct  9 18:00:57 ns1 postfix/smtp[45379]: alt1.gmail-smtp-in.l.google.com[209.85.145.26]:25: depth=0 verify=1 subject=/C=US/ST=California/L=Mountain View/O=Google LLC/CN=mx.google.com
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSLv3/TLS read server certificate
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:TLSv1.3 read server certificate verify
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSLv3/TLS read finished
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSLv3/TLS write change cipher spec
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSLv3/TLS write finished
Oct  9 18:00:57 ns1 postfix/smtp[45379]: alt1.gmail-smtp-in.l.google.com[209.85.145.26]:25: subject_CN=mx.google.com, issuer_CN=GTS CA 1O1, fingerprint=F7:36:85:AB:0E:E5:31:5C:8A:01:DA:E2:DA:3E:AA:A8, pkey_fingerprint=F7:2D:CA:1E:7D:88:F6:30:32:CC:E1:57:D9:09:DB:64
Oct  9 18:00:57 ns1 postfix/smtp[45379]: Trusted TLS connection established to alt1.gmail-smtp-in.l.google.com[209.85.145.26]:25: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSL negotiation finished successfully
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSL negotiation finished successfully
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSLv3/TLS read server session ticket
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSL negotiation finished successfully
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSL negotiation finished successfully
Oct  9 18:00:57 ns1 postfix/smtp[45379]: SSL_connect:SSLv3/TLS read server session ticket
...
Oct  9 18:05:57 ns1 postfix/smtp[45379]: 3CB8062DCE: conversation with alt1.gmail-smtp-in.l.google.com[209.85.145.26] timed out while sending RCPT TO
1 Like

I did a completely fresh setup on a different machine and updated my firewall rules to point to the new server. Same result. I’m at the point where I now think AT&T just randomly blocked outbound traffic on port 25 for my connection. I’ve talked to a few clueless tier 1 and 2 support people and finally got someone who seems to be trying to escalate the issue to even verify that that’s what is happening. At first, he tried to tell me that “We don’t unblock port 25 because of spam.” I told him that was unacceptable and particularly frustrating because I received no notice of any potential change, and I’ve spent an enormous amount of time troubleshooting the issue on my end. I should be getting a call back soon. Fingers crossed. I sure as shit don’t want to have to resort to relaying my smtp traffic through AT&T’s mail server. Fuck that noise.

By the way, huge thanks to @cotton and @nx2l for even attempting to help me sort this out. I honestly expected this thread to sink like a stone.

2 Likes

Update: I looked at the Suricata logs on my firewall and realized that right around the time I stopped receiving email, there started to be a flood of ‘1:2260002 SURICATA Applayer Detect protocol only one direction’ alerts from my mail server on port 25 which resulted in whatever external mail server it was communicating with at the time to be blocked. I suppressed the rule and cleared the block list, and I’m receiving mail again. Still, it’s disconcerting that this started happening all of a sudden.

2 Likes