Unable to renew FreeIPA Certificates, Unable to start pki-tomcated service

Hi All

We are using FreeIPA 4.5 on a CentOS Server. FreeIPA certificates expired in September’19 and they did not get auto renewed. We are unable to renew them.

Below is the output of “getcert list” command

Number of certificates and requests being tracked: 10.
Request ID ‘20170929061357’:
status: MONITORING
stuck: no
key pair storage: type=FILE,location=’/var/lib/ipa/ra-agent.key’
certificate: type=FILE,location=’/var/lib/ipa/ra-agent.pem’
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=BEJN-IDMS.LOCAL
subject: CN=IPA RA,O=BEJN-IDMS.LOCAL
expires: 2019-09-19 06:13:58 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID ‘20170929061419’:
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location=’/etc/pki/pki-tomcat/alias’,nickname=‘auditSigningCert cert-pki-ca’,token=‘NSS Certificate DB’,pin set
certificate: type=NSSDB,location=’/etc/pki/pki-tomcat/alias’,nickname=‘auditSigningCert cert-pki-ca’,token=‘NSS Certificate DB’
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=BEJN-IDMS.LOCAL
subject: CN=CA Audit,O=BEJN-IDMS.LOCAL
expires: 2019-09-19 06:13:47 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert “auditSigningCert cert-pki-ca”
track: yes
auto-renew: yes
Request ID ‘20170929061420’:
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location=’/etc/pki/pki-tomcat/alias’,nickname=‘ocspSigningCert cert-pki-ca’,token=‘NSS Certificate DB’,pin set
certificate: type=NSSDB,location=’/etc/pki/pki-tomcat/alias’,nickname=‘ocspSigningCert cert-pki-ca’,token=‘NSS Certificate DB’
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=BEJN-IDMS.LOCAL
subject: CN=OCSP Subsystem,O=BEJN-IDMS.LOCAL
expires: 2019-09-19 06:13:46 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert “ocspSigningCert cert-pki-ca”
track: yes
auto-renew: yes
Request ID ‘20170929061421’:
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location=’/etc/pki/pki-tomcat/alias’,nickname=‘subsystemCert cert-pki-ca’,token=‘NSS Certificate DB’,pin set
certificate: type=NSSDB,location=’/etc/pki/pki-tomcat/alias’,nickname=‘subsystemCert cert-pki-ca’,token=‘NSS Certificate DB’
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=BEJN-IDMS.LOCAL
subject: CN=CA Subsystem,O=BEJN-IDMS.LOCAL
expires: 2019-09-19 06:13:46 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert “subsystemCert cert-pki-ca”
track: yes
auto-renew: yes
Request ID ‘20170929061423’:
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location=’/etc/pki/pki-tomcat/alias’,nickname=‘Server-Cert cert-pki-ca’,token=‘NSS Certificate DB’,pin set
certificate: type=NSSDB,location=’/etc/pki/pki-tomcat/alias’,nickname=‘Server-Cert cert-pki-ca’,token=‘NSS Certificate DB’
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=BEJN-IDMS.LOCAL
subject: CN=ldap001.bejn-idms.local,O=BEJN-IDMS.LOCAL
expires: 2019-09-19 06:13:46 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert “Server-Cert cert-pki-ca”
track: yes
auto-renew: yes
Request ID ‘20170929061436’:
status: CA_UNREACHABLE
ca-error: Server at https://ldap001.bejn-idms.local/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer’s Certificate has expired.).
stuck: no
key pair storage: type=NSSDB,location=’/etc/dirsrv/slapd-BEJN-IDMS-LOCAL’,nickname=‘Server-Cert’,token=‘NSS Certificate DB’,pinfile=’/etc/dirsrv/slapd-BEJN-IDMS-LOCAL/pwdfile.txt’
certificate: type=NSSDB,location=’/etc/dirsrv/slapd-BEJN-IDMS-LOCAL’,nickname=‘Server-Cert’,token=‘NSS Certificate DB’
CA: IPA
issuer: CN=Certificate Authority,O=BEJN-IDMS.LOCAL
subject: CN=ldap001.bejn-idms.local,O=BEJN-IDMS.LOCAL
expires: 2019-09-30 06:14:36 UTC
dns: ldap001.bejn-idms.local
principal name: ldap/[email protected]
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv BEJN-IDMS-LOCAL
track: yes
auto-renew: yes
Request ID ‘20170929061454’:
status: CA_UNREACHABLE
ca-error: Server at https://ldap001.bejn-idms.local/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer’s Certificate has expired.).
stuck: no
key pair storage: type=NSSDB,location=’/etc/httpd/alias’,nickname=‘Server-Cert’,token=‘NSS Certificate DB’,pinfile=’/etc/httpd/alias/pwdfile.txt’
certificate: type=NSSDB,location=’/etc/httpd/alias’,nickname=‘Server-Cert’,token=‘NSS Certificate DB’
CA: IPA
issuer: CN=Certificate Authority,O=BEJN-IDMS.LOCAL
subject: CN=ldap001.bejn-idms.local,O=BEJN-IDMS.LOCAL
expires: 2019-09-30 06:14:55 UTC
dns: ldap001.bejn-idms.local
principal name: HTTP/[email protected]
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID ‘20170929061502’:
status: NEED_TO_SUBMIT
ca-error: Server at https://ldap001.bejn-idms.local/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer’s Certificate has expired.).
stuck: no
key pair storage: type=FILE,location=’/var/kerberos/krb5kdc/kdc.key’
certificate: type=FILE,location=’/var/kerberos/krb5kdc/kdc.crt’
CA: IPA
issuer: CN=Certificate Authority,O=BEJN-IDMS.LOCAL
subject: CN=ldap001.bejn-idms.local,O=BEJN-IDMS.LOCAL
expires: 2019-09-30 06:15:02 UTC
principal name: krbtgt/[email protected]
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes

pki-tomcatd service is also not starting and getting the error in /var/log/pki/pki-tomcat/ca/debug as:

Internal Database Error encountered: Could not connect to LDAP server host ldap001.bejn-idms.local port 636 Error netscape.ldap.LDAPException: Unable to create socket: org.mozilla.jss.ssl.SSLSocketException: org.mozilla.jss.ssl.SSLSocketException: SSL_ForceHandshake failed: (-8181) Peer’s Certificate has expired.

We tried running “ipa-cacert-manage renew” renew command but that did not work.

Tried updating the certificates with an external certificate. That didn’t work either
[root@ldap001 ~]# ipa-server-certinstall -w -d /home/akanade/star.idmission.com.2019.key /home/akanade/star.idmission.com.2019.crt
Directory Manager password:

Enter private key unlock password:

The full certificate chain is not present in /home/akanade/star.idmission.com.2019.key, /home/akanade/star.idmission.com.2019.crt
The ipa-server-certinstall command failed.
[root@ldap001 ~]#

Tried upgrading FreeIPA with “yum update freeipa-server”.
When we try to start the server, it fails while upgrading the server.

[root@ldap001 ~]# ipactl start
IPA version error: data needs to be upgraded (expected version ‘4.6.5-11.el7.centos.3’, current version ‘4.5.0-21.el7.centos.1.2’)
Automatically running upgrade, for details see /var/log/ipaupgrade.log
Be patient, this may take a few minutes.
Automatic upgrade failed: A certificate with the nickname ‘ipaCert’ exists in the old ‘/etc/httpd/alias’ NSS database as well as in the new PEM file ‘/var/lib/ipa/ra-agent.pem’
Upgrade failed with unknown error exporting pkcs#12 file /etc/httpd/alias/tmpHTwXqq
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
(‘IPA upgrade failed.’, 1)
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

See the upgrade log for more details and/or run /usr/sbin/ipa-server-upgrade again
Aborting ipactl

Regards
Chaitanya