The WildFly Admin Guide covers the reference for the configuration of audit logging framework, but I thought it would be a good idea to show step by step how to configure WildFly to log to a remote syslog server over TLS. I will also throw client certificate authentication into the mix. I have used rsyslog for my experiments since that comes with nice documentation for how to set it up to use TLS. You need to install the rsyslog-gnutls package for the following to work. I do my main work on OS X but have a linux server with rsyslog installed so I will use that as the syslog server. It's address is kabirlinux. Wherever you see the name kabirlinux you should replace it with the host name of your machine running rsyslog.
Generating the server certificates
First I ssh into kabirlinux and generate the keys we need to set up rsyslog to accept remote connections over TLS (taken from http://www.rsyslog.com/tag/tls/ and http://www.rsyslog.com/doc/rsyslog_tls.html). The first thing we need to do is to generate the ca certificate that will be used for signing the other certificates. Please note that wherever I have used sudo, it is very important that you do so too, otherwise rsyslog might not recognise your keys.$ssh kabirlinux Last login: Mon Aug 12 16:15:28 2013 from 192.168.1.102 [kabirlinux ~] $mkdir tmp [kabirlinux ~] $cd tmp [kabirlinux ~/tmp] $sudo certtool --generate-privkey --outfile ca-key.pem [sudo] password for kabir: Generating a 2560 bit RSA private key...Then we generate the ca certificate using the private key:
[kabirlinux ~] $sudo certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem Generating a self signed certificate... Please enter the details of the certificate's distinguished name. Just press enter to ignore a field. Country name (2 chars): UK Organization name: Red Hat Organizational unit name: JBoss Locality name: London State or province name: London Common name: kabirlinux UID: This field should not be used in new certificates. E-mail: me@redhat.com Enter the certificate's serial number in decimal (default: 1376387226): Activation/Expiration time. The certificate will expire in (days): 3650 Extensions. Does the certificate belong to an authority? (y/N): y Path length constraint (decimal, -1 for no constraint): Is this a TLS web client certificate? (y/N): Will the certificate be used for IPsec IKE operations? (y/N): Is this also a TLS web server certificate? (y/N): Enter the e-mail of the subject of the certificate: Will the certificate be used to sign other certificates? (y/N): y Will the certificate be used to sign CRLs? (y/N): Will the certificate be used to sign code? (y/N): Will the certificate be used to sign OCSP requests? (y/N): Will the certificate be used for time stamping? (y/N): Enter the URI of the CRL distribution point: X.509 Certificate Information: Version: 3 ......... Is the above information ok? (y/N): y Signing certificate...Next we generate a certificate request, and the server certificate that will be used to secure TLS.
[kabirlinux ~/tmp] $sudo certtool --generate-privkey --outfile server-key.pem --bits 2048 ** Note: Please use the --sec-param instead of --bits Generating a 2048 bit RSA private key... [kabirlinux ~/tmp] $sudo certtool --generate-request --load-privkey server-key.pem --outfile request.pem Generating a PKCS #10 certificate request... Country name (2 chars): UK Organization name: Red Hat Organizational unit name: JBoss Locality name: London State or province name: London Common name: kabirlinux UID: Enter a dnsName of the subject of the certificate: kabirlinux Enter a dnsName of the subject of the certificate: Enter the IP address of the subject of the certificate: Enter the e-mail of the subject of the certificate: me@redhat.com Enter a challenge password: Does the certificate belong to an authority? (y/N): Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): Will the certificate be used for encryption (RSA ciphersuites)? (y/N): Is this a TLS web client certificate? (y/N): y Is this also a TLS web server certificate? (y/N): y [kabirlinux ~/tmp] $sudo certtool --generate-certificate --load-request request.pem --outfile server-cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem Generating a signed certificate... Enter the certificate's serial number in decimal (default: 1376387546): Activation/Expiration time. The certificate will expire in (days): 3650 Extensions. Do you want to honour the extensions from the request? (y/N): Does the certificate belong to an authority? (y/N): Is this a TLS web client certificate? (y/N): y Will the certificate be used for IPsec IKE operations? (y/N): Is this also a TLS web server certificate? (y/N): y Enter a dnsName of the subject of the certificate: kabirlinux Enter a dnsName of the subject of the certificate: Enter the IP address of the subject of the certificate: Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): Will the certificate be used for encryption (RSA ciphersuites)? (y/N): X.509 Certificate Information: Version: 3 ..... Is the above information ok? (y/N): y Signing certificate... [kabirlinux ~/tmp] $rm request.pemWe then need to protect the ca private key
[kabirlinux ~/tmp] $sudo chmod 400 ca-key.pemNow move the generated files across to /etc/pki/rsyslog, keeping a copy of the server certificate in the tmp directory:
[kabirlinux ~/tmp] $sudo mv ca-key.pem ca.pem server-cert.pem server-key.pem /etc/pki/rsyslog/ [kabirlinux ~/tmp] $sudo cp /etc/pki/rsyslog/server-cert.pem .Then modify the rsyslog configuration file at /etc/rsyslog.conf to allow TLS. In a vanilla installation this is just after the initial ModLoad directives.
.... #### MODULES #### $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # provides kernel logging support (previously done by rklogd) #$ModLoad immark # provides --MARK-- message capability # Provides TLS syslog reception $ModLoad imtcp #Load tcp driver $DefaultNetstreamDriver gtls # Sets the gtls driver as the default # Certificates $DefaultNetstreamDriverCAFile /etc/pki/rsyslog/ca.pem $DefaultNetstreamDriverCertFile /etc/pki/rsyslog/server-cert.pem $DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/server-key.pem # run driver in TLS-only mode $InputTCPServerStreamDriverMode 1 $InputTCPServerStreamDriverAuthMode anon $InputTCPServerRun 514 # start up listener at port 514 ....Make sure the syslog server's firewall is open to TCP requests on port 514. A simple way to get rid of all rules is to run this when ssh'ed into the syslog server (you will of course want to set this up properly once done experimenting):
[kabirlinux ~] $sudo iptables -FThen restart the rsyslog daemon:
[kabirlinux ~] $sudo service rsyslog restart [sudo] password for kabir: Redirecting to /bin/systemctl restart rsyslog.serviceIn the syslog you should then see messages indicating that the service has been restarted
[kabirlinux ~] $sudo tail -f /var/log/messages ..... Aug 12 16:50:32 kabir systemd[1]: Stopping System Logging Service... Aug 12 16:50:32 kabir systemd[1]: Starting System Logging Service... Aug 12 16:50:32 kabir systemd[1]: Started System Logging Service.
Importing the server certificate into a Java truststore
Since the server certificate is not signed by a trusted certificate authorization authority, in my local machine where I will be running WildFly, I need to import the server certificate into a truststore. First I copy it down to my local machine:[~] $mkdir test [~] $cd test [~/test] $sftp kabirlinux Connected to kabirlinux. sftp> cd tmp sftp> ls server-cert.pem sftp> get server-cert.pem Fetching /home/kabir/tmp/server-cert.pem to server-cert.pem /home/kabir/tmp/server-cert.pem 100% 1529 1.5KB/s 00:00 sftp> exit [~/test]Then I import the certificate into a trustore file using the keytool command, and 'test123' as the password:
[~/test] $keytool -import -trustcacerts -file server-cert.pem -alias CA_ALIAS -keystore truststore Enter keystore password: Re-enter new password: ... Trust this certificate? [no]: yes Certificate was added to keystore
Set up WildFly to use syslog with TLS for audit logging
Make sure you are using WildFly 8.0.0.Alpha4 or later! Out of the box wildfly's audit log configuration in its standalone/configuration/standalone.xml will look like this<audit-log>
<formatters>
<json-formatter name="json-formatter"/>
</formatters>
<handlers>
<file-handler name="file" formatter="json-formatter" relative-to="jboss.server.data.dir" path="audit-log.log"/>
</handlers>
<logger log-boot="true" log-read-only="false" enabled="false">
<handlers>
<handler name="file"/>
</handlers>
</logger>
</audit-log>
Change this to <audit-log>
<formatters>
<json-formatter name="json-formatter"/>
</formatters>
<handlers>
<syslog-handler name="syslog-tls" formatter="json-formatter" syslog-format="RFC5424">
<tls host="kabirlinux" port="514" message-transfer="NON_TRANSPARENT_FRAMING">
<truststore path="/Users/kabir/test/truststore" keystore-password="test123"/>
</tls>
</syslog-handler>
</handlers>
<logger log-boot="true" log-read-only="true" enabled="true">
<handlers>
<handler name="syslog-tls"/>
</handlers>
</logger>
</audit-log>
If having the keystore password hard wired in the xml worries you, please take a look at how to use our vault for sensitive information like that. Now if you stop and start WildFly and look in /var/log/messages on the server running rsyslog, you should see audit logged messages on bootup. Also, when executing operations, for example from the CLI you should see those added to the syslog. Client certificate authentication
Rsyslog supports full blown client certificate authentication, however I have not tried setting that up. However, it also has a slightly more permissive mode which simply checks that the client has a certificate signed by the server's ca. So first we generate the client certificate, I do this on my syslog server:[~/test] $ssh kabirlinux Last login: Mon Aug 12 18:02:25 2013 from 192.168.1.102 [kabirlinux ~] $cd tmp/ [kabirlinux ~/tmp] $certtool --generate-privkey --outfile client-key.pem Generating a 2560 bit RSA private key... [kabirlinux ~/tmp] $certtool --generate-request --load-privkey client-key.pem --outfile client-request.pem Generating a PKCS #10 certificate request... Country name (2 chars): UK Organization name: Red Hat Organizational unit name: JBoss Locality name: London State or province name: London Common name: client UID: Enter a dnsName of the subject of the certificate: kabirlinux Enter a dnsName of the subject of the certificate: Enter the IP address of the subject of the certificate: Enter the e-mail of the subject of the certificate: me@redhat.com Enter a challenge password: Does the certificate belong to an authority? (y/N): N Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): Will the certificate be used for encryption (RSA ciphersuites)? (y/N): Is this a TLS web client certificate? (y/N): y Is this also a TLS web server certificate? (y/N): y [kabirlinux ~/tmp] $sudo certtool --generate-certificate --load-request client-request.pem --outfile client-cert.pem --load-ca-certificate /etc/pki/rsyslog/ca.pem --load-ca-privkey /etc/pki/rsyslog/ca-key.pem [sudo] password for kabir: Generating a signed certificate... Enter the certificate's serial number in decimal (default: 1376327802): Activation/Expiration time. The certificate will expire in (days): 3650 Extensions. Do you want to honour the extensions from the request? (y/N): Does the certificate belong to an authority? (y/N): Is this a TLS web client certificate? (y/N): y Will the certificate be used for IPsec IKE operations? (y/N): Is this also a TLS web server certificate? (y/N): y Enter a dnsName of the subject of the certificate: kabirlinux Enter a dnsName of the subject of the certificate: Enter the IP address of the subject of the certificate: Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): Will the certificate be used for encryption (RSA ciphersuites)? (y/N): X.509 Certificate Information: ... Is the above information ok? (y/N): yWe then need to combine the client certificate and key to pkcs12 format to copy across to the client, using 'test123' as the export password
[kabirlinux ~/tmp] $openssl pkcs12 -export -in client-cert.pem -inkey client-key.pem -out client.p12 -name "Whatever" Enter Export Password: Verifying - Enter Export Password: [kabirlinux ~/tmp] $rm -rf client*.pemNow on my Mac I get the client.p12 file and import it into a jks keystore
[~/test] $sftp kabirlinux Connected to kabirlinux. sftp> cd tmp sftp> get client.p12 Fetching /home/kabir/tmp/client.p12 to client.p12 /home/kabir/tmp/client.p12 100% 3094 3.0KB/s 00:00 sftp> exit [~/test] $keytool -v -importkeystore -srckeystore client.p12 -srcstoretype PKCS12 -destkeystore client.jks -deststoretype JKS Enter destination keystore password: Re-enter new password: Enter source keystore password: Entry for alias whatever successfully imported. Import command completed: 1 entries successfully imported, 0 entries failed or cancelled [Storing client.jks]Again, I use 'test123' as the keystore password. Then on the rsyslog server we change /etc/rsyslog.conf to use 'x509/certvalid' instead of 'anon' for the InputTCPServerStreamDriverAuthMode directive, e.g:
$InputTCPServerStreamDriverAuthMode x509/certvalidNext we add the client-certificate-store information to the syslog handler in the audit logging section of standalone/configuration/standalone.xml in the WildFly installation so it looks like:
<audit-log>
<formatters>
<json-formatter name="json-formatter"/>
</formatters>
<handlers>
<syslog-handler name="syslog-tls" formatter="json-formatter" syslog-format="RFC5424">
<tls host="kabirlinux" port="514" message-transfer="NON_TRANSPARENT_FRAMING">
<truststore path="/Users/kabir/test/truststore" keystore-password="test123"/>
<client-certificate-store path="/Users/kabir/test/client.jks" keystore-password="test123"/>
</tls>
</syslog-handler>
</handlers>
<logger log-boot="true" log-read-only="true" enabled="true">
<handlers>
<handler name="syslog-tls"/>
</handlers>
</logger>
</audit-log>
Now if you restart the rsyslog daemon and WildFly you should see the log messages come through again. So this means that you can host rsyslog on a machine not accessible by the people who have access to your WildFly installation. Communication between the WildFly installation and rsyslog is encrypted since it goes over TLS, and only users who have the correct certificate are able to send data to rsyslog.
No comments:
Post a Comment