Tuesday, August 13, 2013

Setting up Secure Audit Logging to a Remote Rsyslog Instance in WildFly 8.0.0.Alpha4

WildFly is the new name for JBoss Application Server. It builds on the same architecture as JBoss Application Server 7. WildFly is the upstream for JBoss Enterprise Application Platform 6, which is our supported middleware offering at Red Hat. Some of our larger customers have asked for additional security features when it comes to managing their installations, and one of the pieces for this is to implement audit logging of the management layer. The initial implementation of audit logging will be part of the WildFly 8.0.0.Alpha4 release, which was released today.

Note that this is a pretty advanced topic, for a simpler overview of audit logging see my follow-up post.

The WildFly Admin Guide covers the reference for the configuration of audit logging framework, but I thought it would be a good idea to show step by step how to configure WildFly to log to a remote syslog server over TLS. I will also throw client certificate authentication into the mix. I have used rsyslog for my experiments since that comes with nice documentation for how to set it up to use TLS. You need to install the rsyslog-gnutls package for the following to work.

I do my main work on OS X but have a linux server with rsyslog installed so I will use that as the syslog server. It's address is kabirlinux. Wherever you see the name kabirlinux you should replace it with the host name of your machine running rsyslog.

Generating the server certificates

First I ssh into kabirlinux and generate the keys we need to set up rsyslog to accept remote connections over TLS (taken from http://www.rsyslog.com/tag/tls/ and http://www.rsyslog.com/doc/rsyslog_tls.html). The first thing we need to do is to generate the ca certificate that will be used for signing the other certificates. Please note that wherever I have used sudo, it is very important that you do so too, otherwise rsyslog might not recognise your keys.
$ssh kabirlinux
Last login: Mon Aug 12 16:15:28 2013 from 192.168.1.102

[kabirlinux ~]
$mkdir tmp

[kabirlinux ~]
$cd tmp

[kabirlinux ~/tmp]
$sudo certtool --generate-privkey --outfile ca-key.pem
[sudo] password for kabir: 
Generating a 2560 bit RSA private key...
Then we generate the ca certificate using the private key:
[kabirlinux ~]
$sudo certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem
Generating a self signed certificate...
Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
Country name (2 chars): UK
Organization name: Red Hat
Organizational unit name: JBoss
Locality name: London
State or province name: London
Common name: kabirlinux
UID: 
This field should not be used in new certificates.
E-mail: me@redhat.com
Enter the certificate's serial number in decimal (default: 1376387226): 


Activation/Expiration time.
The certificate will expire in (days): 3650


Extensions.
Does the certificate belong to an authority? (y/N): y
Path length constraint (decimal, -1 for no constraint): 
Is this a TLS web client certificate? (y/N): 
Will the certificate be used for IPsec IKE operations? (y/N): 
Is this also a TLS web server certificate? (y/N): 
Enter the e-mail of the subject of the certificate:    
Will the certificate be used to sign other certificates? (y/N): y
Will the certificate be used to sign CRLs? (y/N): 
Will the certificate be used to sign code? (y/N): 
Will the certificate be used to sign OCSP requests? (y/N): 
Will the certificate be used for time stamping? (y/N): 
Enter the URI of the CRL distribution point: 
X.509 Certificate Information:
 Version: 3
.........

Is the above information ok? (y/N): y


Signing certificate...
Next we generate a certificate request, and the server certificate that will be used to secure TLS.
[kabirlinux ~/tmp]
$sudo certtool  --generate-privkey --outfile server-key.pem --bits 2048
** Note: Please use the --sec-param instead of --bits
Generating a 2048 bit RSA private key...

[kabirlinux ~/tmp]
$sudo certtool --generate-request --load-privkey server-key.pem --outfile request.pem
Generating a PKCS #10 certificate request...
Country name (2 chars): UK
Organization name: Red Hat
Organizational unit name: JBoss
Locality name: London
State or province name: London
Common name: kabirlinux
UID: 
Enter a dnsName of the subject of the certificate: kabirlinux
Enter a dnsName of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Enter the e-mail of the subject of the certificate: me@redhat.com
Enter a challenge password: 
Does the certificate belong to an authority? (y/N): 
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): 
Will the certificate be used for encryption (RSA ciphersuites)? (y/N): 
Is this a TLS web client certificate? (y/N): y
Is this also a TLS web server certificate? (y/N): y

[kabirlinux ~/tmp]
$sudo certtool --generate-certificate --load-request request.pem --outfile server-cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem
Generating a signed certificate...
Enter the certificate's serial number in decimal (default: 1376387546): 


Activation/Expiration time.
The certificate will expire in (days): 3650


Extensions.
Do you want to honour the extensions from the request? (y/N): 
Does the certificate belong to an authority? (y/N): 
Is this a TLS web client certificate? (y/N): y
Will the certificate be used for IPsec IKE operations? (y/N): 
Is this also a TLS web server certificate? (y/N): y
Enter a dnsName of the subject of the certificate: kabirlinux
Enter a dnsName of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): 
Will the certificate be used for encryption (RSA ciphersuites)? (y/N): 
X.509 Certificate Information:
 Version: 3

.....

Is the above information ok? (y/N): y


Signing certificate...

[kabirlinux ~/tmp]
$rm request.pem 
We then need to protect the ca private key
[kabirlinux ~/tmp]
$sudo chmod 400 ca-key.pem
Now move the generated files across to /etc/pki/rsyslog, keeping a copy of the server certificate in the tmp directory: 
[kabirlinux ~/tmp]
$sudo mv ca-key.pem ca.pem server-cert.pem server-key.pem /etc/pki/rsyslog/

[kabirlinux ~/tmp]
$sudo cp /etc/pki/rsyslog/server-cert.pem .
Then modify the rsyslog configuration file at /etc/rsyslog.conf to allow TLS. In a vanilla installation this is just after the initial ModLoad directives.
....
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides TLS syslog reception
$ModLoad imtcp #Load tcp driver
$DefaultNetstreamDriver gtls # Sets the gtls driver as the default
# Certificates
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/ca.pem
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/server-cert.pem
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/server-key.pem
# run driver in TLS-only mode
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon 
$InputTCPServerRun 514 # start up listener at port 514
....
Make sure the syslog server's firewall is open to TCP requests on port 514. A simple way to get rid of all rules is to run this when ssh'ed into the syslog server (you will of course want to set this up properly once done experimenting):
[kabirlinux ~]
$sudo iptables -F
Then restart the rsyslog daemon:
[kabirlinux ~]
$sudo service rsyslog restart
[sudo] password for kabir: 
Redirecting to /bin/systemctl restart  rsyslog.service
In the syslog you should then see messages indicating that the service has been restarted
[kabirlinux ~]
$sudo tail -f /var/log/messages
.....
Aug 12 16:50:32 kabir systemd[1]: Stopping System Logging Service...
Aug 12 16:50:32 kabir systemd[1]: Starting System Logging Service...
Aug 12 16:50:32 kabir systemd[1]: Started System Logging Service.

Importing the server certificate into a Java truststore

Since the server certificate is not signed by a trusted certificate authorization authority, in my local machine where I will be running WildFly, I need to import the server certificate into a truststore. First I copy it down to my local machine:
[~]
$mkdir test

[~]
$cd test

[~/test]
$sftp kabirlinux
Connected to kabirlinux.
sftp> cd tmp
sftp> ls
server-cert.pem
sftp> get server-cert.pem
Fetching /home/kabir/tmp/server-cert.pem to server-cert.pem
/home/kabir/tmp/server-cert.pem                                                    100% 1529     1.5KB/s   00:00    
sftp> exit

[~/test]
Then I import the certificate into a trustore file using the keytool command, and 'test123' as the password:
[~/test]
$keytool -import -trustcacerts -file server-cert.pem -alias CA_ALIAS -keystore truststore
Enter keystore password:  
Re-enter new password: 
...

Trust this certificate? [no]:  yes
Certificate was added to keystore

Set up WildFly to use syslog with TLS for audit logging

Make sure you are using WildFly 8.0.0.Alpha4 or later! Out of the box wildfly's audit log configuration in its standalone/configuration/standalone.xml will look like this
<audit-log>
            <formatters>
                <json-formatter name="json-formatter"/>
            </formatters>
            <handlers>
                <file-handler name="file" formatter="json-formatter" relative-to="jboss.server.data.dir" path="audit-log.log"/>
            </handlers>
            <logger log-boot="true" log-read-only="false" enabled="false">
                <handlers>
                    <handler name="file"/>
                </handlers>
            </logger>
        </audit-log>
Change this to
<audit-log>
            <formatters>
                <json-formatter name="json-formatter"/>
            </formatters>
            <handlers>
                <syslog-handler name="syslog-tls" formatter="json-formatter" syslog-format="RFC5424">
                    <tls host="kabirlinux" port="514" message-transfer="NON_TRANSPARENT_FRAMING">
                        <truststore path="/Users/kabir/test/truststore" keystore-password="test123"/>
                    </tls>
                </syslog-handler>
            </handlers>
            <logger log-boot="true" log-read-only="true" enabled="true">
                <handlers>
                    <handler name="syslog-tls"/>
                </handlers>
            </logger>
        </audit-log>
If having the keystore password hard wired in the xml worries you, please take a look at how to use our vault for sensitive information like that. Now if you stop and start WildFly and look in /var/log/messages on the server running rsyslog, you should see audit logged messages on bootup. Also, when executing operations, for example from the CLI you should see those added to the syslog.

Client certificate authentication

Rsyslog supports full blown client certificate authentication, however I have not tried setting that up. However, it also has a slightly more permissive mode which simply checks that the client has a certificate signed by the server's ca. So first we generate the client certificate, I do this on my syslog server:
[~/test]
$ssh kabirlinux
Last login: Mon Aug 12 18:02:25 2013 from 192.168.1.102

[kabirlinux ~]
$cd tmp/

[kabirlinux ~/tmp]
$certtool --generate-privkey --outfile client-key.pem
Generating a 2560 bit RSA private key...

[kabirlinux ~/tmp]
$certtool --generate-request --load-privkey client-key.pem --outfile client-request.pem
Generating a PKCS #10 certificate request...
Country name (2 chars): UK
Organization name: Red Hat
Organizational unit name: JBoss
Locality name: London
State or province name: London
Common name: client    
UID: 
Enter a dnsName of the subject of the certificate: kabirlinux
Enter a dnsName of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Enter the e-mail of the subject of the certificate: me@redhat.com
Enter a challenge password: 
Does the certificate belong to an authority? (y/N): N
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): 
Will the certificate be used for encryption (RSA ciphersuites)? (y/N): 
Is this a TLS web client certificate? (y/N): y
Is this also a TLS web server certificate? (y/N): y

[kabirlinux ~/tmp]
$sudo certtool  --generate-certificate --load-request client-request.pem --outfile client-cert.pem  --load-ca-certificate /etc/pki/rsyslog/ca.pem --load-ca-privkey /etc/pki/rsyslog/ca-key.pem
[sudo] password for kabir: 
Generating a signed certificate...
Enter the certificate's serial number in decimal (default: 1376327802): 


Activation/Expiration time.
The certificate will expire in (days): 3650


Extensions.
Do you want to honour the extensions from the request? (y/N): 
Does the certificate belong to an authority? (y/N): 
Is this a TLS web client certificate? (y/N): y
Will the certificate be used for IPsec IKE operations? (y/N): 
Is this also a TLS web server certificate? (y/N): y
Enter a dnsName of the subject of the certificate: kabirlinux
Enter a dnsName of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): 
Will the certificate be used for encryption (RSA ciphersuites)? (y/N): 
X.509 Certificate Information:
...

Is the above information ok? (y/N): y
We then need to combine the client certificate and key to pkcs12 format to copy across to the client, using 'test123' as the export password
[kabirlinux ~/tmp]
$openssl pkcs12  -export -in client-cert.pem -inkey client-key.pem -out client.p12 -name "Whatever"
Enter Export Password:
Verifying - Enter Export Password:

[kabirlinux ~/tmp]
$rm -rf client*.pem
Now on my Mac I get the client.p12 file and import it into a jks keystore
[~/test]
$sftp kabirlinux
Connected to kabirlinux.
sftp> cd tmp 
sftp> get client.p12
Fetching /home/kabir/tmp/client.p12 to client.p12
/home/kabir/tmp/client.p12                                                                                                                                               100% 3094     3.0KB/s   00:00    
sftp> exit

[~/test]
$keytool -v -importkeystore -srckeystore client.p12 -srcstoretype PKCS12 -destkeystore client.jks -deststoretype JKS
Enter destination keystore password:  
Re-enter new password: 
Enter source keystore password:  
Entry for alias whatever successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
[Storing client.jks]
Again, I use 'test123' as the keystore password. Then on the rsyslog server we change /etc/rsyslog.conf to use 'x509/certvalid' instead of 'anon' for the InputTCPServerStreamDriverAuthMode directive, e.g:
$InputTCPServerStreamDriverAuthMode x509/certvalid
Next we add the client-certificate-store information to the syslog handler in the audit logging section of standalone/configuration/standalone.xml in the WildFly installation so it looks like:
<audit-log>
            <formatters>
                <json-formatter name="json-formatter"/>
            </formatters>
            <handlers>
                <syslog-handler name="syslog-tls" formatter="json-formatter" syslog-format="RFC5424">
                    <tls host="kabirlinux" port="514" message-transfer="NON_TRANSPARENT_FRAMING">
                        <truststore path="/Users/kabir/test/truststore" keystore-password="test123"/>
                        <client-certificate-store path="/Users/kabir/test/client.jks" keystore-password="test123"/>
                    </tls>
                </syslog-handler>
            </handlers>
            <logger log-boot="true" log-read-only="true" enabled="true">
                <handlers>
                    <handler name="syslog-tls"/>
                </handlers>
            </logger>
        </audit-log>
Now if you restart the rsyslog daemon and WildFly you should see the log messages come through again. So this means that you can host rsyslog on a machine not accessible by the people who have access to your WildFly installation. Communication between the WildFly installation and rsyslog is encrypted since it goes over TLS, and only users who have the correct certificate are able to send data to rsyslog.

No comments: