Wednesday, August 14, 2013

Simpler Audit Logging Example for WildFly 8.0.0.Alpha4

Tomaz on the WildFly team pointed out that my previous blog post on secure audit logging in WildFly 8.0.0.Alpha4 was a bit too scary :-) So I'd like to point out that that was probably the most complicated setup possible, and to walk through a few simpler examples here while explaining a bit more what goes on. These features are available in WildFly 8.0.0.Alpha4 and later. You can get it from here.

WildFly out of the box comes preconfigured for audit logging to $WILDFLY_HOME/standalone/data/audit-log.log., the only thing you need to do is to enable it. There is a section in $WILDFLY_HOME/standalone/configuration/standalone.xml which contains the configuration for the audit log:
        <audit-log>
            <formatters>
                <json-formatter name="json-formatter">
            </json-formatter></formatters>
            <handlers>
                <file-handler formatter="json-formatter" name="file" path="audit-log.log" relative-to="jboss.server.data.dir">
                </file-handler>
            </handlers>
            <logger enabled="false" log-boot="true" log-read-only="false">
                <handlers>
                    <handler name="file">
                </handler></handlers>
            </logger>
        </audit-log>
This config contains the only audit log formatter we have at the moment, the JSON formatter which formats the audit log records.
It also contains a file-handler called file which points to the file which it should log to.
The file file-handler references the 'json-formatter' which means this handler uses 'json-formatter' to format its log records.
Finally the logger entry references the 'file' handler which means that the 'file' handler will be used for logging audit log records.
The attributes of the logger element mean that we will audit log the operation that happen on boot, that we will not log read-only operations, and that audit logging is not enabled. The not enabled part takes precedence, so we don't log anything at all.

Now if you start up WildFly by running (of course change ~/Downloads/wildfly-8.0.0.Alpha4 to wherever you have installed WildFly)
$cd ~/Downloads/wildfly-8.0.0.Alpha4/bin/
$./standalone.sh
you will see no $WILDFLY_HOME/standalone/data/audit-log.log. file. This is because we need to enable audit logging.



Enable audit logging

In another terminal window go to WildFly's bin directory, start our command-line interface and execute the operation to change the enabled attribute to true:

$cd ~/Downloads/wildfly-8.0.0.Alpha4/bin/

[~/Downloads/wildfly-8.0.0.Alpha4/bin]
$./jboss-cli.sh 
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 
[standalone@localhost:9990 /] /core-service=management/access=audit/logger=audit-log:write-attribute(name=enabled,value=true){"outcome" => "success"}
[standalone@localhost:9990 /] 
If you look in $WILDFLY_HOME/standalone/configuration/standalone.xml again, you will see that the value of the enabled attribute we saw earlier is now true.

Now, if you look, you will see that $WILDFLY_HOME/standalone/data/audit-log.log has been created, and it contains the following:
$more data/audit-log.log
2013-08-14 15:26:16 - {
    "type" : "core",
    "r/o" : false,
    "booting" : false,
    "user" : "$local",
    "domainUUID" : null,
    "access" : "NATIVE",
    "remote-address" : "127.0.0.1/127.0.0.1",
    "success" : true,
    "ops" : [{
        "address" : [
            {
                "core-service" : "management"
            },
            {
                "access" : "audit"
            },
            {
                "logger" : "audit-log"
            }
        ],
        "operation" : "write-attribute",
        "name" : "enabled",
        "value" : true,
        "operation-headers" : {"caller-type" : "user"}
    }]
}

The actual contents of the fields is explained in the documentation but a few things worth pointing out is that access=NATIVE means it came from the native management interface, and ops contains the operation that we executed to enable audit logging.

Enable syslog logging

First you need a syslog server which can accept connections via UDP or TCP/IP. I am on OS X, so I enabled my local syslog server to accept UDP connections on port 514 by following these tips. If you are on linux you probably have rsyslog installed, so you would typically modify /etc/rsyslog.conf to include the following directives
...
$ModLoad imudp.so
$UDPServerRun 514
...

Then restart the syslog service. I think the OS X link shows you how to do that, and for linux you need to restart the rsyslogd service. In the worst case reboot, and start up wildfly and the command-line interface again ;-)

Then in the command-line interface window, execute these following commands to add a syslog handler called mysyslog:
[standalone@localhost:9990 /] batch 
[standalone@localhost:9990 / #] /core-service=management/access=audit/syslog-handler=mysyslog:add(formatter=json-formatter)
#1 /core-service=management/access=audit/syslog-handler=mysyslog:add(formatter=json-formatter)
[standalone@localhost:9990 / #] /core-service=management/access=audit/syslog-handler=mysyslog/protocol=udp:add(host=localhost,port=514)
#2 /core-service=management/access=audit/syslog-handler=mysyslog/protocol=udp:add(host=localhost,port=514)
[standalone@localhost:9990 / #] run-batch 
The batch executed successfully
Then add a reference to mysyslog to the logger:
[standalone@localhost:9990 /] /core-service=management/access=audit/logger=audit-log/handler=mysyslog:add
{"outcome" => "success"}

You should now see the executed commands logged in $WILDFLY_HOME/standalone/data/audit-log.log., e.g.:
2013-08-14 15:55:20 - {
    "type" : "core",
    "r/o" : false,
    "booting" : false,
    "user" : "$local",
    "domainUUID" : null,
    "access" : "NATIVE",
    "remote-address" : "127.0.0.1/127.0.0.1",
    "success" : true,
    "ops" : [{
        "operation" : "composite",
        "address" : [],
        "steps" : [
            {
                "address" : [
                    {
                        "core-service" : "management"
                    },
                    {
                        "access" : "audit"
                    },
                    {
                        "syslog-handler" : "mysyslog"
                    }
                ],
                "operation" : "add",
                "formatter" : "json-formatter",
                "operation-headers" : {"caller-type" : "user"},
                "max-length" : null,
                "syslog-format" : null,
                "truncate" : null,
                "max-failure-count" : null
            },
            {
                "address" : [
                    {
                        "core-service" : "management"
                    },
                    {
                        "access" : "audit"
                    },
                    {
                        "syslog-handler" : "mysyslog"
                    },
                    {
                        "protocol" : "udp"
                    }
                ],
                "operation" : "add",
                "port" : 514,
                "host" : "localhost",
                "operation-headers" : {"caller-type" : "user"}
            }
        ],
        "operation-headers" : {"caller-type" : "user"}
    }]
}
2013-08-14 15:56:34 - {
    "type" : "core",
    "r/o" : false,
    "booting" : false,
    "user" : "$local",
    "domainUUID" : null,
    "access" : "NATIVE",
    "remote-address" : "127.0.0.1/127.0.0.1",
    "success" : true,
    "ops" : [{
        "address" : [
            {
                "core-service" : "management"
            },
            {
                "access" : "audit"
            },
            {
                "logger" : "audit-log"
            },
            {
                "handler" : "mysyslog"
            }
        ],
        "operation" : "add",
        "operation-headers" : {"caller-type" : "user"}
    }]
}
And if you look at syslog,
  • On OS X, open the Console.app and go to 'All Messages'
  • On Linux, look in /var/log/messages
In any case you should see the logger handler reference add in syslog
14/08/2013 15:56:35.000 2013-08-14T15: 56:35.529+01:00 localhost WildFly 2348 - - 2013-08-14 15:56:34 - {
    "type" : "core",
    "r/o" : false,
    "booting" : false,
    "user" : "$local",
    "domainUUID" : null,
    "access" : "NATIVE",
    "remote-address" : "127.0.0.1/127.0.0.1",
    "success" : true,
    "ops" : [{
        "address" : [
            {
                "core-service" : "management"
            },
            {
                "access" : "audit"
            },
            {
                "logger" : "audit-log"
            },
            {
                "handler" : "mysyslog"
            }
        ],
        "operation" : "add",
        "operation-headers" : {"caller-type" : "user"}
    }]
(The reason you did not see the add of the handler is that the audit log subsystem does not actually start using it until the logger handler reference is added). If you execute the following command in the command-line interface
[standalone@localhost:9990 /] /system-property=test123:add(value=testing)
{"outcome" => "success"}
You should see the operation get logged in both syslog and the file.

Finally, if you look at $WILDFLY_HOME/standalone/configuration/standalone.xml again you will see that the syslog handler and reference have been persisted:

        <audit-log>
            <formatters>
                <json-formatter name="json-formatter"/>
            </formatters>
            <handlers>
                <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
                <syslog-handler name="mysyslog" formatter="json-formatter">
                    <udp host="localhost" port="514"/>
                </syslog-handler>
            </handlers>
            <logger log-boot="true" log-read-only="false" enabled="true">
                <handlers>
                    <handler name="file"/>
                    <handler name="mysyslog"/>
                </handlers>
            </logger>
        </audit-log>

Tuesday, August 13, 2013

Setting up Secure Audit Logging to a Remote Rsyslog Instance in WildFly 8.0.0.Alpha4

WildFly is the new name for JBoss Application Server. It builds on the same architecture as JBoss Application Server 7. WildFly is the upstream for JBoss Enterprise Application Platform 6, which is our supported middleware offering at Red Hat. Some of our larger customers have asked for additional security features when it comes to managing their installations, and one of the pieces for this is to implement audit logging of the management layer. The initial implementation of audit logging will be part of the WildFly 8.0.0.Alpha4 release, which was released today.

Note that this is a pretty advanced topic, for a simpler overview of audit logging see my follow-up post.

The WildFly Admin Guide covers the reference for the configuration of audit logging framework, but I thought it would be a good idea to show step by step how to configure WildFly to log to a remote syslog server over TLS. I will also throw client certificate authentication into the mix. I have used rsyslog for my experiments since that comes with nice documentation for how to set it up to use TLS. You need to install the rsyslog-gnutls package for the following to work.

I do my main work on OS X but have a linux server with rsyslog installed so I will use that as the syslog server. It's address is kabirlinux. Wherever you see the name kabirlinux you should replace it with the host name of your machine running rsyslog.

Generating the server certificates

First I ssh into kabirlinux and generate the keys we need to set up rsyslog to accept remote connections over TLS (taken from http://www.rsyslog.com/tag/tls/ and http://www.rsyslog.com/doc/rsyslog_tls.html). The first thing we need to do is to generate the ca certificate that will be used for signing the other certificates. Please note that wherever I have used sudo, it is very important that you do so too, otherwise rsyslog might not recognise your keys.
$ssh kabirlinux
Last login: Mon Aug 12 16:15:28 2013 from 192.168.1.102

[kabirlinux ~]
$mkdir tmp

[kabirlinux ~]
$cd tmp

[kabirlinux ~/tmp]
$sudo certtool --generate-privkey --outfile ca-key.pem
[sudo] password for kabir: 
Generating a 2560 bit RSA private key...
Then we generate the ca certificate using the private key:
[kabirlinux ~]
$sudo certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem
Generating a self signed certificate...
Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
Country name (2 chars): UK
Organization name: Red Hat
Organizational unit name: JBoss
Locality name: London
State or province name: London
Common name: kabirlinux
UID: 
This field should not be used in new certificates.
E-mail: me@redhat.com
Enter the certificate's serial number in decimal (default: 1376387226): 


Activation/Expiration time.
The certificate will expire in (days): 3650


Extensions.
Does the certificate belong to an authority? (y/N): y
Path length constraint (decimal, -1 for no constraint): 
Is this a TLS web client certificate? (y/N): 
Will the certificate be used for IPsec IKE operations? (y/N): 
Is this also a TLS web server certificate? (y/N): 
Enter the e-mail of the subject of the certificate:    
Will the certificate be used to sign other certificates? (y/N): y
Will the certificate be used to sign CRLs? (y/N): 
Will the certificate be used to sign code? (y/N): 
Will the certificate be used to sign OCSP requests? (y/N): 
Will the certificate be used for time stamping? (y/N): 
Enter the URI of the CRL distribution point: 
X.509 Certificate Information:
 Version: 3
.........

Is the above information ok? (y/N): y


Signing certificate...
Next we generate a certificate request, and the server certificate that will be used to secure TLS.
[kabirlinux ~/tmp]
$sudo certtool  --generate-privkey --outfile server-key.pem --bits 2048
** Note: Please use the --sec-param instead of --bits
Generating a 2048 bit RSA private key...

[kabirlinux ~/tmp]
$sudo certtool --generate-request --load-privkey server-key.pem --outfile request.pem
Generating a PKCS #10 certificate request...
Country name (2 chars): UK
Organization name: Red Hat
Organizational unit name: JBoss
Locality name: London
State or province name: London
Common name: kabirlinux
UID: 
Enter a dnsName of the subject of the certificate: kabirlinux
Enter a dnsName of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Enter the e-mail of the subject of the certificate: me@redhat.com
Enter a challenge password: 
Does the certificate belong to an authority? (y/N): 
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): 
Will the certificate be used for encryption (RSA ciphersuites)? (y/N): 
Is this a TLS web client certificate? (y/N): y
Is this also a TLS web server certificate? (y/N): y

[kabirlinux ~/tmp]
$sudo certtool --generate-certificate --load-request request.pem --outfile server-cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem
Generating a signed certificate...
Enter the certificate's serial number in decimal (default: 1376387546): 


Activation/Expiration time.
The certificate will expire in (days): 3650


Extensions.
Do you want to honour the extensions from the request? (y/N): 
Does the certificate belong to an authority? (y/N): 
Is this a TLS web client certificate? (y/N): y
Will the certificate be used for IPsec IKE operations? (y/N): 
Is this also a TLS web server certificate? (y/N): y
Enter a dnsName of the subject of the certificate: kabirlinux
Enter a dnsName of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): 
Will the certificate be used for encryption (RSA ciphersuites)? (y/N): 
X.509 Certificate Information:
 Version: 3

.....

Is the above information ok? (y/N): y


Signing certificate...

[kabirlinux ~/tmp]
$rm request.pem 
We then need to protect the ca private key
[kabirlinux ~/tmp]
$sudo chmod 400 ca-key.pem
Now move the generated files across to /etc/pki/rsyslog, keeping a copy of the server certificate in the tmp directory: 
[kabirlinux ~/tmp]
$sudo mv ca-key.pem ca.pem server-cert.pem server-key.pem /etc/pki/rsyslog/

[kabirlinux ~/tmp]
$sudo cp /etc/pki/rsyslog/server-cert.pem .
Then modify the rsyslog configuration file at /etc/rsyslog.conf to allow TLS. In a vanilla installation this is just after the initial ModLoad directives.
....
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides TLS syslog reception
$ModLoad imtcp #Load tcp driver
$DefaultNetstreamDriver gtls # Sets the gtls driver as the default
# Certificates
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/ca.pem
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/server-cert.pem
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/server-key.pem
# run driver in TLS-only mode
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon 
$InputTCPServerRun 514 # start up listener at port 514
....
Make sure the syslog server's firewall is open to TCP requests on port 514. A simple way to get rid of all rules is to run this when ssh'ed into the syslog server (you will of course want to set this up properly once done experimenting):
[kabirlinux ~]
$sudo iptables -F
Then restart the rsyslog daemon:
[kabirlinux ~]
$sudo service rsyslog restart
[sudo] password for kabir: 
Redirecting to /bin/systemctl restart  rsyslog.service
In the syslog you should then see messages indicating that the service has been restarted
[kabirlinux ~]
$sudo tail -f /var/log/messages
.....
Aug 12 16:50:32 kabir systemd[1]: Stopping System Logging Service...
Aug 12 16:50:32 kabir systemd[1]: Starting System Logging Service...
Aug 12 16:50:32 kabir systemd[1]: Started System Logging Service.

Importing the server certificate into a Java truststore

Since the server certificate is not signed by a trusted certificate authorization authority, in my local machine where I will be running WildFly, I need to import the server certificate into a truststore. First I copy it down to my local machine:
[~]
$mkdir test

[~]
$cd test

[~/test]
$sftp kabirlinux
Connected to kabirlinux.
sftp> cd tmp
sftp> ls
server-cert.pem
sftp> get server-cert.pem
Fetching /home/kabir/tmp/server-cert.pem to server-cert.pem
/home/kabir/tmp/server-cert.pem                                                    100% 1529     1.5KB/s   00:00    
sftp> exit

[~/test]
Then I import the certificate into a trustore file using the keytool command, and 'test123' as the password:
[~/test]
$keytool -import -trustcacerts -file server-cert.pem -alias CA_ALIAS -keystore truststore
Enter keystore password:  
Re-enter new password: 
...

Trust this certificate? [no]:  yes
Certificate was added to keystore

Set up WildFly to use syslog with TLS for audit logging

Make sure you are using WildFly 8.0.0.Alpha4 or later! Out of the box wildfly's audit log configuration in its standalone/configuration/standalone.xml will look like this
<audit-log>
            <formatters>
                <json-formatter name="json-formatter"/>
            </formatters>
            <handlers>
                <file-handler name="file" formatter="json-formatter" relative-to="jboss.server.data.dir" path="audit-log.log"/>
            </handlers>
            <logger log-boot="true" log-read-only="false" enabled="false">
                <handlers>
                    <handler name="file"/>
                </handlers>
            </logger>
        </audit-log>
Change this to
<audit-log>
            <formatters>
                <json-formatter name="json-formatter"/>
            </formatters>
            <handlers>
                <syslog-handler name="syslog-tls" formatter="json-formatter" syslog-format="RFC5424">
                    <tls host="kabirlinux" port="514" message-transfer="NON_TRANSPARENT_FRAMING">
                        <truststore path="/Users/kabir/test/truststore" keystore-password="test123"/>
                    </tls>
                </syslog-handler>
            </handlers>
            <logger log-boot="true" log-read-only="true" enabled="true">
                <handlers>
                    <handler name="syslog-tls"/>
                </handlers>
            </logger>
        </audit-log>
If having the keystore password hard wired in the xml worries you, please take a look at how to use our vault for sensitive information like that. Now if you stop and start WildFly and look in /var/log/messages on the server running rsyslog, you should see audit logged messages on bootup. Also, when executing operations, for example from the CLI you should see those added to the syslog.

Client certificate authentication

Rsyslog supports full blown client certificate authentication, however I have not tried setting that up. However, it also has a slightly more permissive mode which simply checks that the client has a certificate signed by the server's ca. So first we generate the client certificate, I do this on my syslog server:
[~/test]
$ssh kabirlinux
Last login: Mon Aug 12 18:02:25 2013 from 192.168.1.102

[kabirlinux ~]
$cd tmp/

[kabirlinux ~/tmp]
$certtool --generate-privkey --outfile client-key.pem
Generating a 2560 bit RSA private key...

[kabirlinux ~/tmp]
$certtool --generate-request --load-privkey client-key.pem --outfile client-request.pem
Generating a PKCS #10 certificate request...
Country name (2 chars): UK
Organization name: Red Hat
Organizational unit name: JBoss
Locality name: London
State or province name: London
Common name: client    
UID: 
Enter a dnsName of the subject of the certificate: kabirlinux
Enter a dnsName of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Enter the e-mail of the subject of the certificate: me@redhat.com
Enter a challenge password: 
Does the certificate belong to an authority? (y/N): N
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): 
Will the certificate be used for encryption (RSA ciphersuites)? (y/N): 
Is this a TLS web client certificate? (y/N): y
Is this also a TLS web server certificate? (y/N): y

[kabirlinux ~/tmp]
$sudo certtool  --generate-certificate --load-request client-request.pem --outfile client-cert.pem  --load-ca-certificate /etc/pki/rsyslog/ca.pem --load-ca-privkey /etc/pki/rsyslog/ca-key.pem
[sudo] password for kabir: 
Generating a signed certificate...
Enter the certificate's serial number in decimal (default: 1376327802): 


Activation/Expiration time.
The certificate will expire in (days): 3650


Extensions.
Do you want to honour the extensions from the request? (y/N): 
Does the certificate belong to an authority? (y/N): 
Is this a TLS web client certificate? (y/N): y
Will the certificate be used for IPsec IKE operations? (y/N): 
Is this also a TLS web server certificate? (y/N): y
Enter a dnsName of the subject of the certificate: kabirlinux
Enter a dnsName of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): 
Will the certificate be used for encryption (RSA ciphersuites)? (y/N): 
X.509 Certificate Information:
...

Is the above information ok? (y/N): y
We then need to combine the client certificate and key to pkcs12 format to copy across to the client, using 'test123' as the export password
[kabirlinux ~/tmp]
$openssl pkcs12  -export -in client-cert.pem -inkey client-key.pem -out client.p12 -name "Whatever"
Enter Export Password:
Verifying - Enter Export Password:

[kabirlinux ~/tmp]
$rm -rf client*.pem
Now on my Mac I get the client.p12 file and import it into a jks keystore
[~/test]
$sftp kabirlinux
Connected to kabirlinux.
sftp> cd tmp 
sftp> get client.p12
Fetching /home/kabir/tmp/client.p12 to client.p12
/home/kabir/tmp/client.p12                                                                                                                                               100% 3094     3.0KB/s   00:00    
sftp> exit

[~/test]
$keytool -v -importkeystore -srckeystore client.p12 -srcstoretype PKCS12 -destkeystore client.jks -deststoretype JKS
Enter destination keystore password:  
Re-enter new password: 
Enter source keystore password:  
Entry for alias whatever successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
[Storing client.jks]
Again, I use 'test123' as the keystore password. Then on the rsyslog server we change /etc/rsyslog.conf to use 'x509/certvalid' instead of 'anon' for the InputTCPServerStreamDriverAuthMode directive, e.g:
$InputTCPServerStreamDriverAuthMode x509/certvalid
Next we add the client-certificate-store information to the syslog handler in the audit logging section of standalone/configuration/standalone.xml in the WildFly installation so it looks like:
<audit-log>
            <formatters>
                <json-formatter name="json-formatter"/>
            </formatters>
            <handlers>
                <syslog-handler name="syslog-tls" formatter="json-formatter" syslog-format="RFC5424">
                    <tls host="kabirlinux" port="514" message-transfer="NON_TRANSPARENT_FRAMING">
                        <truststore path="/Users/kabir/test/truststore" keystore-password="test123"/>
                        <client-certificate-store path="/Users/kabir/test/client.jks" keystore-password="test123"/>
                    </tls>
                </syslog-handler>
            </handlers>
            <logger log-boot="true" log-read-only="true" enabled="true">
                <handlers>
                    <handler name="syslog-tls"/>
                </handlers>
            </logger>
        </audit-log>
Now if you restart the rsyslog daemon and WildFly you should see the log messages come through again. So this means that you can host rsyslog on a machine not accessible by the people who have access to your WildFly installation. Communication between the WildFly installation and rsyslog is encrypted since it goes over TLS, and only users who have the correct certificate are able to send data to rsyslog.