In many cases, MySQL will run on the local machine. In this case, you can simply use "127.0.0.1" for database-server. This can be especially advisable, if you do not need to expose MySQL to any process outside of the local machine. In this case, you can simply bind it to 127.0.0.1, which provides a quite secure setup. Of course, also supports remote MySQL instances. In that case, use the remote server name (e.g. mysql.example.com) or IP-address. The database-name by default is "syslog". If you have modified the default, use your name here. Database-userid and -password are the credentials used to connect to the database. As they are stored in clear text in rsyslog.conf, that user should have only the least possible privileges. It is sufficient to grant it INSERT privileges to the systemevents table, only. As a side note, it is strongly advisable to make the rsyslog.conf file readable by root only - if you make it world-readable, everybody could obtain the password (and eventually other vital information from it). In our example, let's assume you have created a MySQL user named "syslogwriter" with a password of "topsecret" (just to say it bluntly: such a password is NOT a good idea...). If your MySQL database is on the local machine, your rsyslog.conf line might look like in this sample:
*.* :ommysql:127.0.0.1,Syslog,syslogwriter,topsecret
Save rsyslog.conf, restart rsyslogd - and you should see syslog messages being stored in the "systemevents" table!
The example line stores every message to the database. Especially if you have a high traffic volume, you will probably limit the amount of messages being logged. This is easy to acomplish: the "write database" action is just a regular selector line. As such, you can apply normal selector-line filtering. If, for example, you are only interested in messages from the mail subsystem, you can use the following selector line:
mail.*:ommysql:127.0.0.1,syslog,syslogwriter,topsecret
Review the rsyslog.conf documentation for details on selector lines and their filtering.
You have now completed everything necessary to store syslog messages to the MySQL database. If you would like to try out a front-end, you might want to look at phpLogCon, which displays syslog data in a browser. As of this writing, phpLogCon is not yet a powerful tool, but it's open source, so it might be a starting point for your own solution.
On Reliability...
Rsyslogd writes syslog messages directly to the database. This implies that the database must be available at the time of message arrival. If the database is offline, no space is left or something else goes wrong - rsyslogd can not write the database record. If rsyslogd is unable to store a message, it performs one retry. This is helpful if the database server was restarted. In this case, the previous connection was broken but a reconnect immediately succeeds. However, if the database is down for an extended period of time, an immediate retry does not help.
Message loss in this scenario can easily be prevented with rsyslog. All you need to do is run the database writer in queued mode. This is now described in a generic way and I do not intend to duplicate it here. So please be sure to read "Handling a massive syslog database insert rate with Rsyslog", which describes the scenario and also includes configuration examples.
Conclusion
With minimal effort, you can use rsyslogd to write syslog messages to a MySQL database. You can even make it absolutely fail-safe and protect it against database server downtime. Once the messages are arrived there, you can interactivley review and analyse them. In practice, the messages are also stored in text files for longer-term archival and the databases are cleared out after some time (to avoid becoming too slow). If you expect an extremely high syslog message volume, storing it in real-time to the database may outperform your database server. In such cases, either filter out some messages or used queued mode (which in general is recommended with databases).
The method outlined in this paper provides an easy to setup and maintain solution for most use cases.
Feedback Requested
I would appreciate feedback on this paper. If you have additional ideas, comments or find bugs, please let me know.
References and Additional Material
- www.rsyslog.com - the rsyslog site
- Paper on Syslog Server Optimization
Revision History
- 2005-08-02 * Rainer Gerhards * initial version created
- 2005-08-03 * Rainer Gerhards * added references to demo site
- 2007-06-13 * Rainer Gerhards * removed demo site - was torn down because too expensive for usage count
- 2008-02-21 * Rainer Gerhards * updated reliability section, can now be done with on-demand disk queues
- 2008-02-28 * Rainer Gerhards * added info on other databases, updated syntax to more recent one
Copyright
Copyright (c) 2005-2008 Rainer Gerhards and Adiscon.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at http://www.gnu.org/copyleft/fdl.html.
[manual index] [rsyslog.conf] [rsyslog site]
This documentation is part of the rsyslog project.
Copyright © 2008 by Rainer Gerhards and Adiscon. Released under the GNU GPL version 2 or higher.