Revolution Systems specializes in web application development, Systems Administration services and commercial support of many Open Source Software systems.

Centralized logging for fun and profit!

Setting up a centralized log server using syslog isn't as hard as many may believe. Whether it's logs from Apache, nginx, email services, or even from your own Python applications having a central log server gives you many benefits:

Benefits to a centralized logs

  • Reduces disk space usage and disk I/O on core servers that should be busy doing something else. This is especially true if you want to log all queries to your database. Doing this on the same disk as your actual database creates a write for every read and an extra write for every write.
  • Removes logs from the server in the event of an intrusion or system failure. By having the logs elsewhere you at least have a chance of finding something useful about what happened.
  • All of your logs are in one place, duh! This makes things like grepping through say Apache error logs across multiple webservers easier than bouncing around between boxes. Any log processing and log rotation can also be centralized which may delay your sysadmin from finally snapping and killing everyone.

Syslog Review

In case you aren't terribly familiar with how syslog works, here's a quick primer. Syslog separates out various logs using two items. Facilities and Levels. Here are the standard facilities:

  • 0 kernel messages
  • 1 user-level messages
  • 2 mail system
  • 3 system daemons
  • 4 security/authorization messages
  • 5 messages generated internally by syslogd
  • 6 line printer subsystem
  • 7 network news subsystem
  • 8 UUCP subsystem
  • 9 clock daemon
  • 10 security/authorization messages
  • 11 FTP daemon
  • 12 NTP subsystem
  • 13 log audit
  • 14 log alert
  • 15 clock daemon
  • 16 local use 0 (local0)
  • 17 local use 1 (local1)
  • 18 local use 2 (local2)
  • 19 local use 3 (local3)
  • 20 local use 4 (local4)
  • 21 local use 5 (local5)
  • 22 local use 6 (local6)
  • 23 local use 7 (local7)

For each facility logs are sent using a particular level, the levels are:

  • 0 Emergency: system is unusable
  • 1 Alert: action must be taken immediately
  • 2 Critical: critical conditions
  • 3 Error: error conditions
  • 4 Warning: warning conditions
  • 5 Notice: normal but significant condition
  • 6 Informational: informational messages
  • 7 Debug: debug-level messages

So for any given log message you set these two options to give a hint as to where the logs should be directed. For example, if an email server receives a new message it would likely be sent as mail.info and a kernel panic would be sent using kern.emerg

The receiving syslog server then can be configured to direct log messages of a certain facility and/or log level to various files. For example, a default Ubuntu system has some settings like this: 

daemon.*        /var/log/daemon.log
kern.*          /var/log/kern.log
mail.*          /var/log/mail.log

But you can also do more granular separation for example you might want to log mail.err into a separate file from the main mail logs to make it easier to spot new errors with this: 

mail.*        /var/log/mail.log
mail.err      /var/log/mail-errors.log

Setting up your central server

Configuring the master log server is pretty easy. On Ubuntu the default syslog server is rsyslog and that's what I'll be using as an example here. You'll need to edit /etc/rsyslog.conf and uncomment the UDP module. You could also use the TCP module, but that one binds to all of your interfaces so you will need to restrict access to it with iptables (or some other mechanism) in order to not allow hackers to fill up your disks remotely. So your configuration should now contain these uncommented lines, where 'x' is an internal protected IP address: 

$ModLoad imudp
$UDPServerAddress x.x.x.x
$UDPServerRun 514

And then restart rsyslogd. See that wasn't so hard...

Setting up the remote log sending servers

Setting up your remote servers is even easier. If you want to send ALL of your logs to the central server it's just a matter of adding one line to the top of /etc/rsyslog.d/50-default.conf. That line is: 

 
*.* @x.x.x.x:514

This will send all logs of any facility and any level to the server. Note that the local syslog will, as configured by default, still log locally. So if you don't want that be sure to remove all of the other configuration in this file.

You can also get fancy here and keep some logs on the local server and only send some things remotely. For most of your custom apps and logs you'll want to be using the LOCAL[0-9] facilities. Let's say we're going to want to centrally log our Python logs and Apache error logs. We'll be using LOCAL0 and LOCAL1 for them respectively. That config would look like: 

local0.* @x.x.x.x:514
local1.* @x.x.x.x:514

Keep in mind however that most systems have *.info, *.debug, etc. configurations setup so you might be duplicating your data. If you poke around this file you'll see lots of configurations ending in .none, this instructs rsyslog to not include those facilities in this particular file. So for example, you'd want to edit your /var/log/syslog to resemble this: 

 
*.*;auth,authpriv,local0,local1.none        /var/log/syslog

Additional help and features

While most applications are easy to setup for use with syslog, here are some pointers for more info on the subject:

  • Apache support sending error logs to syslog via the ErrorLog syslog:local1 configuration option. However, it does not support sending access logs directly. To do that you'll need a small script and pipe your access logs through it.
  • For more information on setting up your own Python code to use syslog, check out the logging.handlers.SysLogHandler handler for the logging module.

We've only really scratched the surface of the features of rsyslog with this setup. You can configure it to do some fairly advanced separation of logs based on the sending host, application name, and other various aspects of the message itself. Refer to the rsyslog documentation for more information on that.

Happy Logging!

Posted on August 26th, 2010 at 19:12:23 and has 6 comments

Investing in Yourself - A review of Django 1.1 Testing and Debugging by Karen M. Tracey

Packt Publishing recently asked me to read and review Django 1.1 Testing and Debugging and I have to admit I really enjoyed reading this book. Often I find myself debating whether or not to purchase a new development book. I'm usually thinking "If I spend $XX.XX on this book, will I really learn anything worth that much?". Especially considering most answers are a few Google searches away. I can happily attest this book is definitely worth the cost.

The book starts off with the usual introduction to testing, discussing both Doctests and Unittest, which is obviously required for a book on this subject. However, this book differs greatly from many in that it walks you through testing and debugging your application as you would when building a real application. Many tech books strive to be a great tutorial, but often fall short of the mark and end up just being some verbiage around a rehashing of the available documentation. They end up being more reference than tutorial. Django 1.1 Testing and Debugging however does a great job of walking you through real world scenarios. For example, it covers topics (and in the proper order in my opinion) like:

  • Test coverage
  • Everything you would ever want to know about the default Django debug page
  • How to debug urls.py issues
  • How debugging differs between the development server and what to do in production
  • Getting ahold of what database queries were actually generated on a given page
  • Using the wonderful Django Debug Toolbar
  • Using logging to help debug your code
  • Effectively using the Python Debugger
  • How to report Django bugs

All of these tools and techniques should be in all of our development arsenals, but this is the first book I've seen that puts them all together in a way that is accessible to Django developers of nearly any skill level. Beginners should read the book, even if they don't understand everything right away, just to know what options they do have at their disposal. However, more advanced developers will likely find a few golden nuggets of wisdom that will pay productivity dividends for years to come.

Hence the title of this post Investing in Yourself. While Packt was nice enough to give me a free copy of the book to review, had I paid full list price the time savings from the few little nuggets of wisdom I either didn't know about or for whatever reason never clicked with me before would have paid for the book many times over in just the first few weeks.

I highly encourage you to pick this up and with it improve your code going forward. You can check out a sample chapter of the book to whet your appetite a bit.

Posted on July 6th, 2010 at 13:20:05 and has 0 comments

Early registration for our Advanced Django class ends soon

Early registration ends Friday for the March Advanced Django Class I'm teaching, so if you're planning on coming, you should sign up soon!

I'm really excited about this class: we'll get to dive really deep into the good parts of Django. We'll cover all sorts of advanced usage of Django's APIs, spend a bunch of time playing with all the cool stuff out there in the Django ecosystem, and actually spend a whole day setting up and configuring a real-world deployment stack.

Many more details are over on the class description page, along with contact info if you've got any questions.

Posted on February 16th, 2010 at 16:32:44 and has 0 comments

Django Training in 2010

The new year's shaping up to be a great one for Django: Django 1.2 is on track to ship this March, and there's no doubt in my mind that it'll be the best release ever.

Because of this, we expect to see a lot of new people wanting to learn Django next year, so we're stepping up our training offerings in the new year.

This Friday, I'll be holding a one-day Django workshop in New York City. The class is mostly full, but a few spots are still available. This is the second in a series of workshops we've been holding in conjunction with HoldenWeb, and we plan to offer more (see below)

In March, I'll be teaching a week-long advanced Django class in Kansas City. This class is perfect for folks who know Django but want to really learn all the ins and outs. We'll look at a whole bunch of advanced uses of Django, learn how to use optional add-ons like GeoDjango, Piston, Haystack, and even dig into the internals of Django a bit. We'll also spend a whole day setting up a real-world Django production environment and tuning it for performance.

I'm really looking forward to this class: there's some really fantastic bits buried deep in Django that I don't get much call to talk about.

Finally, we're currently scouting locations and topics for future classes and would like your input! If you'd be interested in attending a future Django class there's a quick questionnaire you can fill out.

Posted on January 18th, 2010 at 17:28:48 and has 3 comments

Django 1.0 Template Development Review

Django 1.0 Template Development by Scott Newman actually surprised me in it's depth of covering the topic.

I assumed it would be written with the absolute Django Template beginner in mind. While it is definitely an appropriate book for beginners, it also covers more advanced topics such as:

  • Writing custom template tags and filters, Chapter 7
  • Covers pagination quite well in Chapter 8
  • Customizing the Django admin's look and feel in Chapter 9
  • And gives a good designer intro to caching in Chapter 10

This book should definitely be required reading for web designers that are looking to use Django. As someone who has read and re-read the wonderful Django Documentation many times this book does a great job of distilling all of that reference knowledge into book form.

Posted on September 17th, 2009 at 12:52:52 and has 0 comments