1) Not mentioned in the syslog-ng documentation on performance is the log_msg_size() setting. When follow_freq() is used on a logfile to emulate tail -f behavior, it is necessary for that log_msg_size() is large enough to accomodate all log messages for the reading interval.

2) I've been toying around with the idea of using syslog-ng as a plumbing tool to route logs as I wish. The flexibility it provides is excellent while retaining a simple configuration file, reminding me of the Exim MTA. Currently, I'm using syslog-ng to aggregate 1600 lines/s worth of squid logs. With this workload it consumes ~15% of one Xeon L5335 core.

It is currently quite cumbersome to pass the logs to external programs, requiring a named socket to be created for IPC. Perhaps a process spawner/manager could be hacked in to facilitate this?

Some possibilities to consider with the aggregated log:

a) Visualization, by piping it to a stream visualization tool, such as glTail.rb or Ganglia (with an appropriate wrapper to gmetric).

b) Out-of-band IDS, by piping it to a PHPIDS loop. The client headers can be extracted from the log line and exploded into the superglobals. An external trigger can be called to firewall off malicious clients.

c) Out-of-band DOS mitigator, by piping it to a script that watches for unusually high number of requests. Action can be taken as with (b).