Postfix/Stomp_logging_patch

Can't find an archive URL :/

Date: Tue, 29 Sep 2009 16:19:25 +0530
Message-ID: <64de5c8b0909290349q108af1c3y47d7821c742abef0@mail.gmail.com>
Subject: Stomp logging for postfix - patch
From: Rajkumar S <rajkumars@gmail.com>
To: Postfix users <postfix-users@postfix.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Sender: owner-postfix-users@postfix.org
Precedence: bulk
List-Post: <mailto:postfix-users@postfix.org>
List-Help: <http://www.postfix.org/lists.html>
List-Unsubscribe: <mailto:majordomo@postfix.org>
List-Subscribe: <mailto:majordomo@postfix.org>

I have couple of postfix servers whose logs I need to parse and store
in a database for generating reports. I have tried parsing individual
logs but it was taking too much time for parsing. Also with the number
of types of logs postfix is generating, parser would break every now
and then when postfix logs an entry with say, malformed email id.

I finally ditched the parsing part and coded up a patch to send the
log entries to a stomp server. I ideally would like to pass around
perfectly valid json entries as logs, but right now I just separate
the entries with \233 (some thing I do not expect to appear in a log)
and I have a program to get the log entries from RabbitMQ split them
and push them to DB.

Some of the advantages here are that there is no parsing, Just split
using delimiter and I get a neat key value pair. Also every log has a
type field with which I can directly switch.

The patch is uploaded at http://pastebin.ca/1584108.The
patch is far from perfect, but I have it running it in a moderately
loaded box for some time.

Postfix has a logging class but I have decided to put a different
function partially because that was the easy way out, and partially
because stomp logging should augment the existing logging than change
it's destination. I am also having difficulty in understanding the
build process. consequently the build process is convoluted.

C stomp library is built up on apache apr library. So (in debian)
dependencies are : libapr1, libapr1-dev libaprutil1, libaprutil1-dev
and libdb-4.6-dev. I am building postfix on a 64bit machine, so a
patch for adding  -D_LARGEFILE64_SOURCE to makefile is necessary (for
use by APR).

--- /dev/null
+++ b/largefile64.patch
@@ -0,0 +1,13 @@
+diff --git a/makedefs b/makedefs
+index cfba0e9..4860358 100644
+--- a/makedefs
++++ b/makedefs
+@@ -450,7 +450,7 @@ case "$CCARGS" in
+  *-DNO_PCRE*) ;;
+            *) pcre_cflags=`(pcre-config --cflags) 2>/dev/null` &&
+                   pcre_libs=`(pcre-config --libs) 2>/dev/null` && {
+-                      CCARGS="$CCARGS -DHAS_PCRE $pcre_cflags"
++                      CCARGS="$CCARGS -DHAS_PCRE $pcre_cflags
-D_LARGEFILE64_SOURCE"
+                       AUXLIBS="$AUXLIBS $pcre_libs"
+               }
+               ;;

I am sure there must be some way to get the build flag automatically,
but I am unable to figure that out. So till then this is ugly hack #1

After executing make at this point, the build process stops after some
time and here I need to do another magic.

Copy src/global/mail_params.h  to include/ and execute the make again
and the build completes successfully. I have no idea what happens
during the build and how to "properly" modify the build scripts for a
flawless build, but my steps works :)

The patch adds 3 entries in config files

stomp_port = 61613
queue = postfix
amq_server = ravanan

queue is the message queue in RabbitMQ (or ActiveMQ) and amq_server is
the ip of the RabbitMQ server.

This is my first jab at postfix source and I only consider the patch a
place to start some discussion. It would be great if some one could go
through/test the code and also suggest fixes to the build problems. I
believe having postfix log to a stomp server using json would be a
great feature for people running multiple postfix serves and need to
analyze the data in a central location.

with warm regards,

raj

Alternative

I'am working in a (python) framework to parse in real-time any log
format and store in a database, it's base in plugins that repsent a
class.  A new plugin is a class with a method "insert", very easy make
anything. I working in postfix plugin but iptables plugin seems OK.

http://groups.google.com/group/post-la/attach/aa63b7c30a0bfc91/processLog-2009092701.tar.gz?part=2

--
Reinaldo de Carvalho