= Postfix/Stomp_logging_patch = Can't find an archive URL :/ ---- {{{ Date: Tue, 29 Sep 2009 16:19:25 +0530 Message-ID: <64de5c8b0909290349q108af1c3y47d7821c742abef0@mail.gmail.com> Subject: Stomp logging for postfix - patch From: Rajkumar S To: Postfix users Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sender: owner-postfix-users@postfix.org Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: I have couple of postfix servers whose logs I need to parse and store in a database for generating reports. I have tried parsing individual logs but it was taking too much time for parsing. Also with the number of types of logs postfix is generating, parser would break every now and then when postfix logs an entry with say, malformed email id. I finally ditched the parsing part and coded up a patch to send the log entries to a stomp server. I ideally would like to pass around perfectly valid json entries as logs, but right now I just separate the entries with \233 (some thing I do not expect to appear in a log) and I have a program to get the log entries from RabbitMQ split them and push them to DB. Some of the advantages here are that there is no parsing, Just split using delimiter and I get a neat key value pair. Also every log has a type field with which I can directly switch. The patch is uploaded at http://pastebin.ca/1584108.The patch is far from perfect, but I have it running it in a moderately loaded box for some time. Postfix has a logging class but I have decided to put a different function partially because that was the easy way out, and partially because stomp logging should augment the existing logging than change it's destination. I am also having difficulty in understanding the build process. consequently the build process is convoluted. C stomp library is built up on apache apr library. So (in debian) dependencies are : libapr1, libapr1-dev libaprutil1, libaprutil1-dev and libdb-4.6-dev. I am building postfix on a 64bit machine, so a patch for adding -D_LARGEFILE64_SOURCE to makefile is necessary (for use by APR). --- /dev/null +++ b/largefile64.patch @@ -0,0 +1,13 @@ +diff --git a/makedefs b/makedefs +index cfba0e9..4860358 100644 +--- a/makedefs ++++ b/makedefs +@@ -450,7 +450,7 @@ case "$CCARGS" in + *-DNO_PCRE*) ;; + *) pcre_cflags=`(pcre-config --cflags) 2>/dev/null` && + pcre_libs=`(pcre-config --libs) 2>/dev/null` && { +- CCARGS="$CCARGS -DHAS_PCRE $pcre_cflags" ++ CCARGS="$CCARGS -DHAS_PCRE $pcre_cflags -D_LARGEFILE64_SOURCE" + AUXLIBS="$AUXLIBS $pcre_libs" + } + ;; I am sure there must be some way to get the build flag automatically, but I am unable to figure that out. So till then this is ugly hack #1 After executing make at this point, the build process stops after some time and here I need to do another magic. Copy src/global/mail_params.h to include/ and execute the make again and the build completes successfully. I have no idea what happens during the build and how to "properly" modify the build scripts for a flawless build, but my steps works :) The patch adds 3 entries in config files stomp_port = 61613 queue = postfix amq_server = ravanan queue is the message queue in RabbitMQ (or ActiveMQ) and amq_server is the ip of the RabbitMQ server. This is my first jab at postfix source and I only consider the patch a place to start some discussion. It would be great if some one could go through/test the code and also suggest fixes to the build problems. I believe having postfix log to a stomp server using json would be a great feature for people running multiple postfix serves and need to analyze the data in a central location. with warm regards, raj }}} Alternative {{{ I'am working in a (python) framework to parse in real-time any log format and store in a database, it's base in plugins that repsent a class. A new plugin is a class with a method "insert", very easy make anything. I working in postfix plugin but iptables plugin seems OK. http://groups.google.com/group/post-la/attach/aa63b7c30a0bfc91/processLog-2009092701.tar.gz?part=2 -- Reinaldo de Carvalho }}}