OpenBSD Journal

Analyzing PF Logs

Contributed by jose on from the integrated-security dept.

(someone who wishes to remain anonymous) asks:
"I know there are quite few XML tutorials on O'Reilly and excellent man pages for pf logging, But I am wondering if anyone had written some tool to convert(offline or seperate process) PF logs to XML?would such work be useful?if combined with Snort's XML then we can have a smarter look at security events.

Many Companies are promising the definitive correlation enginer/intrusion prevention. and I want to know if it is time for an open source intiative? in other words:

I need your tips on what to do after I finished analyzing snort's XML :)"

It shouldn't be too hard to write a pflog reader which outputs XML (possibly sending it to another system for alerting). Has anyone done this?

(Comments are closed)

Comments

  1. By Anonymous Coward () on

    Well we know how the developers would feel about such a tool... it was once suggested to do this to syslogd:

    http://www.sigmasoft.com/~openbsd/archive/openbsd-tech/200211/msg00173.html

    Be sure to read the follow-ups.

    Comments

    1. By Anonymous Coward () on

      This is not the same as changing syslog to log in XML.

      It is simply writing a tool/script/whatever to read PF logs and create an XML log that can be parsed and displayed.

      Big difference. :)

    2. By Anonymous Coward () on

      I know about the syslog issue, I definately remember theo's exact words :) :). But this is a different matter, it is merely to write an awk or perl to convert the entries to XML.
      since we can use tcpdump on pflog0 then this may be the more general task of piping tcpdump to a perl and do a while(){}loop.At least that what I originally was thinking.

      Comments

      1. By Petr R. () on

        over my dead body :o]]]]

  2. By Anonymous Coward () on

    Can you please enlighten me on what those companies are?

    Comments

    1. By Anonymous Coward () on

      I think there is a quite a few


      1-CheckPoint is trying to push their SmartDefense technology.


      2-Symantec with ManHunt.


      I know there are few others, google for it.it is mostly hype, and sending RST , and I am not sure how deep the analysis goes.
      One very interesting project is the Emerald project, this is more of correlation engine that tries to eliminate false positives and beyond.
      it is developed by Stanford Research Institute : http://www.sdl.sri.com/projects/emerald/project.html

  3. By paul weissmann () on

    There is an imho very nice log-analyzing tool available which can create all sorts of summaries (txt, xml, html, pdf) from all sorts of logs (apache, samba, syslog, ipf, blablabla).
    www is at http://logreport.org/lire/ .
    I think it'd be better if someone with time and knowledge would write a pf-backend for this engine, so a lot of duplicate work could be saved.

    Comments

    1. By Anonymous Coward () on

      it seems nice indeed, but as you said , it does all of SUMMARY reports, which is not what I want to do.I see this XML conversion thing to be particularly useful with daniel's new dynamic ruleset modification.so if we have XML from snort, and XML from pf then we can combine both and use SEQUENCE numbers to respond in a specific way to an event.
      what do you think?

      Comments

      1. By paul weissmann () on

        i must say i really did not look deeper in the creation of xml-summaries by lire. i know what you mean, maybe this'd need more serious thinking.
        but i think support for pf in lire would be worthwile and a good start imho

      2. By Can Erkin Acar () on

        You already have binary dumps from pf and snort, which gives you the SAME if not more information using a fraction of space and without the parsing overhead XML would have.

        Comments

        1. By Anonymous Coward () on

          Yes I agree, and possibly you could write something to interact with the dynamic ruleset modification using that.This is probably the way to go since it will be fastest.But would you like to read your logs in that form?

          Comments

          1. By Can Erkin Acar () on

            For log reading tcpdump is usually enough. When you have binary logs, you can analyze them with any of the available tools: ethereal, tcpdump, snort, p0f, even ones that are not written yet. They provide you with complimentary information. When you convert to XML or even ASCII, using _one_ tool you start losing information. What if your (XML/ASCII) conversion tool does not correctly parse some (possibly malicious) packet? Think about the recent tcpdump DoS vulnerability, for instance. I prefer to have the binary logs and convert them to ascii/xml/whatever for visualization purposes only ;)

            Comments

            1. By Anonymous Coward () on

              It probably would be better to do the analysis on the binary level. Any conversions should be for human consumption

  4. By Anonymous Coward () on

    You should try this opensource soft: http://www.conostix.com/ipfc/

    Comments

    1. By Anonymous Coward () on

      I knew about this project, I downloaded it like 7 months ago or something and it was not too useful
      I ll have to download and see how much they got done

  5. By Anonymous Coward () on

    RFC 3252, how to convert TCP packets to XML, once you implement it, be sure to share

    Comments

    1. By Anonymous Coward () on

      aka BLOAT..

      http://www.faqs.org/rfcs/rfc3252.html

    2. By djm () on

      That is an April Fools day joke RFC.

      Comments

      1. By Anonymous Coward () on

        Last years's april's fool becomes today's BLOATed i_need standard... waiting for new april's fool problem

  6. By jose () on http://monkey.org/~jose/

    hi

    i wrote pf2xml for this ... this is more a P0C version at this point, but its at http://monkey.org/~jose/software/pf2xml/pf2xml-0.1 ... a C version should be easy to whip up and remove the hackish dependencies on the right combination of tcpdump flags ... i hope you find it useful, feel free to provide patches and run xmllint on it ...

    Comments

    1. By jose () on http://monkey.org/~jose/

      hehe ... friend who knows XMl had a look at my site and noticed "this doesn't work as XML. not in the least". pf2xml-0.2 is up at the same site and HAS BEEN VALIDATED.

      now to code a version in C.

      Comments

      1. By Anonymous Coward () on

        cool man :)

  7. By djm () on

    What does converting to XML buy you? Buzzword compliance? You can perform most any query on a pflog using tcpdump and bpf filters, converting to a bloated text representation will lose you that immediately.

    Comments

    1. By Anonymous Coward () on

      I'd like to know the same.

      Comments

      1. By Anonymous Coward () on

        Because it is easier to view, analyze, deal with .
        it will provide for a standard format for logging from different device/apps, so one will be able to correlate what actually happened between his IDS,FW,VPN,AV
        how do you currently analyze(if you even analyze it) your pf logs?
        sure, you could grep and awk for things you want, but do you thing making the data more accessible worth anything to you?
        if you have scripts in place that does this, then please share :)

        Comments

        1. By djm () on

          how do you currently analyze(if you even analyze it) your pf logs?

          Like I said, using "tcpdump -r" and filters. Within seconds, I can determine how many packets are blocked from a specific host, or to a specific port.

          How do you do that with XML? Just by changing formats, you have to reengineer your most basic tools.

          it will provide for a standard format for logging from different device/apps

          No it won't. XML isn't some magical format that automatically provides interoperability. Applications need to be taught the schema like any other format. What standard packet format do you propose which retains pf's annotations? RFC3252? :)

          so one will be able to correlate what actually happened between his IDS,FW,VPN,AV

          At the end of the day, you will still be correllating on common data (timestamps, packet headers), so the format makes no difference here.

          Comments

          1. By Anonymous Coward () on

            Again I think people are misunderstanding.
            why do I need to teach PF the XML api?!
            pf will continue to work the same way, or in any way daniel thinks is best


            I dont propose to change PF logging format.I am propsoing offline, or seperate processing of the pflogs to XML.


            Why?because I something related to process snort's XML logs, I want to combine it with pf, and/or netfilter logs.the goal is a library that allows developers to do something like this :


            IDSXMLLOG sLog;
            sLog.open("MyXMLLog");
            Event* p=sLog.GetEvent(15);
            if(Event->GetIP() == Something)
            call_func_x();



            I am not saying this cant be done with awk or perl or sed or whatever.I am saying this is quite some work to be able to combine the data

          2. By Anonymous Coward () on

            Again I think people are misunderstanding.
            why do I need to teach PF the XML api?!
            pf will continue to work the same way, or in any way daniel thinks is best


            I dont propose to change PF logging format.I am propsoing offline, or seperate processing of the pflogs to XML.


            Why?because I something related to process snort's XML logs, I want to combine it with pf, and/or netfilter logs.the goal is a library that allows developers to do something like this :


            IDSXMLLOG sLog;
            sLog.open("MyXMLLog");
            Event* p=sLog.GetEvent(15);
            if(Event->GetIP() == Something)
            call_func_x();



            I am not saying this cant be done with awk or perl or sed or whatever.I am saying this is quite some work to be able to combine the data

      2. By Anonymous Coward () on

        Probably for the same reason I wrote a nasty perl script to parse Cisco, IP Filter, Bay, (pf coming soon) and half a dozen other apps that were of interest to me, logs and dump them into a SQL database. To do correlation using another custom tool I wrote. He has likely written some tool that already reads an XML file and does correlation based on it...why it wouldn't take advantage of a database is beyond me, but maybe it does *shrug*.

        PS. unfortunately my code is not available as it's proprietary, custom written, (and *(&^&^$% ugly, but functional) and I'm not at liberty to disclose it.

        Comments

        1. By Anonymous Coward () on

          Yes, I did wrote a C++ that implements a class for snort XML logs and now I want to correlate pf logs with it.
          Why not use a DB? because whenI started, there were already ACID, which did a good job with Snort DB tables.there were not any that I am aware of to parse pf and import the data into a DB. and because I want to minimize the number of apps involved. That is why what i wrote was a standalone.
          I am going to use jose's script and hopefully soon I ll realease the PF XML logs parser and GUI viewer.

  8. By Paul Hirsch () paul wants no spam at voltagenoir dot org on mailto:paul wants no spam at voltagenoir dot org

    PF logs are just tcpdump format files. You can read them right into Snort using the -r option.
    That is the genius of using tcpdumps as logs: any old program that can process the format has full access to the entire capture, just like it was picked off the wire.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]