Opinion about pflog

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Opinion about pflog

Walter Alejandro Iglesias-2
I know complaining is useless.  Forgive me this time.

I'm about to run my own web server using OpenBSD.  I'm giving my first
steps with pf.  I was very enthusiastic till I got to this point:

https://www.openbsd.org/faq/pf/logging.html

It says:

    The log file written by pflogd is in binary format and cannot be
    read using a text editor.

So, *binary* logs.  Sounds familiar to me.  And then:

   In many situations it is desirable to have the firewall logs available
   in ASCII format

And this "uncommon" practice among unix system administrators (sarcasm),
needs a "workaround".  You end with a file with a curious termination:

    Create the file /var/log/pflog.txt ...


I must confess I'm one among those "run to the hills" paranoids.  I'm
not an expert, perhaps I'm judging pflog wrong but, anyway, I still
prefer the traditional way, using cat, grep and tail.

Reply | Threaded
Open this post in threaded view
|

Re: Opinion about pflog

Martin Brandenburg
On Wed, 28 Sep 2016, Walter Alejandro Iglesias wrote:

> I know complaining is useless.  Forgive me this time.
>
> I'm about to run my own web server using OpenBSD.  I'm giving my first
> steps with pf.  I was very enthusiastic till I got to this point:
>
> https://www.openbsd.org/faq/pf/logging.html
>
> It says:
>
>     The log file written by pflogd is in binary format and cannot be
>     read using a text editor.
>
> So, *binary* logs.  Sounds familiar to me.  And then:
>
>    In many situations it is desirable to have the firewall logs available
>    in ASCII format
>
> And this "uncommon" practice among unix system administrators (sarcasm),
> needs a "workaround".  You end with a file with a curious termination:
>
>     Create the file /var/log/pflog.txt ...
>
>
> I must confess I'm one among those "run to the hills" paranoids.  I'm
> not an expert, perhaps I'm judging pflog wrong but, anyway, I still
> prefer the traditional way, using cat, grep and tail.
>
>

# file /var/log/pflog
/var/log/pflog: tcpdump capture file (little-endian) - version 2.4 (OpenBSD PFLOG, capture length 160)

Would you rather have something convert packets to ASCII arbitrarily
throwing away `unimportant' fields?

Martin

Reply | Threaded
Open this post in threaded view
|

Re: Opinion about pflog

Theo de Raadt-2
In reply to this post by Walter Alejandro Iglesias-2
> I know complaining is useless.  Forgive me this time.
>
> I'm about to run my own web server using OpenBSD.  I'm giving my first
> steps with pf.  I was very enthusiastic till I got to this point:
>
> https://www.openbsd.org/faq/pf/logging.html
>
> It says:
>
>     The log file written by pflogd is in binary format and cannot be
>     read using a text editor.
>
> So, *binary* logs.  Sounds familiar to me.  And then:

Your type of person seems familiar to be me.  Undeducated *check*
opinioned *check*  Contrasting authoritatively without any education
to back it up *check*

pflog generates pcap files.  that is the DEFACTO INDUSTRY format
for packet logs, since they can be generated at extremely high speed
without decomposition, and then can be analysed later, offline, using
the pcap library with a sophisticated grammer and bpf executation
engine.

So now get lost, grow up, go learn something, and stop being a dick
comparing things you don't know anything about to other things you
don't know anything about.

There is no way to forgive people who intentionally step in the shit.

Reply | Threaded
Open this post in threaded view
|

Re: Opinion about pflog

fwsoucy
In reply to this post by Walter Alejandro Iglesias-2
On 09/28/2016 03:25 PM, Walter Alejandro Iglesias wrote:

> I know complaining is useless.  Forgive me this time.
>
> I'm about to run my own web server using OpenBSD.  I'm giving my first
> steps with pf.  I was very enthusiastic till I got to this point:
>
> https://www.openbsd.org/faq/pf/logging.html
>
> It says:
>
>     The log file written by pflogd is in binary format and cannot be
>     read using a text editor.
>
> So, *binary* logs.  Sounds familiar to me.  And then:
>
>    In many situations it is desirable to have the firewall logs available
>    in ASCII format
>
> And this "uncommon" practice among unix system administrators (sarcasm),
> needs a "workaround".  You end with a file with a curious termination:
>
>     Create the file /var/log/pflog.txt ...
>
>
> I must confess I'm one among those "run to the hills" paranoids.  I'm
> not an expert, perhaps I'm judging pflog wrong but, anyway, I still
> prefer the traditional way, using cat, grep and tail.
>
if by *familiar* you are implying systemd than just stop. if you're that
worried about binary logs you should have jumped ship from unix and
unix-like systems decades ago. man utmp(5).

Reply | Threaded
Open this post in threaded view
|

Re: Opinion about pflog

John Jasen
In reply to this post by Walter Alejandro Iglesias-2
On 09/28/2016 04:25 PM, Walter Alejandro Iglesias wrote:

> And this "uncommon" practice among unix system administrators (sarcasm),
> needs a "workaround".  You end with a file with a curious termination:
>
>     Create the file /var/log/pflog.txt ...

You can name it pflog.log versus pflog.txt, if you wish, and the web
page provides a reference implementation to send it to syslog.

I'll note, a decently busy firewall can swamp a remote syslog server.

Reply | Threaded
Open this post in threaded view
|

Re: Opinion about pflog

Peter Nicolai Mathias Hansteen
In reply to this post by Walter Alejandro Iglesias-2
On 09/28/16 22:25, Walter Alejandro Iglesias wrote:

> I'm about to run my own web server using OpenBSD.  I'm giving my first
> steps with pf.  I was very enthusiastic till I got to this point:
>
> https://www.openbsd.org/faq/pf/logging.html
>
> It says:
>
>     The log file written by pflogd is in binary format and cannot be
>     read using a text editor.
>
> So, *binary* logs.  Sounds familiar to me.  And then:

I hope this doesn't discourage you too much.

While I don't have the privilege of reading the developers' minds, there
are a few practical considerations that help make the binary logging
understandable as the default configuration.

One is that firewalls are not necessarily the most capable pieces of
hardware you could put your hands on. Decoding every single packet to
text and writing to syslog or even to console could in itself add
significantly to the load on the system.

I hadn't thought of those incidents for several years, but while writing
this I remember some episodes involving Linux-based firewalls that did
log every packet, decoded to readable text, to the system console and to
syslog. In slightly high-traffic situations -- not actual DDOSes by any
measure -- logging on to the system to fix whatever was not really doable.

Be that as it may, one other thing that comes to mind is that frankly,
most of the traffic is likely to be of no interest whatsoever once it's
been successfully handled according to your PF rules.

> I must confess I'm one among those "run to the hills" paranoids.  I'm
> not an expert, perhaps I'm judging pflog wrong but, anyway, I still
> prefer the traditional way, using cat, grep and tail.

Well, for for generally keeping an eye on things and not putting too
much strain on anything, setting up pflow(4) to export the metadata to
view with something like nfsen is a good option[1].

If you really want to keep a copy of all traffic, well, you need to go
for beefier hardware and set up to keep a copy of your traffic somewhere
and play with whatever combination of snort, bro and friends that fit
your needs. And yes, beefy hardware and sufficient storage will be a
requirement.

Then again, in all likelihood you will not really want to be staring at
all traffic that ever passed through all network interfaces. If and when
your chosen tools show up something you want to look into in more
detail, applying tcpdump to the binary PF logs may very well be
sufficient to find out what happened and make intelligent decisions. If
you want to keep the PF logs around for longer than the defaults, look
into the log rotation settings as your first step.

I remember having a somewhat similar reaction as yours when I first read
about the binary PF logs, but in practical terms the way it's done
actually makes sense.

- P

[1] One such setup is described, with some anecdotes just because, at
http://bsdly.blogspot.com/2014/02/yes-you-too-can-be-evil-network.html

--
Peter N. M. Hansteen, member of the first RFC 1149 implementation team
http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/
"Remember to set the evil bit on all malicious network traffic"
delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.

Reply | Threaded
Open this post in threaded view
|

Re: Opinion about pflog

Walter Alejandro Iglesias-2
In reply to this post by Theo de Raadt-2
On Wed, Sep 28, 2016 at 02:36:10PM -0600, Theo de Raadt wrote:

> > So, *binary* logs.  Sounds familiar to me.  And then:
>
> Your type of person seems familiar to be me.  Undeducated *check*
> opinioned *check*  Contrasting authoritatively without any education
> to back it up *check*
>
> pflog generates pcap files.  that is the DEFACTO INDUSTRY format
> for packet logs, since they can be generated at extremely high speed
> without decomposition, and then can be analysed later, offline, using
> the pcap library with a sophisticated grammer and bpf executation
> engine.
>
> So now get lost, grow up, go learn something,

Too late, I'm 49 years old and spent most of my life being a
professional musician (+20 years playing violoncello).  Being a musician
I had to work a lot for free like FOSS developers, so I think I
understand your bad temper, except I didn't become famous enough to
start being so concerned about the "uneducated opinion" of people about
my work.

I spent only the last six years of my life learning how to administer
unix-like systems.  Obviously not enough to feel myself entitled to give
an opinion here, so you're right.  It won't happen again.

I'll take this opportunity to express my opinion about this project but
from a point of view I think I'm entitled: the human aspect.

Even being myself, as you rightly said, an ignorant in the matter, I
felt treated by OpenBSD developers as an equal.  When I reported a bug
they answered me, and politely, even to personal messages.  Thanks to
all of them for making the difference.


***

Just for fun:

> There is no way to forgive people who intentionally step in the shit.

Breaking news, God isn't Argentinian, is Canadian!

Reply | Threaded
Open this post in threaded view
|

Re: Opinion about pflog

Walter Alejandro Iglesias-2
In reply to this post by Walter Alejandro Iglesias-2
To the other people who answer me here, sorry for the delay, I took some
time to calm down and not degrade myself to the level of discussion some
person here proposed me.


Martin Brandenburg,

I know what pcap files are, I used them.  But, as I said, I'm not an
expert, I didn't take in care that converting them to ASCII could mean
losing information (if I understand you well).

Thanks for the clarification.

***

R0me0 (private) and John Jansen,

I'd read the documentation before posting here.  Thanks anyway.

***

Frederick W. Soucy

You got the "idea behind" my message (by the way, I was aware about
utmp).  Taking in care I'm not in a Linux mailing list I avoided to
mention the abomination by its name :-).  That's why I'm a bit paranoid
and some times I'm sarcastic.  Sorry for that.

The point is, I ask myself the same a lot of unix users probably are
asking themselves, should I invest more time in educating myself in
practices that in two days could be declared obsolete?  Or should I
install MSWindows in my desktop and RedHat in my server and simply use
the casual WYSIWYG interface to read logs (it exists a port called
winpcap)?  Surely there are a lot of system administrators out there
that do this and win the same money than if dealing with pf or iptables
directly.  In theory FOSS projects should be against promoting this
tendency among users (very few understand why) but in practice happens
exactly the opposite, at all levels.


***

Peter Hansteen,

Thanks for your explanation.

As I told you in a private email there aren't the technical details but
some human attitudes what discourages me.  But I won't give up just
because one bad experience.  I'll probably buy your book about pf. ;-)



Thanks to all.


        Walter

Reply | Threaded
Open this post in threaded view
|

Re: Opinion about pflog

lists-2
Fri, 30 Sep 2016 20:43:02 +0200 Walter Alejandro Iglesias
<[hidden email]>
[...]
> The point is, I ask myself the same a lot of unix users probably are
> asking themselves, should I invest more time in educating myself in
> practices that in two days could be declared obsolete?

Hi Walter,

You've come to the right place to ask this question..  In practice, any
intermediate skill OpenBSD user will tell you the answer.  Invest time,
unless you're already burned into something else beyond salvation.  Now
go get some general books on web sites and stop mixing binary and text.
You don't just read pcap files, unless your internal robot says "doit".
I don't believe your story that you're that inexperienced, anyway.  Why
you are concerned with any attitude is your problem, manual pages don't
yell at you.  Neither do books and BSD people are precise and punctual.
I don't see any of your points (being afraid of something) as rational.
OpenBSD makes you more productive, by sparing brain and machine cycles.
The UNIX user is eager to log in, try it, read the manual and do stuff.

OpenBSD: Frequently Asked Questions
[https://www.openbsd.org/faq/]

OpenBSD: PF - User's Guide
[https://www.openbsd.org/faq/pf/]

Hansteen: Firewalling with PF
[https://home.nuug.no/~peter/pf/]

tcpdump - dump traffic on a network
[http://man.openbsd.org/tcpdump]

Kind regards,
Anton