OpenBSD Journal
Home : : Add Story : : Archives : : About : : Create Account : : Login :
NetFlow support with pflow(4)
Contributed by jason on Tue Sep 9 16:31:58 2008 (GMT)
from the don't-forget-to-put-the-seat-down-when-you're-done dept.

Henning Brauer (henning@) has just committed a new pseudo-device to OpenBSD-current. The pflow(4) interface exports IP accounting data over UDP that is compatible with NetFlow version 5. Please continue to read Henning's commit message and a brief introduction to pflow(4) usage.

CVSROOT:	/cvs
Module name:	src
Changes by:	henning@cvs.openbsd.org	2008/09/09 07:56:39

Modified files:
	sbin/ifconfig  : ifconfig.c 
	sbin/pfctl     : parse.y pf_print_state.c pfctl_parser.c 
	sys/conf       : files 
	sys/net        : if_types.h pf.c pfvar.h 
	sys/sys        : sockio.h 
Added files:
	share/man/man4 : pflow.4 
	sys/net        : if_pflow.c if_pflow.h 

Log message:
welcome pflow(4), a netflow v5 compatible flow export interface.
flows export data gathered from pf states.
initial implementation by Joerg Goltermann , guidance and many
changes by me. 'put it in' theo

In typical OpenBSD style, this new pseudo-device works seamlessly with the rest of the networking subsystem. It behaves similarly to other virtual networking devices in that you manage it with ifconfig(8) and can even use tcpdump(8) on the interface to monitor flow exports. Setting it up is very simple:

$ sudo ifconfig pflow0 create
$ sudo ifconfig pflow0 flowsrc 10.0.0.200 flowdst 10.0.0.1:1234
$ ifconfig pflow0
pflow0: flags=41 mtu 1464
        pflow: sender: 10.0.0.200 receiver: 10.0.0.1:1234
        groups: pflow

Flows are tracked using the state-tracking capabilities in pf(4). States that are marked with the pflow state-opts keyword will be exported by the pflow interface once the state is expired from the session table. Here is a sample pf filter rule that enables flow accounting for outbound ICMP traffic:

pass out inet proto icmp keep state (pflow)

Once the ruleset is loaded, pfctl(8) will report which states are being tracked for pflow exports:

$ sudo pfctl -vss | grep -B2 pflow | head -3
all tcp 10.0.0.200:38336 -> 38.68.100.209:22       ESTABLISHED:ESTABLISHED
   [3825225521 + 17376] wscale 0  [3569953586 + 16384] wscale 0
   age 00:32:58, expires in 23:59:57, 888:894 pkts, 75601:180313 bytes, pflow

This is a very useful feature for IP accounting and can even be handy for network troubleshooting. It effectively replaces the userland pfflowd daemon (net/pfflowd) created by Damien Miller (djm@), although we would still need a collector to receive the exports. Fortunately, Damien also created flowd (net/flowd), a secure NetFlow collector, and the perfect complement to pflow(4).

As a self-described network geek, I'm thrilled by this new addition to OpenBSD. Joerg Goltermann created the initial implementation of this feature, while Henning refined it for the initial import. I expect additional enhancements to pflow(4) as it stabilizes in the tree from testing and daily use. Although this won't be included with OpenBSD 4.4, buying a pre-order would be a great way to show our appreciation for this type of work!

[topicpf2]

<< NYCBSDCon 2008 Registration Open | Reply | Threaded | European pre-orders for 4.4 are open! >>

Threshold: Help

Related Links
more by jason


  Re: NetFlow support with pflow(4) (mod -2/36)
by Anonymous Coward (89.104.121.248) on Tue Sep 9 16:17:48 2008 (GMT)
  Long live OpenBSD! (:
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -3/27)
by Anonymous Coward (60.53.30.214) on Tue Sep 9 16:57:33 2008 (GMT)
  Does it support features like to:

1. check ingress traffic only,
2. check egress traffic only,
3. check traffic at specific interface, example a pf-capable FreeBSD router which have 3 interface, em0 (wan),em1 (lan), em2 (dmz), and only capture em2 traffic (which we're interested due to specific reason)?

Banzai OpenBSD. Just ordering 4.4 via Int Order with 1 t-shirt. ;)

-malaysia boleh-
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 4/24)
by Anonymous Coward (60.53.30.214) on Tue Sep 9 17:01:16 2008 (GMT)
  And because it is compatible with NetFlow version 5, is it possible to export it to any netflow collector such as Solarwinds Orion NetFlow Traffic Analyzer, or probably opensource tool like ntop?




  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 0/26)
by jason (jason) on Tue Sep 9 17:12:44 2008 (GMT)
http://www.dixongroup.net/
  > Does it support features like to:
>
> 1. check ingress traffic only,
> 2. check egress traffic only,
> 3. check traffic at specific interface, example a pf-capable FreeBSD router which have 3 interface, em0 (wan),em1 (lan), em2 (dmz), and only capture em2 traffic (which we're interested due to specific reason)?

Sure, just use the "pflow" state-opt keyword only on those filter rules.

> Banzai OpenBSD. Just ordering 4.4 via Int Order with 1 t-shirt. ;)

:)
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 6/34)
by jason (jason) on Tue Sep 9 17:17:30 2008 (GMT)
http://www.dixongroup.net/
 
> And because it is compatible with NetFlow version 5, is it possible to 
> export it to any netflow collector such as Solarwinds Orion NetFlow 
> Traffic Analyzer, or probably opensource tool like ntop?

Sure, as long as the collector is compatible with NetFlow version 5. I use flowd/Flowd.pm for capturing my traffic with [warning: blatant project plug]...

NetFlow Dashboard

:)
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -5/23)
by Anonymous Coward (60.53.30.214) on Tue Sep 9 18:37:06 2008 (GMT)
  > > Does it support features like to:
> >
> > 1. check ingress traffic only,
> > 2. check egress traffic only,
> > 3. check traffic at specific interface, example a pf-capable FreeBSD router which have 3 interface, em0 (wan),em1 (lan), em2 (dmz), and only capture em2 traffic (which we're interested due to specific reason)?
>
> Sure, just use the "pflow" state-opt keyword only on those filter rules.
>
> > Banzai OpenBSD. Just ordering 4.4 via Int Order with 1 t-shirt. ;)
>
> :)


>
> And because it is compatible with NetFlow version 5, is it possible to
> export it to any netflow collector such as Solarwinds Orion NetFlow
> Traffic Analyzer, or probably opensource tool like ntop?
>
>
> Sure, as long as the collector is compatible with NetFlow version 5. I use flowd/Flowd.pm for capturing my traffic with [warning: blatant project plug]...
>
> NetFlow Dashboard
>
> :)


Dear Mr Jason,

Thank you very much for above reply. I'm looking forward for pflow(4) manual to be updated and give it tries in my lab.

Have a nice day!
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -3/23)
by Anonymous Coward (60.53.30.214) on Tue Sep 9 19:18:27 2008 (GMT)
  Last question, are you the one in this video?

http://talks.dixongroup.net/nycbsdcon2006/

It is very nice, funny and interesting video that I've ever seen in BSD world. ;)

  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 0/26)
by jason (jason) on Tue Sep 9 19:28:05 2008 (GMT)
http://www.dixongroup.net/
  > Last question, are you the one in this video?
>
> http://talks.dixongroup.net/nycbsdcon2006/
>
> It is very nice, funny and interesting video that I've ever seen in BSD world. ;)

Guilty as charged. :)
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -1/33)
by Anonymous Coward (24.84.63.46) on Tue Sep 9 19:56:25 2008 (GMT)
  How is this better than the pfflowd util? Cramming more into the kernel is not a good idea.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -1/25)
by jason (jason) on Tue Sep 9 20:18:59 2008 (GMT)
http://www.dixongroup.net/
  > How is this better than the pfflowd util? Cramming more into the kernel is not a good idea.

First, it has full access to the PF state table. Since it only needs information about state deletion, it doesn't have all the overhead that pfflowd has (listening to all state updates). In addition, there are other things planned that will probably require access to the PF internals, but I'm just guessing.

Just because it *can* be done in userland doesn't always mean it must. I think this is a good use of in-kernel functionality. CARP *could* have been done in userland, but nobody is complaining about having that in the kernel. ;)
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Erm.. sure this is a good idea? (mod 3/29)
by Wouter (2001:888:1b6b:b0e::6965:6b73) on Tue Sep 9 21:03:29 2008 (GMT)
  Wasn't Netflow patented?
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Erm.. sure this is a good idea? (mod -1/23)
by jason (jason) on Tue Sep 9 23:10:14 2008 (GMT)
http://www.dixongroup.net/
  > Wasn't Netflow patented?

[IANAL] I believe that refers to sampled NetFlow, which this isn't.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -5/29)
by mpf@ (88.217.158.50) on Wed Sep 10 08:22:18 2008 (GMT)
  > > How is this better than the pfflowd util? Cramming more into the kernel is not a good idea.
>
> First, it has full access to the PF state table. Since it only needs
> information about state deletion, it doesn't have all the overhead that
> pfflowd has (listening to all state updates).

Actually, if you start pfflowd with a pcap filter like: "link[2]==0x03",
the kernel will only pass it state deletions (via pfsync).

  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -6/24)
by ates (78.25.11.204) (ates@ipv6.dp.ua) on Wed Sep 10 08:39:22 2008 (GMT)
  WOW! It's very greate news! OpenBSD best! Thank a lot!
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 0/22)
by Anonymous Coward (81.57.42.108) on Wed Sep 10 11:33:57 2008 (GMT)
  >
> Sure, as long as the collector is compatible with NetFlow version 5. I use flowd/Flowd.pm for capturing my traffic with [warning: blatant project plug]...
>
> NetFlow Dashboard
>
> :)

Cute!
Is this Dashboard distributed?
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 5/29)
by coward1 (81.169.155.246) on Wed Sep 10 13:47:55 2008 (GMT)
  Could kernel recognize traffic streams not only by port but also by application features, then do QoS and/or netflow export? .i.e layer 7 aware?
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 2/24)
by jason (jason) on Wed Sep 10 13:59:37 2008 (GMT)
http://www.dixongroup.net/
  > >
> > Sure, as long as the collector is compatible with NetFlow version 5. I use flowd/Flowd.pm for capturing my traffic with [warning: blatant project plug]...
> >
> > NetFlow Dashboard
> >
> > :)
>
> Cute!
> Is this Dashboard distributed?

It will be soon. I'm wedging it into the chroot with mod_perl, then will announce it officially with an OpenBSD port/package.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 13/41)
by joe (98.97.67.6) on Thu Sep 11 01:05:18 2008 (GMT)
  crud. i was hoping to see this in 4.4!

:(
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -2/22)
by Anonymous Coward (83.88.170.104) on Thu Sep 11 01:08:21 2008 (GMT)
  > Sure, just use the "pflow" state-opt keyword only on those filter rules.

I often miss an alternative to "pass ..." and "block..." that doesn't change whether the packet will pass or not. It wold be also usefull for adding tags, logging and queueing.

Off cause there would be no flow if the packet ended up being blocked.

Maybee a "match" keyword. I know it would simplify my rulesets.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 5/25)
by Loop (203.194.27.78) on Thu Sep 11 05:53:02 2008 (GMT)
  >
> And because it is compatible with NetFlow version 5, is it possible to
> export it to any netflow collector such as Solarwinds Orion NetFlow
> Traffic Analyzer, or probably opensource tool like ntop?
>
>
> Sure, as long as the collector is compatible with NetFlow version 5. I use flowd/Flowd.pm for capturing my traffic with [warning: blatant project plug]...
>
> NetFlow Dashboard
>
> :)

That is dead sexeh!

WANT!!!!
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -1/25)
by Frank DENIS (82.224.188.215) on Thu Sep 11 08:41:51 2008 (GMT)
http://00f.net
  > Just because it *can* be done in userland doesn't always mean it must. I think this is a good use of in-kernel functionality. CARP *could* have been done in userland, but nobody is complaining about having that in the kernel. ;)


Actually some people use the userland version of CARP even on OpenBSD, because they need a custom setup (specific scripts to be spawned when ownership is taken, etc).
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 5/23)
by henning (130.237.95.167) on Thu Sep 11 11:13:18 2008 (GMT)
  > > Sure, just use the "pflow" state-opt keyword only on those filter rules.
>
> I often miss an alternative to "pass ..." and "block..." that doesn't change whether the packet will pass or not. It wold be also usefull for adding tags, logging and queueing.
>
> Off cause there would be no flow if the packet ended up being blocked.
>
> Maybee a "match" keyword. I know it would simplify my rulesets.

that (with exactly that keyword) is on my virtual todo list for ages. 4 years at least. over time my idea on how it is supposed to work got clearer, and be warned: it is entirely nonobvious. well, ok, if you just do match it is easy, but to be really useful you have to think about when what applies. for example, logging only happens on the last matching rule, so a "match log" doesn't help you.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 0/22)
by Brad (2001:470:8802:3:216:41ff:fe17:6933) (brad at comstyle dot com) on Thu Sep 11 16:41:24 2008 (GMT)
  > > Just because it *can* be done in userland doesn't always mean it must. I think this is a good use of in-kernel functionality. CARP *could* have been done in userland, but nobody is complaining about having that in the kernel. ;)
>
>
> Actually some people use the userland version of CARP even on OpenBSD, because they need a custom setup (specific scripts to be spawned when ownership is taken, etc).

You can have scripts spawned without needing UCARP.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod -4/28)
by Anonymous Coward (130.237.95.124) on Fri Sep 12 16:35:19 2008 (GMT)
  The p must flow.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Erm.. sure this is a good idea? (mod -1/21)
by ldso (208.5.80.104) (ldso@sourceforge.net) on Sat Sep 13 15:15:15 2008 (GMT)
http://marmot.dudeabides.net
  > > Wasn't Netflow patented?
>
> [IANAL] I believe that refers to sampled NetFlow, which this isn't.

I would like to hear more about this. So Ciscos and Junipers that we get flows from are sampled. But this isn't, because it shoots out flow data for every connection via pf? I'm interested in what kind of load it puts on various hardware. We are currently using the classic FreeBSD flow-tools solution.

But the fact that it is not sampled doesn't matter I guess, as long as your tools are net-flow v5 capable? I wonder how flow-tools might cope with this, as well as how well our Intel 1U boxes would perform.

Very cool work, good job :) I'll have to play with this when I get back from vacay.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  NetFlow Dashboard question (mod 3/21)
by Terrell Prude' Jr. (151.188.18.46) (tprude@cmosnetworks.com (this is a spamtrap address)) on Sat Oct 18 00:57:45 2008 (GMT)
http://www.cmosnetworks.com/
  > > >
> > > Sure, as long as the collector is compatible with NetFlow version 5. I use flowd/Flowd.pm for capturing my traffic with [warning: blatant project plug]...
> > >
> > > NetFlow Dashboard
> > >
> > > :)
> >
> > Cute!
> > Is this Dashboard distributed?
>
> It will be soon. I'm wedging it into the chroot with mod_perl, then will announce it officially with an OpenBSD port/package.

Will it be available in a traditional tarball as well? We use the Flow-Tools here at work (yes, on Linux, it was pulling teeth to get *that* allowed in our data center). Such a Dashboard would be great.

--TP
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 0/14)
by Jessicaparker (38.103.14.140) on Fri Jan 22 23:37:10 2016 (GMT)
  Are any of these netflow analyzers able to be used with P flow? list of netflow analyzers i dont see any mention in this and nothing is mentioned in this article either: netflow openbsd guide thx
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 2/14)
by &#1575;&#1604;&#1593;&#1575;&#1576; (66.85.185.78) (ztmayto4o@moakt.ws) on Mon Aug 22 12:37:44 2016 (GMT)
&#1576;&#1606;&#1575;&#1578;
  إن ألعاب الفلاش تعرف تطورا كبيرا، خصوصا في مجال الجرافيك والأداء، لقد أصبح الإهتمام بهندسة الصورة من الأولويات، إضافة إلى البحث عن الإمتاع في اللعبة، وهذا ما ستلمسه في لعبة خرجت سنة 2016 وهي لعبة الدبابة المدمرة، إحدى روائع موقع العاب سيارات الحربية. العاب سيارات لعب العاب سيارات 2017 al3ab العاب تلبيس بنات العاب باربي العاب فلاش العاب عربيات
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 0/14)
by jeuxbanat (178.62.31.125) (admin@al3ab.com) on Fri Aug 26 23:17:55 2016 (GMT)
  محبى العاب بنات نقدم لكم اللعبة المميزة وهى لعبة سير استيل المميزة والتى تمكنك من تلبيس بنات بطريقة الشير المميزة والرائعة والجذابة al3ab banat
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: NetFlow support with pflow(4) (mod 0/0)
by mxffiles (218.11.237.74) on Tue Feb 7 06:14:52 2017 (GMT)
  This is a very good post which I really enjoy reading. It is not every day that I have the possibility to see something like this. Software mxf Software mxf converter free download to convert HD camcorder files. ts converter convert ts video files to avi, mp4, wmv, mov mts to avi mp4 mov mkv iMovie, FCP/FCE with mts converter, so to convert mts files for your PC and mobiles. mod converter and convert tod files just free download mod video converter. m2ts
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

[ Home | Add Story | Archives | Polls | About ]

Copyright © 2004-2008 Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to April 2nd 2004 as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. Some icons from slashdot.org used with permission from Kathleen. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. Search engine is ht://Dig. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]