OpenBSD Journal

Kernel options for lots of spamd connections

Contributed by jose on from the spam-tuning dept.

P. Pruett writes: "I would be interested in what kernel tweaks others with openbsd mail servers that use spamd with lots of activity are trying, example of activity: 
  netstat -n | grep 8025 | wc -l
    155
Of note, aggressive tweaking of kernel for us has caused crashes with kmap issues.

Have others tried increasing maxusers?, increasing NMBCLUSTERS, increasing NKMEMPAGES trying twice that of NMBCLUSTERS, maybe setting value for MAXKMAPENT and/or other...

...Then when you get the infamous ddb prompt and if you try boot dump and vmstat -m -M on the core afterwards... what of the indicators in vmstat -m should one pay close attention to if a lot of sendmail and spamd activity?

Thoughts/suggestions on this appreciated."

Having broken lots of kernels with poorly adjusted NMBCLUSTERS and NKMEMPAGES adjustments, I am also curious to see what people suggest.

(Comments are closed)

Comments

  1. By Brian () on


    sevenn@perigee:/home/sevenn> netstat -n | grep 8025 | wc -l
    159
    sevenn@perigee:/home/sevenn>

    I run openbsd's spamd on a router with three mail servers behind it. The only kernel tweaking necessary was boosting nmbclust to 10240, which probably had nothing to do with spamd. I did have to crank the maxconnections in spamd.c though. The default int 3.4 will crash if the number of redirected connections exceeds 200. I believe that it has been raised to 800 in -current.

    Maybe you are asking about spamassassin's spamd?

    I do have "panic: uvm_mapent_alloc: out of static map entries, check MAX_KMAPENT" crash issues on the heavier of the three mail servers. Basically something like 60 spamassassin spamds will launch at once and the box will crash. This happens once a month or so and will continue until I write some sort of wrapper that keeps the number of spamc connections below 40 or so. The crash would happen once a week before increasing max_kmapent to 2000.

    sevenn@gorgimera:/var/crash> sudo ps aux -M bsd.2.core -N bsd.2 | grep -c procmail
    62
    sevenn@gorgimera:/var/crash>

    Comments

    1. By P.Pruett () ppruett@webengr.com on mailto:ppruett@webengr.com

      I have openbsd spamd running on each mail server independently, I am thinking about a transparent bridge for some lans....

      I use spam assassin daemon for some users by procmailrc only on the MX=0 server, but not on secondary MX servers which I have as relays for acceptable domains. Of note on that server MX=0 I also use experimental clamd, and I upped memory and added a spare small gig harddrive as additional swap space just in case needed, and
      also set option=GATEWAY, BUFCACHEPERCENT=20, maxusers 48, option NMBCLUSTERS=4096, option NKMEMPAGES=8192 and for the moment all is okay, except if I am extremely hammered by false smtp auth requests (yep customer had got a bugbear worm) it strains the clamd and may cause its daemon to run out of threads....

      Of note, the crashes from tweaking on the secondary MX mail servers don't say MAX_KMAPENT, rather out of kmap space. Because it was suggested in the openbsd FAQ for performance,
      http://www.openbsd.org/faq/faq11.html
      I had tried option NMBCLUSTERS=8192 with option gateway and option 64 users... that was not good.
      I scaled back to NMBCLUSTER=4094, 48 users, and set option NKMEMPAGES=8192. And it seems more stable.

      Also occassionally spamd on the secondaries dies quietly, maybe the 200 limit, but I susptect its a kernel thing killing it, was worse when NMBCLUSTERS=4094 w/o defining NKMEMPAGES...

      My understanding that if one was really good at interpreting vmstat output, one would just look at hiwater and set kernel options appropriately.
      alas... grasshoper still has not grasped that stone... So how does one become a vmstat guru without all the pain and suffering....





    2. By David Gwynne () loki@animata.net on mailto:loki@animata.net

      I think if you refer to spamassassins manpage for spamd manpage (man 1 spamd) you should not the -m option:

      -m num, --max-children num Allow maximum num children

      I had an issue on a development system that would choke swapping between 30 spamd processes (the box could realistically handle maybe 4, it was very old hardware). Setting the -m flag helped a lot.

  2. By Anonymous Coward () on

    You don't need to do anything special to handle that little. A default GENERIC kernel will work just fine.

    Comments

    1. By P.Pruett () on

      One reason comes to mind;
      If the computer is a gateway also, the generic kernel works but is not optimum.
      option GATEWAY

      Of note, even when using GENERIC, IMHO when doing the updates the kernel should be updated and yes GENERIC config should work. However my experience for high loads on a server dedicated to being a NAT gateway, sendmail relay, spamd trap & named server that had 64meg pentium 333, was that spamd would die silently and sometimes seemed slugish, at least at prompt - perhaps buying a new computer w/ 256meg or 512meg is the answer if using GENERIC.

      Comments

      1. By Anonymous Coward () on

        well, not 'high' loads...
        high load is relative, I probably should have said moderate perhaps... If you are representing a domain name that is one of the ones that mass mail worms direct email to and you have honeypot email addresses that automatically update spamd table, then you will probably see 50+ spamd connections more like 150+

      2. By Anonymous Coward () on

        Personally I would split off some of those functions to separate servers so you don't end up with a single point of failure.

        Unless you are a home user on a shoe string I'd recommend a new server, that one is very old for what you are asking of it. Buy lots of memory, it's cheap. If you cannot convince the bean counters that a new machine is required for the crucial bit of infrastructure you describe then maybe you need to consider a new job ;)

      3. By Anonymous Coward () on

        He's not talking about high loads, he said ~150 spamd connections. There is nothing high loads about that. I have over 200 connections on a GENERIC kernel and it works just fine.

        Comments

        1. By P.Pruett () ppruett@webengr.com on mailto:ppruett@webengr.com

          exceeding 200 connections...

          what type of architecture are you using? i386? supposedly kernels for generic allocate kernel memory differently by architecture? how much physical memory? did you recompile the kernel or is it the openbsd 3.4 kernel from the cdrom, from stable branch and you rebuilt generic kernel?

          According to a previous post "The default int 3.4 will crash if the number of redirected connections exceeds 200"
          You indicate you are running over 200 connections to spamd. If you are using Generic and 3.4 from cdrom, someone has posted conflicting information??? Are you running current in a production environment?

          I would have emailed directly and not posted the question here, however the previous post was anonymous.

          Yes buying better hardware and getting away from i386 archtecture may be the better solution... but costs impact me directly i.e. small biz but
          since my domains have been around since 1996 they are on many spamlists...:(...

          Maybe my machines are nominal ~150 spamd connections with peaking over 200 and that is the hurt... I have five mail servers in MX records for some domains and some are on different lans using spamd to hurt spammers, some of those are also acting as gateways. True I could move the 'spamd' downstream and have the bridge/router redirect to it instead of having the 'spamd' connection on same machine, and will need to do so.

          Comments

          1. By Anonymous Coward () on

            Not current really, but a snapshot yes. I usually install a snapshot, I've never run into any real problems doing so, they are pretty stable. If something was *really* important I'd stick to stable, but for a spamd machine, its not really life or death.

    2. By Chris Nadovich () chris@jtan.com on http://www.jtan.com

      Our mailserver sees about 2 spams per second. Roughly half of these we manage to direct to spamd. We also have mailscanner and sendmail on this box. This results in roughly 400 spamd connections, steady state. With everything, the LA is typically 5.

      It's a thing of beauty. However, we do see occasional crashes.

      We could not use the generic kernel. We saw too many MAX_KMAPENT crashes. We also ran out of files occasionally. Both of these needed tweeking. Now we see crashes no more than about one per month (MAX_KMAPENT crashes, typically).

      One thing that should be pointed out: do please use the -current version of spamd. Some of the older versions had small bugs that made them unstable in certain situations. If you aren't using the latest spamd, then you can't expect it to be stable in all cases.

      We also saw problems with the TCP window size set to 1 (the default in some spamd versions). This setting seemed to hurt us more than it hurt the spammers.

  3. By Anonymous Coward () on

    This is an HTML version of the definitive guide to performance tuning:
    http://www.pantz.org/os/openbsd/runningandtunningopenbsd.shtml

    Here is the original PS version of the same thing:
    http://cvs.openbsd.org/papers/tuning-openbsd.ps

    Comments

    1. By P.Pruett () on

      thats helpfull in pointing out which tools. would like more info on using the tools...
      ... vmstat can be the most helpful tool, especially for looking at a core after dump of the kernel.

      Now for me the question continues, besides highwater for mcpl what tell tale things to look for, Which may require a in depth explanation of all the output of vmstat, like what nominal for pkrkentry? Possibly this and more is clearly documented by existing docs but missed by google searches

    2. By P.Pruett () on

      Another paper, that was mentioned awhile back came out last year and it is useful
      in showing what can be done with the tools also,
      especially vmstat -m
      http://openbsd.underlinux.com.br/papers/tuning.pdf

      Comments

      1. By EN () on

        It's same paper as at http://cvs.openbsd.org/papers/tuning-openbsd.ps

  4. By yusuf () on

    Does spamd use libevent (from Niels Provos), it might be useful if/when OpenBSD gets kqueue support so that spamd can scale to lots of connections

    Combined with pf's excellent table support, spamd with a kqueue based libevent would help throttle spammers who are punishing mail servers with lots of connections

    Comments

    1. By gwyllion () on

      OpenBSD already has kqueue support and libevent in OpenBSD uses it. So you only have to teach spamd to use libevent.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]