spamd-setup fails from cron

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

spamd-setup fails from cron

Jan Stary
Pretty current 5.1-current/amd64.
This is what happens with the following line in root's crontab

        0 * * * * /usr/libexec/spamd-setup -d

On May 29 03:00:02, Cron Daemon wrote:
> Getting http://www.openbsd.org/spamd/traplist.gz
> spamd-setup: Could not add blacklist uatraps: Illegal seek
> Getting http://www.openbsd.org/spamd/nixspam.gz
> ftp: Writing -: Broken pipe
> blacklist nixspam 40000 entries

What is the 'illegal seek' spamd-setup reports?
What is the ftp's broken pipe?

When I run the same command from the command line,
everything goes fine. Is the cron job run in a more
restricted environment?

        Jan

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

David Diggles-2
Change it to this:

<insert non zero number here> * * * * /usr/libexec/spamd-setup -d

It will probably fix the problem.

On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote:

> Pretty current 5.1-current/amd64.
> This is what happens with the following line in root's crontab
>
> 0 * * * * /usr/libexec/spamd-setup -d
>
> On May 29 03:00:02, Cron Daemon wrote:
> > Getting http://www.openbsd.org/spamd/traplist.gz
> > spamd-setup: Could not add blacklist uatraps: Illegal seek
> > Getting http://www.openbsd.org/spamd/nixspam.gz
> > ftp: Writing -: Broken pipe
> > blacklist nixspam 40000 entries
>
> What is the 'illegal seek' spamd-setup reports?
> What is the ftp's broken pipe?
>
> When I run the same command from the command line,
> everything goes fine. Is the cron job run in a more
> restricted environment?
>
> Jan

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

Peter N. M. Hansteen-3
In reply to this post by Jan Stary
On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote:
 
> When I run the same command from the command line,
> everything goes fine. Is the cron job run in a more
> restricted environment?

you could be hitting the 'zero minute rush', where world+dog tries to
connect simultaneously.  try shifting to a few minutes past the hour and
see if that helps.

- Peter
--
Peter N. M. Hansteen, member of the first RFC 1149 implementation team
http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/
"Remember to set the evil bit on all malicious network traffic"
delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

Gilles Chehade-7
On Tue, May 29, 2012 at 09:14:29AM +0200, Peter N. M. Hansteen wrote:

> On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote:
>  
> > When I run the same command from the command line,
> > everything goes fine. Is the cron job run in a more
> > restricted environment?
>
> you could be hitting the 'zero minute rush', where world+dog tries to
> connect simultaneously.  try shifting to a few minutes past the hour and
> see if that helps.
>

Please avoid 15 minutes past the hour ;-)

--
Gilles Chehade

https://www.poolp.org | http://pool.ps                                @poolpOrg

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

David Diggles-2
A random sleep between 0 and 3599 prior to running
spamd-setup in cron would not go astray.

On Tue, May 29, 2012 at 09:23:43AM +0200, Gilles Chehade wrote:

> On Tue, May 29, 2012 at 09:14:29AM +0200, Peter N. M. Hansteen wrote:
> > On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote:
> >  
> > > When I run the same command from the command line,
> > > everything goes fine. Is the cron job run in a more
> > > restricted environment?
> >
> > you could be hitting the 'zero minute rush', where world+dog tries to
> > connect simultaneously.  try shifting to a few minutes past the hour and
> > see if that helps.
> >
>
> Please avoid 15 minutes past the hour ;-)
>
> --
> Gilles Chehade
>
> https://www.poolp.org | http://pool.ps                                @poolpOrg

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

Mitja Muženič
In reply to this post by Gilles Chehade-7
> > you could be hitting the 'zero minute rush', where world+dog tries to
> > connect simultaneously.  try shifting to a few minutes past the hour
> and
> > see if that helps.
> >
>
> Please avoid 15 minutes past the hour ;-)

Years ago I've been toying with the idea of having a flag for random-delay
mode in spamd-setup. So the default cronjob is still set at zero minute, but
spamd-setup then waits for random amount of minutes before hitting the
server. You lose the predictability of when your blacklists get updated but
it spreads the load on the infrastructure without involving the human factor
(the lazy admin that uncomments the cron entry without changing the time).

Mitja

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

Bret S. Lambert-2
In reply to this post by Gilles Chehade-7
> Please avoid 15 minutes past the hour ;-)

sleep $(($RANDOM % 2048)) && /usr/libexec/spamd-setup -d

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

Christer Solskogen-3
In reply to this post by Mitja Muženič
On Tue, May 29, 2012 at 9:50 AM, Mitja MuE>eniD
 <[hidden email]> wrote:
> Years ago I've been toying with the idea of having a flag for random-delay
> mode in spamd-setup. So the default cronjob is still set at zero minute,
but
> spamd-setup then waits for random amount of minutes before hitting the
> server. You lose the predictability of when your blacklists get updated but
> it spreads the load on the infrastructure without involving the human
factor
> (the lazy admin that uncomments the cron entry without changing the time).
>

portsnap, a tool that FreeBSD uses for fetching the ports tree is
using a simular method.
Take a look here:
http://svnweb.freebsd.org/base/head/usr.sbin/portsnap/portsnap/portsnap.sh?re
vision=235208&view=markup


--
chs,

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

Jan Stary
In reply to this post by Peter N. M. Hansteen-3
On May 29 09:14:29, Peter N. M. Hansteen wrote:
> On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote:
>  
> > When I run the same command from the command line,
> > everything goes fine. Is the cron job run in a more
> > restricted environment?
>
> you could be hitting the 'zero minute rush', where world+dog tries to
> connect simultaneously.  try shifting to a few minutes past the hour and
> see if that helps.

Yes it did help.  Thanks!

        Jan

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

David Diggles-2
In reply to this post by Bret S. Lambert-2
On Tue, May 29, 2012 at 09:51:54AM +0200, Bret Lambert wrote:
> > Please avoid 15 minutes past the hour ;-)
>
> sleep $(($RANDOM % 2048)) && /usr/libexec/spamd-setup -d

Tried something like the above, and found it still
fails at peak times, so I am trying something else:

I made a wrapper called ss (spamd sync), to keep
retrying within a diminishing timeframe.

#!/bin/sh
remaining=$1;shift
cmd=$@
lock=/var/run/$(basename $1).lock
[ -f $lock ] || {
  touch $lock
  while [ $remaining -gt 0 ]; do
    seconds=$(($RANDOM % $remaining))
    echo $(date) $seconds >> $lock
    sleep $s
    $cmd && return || remaining=$(($remaining - $seconds))
  done
  rm $lock
}

0 * * * * ss 3600 /usr/libexec/spamd-setup -d

The time overhead of running the command creates a small
possibility for overlapping of cron events, so I created
a lock file.  This also seemed a good place to store the
sleep value(s).

/var/run/spamd-setup.lock

spamd-setup(8) does not say how it behaves when daemonized.
Maybe this is a better option than running from the cron?

.d.d.

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

David Diggles-2
On Fri, Jun 01, 2012 at 03:47:21PM +1000, David Diggles wrote:
[ snip ]
>     sleep $s
[ snip ]

Arghh.. ;-) sleep $seconds here

Anyway, you get the idea.

#!/bin/sh
remaining=$1;shift
cmd=$@
lock=/var/run/$(basename $1).lock
[ -f $lock ] || {
  touch $lock
  while [ $remaining -gt 0 ]; do
    seconds=$(($RANDOM % $remaining))
    echo $(date) $seconds >> $lock
    sleep $seconds
    $cmd && return || remaining=$(($remaining - $seconds))
  done
  rm $lock
}

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

David Diggles-2
> #!/bin/sh
> remaining=$1;shift
> cmd=$@
> lock=/var/run/$(basename $1).lock
> [ -f $lock ] || {
>   touch $lock
>   while [ $remaining -gt 0 ]; do
>     seconds=$(($RANDOM % $remaining))
>     echo $(date) $seconds >> $lock
>     sleep $seconds
>     $cmd && return || remaining=$(($remaining - $seconds))
>   done
>   rm $lock
> }
>

*groan*.. another mistake.. I'm such an idiot sometimes ;-)

I don't recommend running this without checking it first.

#!/bin/sh
remaining=$1;shift
cmd=$@
lock=/var/run/$(basename $1).lock
[ -f $lock ] || {
  touch $lock
  while [ $remaining -gt 0 ]; do
    seconds=$(($RANDOM % $remaining))
    echo $(date) $seconds >> $lock
    sleep $seconds
    $cmd && break || remaining=$(($remaining - $seconds))
  done
  rm $lock
}

Reply | Threaded
Open this post in threaded view
|

Re: spamd-setup fails from cron

David Diggles-2
Ok;

After running that a few days, it works fine, but... the interval between updates
is all over the place.

I rewrote it, to only change the sleep value under 2 circumstances:

First time run, or after a failure.

Now it's updating hourly again.

I will not make the same mistake of posting it to the list, because archiving a
possibly buggy script that someone may copy someday is not a great idea.

However I think the methodology is now sound, so write your own or mail me directly
if you want a copy of it to adopt.

On Fri, Jun 01, 2012 at 04:45:24PM +1000, David Diggles wrote:

> > #!/bin/sh
> > remaining=$1;shift
> > cmd=$@
> > lock=/var/run/$(basename $1).lock
> > [ -f $lock ] || {
> >   touch $lock
> >   while [ $remaining -gt 0 ]; do
> >     seconds=$(($RANDOM % $remaining))
> >     echo $(date) $seconds >> $lock
> >     sleep $seconds
> >     $cmd && return || remaining=$(($remaining - $seconds))
> >   done
> >   rm $lock
> > }
> >
>
> *groan*.. another mistake.. I'm such an idiot sometimes ;-)
>
> I don't recommend running this without checking it first.
>
> #!/bin/sh
> remaining=$1;shift
> cmd=$@
> lock=/var/run/$(basename $1).lock
> [ -f $lock ] || {
>   touch $lock
>   while [ $remaining -gt 0 ]; do
>     seconds=$(($RANDOM % $remaining))
>     echo $(date) $seconds >> $lock
>     sleep $seconds
>     $cmd && break || remaining=$(($remaining - $seconds))
>   done
>   rm $lock
> }