OpenBSD 6.4 smtpd local mail delivery missing "From " when .forward (procmail)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenBSD 6.4 smtpd local mail delivery missing "From " when .forward (procmail)

Steve Williams-16
Hi,

I upgraded from OpenBSD 6.3 to OpenBSD 6.4 today.  I upgraded all
packages, switched to php7, etc.

I've been running OpenBSD since 2.7 so this is a very known process.

The upgrade went quite smoothly and is working fine except for my
email.  I have massaged the smtpd.conf file to comply with the OpenBSD
6.4 grammar.

I run a VERY simple smtpd configuration saving in mbox format.

I am also using procmail to direct emails into various folders, launched
with a .forward.  This has been working since about 2005 :),
historically with sendmail and more recently, smtpd.

Unfortunately, email is being written to both my INBOX and the procmail
the folders **incorrectly** post upgrade.

They are all missing the "From " line that is supposed to indicate the
start of a new email message.

It seems like the email is being passed "raw" to procmail without being
processed by "mail.local" ... or that's my interpretation.

In the /var/mail/steve file, I can see the following lines prior to the
upgrade:
 From steve+caf_=steve=[hidden email] Sat
Jan 26 09:52:48 2019
^^^^^^

After the upgrade, I'm not getting those "From " lines which appear to
be added by mail.local(8):
     Individual mail messages in the mailbox are delimited by an empty line
     followed by a line beginning with the string "From ".  A line
containing
     the string "From ", the sender's name and a timestamp is prepended to
     each delivered mail message.

If there is a .forward, does that preclude mail.local from being called?

When I remove the .forward, incoming mail is written correctly to the INBOX.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I'm hoping I'm missing something in my simple smptd.conf file... but for
the life of me, I cannot figure out what it is.

Here is my smtpd.conf file:
-----------------------------------
# System aliases file
table aliases file:/etc/mail/aliases

# Listen on all interfaces as "mail.williamsitconsulting.com"
listen on all hostname "mail.williamsitconsulting.com"

action "local" mbox alias <aliases>

# Incoming mail for the two domains.
match from any for domain "williamsitconsulting.com" action "local"
match from any for domain "williams-steve.com" action "local"

.forward file
----------------
"|/usr/local/bin/procmail"


Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.4 smtpd local mail delivery missing "From " when .forward (procmail)

Steve Williams-16
On 26/01/2019 11:03 p.m., [hidden email] wrote:

> On Sat, Jan 26, 2019 at 08:53:06PM -0700, Steve Williams wrote:
>> Hi,
>>
>> I upgraded from OpenBSD 6.3 to OpenBSD 6.4 today.  I upgraded all packages,
>> switched to php7, etc.
>>
>> I've been running OpenBSD since 2.7 so this is a very known process.
>>
>> The upgrade went quite smoothly and is working fine except for my email.  I
>> have massaged the smtpd.conf file to comply with the OpenBSD 6.4 grammar.
>>
>> I run a VERY simple smtpd configuration saving in mbox format.
>>
>> I am also using procmail to direct emails into various folders, launched
>> with a .forward.  This has been working since about 2005 :), historically
>> with sendmail and more recently, smtpd.
>>
>> Unfortunately, email is being written to both my INBOX and the procmail the
>> folders **incorrectly** post upgrade.
>>
>> They are all missing the "From " line that is supposed to indicate the start
>> of a new email message.
>>
>> It seems like the email is being passed "raw" to procmail without being
>> processed by "mail.local" ... or that's my interpretation.
>>
>> In the /var/mail/steve file, I can see the following lines prior to the
>> upgrade:
>>  From steve+caf_=steve=[hidden email] Sat Jan 26
>> 09:52:48 2019
>> ^^^^^^
>>
>> After the upgrade, I'm not getting those "From " lines which appear to be
>> added by mail.local(8):
>>      Individual mail messages in the mailbox are delimited by an empty line
>>      followed by a line beginning with the string "From ".  A line containing
>>      the string "From ", the sender's name and a timestamp is prepended to
>>      each delivered mail message.
>>
>> If there is a .forward, does that preclude mail.local from being called?
>>
>> When I remove the .forward, incoming mail is written correctly to the INBOX.
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>
>> I'm hoping I'm missing something in my simple smptd.conf file... but for the
>> life of me, I cannot figure out what it is.
>>
>> Here is my smtpd.conf file:
>> -----------------------------------
>> # System aliases file
>> table aliases file:/etc/mail/aliases
>>
>> # Listen on all interfaces as "mail.williamsitconsulting.com"
>> listen on all hostname "mail.williamsitconsulting.com"
>>
>> action "local" mbox alias <aliases>
>>
>> # Incoming mail for the two domains.
>> match from any for domain "williamsitconsulting.com" action "local"
>> match from any for domain "williams-steve.com" action "local"
>>
>> forward file
>> ----------------
>> "|/usr/local/bin/procmail"
>>
>>
> Have you searched the list? I'm certain something very similar has been asked/reported recently.
>
> I'm curious if the following would work:
>
> action "local" mda "/usr/local/bin/procmail" alias <aliases>
>
> good luck,
>
> Edgar
Hi,

<as a side note, your email deathstar.my.domain doesn't seem to resolve :) >

I've searched for a couple of hours on the mail list, google, undeadly,
etc.  I might just have tunnel vision, but I can't seem to find anything
about this.

It's definitely related to the 6.3->6.4 smtpd changes.

I did what you suggested and a quick test reveals that there is no "From
" line pre-pended to the mail when it's delivered to my INBOX.

Here's the end of my /var/mail/steve after a test.  Your's was the last
email in my inbox prior to running the test
-----------------------------------------------------
Have you searched the list? I'm certain something very similar has been
asked/reported recently.

I'm curious if the following would work:

action "local" mda "/usr/local/bin/procmail" alias <aliases>

good luck,

Edgar

*********** This should be a line that starts with "From "....
Return-Path: <[hidden email]>
Delivered-To: [hidden email]
Received: from localhost (williamsitconsulting.com [local])
         by williamsitconsulting.com (OpenSMTPD) with ESMTPA id cbf61bbc
         for <[hidden email]>;
         Sat, 26 Jan 2019 23:31:01 -0700 (MST)
From: Steve Williams <[hidden email]>
Date: Sat, 26 Jan 2019 23:31:01 -0700 (MST)
To: [hidden email]
Subject: Teest  of procmail as mda
Message-ID: <[hidden email]>

test


Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.4 smtpd local mail delivery missing "From " when .forward (procmail)

Gilles Chehade-7
In reply to this post by Steve Williams-16
On Sat, Jan 26, 2019 at 08:53:06PM -0700, Steve Williams wrote:

> Hi,
>
> I upgraded from OpenBSD 6.3 to OpenBSD 6.4 today.?? I upgraded all packages,
> switched to php7, etc.
>
> I've been running OpenBSD since 2.7 so this is a very known process.
>
> The upgrade went quite smoothly and is working fine except for my email.?? I
> have massaged the smtpd.conf file to comply with the OpenBSD 6.4 grammar.
>
> I run a VERY simple smtpd configuration saving in mbox format.
>
> I am also using procmail to direct emails into various folders, launched
> with a .forward.?? This has been working since about 2005 :), historically
> with sendmail and more recently, smtpd.
>
> Unfortunately, email is being written to both my INBOX and the procmail the
> folders **incorrectly** post upgrade.
>
> They are all missing the "From " line that is supposed to indicate the start
> of a new email message.
>
> It seems like the email is being passed "raw" to procmail without being
> processed by "mail.local" ... or that's my interpretation.
>

you need to tweak your procmail's command so it adds the From delimiter,
there's an option for that.

smtpd used to add the From delimiter for mda, which allowed procmail not
to require that option, however this wasn't correct and when we made the
mda improvements between 6.3 and 6.4, it became impossible to accomodate
procmail without introducing special cases and ugly hacks such as having
explicit search for the string 'procmail' in .forward files.

it wasn't worth it when people can just pass procmail an option.

also don't use procmail, it's trash and there are far better options for
you to use today, fdm being the first to come to mind :-)

--
Gilles Chehade       @poolpOrg

https://www.poolp.org                 tip me: https://paypal.me/poolpOrg

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.4 smtpd local mail delivery missing "From " when .forward (procmail)

Edgar Pettijohn III-2
In reply to this post by Steve Williams-16

On Jan 27, 2019 12:34 AM, Steve Williams <[hidden email]> wrote:

>
> On 26/01/2019 11:03 p.m., [hidden email] wrote:
> > On Sat, Jan 26, 2019 at 08:53:06PM -0700, Steve Williams wrote:
> >> Hi,
> >>
> >> I upgraded from OpenBSD 6.3 to OpenBSD 6.4 today.  I upgraded all packages,
> >> switched to php7, etc.
> >>
> >> I've been running OpenBSD since 2.7 so this is a very known process.
> >>
> >> The upgrade went quite smoothly and is working fine except for my email.  I
> >> have massaged the smtpd.conf file to comply with the OpenBSD 6.4 grammar.
> >>
> >> I run a VERY simple smtpd configuration saving in mbox format.
> >>
> >> I am also using procmail to direct emails into various folders, launched
> >> with a .forward.  This has been working since about 2005 :), historically
> >> with sendmail and more recently, smtpd.
> >>
> >> Unfortunately, email is being written to both my INBOX and the procmail the
> >> folders **incorrectly** post upgrade.
> >>
> >> They are all missing the "From " line that is supposed to indicate the start
> >> of a new email message.
> >>
> >> It seems like the email is being passed "raw" to procmail without being
> >> processed by "mail.local" ... or that's my interpretation.
> >>
> >> In the /var/mail/steve file, I can see the following lines prior to the
> >> upgrade:
> >>  From steve+caf_=steve=[hidden email] Sat Jan 26
> >> 09:52:48 2019
> >> ^^^^^^
> >>
> >> After the upgrade, I'm not getting those "From " lines which appear to be
> >> added by mail.local(8):
> >>      Individual mail messages in the mailbox are delimited by an empty line
> >>      followed by a line beginning with the string "From ".  A line containing
> >>      the string "From ", the sender's name and a timestamp is prepended to
> >>      each delivered mail message.
> >>
> >> If there is a .forward, does that preclude mail.local from being called?
> >>
> >> When I remove the .forward, incoming mail is written correctly to the INBOX.
> >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>
> >> I'm hoping I'm missing something in my simple smptd.conf file... but for the
> >> life of me, I cannot figure out what it is.
> >>
> >> Here is my smtpd.conf file:
> >> -----------------------------------
> >> # System aliases file
> >> table aliases file:/etc/mail/aliases
> >>
> >> # Listen on all interfaces as "mail.williamsitconsulting.com"
> >> listen on all hostname "mail.williamsitconsulting.com"
> >>
> >> action "local" mbox alias <aliases>
> >>
> >> # Incoming mail for the two domains.
> >> match from any for domain "williamsitconsulting.com" action "local"
> >> match from any for domain "williams-steve.com" action "local"
> >>
> >> forward file
> >> ----------------
> >> "|/usr/local/bin/procmail"
> >>
> >>
> > Have you searched the list? I'm certain something very similar has been asked/reported recently.
> >
> > I'm curious if the following would work:
> >
> > action "local" mda "/usr/local/bin/procmail" alias <aliases>
> >
> > good luck,
> >
> > Edgar
> Hi,
>
> <as a side note, your email deathstar.my.domain doesn't seem to resolve :) >
>

Sorry. Playing with my mutt config broke it as usual. :)

Thanks for the heads up before too many emails were sent.

Edgar

> I've searched for a couple of hours on the mail list, google, undeadly,
> etc.  I might just have tunnel vision, but I can't seem to find anything
> about this.
>
> It's definitely related to the 6.3->6.4 smtpd changes.
>
> I did what you suggested and a quick test reveals that there is no "From
> " line pre-pended to the mail when it's delivered to my INBOX.
>
> Here's the end of my /var/mail/steve after a test.  Your's was the last
> email in my inbox prior to running the test
> -----------------------------------------------------
> Have you searched the list? I'm certain something very similar has been
> asked/reported recently.
>
> I'm curious if the following would work:
>
> action "local" mda "/usr/local/bin/procmail" alias <aliases>
>
> good luck,
>
> Edgar
>
> *********** This should be a line that starts with "From "....
> Return-Path: <[hidden email]>
> Delivered-To: [hidden email]
> Received: from localhost (williamsitconsulting.com [local])
>         by williamsitconsulting.com (OpenSMTPD) with ESMTPA id cbf61bbc
>         for <[hidden email]>;
>         Sat, 26 Jan 2019 23:31:01 -0700 (MST)
> From: Steve Williams <[hidden email]>
> Date: Sat, 26 Jan 2019 23:31:01 -0700 (MST)
> To: [hidden email]
> Subject: Teest  of procmail as mda
> Message-ID: <[hidden email]>
>
> test
>
>

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.4 smtpd local mail delivery missing "From " when .forward (procmail)

Steve Williams-16
In reply to this post by Gilles Chehade-7
On 27/01/2019 2:21 a.m., Gilles Chehade wrote:

> On Sat, Jan 26, 2019 at 08:53:06PM -0700, Steve Williams wrote:
>> Hi,
>>
>> I upgraded from OpenBSD 6.3 to OpenBSD 6.4 today.?? I upgraded all packages,
>> switched to php7, etc.
>>
>> I've been running OpenBSD since 2.7 so this is a very known process.
>>
>> The upgrade went quite smoothly and is working fine except for my email.?? I
>> have massaged the smtpd.conf file to comply with the OpenBSD 6.4 grammar.
>>
>> I run a VERY simple smtpd configuration saving in mbox format.
>>
>> I am also using procmail to direct emails into various folders, launched
>> with a .forward.?? This has been working since about 2005 :), historically
>> with sendmail and more recently, smtpd.
>>
>> Unfortunately, email is being written to both my INBOX and the procmail the
>> folders **incorrectly** post upgrade.
>>
>> They are all missing the "From " line that is supposed to indicate the start
>> of a new email message.
>>
>> It seems like the email is being passed "raw" to procmail without being
>> processed by "mail.local" ... or that's my interpretation.
>>
> you need to tweak your procmail's command so it adds the From delimiter,
> there's an option for that.
>
> smtpd used to add the From delimiter for mda, which allowed procmail not
> to require that option, however this wasn't correct and when we made the
> mda improvements between 6.3 and 6.4, it became impossible to accomodate
> procmail without introducing special cases and ugly hacks such as having
> explicit search for the string 'procmail' in .forward files.
>
> it wasn't worth it when people can just pass procmail an option.
>
> also don't use procmail, it's trash and there are far better options for
> you to use today, fdm being the first to come to mind :-)
>

Hi Gilles,

Thanks very much for the reply.  I had looked in for some kind of flag
that could be set in the procmail config file... I never thought it
would be a command line argument to procmail :(  I cannot believe I
missed that.

I had even downloaded the smtpd source and saw that mail.local could
only write to a file (not a stream)... and that's the only place in the
source code where a "From " was written so I knew it wasn't going to be
a smtpd configuration change.

Do you think a hint about the behaviour change might be worth a mention
in the smptd.conf (5) man page?  I've been trying to come up with some
non-procmail specific wording...I am not happy with the following
wording, but something along these lines to give a clue that piping to a
program won't have the same effect as writing to a physical file.

              mbox    Deliver the message to the user's mbox with
                      mail.local(8).  mbox format is only honoured if
final delivery is a file and not a program.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Thanks for the heads up about fdm.  I'll have a look at it.

Also, thanks for such an amazingly simple email program to configure.  I
have spent so many hours over the years researching (scratching my head)
how to configure sendmail!  A 4 line config file to have a functioning
email configuration is pretty staggering!

Cheers,
Steve W.

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.4 smtpd local mail delivery missing "From " when .forward (procmail)

Gilles Chehade-7
On Sun, Jan 27, 2019 at 12:42:23PM -0700, Steve Williams wrote:

>
> Hi Gilles,
>
> Thanks very much for the reply.?? I had looked in for some kind of flag that
> could be set in the procmail config file... I never thought it would be a
> command line argument to procmail :(?? I cannot believe I missed that.
>
> I had even downloaded the smtpd source and saw that mail.local could only
> write to a file (not a stream)... and that's the only place in the source
> code where a "From " was written so I knew it wasn't going to be a smtpd
> configuration change.
>
> Do you think a hint about the behaviour change might be worth a mention in
> the smptd.conf (5) man page??? I've been trying to come up with some
> non-procmail specific wording...I am not happy with the following wording,
> but something along these lines to give a clue that piping to a program
> won't have the same effect as writing to a physical file.
>
> ???????????????????????? mbox?????? Deliver the message to the user's mbox with
> ???????????????????????????????????????? mail.local(8).?? mbox format is
> only honoured if final delivery is a file and not a program.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>

this isn't accurate, smtpd(8) doesn't write to a file.

smtpd doesn't know about an "mbox format", what it does is that it calls
mail.local(8) which is an mda that should really be called mail.mbox and
which itself writes to a file in mbox format.

I don't think its necessary to document this behaviour in smtpd.conf(5),
procmail is the only case I know where you may forget to pass the option
and be bit, which leads to threads like this where i'm given a chance to
discourage you from using procmail.
k

> Thanks for the heads up about fdm.?? I'll have a look at it.
>
> Also, thanks for such an amazingly simple email program to configure.?? I
> have spent so many hours over the years researching (scratching my head) how
> to configure sendmail!?? A 4 line config file to have a functioning email
> configuration is pretty staggering!
>

Cheers :-)


--
Gilles Chehade       @poolpOrg

https://www.poolp.org                 tip me: https://paypal.me/poolpOrg