mail.local(8) corrupts messages where `>From ` follows blank line

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

mail.local(8) corrupts messages where `>From ` follows blank line

Demi M. Obenour
>Synopsis: mail.local(8) corrupts messages where `>From ` follows blank line
>Category: system
>Environment:
        System      : OpenBSD 6.6
        Details     : OpenBSD 6.6 (GENERIC.MP) #5: Sun Feb 16 01:56:11 MST 2020
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:
        If a message contains a blank line followed by a line matching the
        regex `>+From `, mail.local(8) fails to escape the second line.  This
        makes it impossible to distinguish the case where a blank line is
        followed by a line starting with `From ` and where a blank line is
        followed by a line starting with `>From `.
>How-To-Repeat:
        Send a message to oneself containing:

        ```
        some text here

        >From this should be escaped but is not
        ```
>Fix:
        mail.local(8) needs to consider lines that match `^>*From ` to need
        escaping if they follow a blank line, not just lines that start with
        `From `.


dmesg:
OpenBSD 6.6 (GENERIC.MP) #5: Sun Feb 16 01:56:11 MST 2020
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 385871872 (367MB)
avail mem = 361558016 (344MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfc001000 (12 entries)
bios0: vendor Xen version "4.8.5-14.fc25" date 12/11/2019
bios0: Xen HVM domU
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC HPET WAET SSDT SSDT
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 48 pins, remapped
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, 2808.62 MHz, 06-9e-09
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MPX,RDSEED,ADX,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 103MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, 2892.46 MHz, 06-9e-09
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MPX,RDSEED,ADX,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
acpihpet0 at acpi0: 62500000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
cpu0: using VERW MDS workaround (except on vmm entry)
pvbus0 at mainbus0: Xen 4.8
xen0 at pvbus0: features 0x2705, 32 grant table frames, event channel 1
xbf0 at xen0 backend 0 channel 6: disk
scsibus1 at xbf0: 2 targets
sd0 at scsibus1 targ 0 lun 0: <Xen, phy xvda 51712, 0000>
sd0: 10240MB, 512 bytes/sector, 20971520 sectors
xbf1 at xen0 backend 0 channel 7: disk
scsibus2 at xbf1: 2 targets
sd1 at scsibus2 targ 0 lun 0: <Xen, phy xvdb 51728, 0000>
sd1: 2048MB, 512 bytes/sector, 4194304 sectors
xbf2 at xen0 backend 0 channel 8: disk
scsibus3 at xbf2: 2 targets
sd2 at scsibus3 targ 0 lun 0: <Xen, phy xvdc 51744, 0000>
sd2: 10240MB, 512 bytes/sector, 20971520 sectors
xnf0 at xen0 backend 8 channel 9: address 00:16:3e:5e:6c:00
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: SMBus disabled
xspd0 at pci0 dev 2 function 0 "XenSource Platform Device" rev 0x01
vga1 at pci0 dev 3 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ehci0 at pci0 dev 4 function 0 "Intel 82801DB USB" rev 0x10: apic 1 int 35
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
root on sd0a (1869297ee7933887.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 1: density unknown

usbdevs:
Controller /dev/usb0:
addr 01: 8086:0000 Intel, EHCI root hub
         high speed, self powered, config 1, rev 1.00
         driver: uhub0
addr 02: 0627:0001 QEMU, QEMU USB Tablet
         high speed, power 100 mA, config 1, rev 0.00, iSerial 42
         driver: uhidev0

Reply | Threaded
Open this post in threaded view
|

Re: [Released] mail.local(8) corrupts messages where `>From ` follows blank line

guenther
On Thu, 5 Mar 2020, [hidden email] wrote:
> >Synopsis:  mail.local(8) corrupts messages where `>From ` follows blank
...

> >Description:
>         If a message contains a blank line followed by a line matching the
> regex `>+From `, mail.local(8) fails to escape the second line.  This
> makes it impossible to distinguish the case where a blank line is
> followed by a line starting with `From ` and where a blank line is
> followed by a line starting with `>From `.
> >How-To-Repeat:
> Send a message to oneself containing:
>
>         ```
>         some text here
>
>         >From this should be escaped but is not
>         ```
> >Fix:
> mail.local(8) needs to consider lines that match `^>*From ` to need
> escaping if they follow a blank line, not just lines that start with
> `From `.


Nope.  The unix mailbox format** does not reversibly "quote" a "From " but
rather irreversibly mangles them.  A handful of tools have tried changing
the interpretation but that's merely made the situation worse by making
behaviors inconsistent and mangling a different set of messages.

If you need a format that doesn't mangle this sort of content, then Don't
Use UNIX mailbox format.


Philip Guenther
Unix Email Greybeard

** it's sometimes called the BSD format, but it predates BSD

Reply | Threaded
Open this post in threaded view
|

Re: [Released] mail.local(8) corrupts messages where `>From ` follows blank line

Demi M. Obenour
On 2020-03-05 20:27, Philip Guenther wrote:
> Nope.  The unix mailbox format** does not reversibly "quote" a "From " but
> rather irreversibly mangles them.  A handful of tools have tried changing
> the interpretation but that's merely made the situation worse by making
> behaviors inconsistent and mangling a different set of messages.
>
> If you need a format that doesn't mangle this sort of content, then Don't
> Use UNIX mailbox format.

In that case, `mbox` should not be the default configuration, and
`mail(1)` et al should support alternatives.

Unless I am missing something, there are no tools that support reading
from maildir in the base system.  Furthermore, there are variations
of mbox that do not mangle *any* messages, since they use reversible
quoting.  Proper use of MIME encoding also avoids the problem, but
none of the tools in the base system support decoding it, and none
of the programs in the base system that send mail use it (another bug).

I don’t particularly care which of the various variations on mbox
is used, provided that mail(1) and mail.local(8) are consistent with
each other.

Sincerely,

Demi


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Released] mail.local(8) corrupts messages where `>From ` follows blank line

Theo de Raadt-2
Demi M. Obenour <[hidden email]> wrote:

> On 2020-03-05 20:27, Philip Guenther wrote:
> > Nope.  The unix mailbox format** does not reversibly "quote" a "From " but
> > rather irreversibly mangles them.  A handful of tools have tried changing
> > the interpretation but that's merely made the situation worse by making
> > behaviors inconsistent and mangling a different set of messages.
> >
> > If you need a format that doesn't mangle this sort of content, then Don't
> > Use UNIX mailbox format.
>
> In that case, `mbox` should not be the default configuration, and
> `mail(1)` et al should support alternatives.

Did you submit a diff which starts us in that direction?  Or was it
deleted by the mailing list?

> Unless I am missing something, there are no tools that support reading
> from maildir in the base system.  Furthermore, there are variations
> of mbox that do not mangle *any* messages, since they use reversible
> quoting.  Proper use of MIME encoding also avoids the problem, but
> none of the tools in the base system support decoding it, and none
> of the programs in the base system that send mail use it (another bug).
>
> I don’t particularly care which of the various variations on mbox
> is used, provided that mail(1) and mail.local(8) are consistent with
> each other.

Saying so doesn't make it happen.  *ALL* the pieces must be prepared,
then it can change.  If everything isn't ready, then why would we not
stick to the status quo?  In my view, this issue is small compared to
some incoherent mailfile here maildir there and shrug situation.  It
sounds like you have a plan.  Want to execute *ALL OF IT*?

Reply | Threaded
Open this post in threaded view
|

Re: [Released] mail.local(8) corrupts messages where `>From ` follows blank line

Theo de Raadt-2
Demi M. Obenour <[hidden email]> wrote:

> On 2020-03-05 20:58, Theo de Raadt wrote:
> > Saying so doesn't make it happen.  *ALL* the pieces must be prepared,
> > then it can change.  If everything isn't ready, then why would we not
> > stick to the status quo?  In my view, this issue is small compared to
> > some incoherent mailfile here maildir there and shrug situation.  It
> > sounds like you have a plan.  Want to execute *ALL OF IT*?
>
> Adding a flag to mail.local(8) to reversibly quote messages is trivial;
> I can prepare a diff when I have time.  Adding a command to mail(1)
> to undo this quoting should not be hard either.

You are proposing a needless incompatibility for what reason?

> That said, my research indicates that Thunderbird (at least) uses
> quoted-printable encoding to avoid problems with the existing mbox
> format, and mutt can be told to do so.  Furthermore, at least mutt
> would break if this change was made.  Therefore, this flag should be
> disabled by default.

So you want to add something that can't be used?

Grea.t....