smtpd - help needed tranlsating to new virtual map syntax

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

smtpd - help needed tranlsating to new virtual map syntax

Adam Thompson
[Cross-posting here before I give up and switch to Postfix  -Adam]


I have an old instance that uses smtpd's virtual <vmap> to rewrite *sender* addresses.
Reading the 6.4-STABLE version of the smtpd.conf(5) manpage, I can't see how to accomplish my goal any more - it looks impossible.

I don't want to upgrade a working mail relay server to something that might be broken, so I'm seeking assistance first.

The purpose of this system is purely to relay mail from internal, semi-broken-ish systems out to our Office365 tenant, but I need to clean up bogus MAIL FROM / "From:" headers first, lest they be flagged as spam.

In general, I think I'm asking how to use virtual <map> with the "relay" action in the new syntax - the manual tells me this is impossible!?

Thanks,
-Adam


Old smtpd.conf:

===start===
listen on 0.0.0.0
listen on ::
table aliases db:/etc/opensmtpd/aliases.db
table vmap db:/etc/opensmtpd/vmap.db
table localnets { 192.168.10.0/24, 192.168.100.0/24, 192.168.157.0/24, 192.168.158.0/24, 192.168.101.0/24, 10.158.0.0/16 }
accept from local                 for any                relay via smtp://XXX-ca.mail.protection.outlook.com hostname remote.XXX.ca
accept from source 192.168.158.63 for domain 192.168.158.63 virtual <vmap> deliver to lmtp localhost:25
accept from source 192.168.100.63 for domain 192.168.100.63 virtual <vmap> deliver to lmtp localhost:25
accept from source 192.168.158.63 for any                relay via smtp://XXX-ca.mail.protection.outlook.com as [hidden email] hostname
remote.XXX.ca
accept from source 192.168.100.63 for any                relay via smtp://XXX-ca.mail.protection.outlook.com as [hidden email] hostname
remote.XXX.ca
accept from source <localnets>    for any                relay via smtp://XXX-ca.mail.protection.outlook.com hostname remote.XXX.ca
===end===

old vmap:

===start===
ilom-alert@192.168.100.63:      [hidden email]
[hidden email]:   [hidden email]
[hidden email]: [hidden email]
[hidden email]: [hidden email]
===end===


Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax

Edgar Pettijohn III-2
It would be helpful if you show what you have tried.

Should be as simple as:

action "relay-01" lmtp /var/run/lmtp.sock virtual <vmap>

match from src <localnets> action "relay-01"

Edgar
On Jan 16, 2019 7:37 AM, Adam Thompson <[hidden email]> wrote:

>
> [Cross-posting here before I give up and switch to Postfix  -Adam]
>
>
> I have an old instance that uses smtpd's virtual <vmap> to rewrite *sender* addresses.
> Reading the 6.4-STABLE version of the smtpd.conf(5) manpage, I can't see how to accomplish my goal any more - it looks impossible.
>
> I don't want to upgrade a working mail relay server to something that might be broken, so I'm seeking assistance first.
>
> The purpose of this system is purely to relay mail from internal, semi-broken-ish systems out to our Office365 tenant, but I need to clean up bogus MAIL FROM / "From:" headers first, lest they be flagged as spam.
>
> In general, I think I'm asking how to use virtual <map> with the "relay" action in the new syntax - the manual tells me this is impossible!?
>
> Thanks,
> -Adam
>
>
> Old smtpd.conf:
>
> ===start===
> listen on 0.0.0.0
> listen on ::
> table aliases db:/etc/opensmtpd/aliases.db
> table vmap db:/etc/opensmtpd/vmap.db
> table localnets { 192.168.10.0/24, 192.168.100.0/24, 192.168.157.0/24, 192.168.158.0/24, 192.168.101.0/24, 10.158.0.0/16 }
> accept from local                 for any                relay via smtp://XXX-ca.mail.protection.outlook.com hostname remote.XXX.ca
> accept from source 192.168.158.63 for domain 192.168.158.63 virtual <vmap> deliver to lmtp localhost:25
> accept from source 192.168.100.63 for domain 192.168.100.63 virtual <vmap> deliver to lmtp localhost:25
> accept from source 192.168.158.63 for any                relay via smtp://XXX-ca.mail.protection.outlook.com as [hidden email] hostname
> remote.XXX.ca
> accept from source 192.168.100.63 for any                relay via smtp://XXX-ca.mail.protection.outlook.com as [hidden email] hostname
> remote.XXX.ca
> accept from source <localnets>    for any                relay via smtp://XXX-ca.mail.protection.outlook.com hostname remote.XXX.ca
> ===end===
>
> old vmap:
>
> ===start===
> ilom-alert@192.168.100.63:      [hidden email]
> [hidden email]:   [hidden email]
> [hidden email]: [hidden email]
> [hidden email]: [hidden email]
> ===end===
>
>

Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax

Adam Thompson
In reply to this post by Adam Thompson
As I said, I haven't tried anything yet as I don't want to break a working system, and I don't have a good way to test this in parallel right now.

The manpage says "The local delivery methods support additional options: [...] virtual" without specifying which delivery methods are "local".  My assumption was that only "mbox" and "mda" were local, as lmtp can, and often does, point to another server.

Some brief experiments with a VM only got me syntax errors, so I didn't pursue that very thoroughly before asking for clarification.

-Adam

-----Original Message-----
From: Edgar Pettijohn <[hidden email]>
Sent: Wednesday, January 16, 2019 8:12 AM
To: Adam Thompson <[hidden email]>; [hidden email]
Subject: Re: smtpd - help needed tranlsating to new virtual map syntax

It would be helpful if you show what you have tried.

Should be as simple as:

action "relay-01" lmtp /var/run/lmtp.sock virtual <vmap>

match from src <localnets> action "relay-01"

Edgar
On Jan 16, 2019 7:37 AM, Adam Thompson <[hidden email]> wrote:

>
> [Cross-posting here before I give up and switch to Postfix  -Adam]
>
>
> I have an old instance that uses smtpd's virtual <vmap> to rewrite *sender* addresses.
> Reading the 6.4-STABLE version of the smtpd.conf(5) manpage, I can't see how to accomplish my goal any more - it looks impossible.
>
> I don't want to upgrade a working mail relay server to something that might be broken, so I'm seeking assistance first.
>
> The purpose of this system is purely to relay mail from internal, semi-broken-ish systems out to our Office365 tenant, but I need to clean up bogus MAIL FROM / "From:" headers first, lest they be flagged as spam.
>
> In general, I think I'm asking how to use virtual <map> with the "relay" action in the new syntax - the manual tells me this is impossible!?
>
> Thanks,
> -Adam
>
>
> Old smtpd.conf:
>
> ===start===
> listen on 0.0.0.0
> listen on ::
> table aliases db:/etc/opensmtpd/aliases.db table vmap
> db:/etc/opensmtpd/vmap.db table localnets { 192.168.10.0/24,
> 192.168.100.0/24, 192.168.157.0/24, 192.168.158.0/24,
> 192.168.101.0/24, 10.158.0.0/16 } accept from local                
> for any                relay via
> smtp://XXX-ca.mail.protection.outlook.com hostname remote.XXX.ca
> accept from source 192.168.158.63 for domain 192.168.158.63 virtual
> <vmap> deliver to lmtp localhost:25 accept from source 192.168.100.63
> for domain 192.168.100.63 virtual <vmap> deliver to lmtp localhost:25
> accept from source 192.168.158.63 for any                relay via
> smtp://XXX-ca.mail.protection.outlook.com as [hidden email] hostname
> remote.XXX.ca accept from source 192.168.100.63 for any                
> relay via smtp://XXX-ca.mail.protection.outlook.com as [hidden email]
> hostname remote.XXX.ca accept from source <localnets>    for any                
> relay via smtp://XXX-ca.mail.protection.outlook.com hostname
> remote.XXX.ca ===end===
>
> old vmap:
>
> ===start===
> ilom-alert@192.168.100.63:      [hidden email]
> [hidden email]:   [hidden email]
> [hidden email]: [hidden email]
> [hidden email]: [hidden email]
> ===end===
>
>

Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax

Adam Thompson
As it turns out, no, that doesn't work.
Trying to fix up broken sender mail domain-parts only simply gets me a "5.2.4 Mailing list expansion problem" error, with no debug output to suggest why.

In this test case, my translations map had:

        @bad.athompso.net @good.athompso.net

in it.  Obviously, this is a test setup :).
Smtpd.conf itself consisted of:

        listen on all received-auth
        smtp max-message-size 100M
        table translations file:/etc/mail/translations          # ORIG->NEW mappings
        table allowed-hosts file:/etc/mail/allowed-hosts        # Who can connect?  (bare IP addresses or CIDR subnets)
        action translate lmtp "/var/run/lmtp.sock" virtual <translations>       # 1st pass on allowed rewrite mail
        action forward forward-only                                             # and now it's not our problem anymore
        match for any from local action forward                 # 2nd pass for reinjected mail, this time just forward it
        match for any from src <allowed-hosts> action translate # inbound mail - hand it to LMTP, translating as we go

A cursory glance at the source code (yikes, it's been a long time since I was a programmer) suggests that virtual now only works on recipients, not senders.  Which is too bad for me, as that means I'll have to switch at least one box to use Postfix.

I'm not convinced the new smtpd.conf grammar improves anything at all, but I assume it must help someone or it wouldn't have changed... but I believe my use case got thrown out with the bathwater, so to speak.  Oh, well.  :-(

(If anyone cares, the bad sender addresses are mostly alerts coming from older Sun ALOMs and at least one Lexmark printer that also sends email with broken From addresses.)

-Adam

 
-----Original Message-----
From: [hidden email] <[hidden email]> On Behalf Of Adam Thompson
Sent: Wednesday, January 16, 2019 8:26 AM
To: 'Edgar Pettijohn' <[hidden email]>; [hidden email]
Subject: Re: smtpd - help needed tranlsating to new virtual map syntax

As I said, I haven't tried anything yet as I don't want to break a working system, and I don't have a good way to test this in parallel right now.

The manpage says "The local delivery methods support additional options: [...] virtual" without specifying which delivery methods are "local".  My assumption was that only "mbox" and "mda" were local, as lmtp can, and often does, point to another server.

Some brief experiments with a VM only got me syntax errors, so I didn't pursue that very thoroughly before asking for clarification.

-Adam

-----Original Message-----
From: Edgar Pettijohn <[hidden email]>
Sent: Wednesday, January 16, 2019 8:12 AM
To: Adam Thompson <[hidden email]>; [hidden email]
Subject: Re: smtpd - help needed tranlsating to new virtual map syntax

It would be helpful if you show what you have tried.

Should be as simple as:

action "relay-01" lmtp /var/run/lmtp.sock virtual <vmap>

match from src <localnets> action "relay-01"

Edgar
On Jan 16, 2019 7:37 AM, Adam Thompson <[hidden email]> wrote:

>
> [Cross-posting here before I give up and switch to Postfix  -Adam]
>
>
> I have an old instance that uses smtpd's virtual <vmap> to rewrite *sender* addresses.
> Reading the 6.4-STABLE version of the smtpd.conf(5) manpage, I can't see how to accomplish my goal any more - it looks impossible.
>
> I don't want to upgrade a working mail relay server to something that might be broken, so I'm seeking assistance first.
>
> The purpose of this system is purely to relay mail from internal, semi-broken-ish systems out to our Office365 tenant, but I need to clean up bogus MAIL FROM / "From:" headers first, lest they be flagged as spam.
>
> In general, I think I'm asking how to use virtual <map> with the "relay" action in the new syntax - the manual tells me this is impossible!?
>
> Thanks,
> -Adam
>
>
> Old smtpd.conf:
>
> ===start===
> listen on 0.0.0.0
> listen on ::
> table aliases db:/etc/opensmtpd/aliases.db table vmap
> db:/etc/opensmtpd/vmap.db table localnets { 192.168.10.0/24,
> 192.168.100.0/24, 192.168.157.0/24, 192.168.158.0/24,
> 192.168.101.0/24, 10.158.0.0/16 } accept from local                
> for any                relay via
> smtp://XXX-ca.mail.protection.outlook.com hostname remote.XXX.ca
> accept from source 192.168.158.63 for domain 192.168.158.63 virtual
> <vmap> deliver to lmtp localhost:25 accept from source 192.168.100.63
> for domain 192.168.100.63 virtual <vmap> deliver to lmtp localhost:25
> accept from source 192.168.158.63 for any                relay via
> smtp://XXX-ca.mail.protection.outlook.com as [hidden email] hostname
> remote.XXX.ca accept from source 192.168.100.63 for any                
> relay via smtp://XXX-ca.mail.protection.outlook.com as [hidden email]
> hostname remote.XXX.ca accept from source <localnets>    for any                
> relay via smtp://XXX-ca.mail.protection.outlook.com hostname
> remote.XXX.ca ===end===
>
> old vmap:
>
> ===start===
> ilom-alert@192.168.100.63:      [hidden email]
> [hidden email]:   [hidden email]
> [hidden email]: [hidden email]
> [hidden email]: [hidden email]
> ===end===
>
>


Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax

Adam Thompson
In reply to this post by Adam Thompson
I found the "-T" (trace) flag to smtpd(8), and it gives me this, which AFAICT confirms my suspicions:
        [...]
        rule #2 matched: match from src allowed-hosts for any => translate
        lookup: lookup "[hidden email]" as ALIAS in table static:translations -> 0
        lookup: lookup "athompso" as ALIAS in table static:translations -> 0
        lookup: lookup "@athompso.net" as ALIAS in table static:translations -> 0
        lookup: lookup "@" as ALIAS in table static:translations -> 0
        expand: lka_expand: no aliases for virtual
        mproc: lka -> pony : 53 IMSG_SMTP_EXPAND_RCPT
        expand: 0x154201b89018: clearing expand tree
        imsg: pony <- lka: IMSG_SMTP_EXPAND_RCPT (len=53)
        smtp: 0x1127a72e6000: >>> 524 5.2.4 Mailing list expansion problem
        [...]

complete output attached below, I've changed to @old.athompso.net and @new.athompso.net during testing since the last email.

On the sending side, from another host (listed in <allowed-hosts>), I'm running:
        swaks --to [hidden email] --from [hidden email] --server XXXX
which faithfully reports the 5.2.4 error.

I'm slightly disappointed, I still like OpenSMTPd's concise configuration syntax.  Postfix could still rewrite source addresses last time I checked, I hope it's still there - I do NOT want to run sendmail, thank you very much.

-Adam



Smtpd(8) trace output including invocation:
===from here to end of message===
bhs# smtpd -d -T all -v -d
debug: init ssl-tree
debug: init ca-tree
debug: init ssl-tree
debug: using "fs" queue backend
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
info: OpenSMTPD 6.4.0 starting
debug: init ssl-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ca-tree
debug: init ca-tree
debug: init ca-tree
debug: init ca-tree
debug: init ca-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ssl-tree
debug: init ca-tree
debug: using "fs" queue backend
debug: using "fs" queue backend
debug: using "fs" queue backend
debug: using "fs" queue backend
debug: using "fs" queue backend
debug: init ssl-tree
debug: using "ramqueue" scheduler backend
debug: using "ramqueue" scheduler backend
debug: using "ramqueue" scheduler backend
debug: using "ramqueue" scheduler backend
debug: using "ramqueue" scheduler backend
debug: using "fs" queue backend
debug: using "ram" stat backend
setup_peer: klondike -> control[63932] fd=4
setup_peer: klondike -> pony express[40666] fd=5
setup_done: ca[35575] done
debug: using "ram" stat backend
setup_peer: control -> klondike[35575] fd=4
setup_peer: control -> lookup[49698] fd=5
setup_peer: control -> pony express[40666] fd=6
setup_peer: control -> queue[21502] fd=7
setup_peer: control -> scheduler[14152] fd=8
setup_done: control[63932] done
debug: using "ram" stat backend
setup_peer: lookup -> control[63932] fd=4
setup_peer: lookup -> pony express[40666] fd=5
setup_peer: lookup -> queue[21502] fd=6
setup_done: lka[49698] done
debug: using "ram" stat backend
setup_peer: pony express -> control[63932] fd=4
setup_peer: pony express -> klondike[35575] fd=5
setup_peer: pony express -> lookup[49698] fd=6
setup_peer: pony express -> queue[21502] fd=7
setup_done: pony[40666] done
debug: using "ram" stat backend
setup_peer: queue -> control[63932] fd=4
setup_peer: queue -> pony express[40666] fd=5
setup_peer: queue -> lookup[49698] fd=6
setup_peer: queue -> scheduler[14152] fd=7
setup_done: queue[21502] done
debug: using "ramqueue" scheduler backend
debug: using "ram" stat backend
setup_peer: scheduler -> control[63932] fd=4
setup_peer: scheduler -> queue[21502] fd=5
setup_done: scheduler[14152] done
smtpd: setup done
mproc: parent -> control: enabled
mproc: parent -> lka: enabled
mproc: parent -> queue: enabled
mproc: parent -> ca: enabled
mproc: parent -> pony: enabled
debug: parent_send_config_ruleset: reloading
mproc: parent -> lka : 0 IMSG_CONF_START
mproc: parent -> lka : 0 IMSG_CONF_END
debug: parent_send_config: configuring pony process
mproc: parent -> pony : 0 IMSG_CONF_START
mproc: parent -> pony : 0 IMSG_CONF_END
debug: parent_send_config: configuring ca process
mproc: parent -> ca : 0 IMSG_CONF_START
mproc: parent -> ca : 0 IMSG_CONF_END
setup_proc: klondike done
setup_proc: control done
setup_proc: lookup done
setup_proc: pony express done
setup_proc: queue done
setup_proc: scheduler done
mproc: ca -> control: enabled
debug: bounce warning after 4h
mproc: ca -> parent: enabled
mproc: ca -> pony: enabled
mproc: ca -> pony: disabled
mproc: pony -> parent: enabled
mproc: scheduler -> control: enabled
imsg: ca <- parent: IMSG_CONF_START (len=0)
mproc: lka -> parent: enabled
mproc: pony -> queue: enabled
mproc: scheduler -> queue: enabled
scheduler: getting batch: mask=0x3f, count=10
debug: /--- ramqueue: scheduler_ram_batch()
debug: \---
scheduler: got r=0, delay=-1, count=10
scheduler: sleeping
imsg: ca <- parent: IMSG_CONF_END (len=0)
debug: init private ssl-tree
mproc: ca -> pony: enabled
ramstat: init
ramstat: set: uptime
ramstat: uptime: n/a -> n/a
mproc: lka -> queue: enabled
mproc: lka -> control: enabled
mproc: lka -> pony: enabled
mproc: lka -> pony: disabled
imsg: lka <- parent: IMSG_CONF_START (len=0)
imsg: lka <- parent: IMSG_CONF_END (len=0)
TABLE "<anydestination>" type=LIST config=""
        "*"
        "bhs"
TABLE "<anyhost>" type= config=""
        "0.0.0.0/0"
        "::/0"
        "local"
TABLE "<getpwnam>" type=DYNAMIC config=""
TABLE "<localhost>" type= config=""
        "127.0.0.1"
        "144.217.89.103"
        "ipv6:2607:5300:201:3100::6893"
        "ipv6:::1"
        "ipv6:fe80::1%lo0"
        "ipv6:fe80::ecd0:b4f4:9b5:1638%vio0"
        "local"
TABLE "<localnames>" type=LIST config=""
        "127.0.0.1"
        "144.217.89.103"
        "[127.0.0.1]"
        "[144.217.89.103]"
        "[ipv6:2607:5300:201:3100::6893]"
        "[ipv6:::1]"
        "[ipv6:fe80::1%lo0]"
        "[ipv6:fe80::ecd0:b4f4:9b5:1638%vio0]"
        "[ipv6:ipv6:2607:5300:201:3100::6893]"
        "[ipv6:ipv6:::1]"
        "[ipv6:ipv6:fe80::1%lo0]"
        "[ipv6:ipv6:fe80::ecd0:b4f4:9b5:1638%vio0]"
        "bhs.header-rewrite.net"
        "ipv6:2607:5300:201:3100::6893"
        "ipv6:::1"
        "ipv6:fe80::1%lo0"
        "ipv6:fe80::ecd0:b4f4:9b5:1638%vio0"
        "localhost"
TABLE "allowed-hosts" type=LIST config="/etc/mail/allowed-hosts"
        "204.16.144.114"
        "2607:5300:201:2100::9a6"
        "54.39.183.132"
        "54.39.85.16"
        "home.athompso.net"
        "mail.athompso.net"
TABLE "translations" type=HASH config="/etc/mail/translations"
        "@old.athompso.net" -> "new.athompso.net"
mproc: lka -> pony: enabled
mproc: pony -> lka: enabled
mproc: pony -> control: enabled
mproc: pony -> ca: enabled
debug: ca_engine_init: using RSA privsep engine
imsg: pony <- parent: IMSG_CONF_START (len=0)
imsg: pony <- parent: IMSG_CONF_END (len=0)
debug: smtp: listen on IPv6:::1 port 25 flags 0xc00 pki "" ca ""
debug: smtp: listen on IPv6:fe80::1%lo0 port 25 flags 0xc00 pki "" ca ""
debug: smtp: listen on 127.0.0.1 port 25 flags 0xc00 pki "" ca ""
debug: smtp: listen on 144.217.89.103 port 25 flags 0xc00 pki "" ca ""
debug: smtp: listen on IPv6:fe80::ecd0:b4f4:9b5:1638%vio0 port 25 flags 0xc00 pki "" ca ""
debug: smtp: listen on IPv6:2607:5300:201:3100::6893 port 25 flags 0xc00 pki "" ca ""
debug: smtp: will accept at most 499 clients
mproc: control -> scheduler: enabled
mproc: control -> queue: enabled
mproc: control -> parent: enabled
mproc: control -> lka: enabled
mproc: control -> pony: enabled
mproc: control -> ca: enabled
queue-backend: queue_init(1) -> 1
mproc: queue -> parent: enabled
mproc: queue -> control: enabled
mproc: queue -> lka: enabled
mproc: queue -> scheduler: enabled
mproc: queue -> pony: enabled
queue-backend: queue_envelope_walk() -> -1 (0000000000000000)
debug: queue: done loading queue into scheduler
debug: smtpd: scanning offline queue...
debug: smtpd: offline scanning done






smtp: 0x1127a72e6000: connected to listener 0x112862dcc000 [hostname=bhs.header-rewrite.net, port=25, tag=]
mproc: pony -> lka: realloc 0 -> 128
mproc: pony -> lka : 28 IMSG_GETNAMEINFO
mproc: pony -> control: realloc 0 -> 128
mproc: pony -> control : 45 IMSG_STAT_INCREMENT
mproc: pony -> control : 51 IMSG_STAT_INCREMENT
imsg: lka <- pony: IMSG_GETNAMEINFO (len=28)
imsg: control <- pony: IMSG_STAT_INCREMENT (len=45)
ramstat: increment: smtp.session
mproc: lka -> pony: realloc 0 -> 128
mproc: lka -> pony : 31 IMSG_GETNAMEINFO
imsg: pony <- lka: IMSG_GETNAMEINFO (len=31)
smtp: 0x1127a72e6000: STATE_NEW -> STATE_CONNECTED
d3a5ff63e6a0e04e smtp connected address=54.39.183.132 host=mail.athompso.net
smtp: 0x1127a72e6000: >>> 220 bhs.header-rewrite.net ESMTP OpenSMTPD
smtp: 0x1127a72e6000: IO_LOWAT <io:0x11284ad86400 fd=15 to=300000 fl=W ib=0 ob=0>
ramstat: smtp.session (0x1d3c3a072500): 0 -> 1
imsg: control <- pony: IMSG_STAT_INCREMENT (len=51)
ramstat: increment: smtp.session.inet4
ramstat: smtp.session.inet4 (0x1d3c616c4640): 0 -> 1
smtp: 0x1127a72e6000: IO_DATAIN <io:0x11284ad86400 fd=15 to=300000 fl=R ib=24 ob=0>
smtp: 0x1127a72e6000: <<< EHLO mail.athompso.net
smtp: 0x1127a72e6000: STATE_CONNECTED -> STATE_HELO
smtp: 0x1127a72e6000: >>> 250-bhs.header-rewrite.net Hello mail.athompso.net [54.39.183.132], pleased to meet you
smtp: 0x1127a72e6000: >>> 250-8BITMIME
smtp: 0x1127a72e6000: >>> 250-ENHANCEDSTATUSCODES
smtp: 0x1127a72e6000: >>> 250-SIZE 104857600
smtp: 0x1127a72e6000: >>> 250-DSN
smtp: 0x1127a72e6000: >>> 250 HELP
smtp: 0x1127a72e6000: IO_LOWAT <io:0x11284ad86400 fd=15 to=300000 fl=W ib=0 ob=0>
smtp: 0x1127a72e6000: IO_DATAIN <io:0x11284ad86400 fd=15 to=300000 fl=R ib=39 ob=0>
smtp: 0x1127a72e6000: <<< MAIL FROM:<[hidden email]>
mproc: pony -> queue: realloc 0 -> 128
mproc: pony -> queue : 8 IMSG_SMTP_MESSAGE_CREATE
imsg: queue <- pony: IMSG_SMTP_MESSAGE_CREATE (len=8)
queue-backend: queue_message_create() -> 1 (d7bf0ff6)
mproc: queue -> pony: realloc 0 -> 128
mproc: queue -> pony : 16 IMSG_SMTP_MESSAGE_CREATE
imsg: pony <- queue: IMSG_SMTP_MESSAGE_CREATE (len=16)
smtp: 0x1127a72e6000: >>> 250 2.0.0: Ok
smtp: 0x1127a72e6000: IO_LOWAT <io:0x11284ad86400 fd=15 to=300000 fl=W ib=0 ob=0>
smtp: 0x1127a72e6000: IO_DATAIN <io:0x11284ad86400 fd=15 to=300000 fl=R ib=33 ob=0>
smtp: 0x1127a72e6000: <<< RCPT TO:<[hidden email]>
mproc: pony -> lka: realloc 128 -> 512
mproc: pony -> lka : 283 IMSG_SMTP_EXPAND_RCPT
imsg: lka <- pony: IMSG_SMTP_EXPAND_RCPT (len=283)
expand: 0x154201b89018: expand_insert() called for address:[hidden email][parent=0x0, rule=0x0]
expand: 0x154201b89018: inserted node 0x1541802cb000
expand: lka_expand: address: [hidden email] [depth=0]
lookup: check "54.39.183.132" as NETADDR in table static:<localhost> -> 0
lookup: check "54.39.183.132" as NETADDR in table static:allowed-hosts -> found
lookup: check "athompso.net" as DOMAIN in table static:<anydestination> -> found
rule #2 matched: match from src allowed-hosts for any => translate
lookup: lookup "[hidden email]" as ALIAS in table static:translations -> 0
lookup: lookup "athompso" as ALIAS in table static:translations -> 0
lookup: lookup "@athompso.net" as ALIAS in table static:translations -> 0
lookup: lookup "@" as ALIAS in table static:translations -> 0
expand: lka_expand: no aliases for virtual
mproc: lka -> pony : 53 IMSG_SMTP_EXPAND_RCPT
expand: 0x154201b89018: clearing expand tree
imsg: pony <- lka: IMSG_SMTP_EXPAND_RCPT (len=53)
smtp: 0x1127a72e6000: >>> 524 5.2.4 Mailing list expansion problem
d3a5ff63e6a0e04e smtp failed-command address=54.39.183.132 host=mail.athompso.net command="RCPT TO:<[hidden email]>" result="524 5.2.4 Mailing list expansion problem"
smtp: 0x1127a72e6000: IO_LOWAT <io:0x11284ad86400 fd=15 to=300000 fl=W ib=0 ob=0>
smtp: 0x1127a72e6000: IO_DATAIN <io:0x11284ad86400 fd=15 to=300000 fl=R ib=6 ob=0>
smtp: 0x1127a72e6000: <<< QUIT
smtp: 0x1127a72e6000: >>> 221 2.0.0: Bye
smtp: 0x1127a72e6000: STATE_HELO -> STATE_QUIT
smtp: 0x1127a72e6000: IO_LOWAT <io:0x11284ad86400 fd=15 to=300000 fl=W ib=0 ob=0>
d3a5ff63e6a0e04e smtp disconnected address=54.39.183.132 host=mail.athompso.net reason=quit
mproc: pony -> queue : 4 IMSG_SMTP_MESSAGE_ROLLBACK
mproc: pony -> control : 45 IMSG_STAT_DECREMENT
imsg: queue <- pony: IMSG_SMTP_MESSAGE_ROLLBACK (len=4)
imsg: control <- pony: IMSG_STAT_DECREMENT (len=45)
ramstat: decrement: smtp.session
ramstat: smtp.session (0x1d3c1e55aec0): 1 -> 0
queue-backend: queue_message_delete(d7bf0ff6) -> 1
mproc: queue -> scheduler: realloc 0 -> 128
mproc: queue -> scheduler : 4 IMSG_QUEUE_MESSAGE_ROLLBACK
imsg: scheduler <- queue: IMSG_QUEUE_MESSAGE_ROLLBACK (len=4)
scheduler: aborting msg:d7bf0ff6
scheduler: getting batch: mask=0x3f, count=10
debug: /--- ramqueue: scheduler_ram_batch()
debug: \---
scheduler: got r=0, delay=-1, count=10
scheduler: sleeping


^Cdebug: got signal 2
debug: clearing p=ca, fd=4, pid=35575
debug: ca -> parent: pipe closed
debug: ca agent exiting
debug: control -> klondike: pipe closed
debug: control agent exiting
debug: pony -> klondike: pipe closed
debug: pony agent exiting
debug: lka -> control: pipe closed
debug: lookup agent exiting
debug: queue -> control: pipe closed
debug: queue agent exiting
debug: scheduler -> control: pipe closed
debug: scheduler agent exiting
debug: clearing p=pony, fd=7, pid=40666
debug: clearing p=control, fd=5, pid=63932
debug: clearing p=lka, fd=6, pid=49698
debug: clearing p=scheduler, fd=9, pid=14152
debug: clearing p=queue, fd=8, pid=21502
Exiting
bhs#


Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax

Gilles Chehade-7
In reply to this post by Adam Thompson
On Sun, Jan 20, 2019 at 04:14:05PM -0600, Adam Thompson wrote:
> As it turns out, no, that doesn't work.
> Trying to fix up broken sender mail domain-parts only simply gets me a "5.2.4 Mailing list expansion problem" error, with no debug output to suggest why.
>
> In this test case, my translations map had:
>
> @bad.athompso.net @good.athompso.net
>

What is a translation map ?

There is no such thing in OpenSMTPD (as of today).


> in it.  Obviously, this is a test setup :).
> Smtpd.conf itself consisted of:
>
> listen on all received-auth
> smtp max-message-size 100M
> table translations file:/etc/mail/translations          # ORIG->NEW mappings
> table allowed-hosts file:/etc/mail/allowed-hosts        # Who can connect?  (bare IP addresses or CIDR subnets)
> action translate lmtp "/var/run/lmtp.sock" virtual <translations>       # 1st pass on allowed rewrite mail
> action forward forward-only                                             # and now it's not our problem anymore
> match for any from local action forward                 # 2nd pass for reinjected mail, this time just forward it
> match for any from src <allowed-hosts> action translate # inbound mail - hand it to LMTP, translating as we go
>
>


from table(5):
 then tell the first people who attempts to help that yu/uuuu///yyyyyyyyyyyy
     Aliasing tables
     
     Aliasing tables are mappings that associate a recipient to one or many
     destinations.  They can be used in two contexts: primary domain aliases
     and virtual domain mapping.
     
     [...]
     
     In a virtual domain context, the key is either a user part, a full email
     address or a catch all, following selection rules described in
     smtpd.conf(5), and the value is one or many recipients as described in
     aliases(5):

           user1                   otheruser
           [hidden email]       otheruser1,otheruser2
           @example.org            [hidden email]
           @                       [hidden email]


You're feeding the virtual table with invalid values.

Also, this is a recipient translation mechanism, similar to aliases, and
not a sender rewriting mechanism which we do not have at this point.


> A cursory glance at the source code (yikes, it's been a long time since I was a programmer) suggests that virtual now only works on recipients, not senders.  Which is too bad for me, as that means I'll have to switch at least one box to use Postfix.
>

virtual _now_ only works on recipients, not senders ?

the virtual code hasn't changed, it works the way it always did.

there is no way it could ever do what you're describing or attempting to
do given that it doesn't operate at all anywhere near the message. there
is no way it has ever parsed:

  @bad.athompso.net @good.athompso.net

and the only thing that changed is that such errors are now visible from
the session as:

    5.2.4 Mailing list expansion problem

instead of an invalid recipient error like it probably did in 6.3


> I'm not convinced the new smtpd.conf grammar improves anything at all, but I assume it must help someone or it wouldn't have changed... but I believe my use case got thrown out with the bathwater, so to speak.  Oh, well.  :-(
>

This is bullshit.

The grammar doesn't reduce the functional scope, it can only expand it.

What you are describing has never existed in smtpd, there's never been
code to translate sender addresses and there's a good reason for that:

it not considered doable before the grammar change...

But sure, blame it on the grammar.


> (If anyone cares, the bad sender addresses are mostly alerts coming from older Sun ALOMs and at least one Lexmark printer that also sends email with broken From addresses.)
>


I may sound a bit harsh, but starting a thread with "this is my last try
or I'll switch"
     Aliasing tables
     
     Aliasing tables are mappings that associate a recipient to one or many
     destinations.  They can be used in two contexts: primary domain aliases
     and virtual domain mapping.
     
     [...]
     
     In a virtual domain context, the key is either a user part, a full email
     address or a catch all, following selection rules described in
     smtpd.conf(5), and the value is one or many recipients as described in
     aliases(5):

           user1                   otheruser
           [hidden email]       otheruser1,otheruser2
           @example.org            [hidden email]
           @                       [hidden email]


You're feeding the virtual table with invalid values.

Also, this is a recipient translation mechanism, similar to aliases, and
not a sender rewriting mechanism which we do not have at this point.


> A cursory glance at the source code (yikes, it's been a long time since I was a programmer) suggests that virtual now only works on recipients, not senders.  Which is too bad for me, as that means I'll have to switch at least one box to use Postfix.
>

virtual _now_ only works on recipients, not senders ?

the virtual code hasn't changed, it works the way it always did.

there is no way it could ever do what you're describing or attempting to
do given that it doesn't operate at all anywhere near the message. there
is no way it has ever parsed:

  @bad.athompso.net @good.athompso.net

and the only thing that changed is that such errors are now visible from
the session as:

    5.2.4 Mailing list expansion problem

instead of an invalid recipient error like it probably did in 6.3


> I'm not convinced the new smtpd.conf grammar improves anything at all, but I assume it must help someone or it wouldn't have changed... but I believe my use case got thrown out with the bathwater, so to speak.  Oh, well.  :-(
>

This is bullshit.

The grammar doesn't reduce the functional scope, it can only expand it.

What you are describing has never existed in smtpd, there's never been
code to translate sender addresses and there's a good reason for that:

it not considered doable before the grammar change...

But sure, blame it on the grammar.


> (If anyone cares, the bad sender addresses are mostly alerts coming from older Sun ALOMs and at least one Lexmark printer that also sends email with broken From addresses.)
>


I may sound a bit harsh, but starting a thread with "this is my last try
or I'll switch"
     Aliasing tables
     
     Aliasing tables are mappings that associate a recipient to one or many
     destinations.  They can be used in two contexts: primary domain aliases
     and virtual domain mapping.
     
     [...]
     
     In a virtual domain context, the key is either a user part, a full email
     address or a catch all, following selection rules described in
     smtpd.conf(5), and the value is one or many recipients as described in
     aliases(5):

           user1                   otheruser
           [hidden email]       otheruser1,otheruser2
           @example.org            [hidden email]
           @                       [hidden email]


You're feeding the virtual table with invalid values.

Also, this is a recipient translation mechanism, similar to aliases, and
not a sender rewriting mechanism which we do not have at this point.


> A cursory glance at the source code (yikes, it's been a long time since I was a programmer) suggests that virtual now only works on recipients, not senders.  Which is too bad for me, as that means I'll have to switch at least one box to use Postfix.
>

virtual _now_ only works on recipients, not senders ?

the virtual code hasn't changed, it works the way it always did.

there is no way it could ever do what you're describing or attempting to
do given that it doesn't operate at all anywhere near the message. there
is no way it has ever parsed:

  @bad.athompso.net @good.athompso.net

and the only thing that changed is that such errors are now visible from
the session as:

    5.2.4 Mailing list expansion problem

instead of an invalid recipient error like it probably did in 6.3


> I'm not convinced the new smtpd.conf grammar improves anything at all, but I assume it must help someone or it wouldn't have changed... but I believe my use case got thrown out with the bathwater, so to speak.  Oh, well.  :-(
>

This is bullshit.

The grammar doesn't reduce the functional scope, it can only expand it.

What you are describing has never existed in smtpd, there's never been
code to translate sender addresses and there's a good reason for that:

it not considered doable before the grammar change...

But sure, blame it on the grammar.


> (If anyone cares, the bad sender addresses are mostly alerts coming from older Sun ALOMs and at least one Lexmark printer that also sends email with broken From addresses.)
>


I may sound a bit harsh, but starting a thread with "this is my last try
or I'll switch"
     Aliasing tables
     
     Aliasing tables are mappings that associate a recipient to one or many
     destinations.  They can be used in two contexts: primary domain aliases
     and virtual domain mapping.
     
     [...]
     
     In a virtual domain context, the key is either a user part, a full email
     address or a catch all, following selection rules described in
     smtpd.conf(5), and the value is one or many recipients as described in
     aliases(5):

           user1                   otheruser
           [hidden email]       otheruser1,otheruser2
           @example.org            [hidden email]
           @                       [hidden email]


You're feeding the virtual table with invalid values.

Also, this is a recipient translation mechanism, similar to aliases, and
not a sender rewriting mechanism which we do not have at this point.


> A cursory glance at the source code (yikes, it's been a long time since I was a programmer) suggests that virtual now only works on recipients, not senders.  Which is too bad for me, as that means I'll have to switch at least one box to use Postfix.
>

virtual _now_ only works on recipients, not senders ?

the virtual code hasn't changed, it works the way it always did.

there is no way it could ever do what you're describing or attempting to
do given that it doesn't operate at all anywhere near the message. there
is no way it has ever parsed:

  @bad.athompso.net @good.athompso.net

and the only thing that changed is that such errors are now visible from
the session as:

    5.2.4 Mailing list expansion problem

instead of an invalid recipient error like it probably did in 6.3


> I'm not convinced the new smtpd.conf grammar improves anything at all, but I assume it must help someone or it wouldn't have changed... but I believe my use case got thrown out with the bathwater, so to speak.  Oh, well.  :-(
>

This is bullshit.

The grammar doesn't reduce the functional scope, it can only expand it.

What you are describing has never existed in smtpd, there's never been
code to translate sender addresses and there's a good reason for that:

it not considered doable before the grammar change...

But sure, blame it on the grammar.


> (If anyone cares, the bad sender addresses are mostly alerts coming from older Sun ALOMs and at least one Lexmark printer that also sends email with broken From addresses.)
>


I may sound a bit harsh, but starting a thread with "this is my last try
or I'll switch"
     Aliasing tables
     
     Aliasing tables are mappings that associate a recipient to one or many
     destinations.  They can be used in two contexts: primary domain aliases
     and virtual domain mapping.
     
     [...]
     
     In a virtual domain context, the key is either a user part, a full email
     address or a catch all, following selection rules described in
     smtpd.conf(5), and the value is one or many recipients as described in
     aliases(5):

           user1                   otheruser
           [hidden email]       otheruser1,otheruser2
           @example.org            [hidden email]
           @                       [hidden email]


You're feeding the virtual table with invalid values.

Also, this is a recipient translation mechanism, similar to aliases, and
not a sender rewriting mechanism which we do not have at this point.


> A cursory glance at the source code (yikes, it's been a long time since I was a programmer) suggests that virtual now only works on recipients, not senders.  Which is too bad for me, as that means I'll have to switch at least one box to use Postfix.
>

virtual _now_ only works on recipients, not senders ?

the virtual code hasn't changed, it works the way it always did.

there is no way it could ever do what you're describing or attempting to
do given that it doesn't operate at all anywhere near the message. there
is no way it has ever parsed:

  @bad.athompso.net @good.athompso.net

and the only thing that changed is that such errors are now visible from
the session as:

    5.2.4 Mailing list expansion problem

instead of an invalid recipient error like it probably did in 6.3


> I'm not convinced the new smtpd.conf grammar improves anything at all, but I assume it must help someone or it wouldn't have changed... but I believe my use case got thrown out with the bathwater, so to speak.  Oh, well.  :-(
>

This is bullshit.

The grammar doesn't reduce the functional scope, it can only expand it.

What you are describing has never existed in smtpd, there's never been
code to translate sender addresses and there's a good reason for that:

it not considered doable before the grammar change...

But sure, blame it on the grammar.


> (If anyone cares, the bad sender addresses are mostly alerts coming from older Sun ALOMs and at least one Lexmark printer that also sends email with broken From addresses.)
>


I may sound a bit harsh, but starting a thread with "this is my last try
or I'll switch"
     Aliasing tables
     
     Aliasing tables are mappings that associate a recipient to one or many
     destinations.  They can be used in two contexts: primary domain aliases
     and virtual domain mapping.
     
     [...]
     
     In a virtual domain context, the key is either a user part, a full email
     address or a catch all, following selection rules described in
     smtpd.conf(5), and the value is one or many recipients as described in
     aliases(5):

           user1                   otheruser
           [hidden email]       otheruser1,otheruser2
           @example.org            [hidden email]
           @                       [hidden email]


You're feeding the virtual table with invalid values.

Also, this is a recipient translation mechanism, similar to aliases, and
not a sender rewriting mechanism which we do not have at this point.


> A cursory glance at the source code (yikes, it's been a long time since I was a programmer) suggests that virtual now only works on recipients, not senders.  Which is too bad for me, as that means I'll have to switch at least one box to use Postfix.
>

virtual _now_ only works on recipients, not senders ?

the virtual code hasn't changed, it works the way it always did.

there is no way it could ever do what you're describing or attempting to
do given that it doesn't operate at all anywhere near the message. there
is no way it has ever parsed:

  @bad.athompso.net @good.athompso.net

and the only thing that changed is that such errors are now visible from
the session as:

    5.2.4 Mailing list expansion problem

instead of an invalid recipient error like it probably did in 6.3


> I'm not convinced the new smtpd.conf grammar improves anything at all, but I assume it must help someone or it wouldn't have changed... but I believe my use case got thrown out with the bathwater, so to speak.  Oh, well.  :-(
>

This is bullshit.

The grammar doesn't reduce the functional scope, it can only expand it.

What you are describing has never existed in smtpd, there's never been
code to translate sender addresses and there's a good reason for that:

it not considered doable before the grammar change...

But sure, blame it on the grammar.


> (If anyone cares, the bad sender addresses are mostly alerts coming from older Sun ALOMs and at least one Lexmark printer that also sends email with broken From addresses.)
>


I may sound a bit harsh, but starting a thread with "this is my last try
or I'll switch" (as if it actually matters) right before telling someone
who wants to help you that you actually tried _nothing_ then blaming the
code improvements for a use-case that could have never worked because it
not only uses the wrong _documented_ mechanism but also because the code
to make your use-case work has never existed, kinds of irritates me.

I don't get royalties on smtpd install, please install whatever software
fits your use case, this is how proper engineering works.

--
Gilles Chehade       @poolpOrg

https://www.poolp.org                 tip me: https://paypal.me/poolpOrg

Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax [FIXED]

Gilles Chehade-7
In reply to this post by Adam Thompson
sorry, I obviously f-up my last mail, this one is fixed ;-)


On Sun, Jan 20, 2019 at 04:14:05PM -0600, Adam Thompson wrote:
> As it turns out, no, that doesn't work.
> Trying to fix up broken sender mail domain-parts only simply gets me a "5.2.4 Mailing list expansion problem" error, with no debug output to suggest why.
>
> In this test case, my translations map had:
>
> @bad.athompso.net @good.athompso.net
>

What is a translation map ?

There is no such thing in OpenSMTPD (as of today).


> in it.  Obviously, this is a test setup :).
> Smtpd.conf itself consisted of:
>
> listen on all received-auth
> smtp max-message-size 100M
> table translations file:/etc/mail/translations          # ORIG->NEW mappings
> table allowed-hosts file:/etc/mail/allowed-hosts        # Who can connect?  (bare IP addresses or CIDR subnets)
> action translate lmtp "/var/run/lmtp.sock" virtual <translations>       # 1st pass on allowed rewrite mail
> action forward forward-only                                             # and now it's not our problem anymore
> match for any from local action forward                 # 2nd pass for reinjected mail, this time just forward it
> match for any from src <allowed-hosts> action translate # inbound mail - hand it to LMTP, translating as we go
>
>


from table(5):

     Aliasing tables
     
     Aliasing tables are mappings that associate a recipient to one or many
     destinations.  They can be used in two contexts: primary domain aliases
     and virtual domain mapping.
     
     [...]
     
     In a virtual domain context, the key is either a user part, a full email
     address or a catch all, following selection rules described in
     smtpd.conf(5), and the value is one or many recipients as described in
     aliases(5):

           user1                   otheruser
           [hidden email]       otheruser1,otheruser2
           @example.org            [hidden email]
           @                       [hidden email]


You're feeding the virtual table with invalid values.

Also, this is a recipient translation mechanism, similar to aliases, and
not a sender rewriting mechanism which we do not have at this point.


> A cursory glance at the source code (yikes, it's been a long time since I was a programmer) suggests that virtual now only works on recipients, not senders.  Which is too bad for me, as that means I'll have to switch at least one box to use Postfix.
>

virtual _now_ only works on recipients, not senders ?

the virtual code hasn't changed, it works the way it always did.

there is no way it could ever do what you're describing or attempting to
do given that it doesn't operate at all anywhere near the message. there
is no way it has ever parsed:

  @bad.athompso.net @good.athompso.net

and the only thing that changed is that such errors are now visible from
the session as:

    5.2.4 Mailing list expansion problem

instead of an invalid recipient error like it probably did in 6.3


> I'm not convinced the new smtpd.conf grammar improves anything at all, but I assume it must help someone or it wouldn't have changed... but I believe my use case got thrown out with the bathwater, so to speak.  Oh, well.  :-(
>

This is bullshit.

The grammar doesn't reduce the functional scope, it can only expand it.

What you are describing has never existed in smtpd, there's never been
code to translate sender addresses and there's a good reason for that:

it not considered doable before the grammar change...

But sure, blame it on the grammar.


> (If anyone cares, the bad sender addresses are mostly alerts coming from older Sun ALOMs and at least one Lexmark printer that also sends email with broken From addresses.)
>

I may sound a bit harsh, but starting a thread with "this is my last try
or I'll switch" (as if it actually matters) right before telling someone
who wants to help you that you actually tried _nothing_ then blaming the
code improvements for a use-case that could have never worked because it
not only uses the wrong _documented_ mechanism but also because the code
to make your use-case work has never existed, kinds of irritates me.

I don't get royalties on smtpd install, please install whatever software
fits your use case, this is how proper engineering works.

--
Gilles Chehade       @poolpOrg

https://www.poolp.org                 tip me: https://paypal.me/poolpOrg

Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax [FIXED]

Adam Thompson
On 2019-01-21 04:08, Gilles Chehade wrote:
>> In this test case, my translations map had:
> What is a translation map ?
> There is no such thing in OpenSMTPD (as of today).

A virtual map that happened to be called <translation>.


> You're feeding the virtual table with invalid values.

Apparently, yes.


> Also, this is a recipient translation mechanism, similar to aliases,
> and
> not a sender rewriting mechanism which we do not have at this point.
> [...]
> virtual _now_ only works on recipients, not senders ?
> the virtual code hasn't changed, it works the way it always did.
>
> there is no way it could ever do what you're describing or attempting
> to
> do given that it doesn't operate at all anywhere near the message.
> there
> is no way it has ever parsed:

This is all very surprising to hear.  The existing system works
(somehow).  So I am apparently misunderstanding what is happening,
because with the configuration as shown, telling the various broken
email senders to use that box as their mailhost _somehow_ fixes the
bogus From: headers and envelopes.

Oh, this just occurred to me as I'm writing:  I really hope I didn't
switch to a different MTA on that system years ago, and then just forgot
to check which MTA was actually running.  If that's the case, I'm not
going to bother posting an update, because I'll be busy banging my head
on the wall and then hiding in shame.


>> I'm not convinced the new smtpd.conf grammar improves anything at all,
>> but I assume it must help someone or it wouldn't have changed... but I
>> believe my use case got thrown out with the bathwater, so to speak.  
>> Oh, well.  :-(
> This is bullshit.
> The grammar doesn't reduce the functional scope, it can only expand it.

I'm taking your word for it - you will know far better than I do!


> What you are describing has never existed in smtpd, there's never been
> code to translate sender addresses and there's a good reason for that:

Good reasons aside, I still need to accommodate other vendor's broken
mail implementations, because I can't fix them.  I know of multiple
reasons source rewriting is a bad idea, in general, but I get paid to
make stuff work, not just say that it's broken.


> it not considered doable before the grammar change...
> But sure, blame it on the grammar.

I believed that the grammar change had rendered my use case impossible
because <virtual> was now limited to local delivery methods.  Clearly I
was wrong... and not even in the way I thought I might be wrong.


> I may sound a bit harsh, but starting a thread with "this is my last
> try
> or I'll switch" (as if it actually matters)

My apologies - that was meant to sound more like "I have a plan B so if
this isn't possible, that's OK but I've wasted so much time on this I'm
kinda running out of time, please tell me if I should just stop now and
switch".  I know *exactly* how much OpenBSD devs care if I use their
code or not!  I do not want to be "that asshole", although it seems I've
succeeded again - sorry.

Thank you for taking the time to reply.  Now I'm going to go check that
mail server a 7,000,000th time, this time to see what MTA is actually
*running*, not just *configured*.  I'm not sure whether I want it to be
such a blatant mistake on my part or not... if yes, this all makes sense
but I'm an idiot, whereas if no, then WTF, how is it working at all?

FWIW: I am much happier with OpenSMTPd than with other MTAs because of
its forward-declarative configuration syntax.  Thank you for your work
on bringing a modern, lean, secure(-er) MTA into existence.

-Adam

Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax [FIXED]

Eric Elena-3
In reply to this post by Gilles Chehade-7
On Mon, 21 Jan 2019 11:08:02 +0100 Gilles Chehade wrote:
> I may sound a bit harsh, but starting a thread with "this is my last try
> or I'll switch" (as if it actually matters) right before telling someone
> who wants to help you that you actually tried _nothing_ then blaming the
> code improvements for a use-case that could have never worked because it
> not only uses the wrong _documented_ mechanism but also because the code
> to make your use-case work has never existed, kinds of irritates me.
>
> I don't get royalties on smtpd install, please install whatever software
> fits your use case, this is how proper engineering works.

First of all thank you Gilles (and all the others who contributed to
this project) for your amazing work on OpenSMTPD!

That said, there is a kind of sender rewriting mechanism in OpenSMTP.
Well, it works for me (tm) I'm not saying it's perfect, it might be an
overkill but at least it does what I want it to do. The conf is
included below (only the part for rewriting the sender
address):
o /etc/mail/smtpd.conf
listen on all tls pki my.domain auth-optional
listen on lo0 port 10030 smtps pki my.domain tag MASQ auth senders { foo = [hidden email] } masquerade

table masquser          { "[hidden email]" }
table masq-alias        { "[hidden email]" = "[hidden email]" }

table secrets           file:/etc/mail/secrets

action masq01 mbox virtual <masq-alias>
action masq02 relay host smtps://masqlabel@127.0.0.1:10030 auth <secrets> mail-from "[hidden email]"

<match ... reject here>

match tag MASQ rcpt-to <masquser> action masq01
match from any rcpt-to <masquser> action masq02

<match ... relay here>

o /etc/mail/secrets
masqlabel foo:asuperpassword

When a mail is received (listen on all):
- check if it is rejected
- if not, if the email if for [hidden email], forward it to the very
same OpenSMTP daemon on port 10030 using the authenticated user foo and
using [hidden email] as the MAIL-FROM in the SMTP session (enveloppe)
- when an email is received on port 10030, tag it with the label MASQ.
The authenticated user is allowed to send an email as the user
[hidden email]. The keyword masquerade modifies the From header (the
message itself) to match the address given in the SMTP session
- at that point, the sender address is rewritten both in the SMTP
session and the headers
- if the email is for [hidden email] and is tagged with the label MASQ,
the virtual user address is expanded to the real email address
- continue like a normal message

There is probably room for improvement but I hope this helps.

Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax [FIXED]

Gilles Chehade-7
In reply to this post by Adam Thompson
On Mon, Jan 21, 2019 at 01:04:16PM -0600, Adam Thompson wrote:

>
> > Also, this is a recipient translation mechanism, similar to aliases, and
> > not a sender rewriting mechanism which we do not have at this point.
> > [...]
> > virtual _now_ only works on recipients, not senders ?
> > the virtual code hasn't changed, it works the way it always did.
> >
> > there is no way it could ever do what you're describing or attempting to
> > do given that it doesn't operate at all anywhere near the message. there
> > is no way it has ever parsed:
>
> This is all very surprising to hear.  The existing system works (somehow).
> So I am apparently misunderstanding what is happening, because with the
> configuration as shown, telling the various broken email senders to use that
> box as their mailhost _somehow_ fixes the bogus From: headers and envelopes.
>

the entire virtual expansion happens between the client sending RCPT TO,
and the server responding Ok to that RCPT TO. virtual does not know of a
sender, never, and it is done before the message is actually received so
it doesn't know headers, which is why i'm 100% confident there isn't one
chance it could ever do what you describe.


> Oh, this just occurred to me as I'm writing:  I really hope I didn't switch
> to a different MTA on that system years ago, and then just forgot to check
> which MTA was actually running.  If that's the case, I'm not going to bother
> posting an update, because I'll be busy banging my head on the wall and then
> hiding in shame.
>

that is a more likely possibility.


> > > I'm not convinced the new smtpd.conf grammar improves anything at
> > > all, but I assume it must help someone or it wouldn't have
> > > changed... but I believe my use case got thrown out with the
> > > bathwater, so to speak.  Oh, well.  :-(
> > This is bullshit.
> > The grammar doesn't reduce the functional scope, it can only expand it.
>
> I'm taking your word for it - you will know far better than I do!
>
>
> > What you are describing has never existed in smtpd, there's never been
> > code to translate sender addresses and there's a good reason for that:
>
> Good reasons aside, I still need to accommodate other vendor's broken mail
> implementations, because I can't fix them.  I know of multiple reasons
> source rewriting is a bad idea, in general, but I get paid to make stuff
> work, not just say that it's broken.
>

oh, don't get me wrong, i'm not saying there's a good reason not to have
this rewriting, what i was saying is that there was a good reason why it
was not doable before the grammar change.

it is a useful feature which is part of my todo and which i will work on
as time allows.


> > it not considered doable before the grammar change...
> > But sure, blame it on the grammar.
>
> I believed that the grammar change had rendered my use case impossible
> because <virtual> was now limited to local delivery methods.  Clearly I was
> wrong... and not even in the way I thought I might be wrong.
>

yes, that's true.

using 'virtual' on relay rules didn't transform anything whatsoever, the
code had an explicit check to not enter the transformation lookups if we
were in a relay rule.

the new grammar just made it clear that what you were trying to do could
not work rather than accepting the criteria and disregarding it.


> > I may sound a bit harsh, but starting a thread with "this is my last try
> > or I'll switch" (as if it actually matters)
>
> My apologies - that was meant to sound more like "I have a plan B so if this
> isn't possible, that's OK but I've wasted so much time on this I'm kinda
> running out of time, please tell me if I should just stop now and switch".
> I know *exactly* how much OpenBSD devs care if I use their code or not!  I
> do not want to be "that asshole", although it seems I've succeeded again -
> sorry.
>
> Thank you for taking the time to reply.  Now I'm going to go check that mail
> server a 7,000,000th time, this time to see what MTA is actually *running*,
> not just *configured*.  I'm not sure whether I want it to be such a blatant
> mistake on my part or not... if yes, this all makes sense but I'm an idiot,
> whereas if no, then WTF, how is it working at all?
>
> FWIW: I am much happier with OpenSMTPd than with other MTAs because of its
> forward-declarative configuration syntax.  Thank you for your work on
> bringing a modern, lean, secure(-er) MTA into existence.
>

np ;-)



--
Gilles Chehade       @poolpOrg

https://www.poolp.org                 tip me: https://paypal.me/poolpOrg

Reply | Threaded
Open this post in threaded view
|

Re: smtpd - help needed tranlsating to new virtual map syntax [FIXED]

Gilles Chehade-7
In reply to this post by Eric Elena-3
On Tue, Jan 22, 2019 at 01:11:44AM +0100, Eric Elena wrote:

> On Mon, 21 Jan 2019 11:08:02 +0100 Gilles Chehade wrote:
> > I may sound a bit harsh, but starting a thread with "this is my last try
> > or I'll switch" (as if it actually matters) right before telling someone
> > who wants to help you that you actually tried _nothing_ then blaming the
> > code improvements for a use-case that could have never worked because it
> > not only uses the wrong _documented_ mechanism but also because the code
> > to make your use-case work has never existed, kinds of irritates me.
> >
> > I don't get royalties on smtpd install, please install whatever software
> > fits your use case, this is how proper engineering works.
>
> First of all thank you Gilles (and all the others who contributed to
> this project) for your amazing work on OpenSMTPD!
>
> That said, there is a kind of sender rewriting mechanism in OpenSMTP.
> Well, it works for me (tm) I'm not saying it's perfect, it might be an
> overkill but at least it does what I want it to do. The conf is
> included below (only the part for rewriting the sender
> address):
>
> [...]
>
> When a mail is received (listen on all):
> - check if it is rejected
> - if not, if the email if for [hidden email], forward it to the very
> same OpenSMTP daemon on port 10030 using the authenticated user foo and
> using [hidden email] as the MAIL-FROM in the SMTP session (enveloppe)
> - when an email is received on port 10030, tag it with the label MASQ.
> The authenticated user is allowed to send an email as the user
> [hidden email]. The keyword masquerade modifies the From header (the
> message itself) to match the address given in the SMTP session
> - at that point, the sender address is rewritten both in the SMTP
> session and the headers
> - if the email is for [hidden email] and is tagged with the label MASQ,
> the virtual user address is expanded to the real email address
> - continue like a normal message
>
> There is probably room for improvement but I hope this helps.
>

indeed, a bit overkill and now that we have removed the blockers we must
come up with a simpler way to achieve that...

but what you did, that's smart :-)


--
Gilles Chehade       @poolpOrg

https://www.poolp.org                 tip me: https://paypal.me/poolpOrg