relayd confuses 'with tls' relays

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

relayd confuses 'with tls' relays

Nick Guenther-2
>Synopsis: relayd applies 'with tls' to all relay backends if any have it
>Category: system
>Environment:
        System      : OpenBSD 6.6
        Details     : OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT 2019
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64

>Description:

        I put relayd in front of httpd and some webapps (as in my previous bug report:
        https://marc.info/?l=openbsd-bugs&m=158130991528086&w=2) and discovered connections
        to the webapps breaking when (the backend) httpd uses TLS.
       
        The trouble turned out to be that relayd is applying my `with tls` on the httpd relay
        to the webapps as well, even though I'm not telling it to do any such thing.

>How-To-Repeat:
       
        # Make an empty test folder
       
        ```
        $ mkdir relayd-with-tls; cd relayd-with-tls
        ```

        ## Make a cert for TLS

        This makes a self-signed cert so we'll have to `curl --cacert $(hostname).pem` (or just `curl -k`) to use it;
        if you have a live server you can experiment on, use acme-client to get a "real" cert, or reuse any cert you have.

        ```
        $ openssl req -x509 -newkey rsa:4096 -subj '/CN='"localhost" -nodes -keyout localhost.key -out localhost.pem -days 3
        Generating a 4096 bit RSA private key
        ...........++++
        .....................................................................................................................................................................................................................................................................................................................++++
        writing new private key to 'localhost.key'
        -----
        ```

        ## Set up httpd and a "webapp"

        ```
        $ mkdir -p site
        $ mkdir -p logs # httpd insists on this
        $ echo "Static Site" > site/index.html
        $ cat > httpd.conf <<EOF                                                                  
        chroot "."

        server "site" {
                listen on localhost tls port 8443
                tls {
                    certificate "localhost.pem"
                    key "localhost.key"
                }
                root "site"
                directory auto index
        }
        EOF
        $ doas httpd -f httpd.conf
        $ curl --cacert localhost.pem  https://localhost:8443 # test httpd came up right                                                          
        Static Site
        ```
       
        In a parallel terminal, run the "webapp":
       
        ```
        $ nc -l -k -v 8082 | hexdump -C # our 'webapp'; leave this running.
        Listening on 0.0.0.0 8082
        ```
       
        ## Set up relayd
       
        relayd needs certs to be global:
       
        ```
        $ doas cp localhost.pem /etc/ssl/localhost.crt
        $ doas cp localhost.key /etc/ssl/private/localhost.key
        ```
       
        ```
        $ cat > relayd.conf <<EOF                                                                                                        
        table <web> { "127.0.0.1" }
        table <app> { "127.0.0.1" }
       
        http protocol web {
                # Return HTTP/HTML error pages to the client
                return error
       
                tls keypair localhost
                tls ca file "localhost.pem"
       
                match request path "/app/*" forward to <app>
        }
       
        relay https_proxy {
                listen on 0.0.0.0 port 443 tls
       
                protocol web
                forward with tls to <web> port 8443
                forward to <app> port 8082
        }
        EOF
        $ doas relayd -f relayd.conf
        $ curl --cacert localhost.pem  https://localhost # test that relayd and its `forward to <web>` relay came up
        Static Site
        ```
       
        ## Tests
       
        But the problem is when trying to relay to the "webapp":

        ```
        $ curl --cacert localhost.pem  https://localhost/app/
        ```
       
        the app receives encrypted traffic:
       
        ```
        $ nc -l -k -v 8082 | hexdump -C # our 'webapp' (that we left running)
        Listening on 0.0.0.0 8082
        Connection received on localhost 37169
        00000000  16 03 01 00 b0 01 00 00  ac 03 03 4e bf d5 f5 1a  |...........N....|
        00000010  e3 16 73 40 c5 2f d6 00  76 8d 2d 35 c9 55 50 34  |..s@./..v.-5.UP4|
        00000020  84 b1 eb 84 5f 31 cc b4  eb 8f 21 00 00 4c c0 30  |...._1....!..L.0|
        00000030  c0 2c c0 28 c0 24 c0 14  c0 0a 00 9f 00 6b 00 39  |.,.(.$.......k.9|
        00000040  cc a9 cc a8 cc aa ff 85  00 c4 00 88 00 81 00 9d  |................|
        00000050  00 3d 00 35 00 c0 00 84  c0 2f c0 2b c0 27 c0 23  |.=.5...../.+.'.#|
        00000060  c0 13 c0 09 00 9e 00 67  00 33 00 be 00 45 00 9c  |.......g.3...E..|
        00000070  00 3c 00 2f 00 ba 00 41  00 ff 01 00 00 37 00 05  |.<./...A.....7..|
        00000080  00 05 01 00 00 00 00 00  0b 00 02 01 00 00 0a 00  |................|
        00000090  08 00 06 00 1d 00 17 00  18 00 0d 00 18 00 16 08  |................|
        000000a0  06 06 01 06 03 08 05 05  01 05 03 08 04 04 01 04  |................|
        ```
       
        It doesn't work any better if we switch `nc` for an actual http server:
        ```
        $ nc -l -k -v 8082 | hexdump -C
        ...
        ^C
        $ python -m http.server 8082
        Serving HTTP on 0.0.0.0 port 8082 (http://0.0.0.0:8082/) ...
        ```
       
        Then test again:

        ```
        $ curl --cacert localhost.pem  https://localhost/app/
        curl: (52) Empty reply from server
        ```
       
        ```
        $ python -m http.server 8082
        Serving HTTP on 0.0.0.0 port 8082 (http://0.0.0.0:8082/) ...
        127.0.0.1 - - [10/Feb/2020 15:27:49] code 400, message Bad HTTP/0.9 request type ('\x16\x03\x01\x00°\x01\x00\x00¬\x03\x03åÌH\x00ù°fõ÷3t\x9c\x99r5Íþ\x14¼±')
        127.0.0.1 - - [10/Feb/2020 15:27:49] "°¬åÌHù°fõ÷3tr5Íþ¼±Lñ¥+_qÁåLÀ0À,À(À$ÀÀ" 400 -
        ```

>Fix:

        The quickest workaround is to disable the backend TLS. Since relayd is running on the
        same server as httpd encrypting the second hop doesn't give much (any?) security. But
        if that's not possible -- say, if the backend servers are distributed or you don't
        want to rely entirely on a VPN for encryption -- then you would have to go the other
        way and make sure every webapp has its own internally valid cert.

        That is, do this:
       
        ```
        $ doas pkill httpd relayd
        $ cat > httpd.conf <<EOF                                                                                                        
        chroot "."
       
        server "site" {
                listen on localhost tls port 8443
                listen on localhost port 8080
                tls {
                    certificate "localhost.pem"
                    key "localhost.key"
                }
                root "site"
                directory auto index
        }
        EOF
        $ cat > relayd.conf <<EOF                                                                                                  
        table <web> { "127.0.0.1" }
        table <app> { "127.0.0.1" }
       
        http protocol web {
                # Return HTTP/HTML error pages to the client
                return error
       
                tls keypair localhost
                tls ca file "localhost.pem"
       
        match request path "/app/*" forward to <app>
        }
       
        relay https_proxy {
                listen on 0.0.0.0 port 443 tls
       
                protocol web
                #forward with tls to <web> port 8443
                forward to <web> port 8080
                forward to <app> port 8082
        }
        EOF
        $ doas httpd -f httpd.conf
        $ doas relayd -f relayd.conf
        ```
       
        Now
       
        ```
        $ curl --cacert localhost.pem  https://localhost/ 
        Static Site
        $ curl --cacert localhost.pem  https://localhost/app/
        <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
                "http://www.w3.org/TR/html4/strict.dtd">
        <html>
            <head>
                <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
                <title>Error response</title>
            </head>
            <body>
                <h1>Error response</h1>
                <p>Error code: 404</p>
                <p>Message: File not found.</p>
                <p>Error code explanation: HTTPStatus.NOT_FOUND - Nothing matches the given URI.</p>
            </body>
        </html>
        ```
       
        and the "webapp"'s log:
       
        ```
        $ python -m http.server 8082
        Serving HTTP on 0.0.0.0 port 8082 (http://0.0.0.0:8082/) ...
        127.0.0.1 - - [10/Feb/2020 18:28:47] code 404, message File not found
        127.0.0.1 - - [10/Feb/2020 18:28:47] "GET /app/ HTTP/1.1" 404 -
        ```

Reply | Threaded
Open this post in threaded view
|

Re: relayd confuses 'with tls' relays

Nick Guenther-2
February 10, 2020 6:31 PM, "Nick" <[hidden email]> wrote:

>> Synopsis: relayd applies 'with tls' to all relay backends if any have it
>> Category: system

In light of looking into <https://marc.info/?l=openbsd-bugs&m=158140337725409&w=2>, it occurs to me
that relayd's logic is that each relay only operates a single way: based on the active protocol.

But if that's the case then the `with tls` option belongs in the `protocol` blocks,
and so too the `listen .. tls` option.

Thanks for the work, it is still so refreshing to use BSD.
-Nick