relayd confuses filter rules

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

relayd confuses filter rules

Nick Guenther-2
>Synopsis: relayd mixes up filter rules when a protocol is reused in multiple relays
>Category: system
>Environment:
        System      : OpenBSD 6.6
        Details     : OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT 2019
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:
        I set up relayd in front of httpd, so that http://$(hostname)/* should redirect to
        https://$(hostname)/*, https://$(hostname)/ is the main site but https://$(hostname)/app is a
        embedded webapp. This sort of reverse proxy set up is common with webapps (like NextCloud,
        Mattermost, Gitea, Radicale, etc) to avoid having to fight the same-origin policy: it allows
        /app and the main site to communicate via cookies, websockets, etc. It also avoids firewalls
        that otherwise block webapps run on unusual ports (e.g. the way cPanel is usually installed)
        by making all apps share a single port: 443.

        But relayd applies the /app proxy I asked of it *to the http:// site*, which means people can
        get at my webapp without going through TLS.

        # Afterthought

  After working with this for a bit it's obvious to me that relayd was not meant for doing this
        kind of sub-domain/sub-path reverse proxying. What I really want is httpd's location blocks,
        which are simple and easy, and importantly feature the `strip` command, but to be able to apply
        them to a TCP proxy and not just to FastCGI.
>How-To-Repeat:

0) Make an empty test folder
```
$ mkdir t1; cd t1
```

1) Set up httpd with two vhosts plus a "bounce to https". The first vhost is our main site, the second, running on a different port, represents a backend webapp that runs its own HTTP server

```
$ mkdir -p site app
$ mkdir -p logs # httpd insists on this
$ echo "Static Site" > site/index.html
$ echo "WebApp" > app/index.html
$ cat > httpd.conf <<EOF
# httpd.conf
                                                                                                       
chroot "."

server "default" {
        listen on localhost port 8080
        location * {
            block return 302 "https://\$HTTP_HOST\$REQUEST_URI"
        }
}

server "site" {
        listen on localhost port 8081
        root "site"
        directory auto index
}

server "app" {
        listen on localhost port 8082
        root "app"
        request strip 1 # this site will be mounted at /app but relayd doesn't have (obvious?) URL rewriting
        directory auto index
}
EOF
```

```
$ doas httpd -f httpd.conf # httpd demands root, even though it doesn't need it in this case
```

Test:
```
$ curl http://localhost:8081/         
Static Site
$ curl http://localhost:8082/
WebApp
```

2) Make a cert for TLS

This will be used to set up relayd listening on two ports -- http and https -- and demonstrate
 
This makes a self-signed cert so it'll have to `curl --cacert $(hostname).pem` (or just `curl -k`) to use it; if you have a live server you can experiment on, use acme-client to get a "real" cert, or reuse a valid cert you have.

```
$ openssl req -x509 -newkey rsa:4096 -subj '/CN='"$(hostname)" -nodes -keyout $(hostname).key -out $(hostname).pem -days 3
Generating a 4096 bit RSA private key
...................................................................++++
................................................................................................................................++++
writing new private key to 'matriculate.lan.key'
-----
```

# relayd config

3) Set up relayd with rules to rewrite the URLs.

```
$ cat > relayd.conf <<EOF
table <web> { "127.0.0.1" }
table <app> { "127.0.0.1" }

http protocol web {
        # Return HTTP/HTML error pages to the client
        return error

        tls keypair $(hostname)

        #match request path "/app{,/*}" tag app # want this, but this glob syntax isn't supported?
        match request path "/app" tag app
        match request path "/app/*" tag app
        match request tagged app header set "X-Script-Name" value "/app" # has to match the above!
        match request tagged app forward to <app>
}

# http:// should bounce to the "default" vhost, which should bounce the client to https://
relay http_proxy {
        listen on * port 80

        protocol web
        forward to <web> port 8080
}

# https:// should handle
relay https_proxy {
        listen on * port 443 tls

        protocol web
        forward to <web> port 8081
        forward to <app> port 8082
}
EOF
```


relayd insists you need the certs to be global:

```
$ doas cp $(pwd)/$(hostname).pem /etc/ssl/$(hostname).crt
$ doas cp $(pwd)/$(hostname).key /etc/ssl/private/$(hostname).key
```


# Testing

In one shell:

```
$ doas relayd -d -v -f relayd.conf                        
startup
adding 1 hosts from table web:8080 (no check)
adding 1 hosts from table web:8080 (no check)
adding 1 hosts from table web:8080 (no check)
adding 1 hosts from table web:8081 (no check)
adding 1 hosts from table web:8081 (no check)
adding 1 hosts from table app:8082 (no check)
adding 1 hosts from table app:8082 (no check)
adding 1 hosts from table web:8081 (no check)
adding 1 hosts from table app:8082 (no check)
```

in a parallel shell:

a) demonstrate that http://$(hostname)/, https://$(hostname)/ and https://$(hostname)/app are working as expected

```
$ curl -v http://$(hostname)/
*   Trying 127.0.0.1:80...
* TCP_NODELAY set
* Connected to matriculate.lan (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: matriculate.lan
> User-Agent: curl/7.66.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 302 Found
< Connection: close
< Content-Length: 419
< Content-Type: text/html
< Date: Mon, 10 Feb 2020 02:19:00 GMT
< Location: https://matriculate.lan/
< Server: OpenBSD httpd
<
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>302 Found</title>
<style type="text/css"><!--
body { background-color: white; color: black; font-family: 'Comic Sans MS', 'Chalkboard SE', 'Comic Neue', sans-serif; }
hr { border: 0; border-bottom: 1px dashed; }

--></style>
</head>
<body>
<h1>302 Found</h1>
<hr>
<address>OpenBSD httpd</address>
</body>
</html>
* Closing connection 0

$ curl --cacert /etc/ssl/$(hostname).crt https://$(hostname)/   
Static Site
$ curl --cacert /etc/ssl/$(hostname).crt https://$(hostname)/app/
WebApp
```

b) But http://$(hostname)/app is *wrong*: it passes through directly to the backend without hitting the 302 Redirect:

```
$ curl http://$(hostname)/app/   
WebApp
```

The WebApp, like the Static site, are supposed to be *inaccessible* over http.

So what went wrong here?


Appendix: repro v2 (without TLS)
--------------------------------

The bug doesn't depend on TLS, it was just a bit more realistic and motivating to demonstrate it that way.
It still appears if we replace https://$(hostname)/ with http://$(hostname):8080/

```
$ mkdir t2; cd t2
$ mkdir -p site app
$ mkdir -p logs # httpd insists on this
$ echo "Static Site" > site/index.html
$ echo "WebApp" > app/index.html
$
$ cat > httpd.conf <<EOF                                                                                      
chroot "."

server "site" {
        listen on localhost port 8081
        root "site"
        directory auto index
}

server "app" {
        listen on localhost port 8082
        root "app"
        request strip 1
        directory auto index
}
EOF
$ doas httpd -f httpd.conf
$
$ cat > relayd.conf <<EOF                                                                                                        
table <web> { "127.0.0.1" }
table <app> { "127.0.0.1" }

http protocol web {
        # Return HTTP/HTML error pages to the client
        return error

        #match request path "/app{,/*}" tag app # want this, but this glob syntax isn't supported?
        match request path "/app" tag app
        match request path "/app/*" tag app
        match request tagged app forward to <app>
}

relay http_proxy {
        listen on 0.0.0.0 port 80

        protocol web
        forward to <web> port 8081
}

relay http_alt_proxy {
        listen on 0.0.0.0 port 8080

        protocol web
        forward to <web> port 8081
        forward to <app> port 8082
}
EOF
$ doas relayd -f relayd.conf
```

Then:

```
$ curl http://$(hostname)/app/ 
WebApp
```

even though the port 80 relay isn't supposed to be forwarding to the webapp.



If we add more "webapps":

```
$ mkdir app2; echo "WebApp 2" > app2/index.html
$ cat >> httpd.conf <EOF

server "app2" {
        listen on localhost port 8083
        root "app2"
        request strip 1
        directory auto index
}
EOF
$ vi relayd.conf # ... edit to say:
$ cat relayd.conf
table <web> { "127.0.0.1" }
table <app> { "127.0.0.1" }
table <app2> { "127.0.0.1" }

http protocol web {
        # Return HTTP/HTML error pages to the client
        return error

        #match request path "/app{,/*}" tag app # want this, but this glob syntax isn't supported?
        match request path "/app" tag app
        match request path "/app/*" tag app
        match request tagged app forward to <app>

        #match request path "/app2{,/*}" tag app2 # want this, but this glob syntax isn't supported?
        match request path "/app2" tag app2
        match request path "/app2/*" tag app2
        match request tagged app2 forward to <app2>
}

relay http_proxy {
        listen on 0.0.0.0 port 80

        protocol web
        forward to <web> port 8081
}

relay http_alt_proxy {
        listen on 0.0.0.0 port 8080

        protocol web
        forward to <web> port 8081
        forward to <app> port 8082
        forward to <app2> port 8083
}
$ doas pkill httpd
$ doas pkill relayd
$ doas httpd -f httpd.conf
$ doas relayd -f relayd.conf
```

The problem continues:

```
$ curl http://$(hostname):8080/  # expected
Static Site
$ curl http://$(hostname):8080/app/   # expected
WebApp
$ curl http://$(hostname):8080/app2/   # expected
WebApp 2
$
$ curl http://$(hostname)/   # expected
Static Site
$ curl http://$(hostname)/app/ # BUG
WebApp
$ curl http://$(hostname)/app2/ # BUG
WebApp 2
```


>Fix:
        Adding this to relayd.conf:

```
http protocol web80 {
        return error
}
```

and making `relay http_proxy` use `protocol web80` makes the problem go away.
But if you can't reuse protocols why distinguish them from relays at all?

Reply | Threaded
Open this post in threaded view
|

Re: relayd confuses filter rules

Nick Guenther-2
Ah I step in a giant puddle of egg for my first post here. My lines were all too long and my sample code had typos. Let's try this again:

> How-To-Repeat:

        ```
        $ mkdir relayd-confused-filters; cd relayd-confused-filters
        ```

        ```
        $ openssl req -x509 -newkey rsa:4096 -subj '/CN='"localhost" -nodes \
                  -keyout localhost.key -out localhost.pem -days 3
        $ doas cp localhost.pem /etc/ssl/localhost.crt
        $ doas cp localhost.key /etc/ssl/private/localhost.key
        ```

        ```
        $ mkdir -p site app
        $ echo "Static Site" > site/index.html
        $ echo "WebApp" > app/index.html
        $ mkdir -p logs
        ```

        ```
        # httpd.conf
        chroot "."

        server "default" {
                listen on localhost port 8080
                location * {
                    block return 302 "https://$HTTP_HOST$REQUEST_URI"
                }
        }

        server "site" {
                listen on localhost port 8081
                root "site"
                directory auto index
        }

        server "app" {
                listen on localhost port 8082
                root "app"
                request strip 1
                directory auto index
        }
        ```

        ```
        # relayd.conf
        table <web> { "127.0.0.1" }
        table <app> { "127.0.0.1" }

        http protocol web {
                # Return HTTP/HTML error pages to the client
                return error

                tls keypair localhost

                match request path "/app/*" forward to <app>
        }

        relay http_proxy {
                listen on 0.0.0.0 port 80

                protocol web
                forward to <web> port 8080
        }

        relay https_proxy {
                listen on 0.0.0.0 port 443 tls

                protocol web
                forward to <web> port 8081
                forward to <app> port 8082
        }
        ```


        Forced-https seems to be working:

        ```
        $ curl -v http://localhost/
        *   Trying 127.0.0.1:80...
        * TCP_NODELAY set
        * Connected to localhost (127.0.0.1) port 80 (#0)
        > GET / HTTP/1.1
        > Host: localhost
        > User-Agent: curl/7.66.0
        > Accept: */*
        >
        * Mark bundle as not supporting multiuse
        * HTTP 1.0, assume close after body
        < HTTP/1.0 302 Found
        < Connection: close
        < Content-Length: 419
        < Content-Type: text/html
        < Date: Tue, 11 Feb 2020 12:06:07 GMT
        < Location: https://localhost/
        < Server: OpenBSD httpd
        <
        <!DOCTYPE html>
        <html>
        <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
        <title>302 Found</title>
        <style type="text/css"><!--
        body { background-color: white; color: black; font-family: 'Comic Sans MS', 'Chalkboard SE', 'Comic Neue', sans-serif; }
        hr { border: 0; border-bottom: 1px dashed; }

        --></style>
        </head>
        <body>
        <h1>302 Found</h1>
        <hr>
        <address>OpenBSD httpd</address>
        </body>
        </html>
        ```

        The unified https:// site works too:

        ```
        $ curl --cacert /etc/ssl/localhost.crt  https://localhost/     
        Static Site
        $ curl --cacert /etc/ssl/localhost.crt  https://localhost/app/
        WebApp
        ```

        But here's the bug(!): I can talk to /app *unencrypted*, because the filter rule
        meant for the *other* relay is triggering in http_proxy:

        ```
        $ curl http://localhost/app/                                   
        WebApp
        ```

Reply | Threaded
Open this post in threaded view
|

Re: relayd confuses filter rules

Sebastian Benoit-3
Hi,

Nick([hidden email]) on 2020.02.11 12:15:48 +0000:
> Ah I step in a giant puddle of egg for my first post here. My lines were all too long and my sample code had typos. Let's try this again:

To me it looks like the obvious solution is not to use a protocol intended
for tls connections with a protocol for http connections.

Your reports are a mix of misunderstanding the configuration, some valid
remarks about possible documentation improvements and some real bugs.

Because its getting confusing here what you actually try to do, I'm going to
disregard this and the other reports i have not replied to. At least until
the ones i replied to are resolved.

Thanks,
B.


> > How-To-Repeat:
>
> ```
> $ mkdir relayd-confused-filters; cd relayd-confused-filters
> ```
>
> ```
> $ openssl req -x509 -newkey rsa:4096 -subj '/CN='"localhost" -nodes \
>  -keyout localhost.key -out localhost.pem -days 3
> $ doas cp localhost.pem /etc/ssl/localhost.crt
> $ doas cp localhost.key /etc/ssl/private/localhost.key
> ```
>
> ```
> $ mkdir -p site app
> $ echo "Static Site" > site/index.html
> $ echo "WebApp" > app/index.html
> $ mkdir -p logs
> ```
>
> ```
> # httpd.conf
> chroot "."
>
> server "default" {
> listen on localhost port 8080
> location * {
>    block return 302 "https://$HTTP_HOST$REQUEST_URI"
> }
> }
>
> server "site" {
> listen on localhost port 8081
> root "site"
> directory auto index
> }
>
> server "app" {
> listen on localhost port 8082
> root "app"
> request strip 1
> directory auto index
> }
> ```
>
> ```
> # relayd.conf
> table <web> { "127.0.0.1" }
> table <app> { "127.0.0.1" }
>
> http protocol web {
> # Return HTTP/HTML error pages to the client
> return error
>
> tls keypair localhost
>
> match request path "/app/*" forward to <app>
> }
>
> relay http_proxy {
> listen on 0.0.0.0 port 80
>
> protocol web
> forward to <web> port 8080
> }
>
> relay https_proxy {
> listen on 0.0.0.0 port 443 tls
>
> protocol web
> forward to <web> port 8081
> forward to <app> port 8082
> }
> ```
>
>
> Forced-https seems to be working:
>
> ```
> $ curl -v http://localhost/
> *   Trying 127.0.0.1:80...
> * TCP_NODELAY set
> * Connected to localhost (127.0.0.1) port 80 (#0)
> > GET / HTTP/1.1
> > Host: localhost
> > User-Agent: curl/7.66.0
> > Accept: */*
> >
> * Mark bundle as not supporting multiuse
> * HTTP 1.0, assume close after body
> < HTTP/1.0 302 Found
> < Connection: close
> < Content-Length: 419
> < Content-Type: text/html
> < Date: Tue, 11 Feb 2020 12:06:07 GMT
> < Location: https://localhost/
> < Server: OpenBSD httpd
> <
> <!DOCTYPE html>
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
> <title>302 Found</title>
> <style type="text/css"><!--
> body { background-color: white; color: black; font-family: 'Comic Sans MS', 'Chalkboard SE', 'Comic Neue', sans-serif; }
> hr { border: 0; border-bottom: 1px dashed; }
>
> --></style>
> </head>
> <body>
> <h1>302 Found</h1>
> <hr>
> <address>OpenBSD httpd</address>
> </body>
> </html>
> ```
>
> The unified https:// site works too:
>
> ```
> $ curl --cacert /etc/ssl/localhost.crt  https://localhost/     
> Static Site
> $ curl --cacert /etc/ssl/localhost.crt  https://localhost/app/
> WebApp
> ```
>
> But here's the bug(!): I can talk to /app *unencrypted*, because the filter rule
> meant for the *other* relay is triggering in http_proxy:
>
> ```
> $ curl http://localhost/app/                                   
> WebApp
> ```
>

Reply | Threaded
Open this post in threaded view
|

Re: relayd confuses filter rules

Sebastian Benoit-3
[hidden email]([hidden email]) on 2020.02.12 00:07:19 -0500:

> Hey Benoit,
>
> Let me start by saying I know you're doing this as volunteer work and I hardly need or expect them to be fixed any time soon; thank you again for taking a look. Open source maintenance is a bigger burden than we usually acknowledge.
>
> On February 11, 2020 6:49:12 PM EST, Sebastian Benoit <[hidden email]> wrote:
> >Hi,
> >
> >Nick([hidden email]) on 2020.02.11 12:15:48 +0000:
> >> Ah I step in a giant puddle of egg for my first post here. My lines
> >were all too long and my sample code had typos. Let's try this again:
> >
> >To me it looks like the obvious solution is not to use a protocol
> >intended
> >for tls connections with a protocol for http connections.
> >
> >Your reports are a mix of misunderstanding the configuration, some
> >valid
> >remarks about possible documentation improvements and some real bugs.
>
> Of course I misunderstood. That's kind of my whole point. It would have been immediately cleared up if relayd noticed I was trying to mix HTTPS and HTTP backends and told me that that wasn't allowed, or noticed that I had used a protocol template in two relays, one of which it didn't really make sense in, and stopped me.
>
> But it didn't stop me; in this particular thread, it instead guesses the best it can and ends up mixing the `forward to <app>` from the second relay with the `listen on 0.0.0.0 port 80` from the first. In the other thread , it routes TLS traffic to a plaintext HTTP backend without warning.
>
> I'm asking for these errors to be explicit instead of leaving subtle misconfigurations.
>
> >Because its getting confusing here what you actually try to do, I'm
> >going to
> >disregard this and the other reports i have not replied to. At least
> >until
> >the ones i replied to are resolved.
> >
>
> I'm trying to hook together several vhosts and webapps onto a single port,
> so I don't have ugly :port parts in my URLs. The webapps that speak
> FastCGI I can put behind httpd location or server blocks, but the others I
> can't, so I turned to relayd.
>
> But regardless, I gave isolated test cases for all the issues I ran into
> trying to teach myself, it shouldn't matter what my original goal was,
> unless you think I'm stretching relayd beyond it's intended use.


Yes, thanks for the detailed reports. It's just too much information to work
on all of them at once. I will go through all of them, but not right away.

Also i don't know if you got any useful feedback on reddit. IF you have a
configuration question, [hidden email] is usually a good place to ask,

Best regards,
Benno