pf, OpenVPN, Squid and dummynet

[Mazandar Wiki]

Hello.

I have to limit the Download/Upload speed of my OpenVPN users. I use Squid (in transparent mode) for caching and logging their web traffic. I use pfSense, which is based on FreeBSD 11.2-RELEASE.

pf, Squid and OpenVPN work great, the problem starts when dummynet comes into play. dummynet just doesn't work when Squid is enabled, but it works great when I disable squid. I've included my pf.conf. It seems OK to me, but I am very new to pf.

I tried to pass the traffic to dummynet immediately after redirecting HTTP and HTTPS traffics to ports 3128 and 3129 respectively, but it didn't work:

pass  in  quick  on $OpenVPN inet proto tcp  from any to any port 3128 tracker 1560924429 flags S/SA keep state  dnpipe ( 1,2)

pass  in  quick  on $OpenVPN inet proto tcp  from any to any port 3129 tracker 1560924399 flags S/SA keep state  dnpipe ( 1,2)

(changing the interface to $loopback also didn't help)

I also tried to pass packets destined for 127.0.0.1:3128 and 3129 to dummynet with these rules, but it also didn't work (placing them in appropriate section, before other rules).

pass in  on $loopback inet proto tcp from any to any port 3128 dnpipe(1,2)

pass in  on $loopback inet proto tcp from any to any port 3129 dnpipe(1,2)

I even changed the last rule of the file to this, but it also didn't work either:

pass in quick on ovpns1 proto tcp from any to (ovpns1) port {3128,3129} flags S/SA keep state dnpipe(1,2)

I tried a lot of other stuffs, none of them worked when squid was enabled. Anyway, this is the whole pf.conf, generated by pfSense.

Could you please help me find the problem?

Thank you very much.

set limit table-entries 400000
set optimization normal
set limit states 97000
set limit src-nodes 97000

#System aliases
  
loopback = "{ lo0 }"
WAN = "{ em0 }"
LAN = "{ ovpns1 }"
OpenVPN = "{ openvpn }"

#SSH Lockout Table
table <sshguard> persist
#Snort tables
table <snort2c>
table <virusprot>
table <bogons> persist file "/etc/bogons"
table <vpn_networks> { 192.168.10.0/24 }
table <negate_networks> { 192.168.10.0/24 }

# User Aliases 
 
# Gateways
GWWANGW = " route-to ( em0 172.19.20.1 ) "
GWWAN_DHCP = " route-to ( em0 172.19.20.1 ) "
GWLAN_VPNV4 = " route-to ( ovpns1 192.168.10.1 ) "
GWLAN_VPNV6 = " route-to ( ovpns1 192.168.10.1 ) "

 
set loginterface ovpns1

set skip on pfsync0

scrub on $WAN all    fragment reassemble
scrub on $LAN all    fragment reassemble


no nat proto carp
no rdr proto carp
nat-anchor "natearly/*"
nat-anchor "natrules/*"


# Outbound NAT rules (automatic)

# Subnets to NAT 
tonatsubnets = "{ 127.0.0.0/8 ::1/128 192.168.10.0/24 }"
nat on $WAN inet from $tonatsubnets to any port 500 -> 172.19.20.101/32  static-port
nat on $WAN inet6 from $tonatsubnets to any port 500 -> (em0)  static-port
nat on $WAN inet from $tonatsubnets to any -> 172.19.20.101/32 port 1024:65535 
nat on $WAN inet6 from $tonatsubnets to any -> (em0) port 1024:65535 

# Load balancing anchor
rdr-anchor "relayd/*"
# TFTP proxy
rdr-anchor "tftp-proxy/*"

# Setup Squid proxy redirect
rdr pass on ovpns1 proto tcp from any to !(ovpns1) port 80 -> 127.0.0.1 port 3128
rdr pass on ovpns1 proto tcp from any to !(ovpns1) port 443 -> 127.0.0.1 port 3129

# UPnPd rdr anchor
rdr-anchor "miniupnpd"

anchor "relayd/*"
anchor "openvpn/*"
anchor "ipsec/*"
# block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device,
# and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but
# route-to can override that, causing problems such as in redmine #2073
block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local"
block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local"
#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in log inet all tracker 1000000103 label "Default deny rule IPv4"
block out log inet all tracker 1000000104 label "Default deny rule IPv4"
block in log inet6 all tracker 1000000105 label "Default deny rule IPv6"
block out log inet6 all tracker 1000000106 label "Default deny rule IPv6"

# IPv6 ICMP is not auxilary, it is required for operation
# See man icmp6(4)
# 1    unreach         Destination unreachable
# 2    toobig          Packet too big
# 128  echoreq         Echo service request
# 129  echorep         Echo service reply
# 133  routersol       Router solicitation
# 134  routeradv       Router advertisement
# 135  neighbrsol      Neighbor solicitation
# 136  neighbradv      Neighbor advertisement
pass  quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state

# Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
pass out  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state
pass out  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state
pass in  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state
pass in  quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state
pass in  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state
pass in  quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000113 keep state
# We use the mighty pf, we cannot be fooled.
block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000114 label "Block traffic from port 0"
block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000115 label "Block traffic to port 0"
block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000116 label "Block traffic from port 0"
block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000117 label "Block traffic to port 0"

# Snort package
block log quick from <snort2c> to any tracker 1000000118 label "Block snort2c hosts"
block log quick from any to <snort2c> tracker 1000000119 label "Block snort2c hosts"

# SSH lockout
block in log quick proto tcp from <sshguard> to (self) port 22 tracker 1000000301 label "sshguard"

# webConfigurator lockout
block in log quick proto tcp from <sshguard> to (self) port 443 tracker 1000000351 label "GUI Lockout"
block in log quick from <virusprot> to any tracker 1000000400 label "virusprot overload table"
pass in  quick on { em0 } proto tcp from any to { 172.19.20.101 } port { 8003 8002 } tracker 1000000551 keep state(sloppy)
pass out  quick on { em0 } proto tcp from any to any flags any tracker 1000000552 keep state(sloppy)
# allow our DHCP client out to the WAN
pass in  quick on $WAN proto udp from any port = 67 to any port = 68 tracker 1000000561 label "allow dhcp client out WAN"
pass out  quick on $WAN proto udp from any port = 68 to any port = 67 tracker 1000000562 label "allow dhcp client out WAN" 
# Not installing DHCP server firewall rules for WAN which is configured for DHCP.
antispoof log for $WAN tracker 1000001570
antispoof log for $LAN tracker 1000002620

# loopback
pass in  on $loopback inet all tracker 1000003711 label "pass IPv4 loopback"
pass out  on $loopback inet all tracker 1000003712 label "pass IPv4 loopback"
pass in  on $loopback inet6 all tracker 1000003713 label "pass IPv6 loopback"
pass out  on $loopback inet6 all tracker 1000003714 label "pass IPv6 loopback"
# let out anything from the firewall host itself and decrypted IPsec traffic
pass out  inet all keep state allow-opts tracker 1000003715 label "let out anything IPv4 from firewall host itself"
pass out  inet6 all keep state allow-opts tracker 1000003716 label "let out anything IPv6 from firewall host itself"

pass out  route-to ( em0 172.19.20.1 ) from 172.19.20.101 to !172.19.20.0/24 tracker 1000003811 keep state allow-opts label "let out anything from firewall host itself"
pass out  route-to ( ovpns1 192.168.10.1 ) from 192.168.10.1 to !192.168.10.0/24 tracker 1000003812 keep state allow-opts label "let out anything from firewall host itself"
# make sure the user cannot lock himself out of the webConfigurator or SSH
pass in  quick on ovpns1 proto tcp from any to (ovpns1) port { 443 80 22 } tracker 10000 keep state label "anti-lockout rule"

# User-defined rules follow

anchor "userrules/*"
# array key "enc0" does not exist for "" in array: {WAN LAN OpenVPN } label "USER_RULE"
# array key "l2tp" does not exist for "" in array: {WAN LAN OpenVPN } label "USER_RULE"
pass  in  quick  on $OpenVPN inet proto udp  from any to any port 53 tracker 1560852911 keep state  label "USER_RULE"
pass  in  quick  on $OpenVPN inet proto tcp  from any to any port 53 tracker 1560852890 flags S/SA keep state  label "USER_RULE"
pass  in  quick  on $OpenVPN inet proto tcp  from any to any port 3128 tracker 1560924429 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE"
pass  in  quick  on $OpenVPN inet proto tcp  from any to any port 3129 tracker 1560924399 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE"
block  in  quick  on $OpenVPN inet proto tcp  from any to any tracker 1560924366 flags S/SA  label "USER_RULE"
pass  in  quick  on $WAN reply-to ( em0 172.19.20.1 ) inet proto tcp  from any to any tracker 1560337356 flags S/SA keep state  label "USER_RULE"
pass  in  quick  on $WAN reply-to ( em0 172.19.20.1 ) inet proto udp  from any to any port 1194 tracker 1559381581 keep state  label "USER_RULE: Allow OpenVPN to this firewall1"

# VPN Rules

anchor "tftp-proxy/*"

# Setup squid pass rules for proxy
pass in quick on ovpns1 proto tcp from any to (ovpns1) port {3128,3129} flags S/SA keep state