npppd pptp hangs

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

npppd pptp hangs

Marko Cupać
Hi,

my npppd pptp server has recently got increase from ~20 to >200
concurrent users. So far it worked flawlessly for years, but before few
minutes it become unresponsive.

It stopped logging at one point (I have log redirected to its own file,
/var/log/npppd). npppctl also hanged, returning nothing. I couldn't
restart it with rcctl, or kill it with HUP. I had to resort to `kill
-9', and it started fine afterwards.

It appears that already established sessions worked, but with poor
performance.

I have lots of these in log (I saw them earlier as well but they
weren't causing problems AFAIK):

Mar 23 12:03:26 nat1 /bsd: pipex: ppp=1869 iface=tun1 protocol=PPTP id=45012 Received bad data packet: out of sequence: seq=1266880(1266946-1267010) ack=1915237(1915368-1915471)
Mar 23 12:03:26 nat1 /bsd: pipex: ppp=1869 iface=tun1 protocol=PPTP id=45012 Received bad data packet: out of sequence: seq=1266881(1266946-1267010) ack=1915239(1915368-1915472)
Mar 23 12:03:26 nat1 /bsd: pipex: ppp=1869 iface=tun1 protocol=PPTP id=45012 Workaround the out-of-sequence PPP framing problem: 1215 => 1151
Mar 23 12:06:59 nat1 /bsd: pipex: ppp=1847 iface=tun1 protocol=PPTP id=45439 received packet caused window overflow. seq=218469(218273-218337)may lost 196 packets.

Also, at the time before killing it there's:

Mar 23 13:13:37 nat1 /bsd: splassert: pipex_destroy_session: want 2 have 0
Mar 23 13:13:37 nat1 last message repeated 95 times


Anything I can do to avoid future hangs?

Thank you in advance,

--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Stuart Longland
On 23/3/20 10:26 pm, Marko Cupać wrote:
> Anything I can do to avoid future hangs?

Whilst probably not the answer you're looking for: moving away from PPTP
would be a good start.

The MSCHAPv2 authentication used in PPTP is vulnerable to dictionary
attacks and the RC4 cipher used in MPPE (the security layer of PPTP) is
laughably weak in today's security context.  Whilst MSCHAPv2 can be
replaced with EAP-TLS, there's no fix for MPPE.

IPSec (which is built into OpenBSD) or OpenVPN (in ports) would be
vastly superior options.
--
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
  ...it's backed up on a tape somewhere.

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Marko Cupać
On Tue, 24 Mar 2020 07:13:27 +1000
Stuart Longland <[hidden email]> wrote:

> On 23/3/20 10:26 pm, Marko Cupać wrote:
> > Anything I can do to avoid future hangs?
>
> Whilst probably not the answer you're looking for: moving away from
> PPTP would be a good start.
>
> The MSCHAPv2 authentication used in PPTP is vulnerable to dictionary
> attacks and the RC4 cipher used in MPPE (the security layer of PPTP)
> is laughably weak in today's security context.  Whilst MSCHAPv2 can be
> replaced with EAP-TLS, there's no fix for MPPE.
>
> IPSec (which is built into OpenBSD) or OpenVPN (in ports) would be
> vastly superior options.

Indeed, I am also waiting for the day when I'll be able to point iked
to Microsoft's implementation of a RADIUS server (NPS), which will
authenticate Active Directory domain-joined machines by their machine
certificate and hopefully with additional domain user password for 2FA,
authorise them by Active Directory group membership, and log their
accounting in format which can be easily parsed and converted into
human-readable statistics with currently available parsers.

Uh, that sounded like I'm some kind of Microsoft fanboy, but I'm not. I
just have to provide hundreds of Windows users a way to access resources
on a corporate network in order to keep my bills paid. npppd's pptp
helps me brilliantly (anyone remember poptop? that was hell :)

Anyway, I use IPSec extensively to connect branch office routers, both
in tunnel mode for passive clients with dynamic IPs, and in transport
mode for protecting GRE tunnels (OSPF). Lately I'm adding multipath
redundancy over multiple ISPs using rdomains. OpenVPN also has a place
on my network. OpenBSD is a miracle :)

Pardon my blatant self-promotion on link below, but I think it's a
win-win situation - I get eternal fame and glory on the Internet, and
list readers get copy/paste howto set up npppd pptp server with RADIUS
authentication. Could come handy in this "end of days" situation where
everyone works remotely :D

https://www.mimar.rs/blog/how-to-set-up-pptp-vpn-server-with-openbsd-and-npppd

Best regards,

--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Marko Cupać
On Tue, 24 Mar 2020 09:34:09 +0100
Marko Cupać <[hidden email]> wrote:

> On Tue, 24 Mar 2020 07:13:27 +1000
> Stuart Longland <[hidden email]> wrote:
>
> > On 23/3/20 10:26 pm, Marko Cupać wrote:
> > > Anything I can do to avoid future hangs?

I got another hang, this time killing npppd process crashed complete OS
(sorry for photo, I don't have serial console set up):

https://oblak.mimar.rs/index.php/s/Cc9J745jH93RK6j

At the time when npppd wouldn't accept new connections, and npppctl
won't return anything, but before the crash, i noticed high CPU usage
in top:

45125 _ppp      64    0 3128K 6340K onproc/3  -        39:05 99.85% npppd

Pehaps bugs@ would be more appropriate list?

--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Vitaliy Makkoveev


> On 24 Mar 2020, at 12:09, Marko Cupać <[hidden email]> wrote:
>
> On Tue, 24 Mar 2020 09:34:09 +0100
> Marko Cupać <[hidden email]> wrote:
>
>> On Tue, 24 Mar 2020 07:13:27 +1000
>> Stuart Longland <[hidden email]> wrote:
>>
>>> On 23/3/20 10:26 pm, Marko Cupać wrote:
>>>> Anything I can do to avoid future hangs?
>
> I got another hang, this time killing npppd process crashed complete OS
> (sorry for photo, I don't have serial console set up):
>
> https://oblak.mimar.rs/index.php/s/Cc9J745jH93RK6j
>
> At the time when npppd wouldn't accept new connections, and npppctl
> won't return anything, but before the crash, i noticed high CPU usage
> in top:
>
> 45125 _ppp      64    0 3128K 6340K onproc/3  -        39:05 99.85% npppd
>
> Pehaps bugs@ would be more appropriate list?
>
> --
> Before enlightenment - chop wood, draw water.
> After  enlightenment - chop wood, draw water.
>
> Marko Cupać
> https://www.mimar.rs/
>
Can you try latest snapshot? Can you share your npppd.conf?

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Marko Cupać
On Sat, 28 Mar 2020 01:46:41 +0300
Vitaliy Makkoveev <[hidden email]> wrote:

> Can you try latest snapshot?

Unfortunately, the box that runs npppd is the most important machine on
my network (GRE/IPsec hub for multiple branch offices), I can't take the
risk.

> Can you share your npppd.conf?

Below, I have redacted sensitive information. Perhaps it is worth
mentioning that npppd listens on IP address of CARP interface.

---npppd.conf.start---
# GLOBAL
set max-session 200
set user-max-session 1

# TUNNEL
tunnel EXAMPLEORG protocol pptp {
        listen on IP.ADD.RE.SS
        pptp-hostname vpn.example.org
        pptp-vendor-name "openbsd-npppd"
        ingress-filter yes
        pipex no
        mppe required
        mppe-key-length 128
        mppe-key-state stateless
        idle-timeout 1800
}

# IPCP
ipcp KAPPASTAR {
        pool-address "IP.ADD.RE.SS/24"
        dns-servers IP.ADD.RE.SS
        allow-user-selected-address no
}

# INTERFACE
interface tun1 address IP.ADD.RE.SS ipcp EXAMPLEORG

# AUTHENTICATION
authentication RADIUS type radius {
        strip-nt-domain yes
        strip-atmark-realm yes
        authentication-server {
                address IP.ADD.RE.SS secret "ThisIsNotRealPassword"
        }
        accounting-server {
                address IP.ADD.RE.SS secret "ThisIsNotRealPassword"
        }
}

bind tunnel from EXAMPLEORG authenticated by RADIUS to tun1
---npppd.conf.end---

Thank you in advance for looking into it.
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Vitaliy Makkoveev
On Mon, Mar 30, 2020 at 12:47:13PM +0200, Marko Cupać wrote:

> On Sat, 28 Mar 2020 01:46:41 +0300
> Vitaliy Makkoveev <[hidden email]> wrote:
>
> > Can you try latest snapshot?
>
> Unfortunately, the box that runs npppd is the most important machine on
> my network (GRE/IPsec hub for multiple branch offices), I can't take the
> risk.
>
> > Can you share your npppd.conf?
>
> Below, I have redacted sensitive information. Perhaps it is worth
> mentioning that npppd listens on IP address of CARP interface.
>
> ---npppd.conf.start---
> # GLOBAL
> set max-session 200
> set user-max-session 1
>
> # TUNNEL
> tunnel EXAMPLEORG protocol pptp {
>         listen on IP.ADD.RE.SS
>         pptp-hostname vpn.example.org
>         pptp-vendor-name "openbsd-npppd"
> ingress-filter yes
> pipex no
>         mppe required
>         mppe-key-length 128
>         mppe-key-state stateless
>         idle-timeout 1800
> }
>
> # IPCP
> ipcp KAPPASTAR {
>         pool-address "IP.ADD.RE.SS/24"
>         dns-servers IP.ADD.RE.SS
>         allow-user-selected-address no
> }
>
> # INTERFACE
> interface tun1 address IP.ADD.RE.SS ipcp EXAMPLEORG
>
> # AUTHENTICATION
> authentication RADIUS type radius {
>         strip-nt-domain yes
>         strip-atmark-realm yes
>         authentication-server {
>                 address IP.ADD.RE.SS secret "ThisIsNotRealPassword"
>         }
>         accounting-server {
>                 address IP.ADD.RE.SS secret "ThisIsNotRealPassword"
>         }
> }
>
> bind tunnel from EXAMPLEORG authenticated by RADIUS to tun1
> ---npppd.conf.end---
>
> Thank you in advance for looking into it.
> --
> Before enlightenment - chop wood, draw water.
> After  enlightenment - chop wood, draw water.
>
> Marko Cupać
> https://www.mimar.rs/
>

You have pipex(4) disabled. Is it still hangs with disabled pipex(4)?
As I discovered (https://marc.info/?t=158529976800001&r=1&w=2), npppd
with pipex(4) enabled and non-NULL "idle-timeout" option will crash
kernel. You can disable this option in yout npppd.conf an reenable
pipex(4). Looks like crashes should gone.

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Vitaliy Makkoveev
On Mon, Mar 30, 2020 at 02:28:08PM +0300, Vitaliy Makkoveev wrote:
> You have pipex(4) disabled. Is it still hangs with disabled pipex(4)?
> As I discovered (https://marc.info/?t=158529976800001&r=1&w=2), npppd
> with pipex(4) enabled and non-NULL "idle-timeout" option will crash
> kernel. You can disable this option in yout npppd.conf an reenable
> pipex(4). Looks like crashes should gone.
And don't use pppac(4) with pipex enabled, use pppx(4). Crash you
reported https://marc.info/?t=158506225900002&r=1&w=2 is actual for
pppac(4).

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Marko Cupać
On Mon, 30 Mar 2020 14:33:46 +0300
Vitaliy Makkoveev <[hidden email]> wrote:

> On Mon, Mar 30, 2020 at 02:28:08PM +0300, Vitaliy Makkoveev wrote:
> > You have pipex(4) disabled. Is it still hangs with disabled
> > pipex(4)? As I discovered
> > (https://marc.info/?t=158529976800001&r=1&w=2), npppd with pipex(4)
> > enabled and non-NULL "idle-timeout" option will crash kernel. You
> > can disable this option in yout npppd.conf an reenable pipex(4).
> > Looks like crashes should gone.
> And don't use pppac(4) with pipex enabled, use pppx(4). Crash you
> reported https://marc.info/?t=158506225900002&r=1&w=2 is actual for
> pppac(4).
>

Thanks for the instruction.

I have:
 - left net.pipex.enable=1
 - replaced tun1 with pppx0 in npppd.conf
 - removed 'pipex no' from npppd.conf

So far so good, I'll send update if I experience further hangs.

The instruction on pppx(4) you gave me came as surprising news to me. I
have been using npppd since it still had undocumented 'old style'
config. Once npppd.conf got its manpage (was it 5.3?) I've set up tun1
as PPTP interface and it worked great with up to ~20 clients all these
years. I was very satisfied that all PPTP traffic went through single
interface (as opposed to my previous setup with poptop which created
separate tun interface for each session), as I had the ability to graph
its traffic from SNMP data.

I guess I was 'holding it wrong' all this time, and yet it worked well
:)

Thank you once again.

--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Marko Cupać
On 2020-03-31 10:07, Marko Cupać wrote:

> On Mon, 30 Mar 2020 14:33:46 +0300
> Vitaliy Makkoveev <[hidden email]> wrote:
>
>> On Mon, Mar 30, 2020 at 02:28:08PM +0300, Vitaliy Makkoveev wrote:
>> > You have pipex(4) disabled. Is it still hangs with disabled
>> > pipex(4)? As I discovered
>> > (https://marc.info/?t=158529976800001&r=1&w=2), npppd with pipex(4)
>> > enabled and non-NULL "idle-timeout" option will crash kernel. You
>> > can disable this option in yout npppd.conf an reenable pipex(4).
>> > Looks like crashes should gone.
>> And don't use pppac(4) with pipex enabled, use pppx(4). Crash you
>> reported https://marc.info/?t=158506225900002&r=1&w=2 is actual for
>> pppac(4).
>>
>
> Thanks for the instruction.
>
> I have:
>  - left net.pipex.enable=1
>  - replaced tun1 with pppx0 in npppd.conf
>  - removed 'pipex no' from npppd.conf
>
> So far so good, I'll send update if I experience further hangs.

No crash since changing interface from tun to pppx.

Thanx!

--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Reply | Threaded
Open this post in threaded view
|

Re: npppd pptp hangs

Vitaliy Makkoveev
On Sat, Apr 04, 2020 at 08:22:27PM +0200, Marko Cupać wrote:

> On 2020-03-31 10:07, Marko Cupać wrote:
> > On Mon, 30 Mar 2020 14:33:46 +0300
> > Vitaliy Makkoveev <[hidden email]> wrote:
> >
> > > On Mon, Mar 30, 2020 at 02:28:08PM +0300, Vitaliy Makkoveev wrote:
> > > > You have pipex(4) disabled. Is it still hangs with disabled
> > > > pipex(4)? As I discovered
> > > > (https://marc.info/?t=158529976800001&r=1&w=2), npppd with pipex(4)
> > > > enabled and non-NULL "idle-timeout" option will crash kernel. You
> > > > can disable this option in yout npppd.conf an reenable pipex(4).
> > > > Looks like crashes should gone.
> > > And don't use pppac(4) with pipex enabled, use pppx(4). Crash you
> > > reported https://marc.info/?t=158506225900002&r=1&w=2 is actual for
> > > pppac(4).
> > >
> >
> > Thanks for the instruction.
> >
> > I have:
> >  - left net.pipex.enable=1
> >  - replaced tun1 with pppx0 in npppd.conf
> >  - removed 'pipex no' from npppd.conf
> >
> > So far so good, I'll send update if I experience further hangs.
>
> No crash since changing interface from tun to pppx.
>
> Thanx!
Ok :)
>
> --
> Before enlightenment - chop wood, draw water.
> After  enlightenment - chop wood, draw water.
>
> Marko Cupać
> https://www.mimar.rs/
>