Re: NFS keeps crashing

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Matt Bettinger-2
Why are you using ooenbsd for anything but a firewall.  Even then its
lagging way behind unless you deploying in a dentist office. Openbsd sucks
at pretty much everything else.  Sorry I come from corporate real world
experience not Soho stuff.  Use Linux.

On Sat, Apr 21, 2018, 1:24 AM Rupert Gallagher <[hidden email]> wrote:

> This is what I observed on a controlled environment of three "windows 10
> pro" 1709 clients.
>
> The obsd nfs server had a single share:
>
> /path/to/folder -network 192.168.1 -mask 255.255.255.0
>
> When mounting a share for the first time, Windows allows browsing the
> network to find the resource. This is what happens:
>
> 1. The client asks for the list of nfs resources;
> 2. the server shows a stream of accepted mounts, no warnings, no errors;
> 3. while 2 happens, the client shows a warning that the server is not
> responding;
> 4. when eventually the client returns the list of nfs folders, the server
> crashes.
>
> The above occurs systematically. Restarting the server and repeating the
> client steps lead to a new server crash. The only way to mount the share is
> to type in the path, without browsing.
>
> When the server crashes, the debug shows no warnings and no errors.
>
> The problem did not occur with W10Pro 1703. However, the server should not
> crash, and if it does, it should report useful diagnostics.
>
> R
>
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Matt Bettinger-2
Invest in a NetApp filer and do it the right way.  Plenty of options
spectrascale glfs/cnfs lustre with DNE/IME why struggle with this hobby OS,
seriously?

On Sat, Apr 21, 2018, 1:31 AM MB <[hidden email]> wrote:

>
> Why are you using ooenbsd for anything but a firewall.  Even then its
> lagging way behind unless you deploying in a dentist office. Openbsd sucks
> at pretty much everything else.  Sorry I come from corporate real world
> experience not Soho stuff.  Use Linux.
>
> On Sat, Apr 21, 2018, 1:24 AM Rupert Gallagher <[hidden email]>
> wrote:
>
>> This is what I observed on a controlled environment of three "windows 10
>> pro" 1709 clients.
>>
>> The obsd nfs server had a single share:
>>
>> /path/to/folder -network 192.168.1 -mask 255.255.255.0
>>
>> When mounting a share for the first time, Windows allows browsing the
>> network to find the resource. This is what happens:
>>
>> 1. The client asks for the list of nfs resources;
>> 2. the server shows a stream of accepted mounts, no warnings, no errors;
>> 3. while 2 happens, the client shows a warning that the server is not
>> responding;
>> 4. when eventually the client returns the list of nfs folders, the server
>> crashes.
>>
>> The above occurs systematically. Restarting the server and repeating the
>> client steps lead to a new server crash. The only way to mount the share is
>> to type in the path, without browsing.
>>
>> When the server crashes, the debug shows no warnings and no errors.
>>
>> The problem did not occur with W10Pro 1703. However, the server should
>> not crash, and if it does, it should report useful diagnostics.
>>
>> R
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Andy Kosela
MB <[hidden email]> wrote:

> Invest in a NetApp filer and do it the right way.  Plenty of options
> spectrascale glfs/cnfs lustre with DNE/IME why struggle with this hobby OS,
> seriously?
>
> On Sat, Apr 21, 2018, 1:31 AM MB <[hidden email]> wrote:
>
> >
> > Why are you using ooenbsd for anything but a firewall.  Even then its
> > lagging way behind unless you deploying in a dentist office. Openbsd sucks
> > at pretty much everything else.  Sorry I come from corporate real world
> > experience not Soho stuff.  Use Linux.
> >

A couple of points:

  (1) Stop top-posting.
  (2) NetApp is using BSD nfs code.
  (3) I have run OpenBSD in a "corporate world" on a global scale and it
      usually outperforms everything else, including Linux, and
      definitely is much more stable and secure.
  (4) If OpenBSD "sucks" and you are on a mailing list that "sucks", your
      life must be truly miserable.

To the original author of this thread -- nfs is a UNIX technology,
originally made for Unix to Unix communication on a network.  Windows
client support came later and it is still not stellar.  Samba/CIFS is
what is a usual scenario here.  

But if you are serious about this and want to attract developers'
attention then please become familiar with sendbug(1).

  http://www.openbsd.org/report.html



--Andy

Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Matt Bettinger-2
Sure you ran on a global scale.  Uh huh.  How's that working for you?  I'm
in o&g in one of the top 3 and the only openbsd you see never maybe someone
like you who we break off a VM for to make them happy so they can run their
silly ntp server for their 3 person OU.  Come on.  I'd love to hear more
about your corporate NFS openbsd servers .  Please.

On Sat, Apr 21, 2018, 3:04 AM Andy Kosela <[hidden email]> wrote:

> MB <[hidden email]> wrote:
>
> > Invest in a NetApp filer and do it the right way.  Plenty of options
> > spectrascale glfs/cnfs lustre with DNE/IME why struggle with this hobby
> OS,
> > seriously?
> >
> > On Sat, Apr 21, 2018, 1:31 AM MB <[hidden email]> wrote:
> >
> > >
> > > Why are you using ooenbsd for anything but a firewall.  Even then its
> > > lagging way behind unless you deploying in a dentist office. Openbsd
> sucks
> > > at pretty much everything else.  Sorry I come from corporate real world
> > > experience not Soho stuff.  Use Linux.
> > >
>
> A couple of points:
>
>   (1) Stop top-posting.
>   (2) NetApp is using BSD nfs code.
>   (3) I have run OpenBSD in a "corporate world" on a global scale and it
>       usually outperforms everything else, including Linux, and
>       definitely is much more stable and secure.
>   (4) If OpenBSD "sucks" and you are on a mailing list that "sucks", your
>       life must be truly miserable.
>
> To the original author of this thread -- nfs is a UNIX technology,
> originally made for Unix to Unix communication on a network.  Windows
> client support came later and it is still not stellar.  Samba/CIFS is
> what is a usual scenario here.
>
> But if you are serious about this and want to attract developers'
> attention then please become familiar with sendbug(1).
>
>   http://www.openbsd.org/report.html
>
>
>
> --Andy
>
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Stuart Henderson
In reply to this post by Matt Bettinger-2
> On Sat, Apr 21, 2018, 1:24 AM Rupert Gallagher <[hidden email]> wrote:
>
> > This is what I observed on a controlled environment of three "windows 10
> > pro" 1709 clients.
> >
> > The obsd nfs server had a single share:
> >
> > /path/to/folder -network 192.168.1 -mask 255.255.255.0
> >
> > When mounting a share for the first time, Windows allows browsing the
> > network to find the resource. This is what happens:
> >
> > 1. The client asks for the list of nfs resources;
> > 2. the server shows a stream of accepted mounts, no warnings, no errors;
> > 3. while 2 happens, the client shows a warning that the server is not
> > responding;
> > 4. when eventually the client returns the list of nfs folders, the server
> > crashes.
> >
> > The above occurs systematically. Restarting the server and repeating the
> > client steps lead to a new server crash. The only way to mount the share is
> > to type in the path, without browsing.
> >
> > When the server crashes, the debug shows no warnings and no errors.
> >
> > The problem did not occur with W10Pro 1703. However, the server should not
> > crash, and if it does, it should report useful diagnostics.

Packet captures from broken and working clients would be a good start
to figuring out what's going on.

On 2018/04/21 06:31, MB wrote:
> Why are you using ooenbsd for anything but a firewall.  Even then its
> lagging way behind unless you deploying in a dentist office. Openbsd sucks
> at pretty much everything else.  Sorry I come from corporate real world
> experience not Soho stuff.  Use Linux.

Or Windows, you might as well get some nice features to go with the
complex mess.

Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Otto Moerbeek
On Sat, Apr 21, 2018 at 10:09:38AM +0100, Stuart Henderson wrote:

> > On Sat, Apr 21, 2018, 1:24 AM Rupert Gallagher <[hidden email]> wrote:
> >
> > > This is what I observed on a controlled environment of three "windows 10
> > > pro" 1709 clients.
> > >
> > > The obsd nfs server had a single share:
> > >
> > > /path/to/folder -network 192.168.1 -mask 255.255.255.0
> > >
> > > When mounting a share for the first time, Windows allows browsing the
> > > network to find the resource. This is what happens:
> > >
> > > 1. The client asks for the list of nfs resources;
> > > 2. the server shows a stream of accepted mounts, no warnings, no errors;
> > > 3. while 2 happens, the client shows a warning that the server is not
> > > responding;
> > > 4. when eventually the client returns the list of nfs folders, the server
> > > crashes.
> > >
> > > The above occurs systematically. Restarting the server and repeating the
> > > client steps lead to a new server crash. The only way to mount the share is
> > > to type in the path, without browsing.
> > >
> > > When the server crashes, the debug shows no warnings and no errors.
> > >
> > > The problem did not occur with W10Pro 1703. However, the server should not
> > > crash, and if it does, it should report useful diagnostics.
>
> Packet captures from broken and working clients would be a good start
> to figuring out what's going on.

What do you mean by "the server crashes"? Does the complete OS freeze?
Or is the OS still working apart from NFS?  Did one of te NFS related
daemons (nfsd, mountd, portmap) die?

        -Otto




Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Rupert Gallagher
On Sun, Apr 22, 2018 at 08:17, Otto Moerbeek <[hidden email]> wrote:

> In that case ktracing mountd might help (as well as (packet captures) to see what is going on.

Will get back with the results.
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Rupert Gallagher
In reply to this post by Matt Bettinger-2
The following patch from Microsoft seems to restore functionality. Will see in the following hours. The denial of service remains as a problem on mountd. Will get a spare pc asap and check ktrace.

https://support.microsoft.com/en-us/help/4088776/windows-10-update-kb4088776
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Rupert Gallagher
Keeps crashing.

ktrace attached.
​​

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On 26 April 2018 12:58 PM, Rupert Gallagher <[hidden email]> wrote:

> The following patch from Microsoft seems to restore functionality. Will see in the following hours. The denial of service remains as a problem on mountd. Will get a spare pc asap and check ktrace. 
>
> https://support.microsoft.com/en-us/help/4088776/windows-10-update-kb4088776

ktrace.out (347K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Otto Moerbeek
On Fri, Apr 27, 2018 at 02:04:35PM -0400, Rupert Gallagher wrote:

> Keeps crashing.
>
> ktrace attached.

No immediate clue pops up from the ktrace. I don't think I wil find
the time to debug this futher the coming period.

        -Otto

> ​​
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On 26 April 2018 12:58 PM, Rupert Gallagher <[hidden email]> wrote:
>
> > The following patch from Microsoft seems to restore functionality. Will see in the following hours. The denial of service remains as a problem on mountd. Will get a spare pc asap and check ktrace. 
> >
> > https://support.microsoft.com/en-us/help/4088776/windows-10-update-kb4088776


Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Rupert Gallagher
The lack of information originates from "mountd -d". When it terminates because of an error, it should log the name of the last function and its parameters.

----

On Fri, Apr 27, 2018 at 02:04:35PM -0400, Rupert Gallagher wrote:

> Keeps crashing.
>
> ktrace attached.

No immediate clue pops up from the ktrace. I don't think I wil find
the time to debug this futher the coming period.

-Otto

> ​​
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On 26 April 2018 12:58 PM, Rupert Gallagher <[hidden email]> wrote:
>
> > The following patch from Microsoft seems to restore functionality. Will see in the following hours. The denial of service remains as a problem on mountd. Will get a spare pc asap and check ktrace.
> >
> > https://support.microsoft.com/en-us/help/4088776/windows-10-update-kb4088776
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Theo de Raadt-2
For a while it was amazing watching someone completely misunderstand
the Zeitgeist of the circumstances ---- this is open source, you get
all the pieces from people who largely care, but you also get all
the pieces so that you can DECIDE to care and FIX them when you find
problems and participate in a commonwealth of shared uplift.

But no.  You just bitch about it.  You do no self learning.  You are
simply demanding someone solve your problem because you are noisy.  Yes
self-learning is hard.  That's precisely what all of us did to be able
to maintain the software this far.  Now it is in your ballcourt.  If you
don't do so, that speaks much more to your character than to our
character.

Watching this isn't amazing anymore.  It is seriously sad.

I've come to the conclusion that if you were walking down the
sidewalk and found a poop in your way, you'd pull out your phone and
mail misc about it.  You don't know how to self-help.

Rupert - you got all the pieces, source code and all.  Without
warrantees, from volunteers.  Make them work, or use something else.  In
particular, I have to advice you on a bigger picture component: you need
to grow up.

Rupert Gallagher <[hidden email]> wrote:

> The lack of information originates from "mountd -d". When it terminates because of an error, it should log the name of the last function and its parameters.
>
> ----
>
> On Fri, Apr 27, 2018 at 02:04:35PM -0400, Rupert Gallagher wrote:
>
> > Keeps crashing.
> >
> > ktrace attached.
>
> No immediate clue pops up from the ktrace. I don't think I wil find
> the time to debug this futher the coming period.
>
> -Otto
>
> > ​​
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >
> > On 26 April 2018 12:58 PM, Rupert Gallagher <[hidden email]> wrote:
> >
> > > The following patch from Microsoft seems to restore functionality. Will see in the following hours. The denial of service remains as a problem on mountd. Will get a spare pc asap and check ktrace.
> > >
> > > https://support.microsoft.com/en-us/help/4088776/windows-10-update-kb4088776

Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Otto Moerbeek
In reply to this post by Rupert Gallagher
On Fri, Apr 27, 2018 at 09:12:36PM -0400, Rupert Gallagher wrote:

> The lack of information originates from "mountd -d". When it terminates because of an error, it should log the name of the last function and its parameters.

Wrap your lines.

Well, ktrace does that for system calls. What I'm seeing is that a
socket is closed and mountd tries to write to it and then gets
SIGPIPE because of that. But I do not know yet what's cause and effect.

        -Otto

>
> ----
>
> On Fri, Apr 27, 2018 at 02:04:35PM -0400, Rupert Gallagher wrote:
>
> > Keeps crashing.
> >
> > ktrace attached.
>
> No immediate clue pops up from the ktrace. I don't think I wil find
> the time to debug this futher the coming period.
>
> -Otto
>
> > ​​
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >
> > On 26 April 2018 12:58 PM, Rupert Gallagher <[hidden email]> wrote:
> >
> > > The following patch from Microsoft seems to restore functionality. Will see in the following hours. The denial of service remains as a problem on mountd. Will get a spare pc asap and check ktrace.
> > >
> > > https://support.microsoft.com/en-us/help/4088776/windows-10-update-kb4088776

Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Otto Moerbeek
In reply to this post by Theo de Raadt-2
On Fri, Apr 27, 2018 at 07:29:50PM -0600, Theo de Raadt wrote:

> For a while it was amazing watching someone completely misunderstand
> the Zeitgeist of the circumstances ---- this is open source, you get
> all the pieces from people who largely care, but you also get all
> the pieces so that you can DECIDE to care and FIX them when you find
> problems and participate in a commonwealth of shared uplift.
>
> But no.  You just bitch about it.  You do no self learning.  You are
> simply demanding someone solve your problem because you are noisy.  Yes
> self-learning is hard.  That's precisely what all of us did to be able
> to maintain the software this far.  Now it is in your ballcourt.  If you
> don't do so, that speaks much more to your character than to our
> character.
>
> Watching this isn't amazing anymore.  It is seriously sad.
>
> I've come to the conclusion that if you were walking down the
> sidewalk and found a poop in your way, you'd pull out your phone and
> mail misc about it.  You don't know how to self-help.
>
> Rupert - you got all the pieces, source code and all.  Without
> warrantees, from volunteers.  Make them work, or use something else.  In
> particular, I have to advice you on a bigger picture component: you need
> to grow up.

Theo's right. To quote a piece of license text governining most of
the code:

 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS.

*I* only care about this (a bit) because I might run into this problem
myself some day. I decided already that trying to reproduce this is
likely much more effective than interacting with Rupert. But that will
take some time.

        -Otto

Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Rupert Gallagher
In reply to this post by Theo de Raadt-2
From my standpoint, I encountered a programming error that leads to denial of service, and returned the info you asked for. Then I meet you moaning about it, with your usual dump of negative energy. Are you depressed? I am not a social assistant.
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Philip Guenther
In reply to this post by Otto Moerbeek
On Sat, 28 Apr 2018, Otto Moerbeek wrote:
> On Fri, Apr 27, 2018 at 09:12:36PM -0400, Rupert Gallagher wrote:
>
> > The lack of information originates from "mountd -d". When it terminates because of an error, it should log the name of the last function and its parameters.
>
> Wrap your lines.
>
> Well, ktrace does that for system calls. What I'm seeing is that a
> socket is closed and mountd tries to write to it and then gets
> SIGPIPE because of that. But I do not know yet what's cause and effect.

Well, since it's handling TCP connections mountd should be ignoring
SIGPIPE, no?

Index: mountd.c
===================================================================
RCS file: /data/src/openbsd/src/sbin/mountd/mountd.c,v
retrieving revision 1.85
diff -u -p -r1.85 mountd.c
--- mountd.c 23 Dec 2015 21:32:52 -0000 1.85
+++ mountd.c 28 Apr 2018 08:39:46 -0000
@@ -295,6 +295,7 @@ main(int argc, char *argv[])
 
  signal(SIGCHLD, (void (*)(int)) check_child);
  signal(SIGHUP, (void (*)(int)) new_exportlist);
+ signal(SIGPIPE, SIG_IGN);
 
  if (socketpair(AF_UNIX, SOCK_STREAM, PF_UNSPEC, socks) == -1) {
  syslog(LOG_ERR, "socketpair: %m");

Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Otto Moerbeek
On Sat, Apr 28, 2018 at 10:40:01AM +0200, Philip Guenther wrote:

> On Sat, 28 Apr 2018, Otto Moerbeek wrote:
> > On Fri, Apr 27, 2018 at 09:12:36PM -0400, Rupert Gallagher wrote:
> >
> > > The lack of information originates from "mountd -d". When it terminates because of an error, it should log the name of the last function and its parameters.
> >
> > Wrap your lines.
> >
> > Well, ktrace does that for system calls. What I'm seeing is that a
> > socket is closed and mountd tries to write to it and then gets
> > SIGPIPE because of that. But I do not know yet what's cause and effect.
>
> Well, since it's handling TCP connections mountd should be ignoring
> SIGPIPE, no?

Yes, I was pondering that already.

Worth a try (I cannot reproduce, not having Windows 10 Pro),

        -Otto

>
> Index: mountd.c
> ===================================================================
> RCS file: /data/src/openbsd/src/sbin/mountd/mountd.c,v
> retrieving revision 1.85
> diff -u -p -r1.85 mountd.c
> --- mountd.c 23 Dec 2015 21:32:52 -0000 1.85
> +++ mountd.c 28 Apr 2018 08:39:46 -0000
> @@ -295,6 +295,7 @@ main(int argc, char *argv[])
>  
>   signal(SIGCHLD, (void (*)(int)) check_child);
>   signal(SIGHUP, (void (*)(int)) new_exportlist);
> + signal(SIGPIPE, SIG_IGN);
>  
>   if (socketpair(AF_UNIX, SOCK_STREAM, PF_UNSPEC, socks) == -1) {
>   syslog(LOG_ERR, "socketpair: %m");

Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Otto Moerbeek
On Sat, Apr 28, 2018 at 11:28:50AM +0200, Otto Moerbeek wrote:

> On Sat, Apr 28, 2018 at 10:40:01AM +0200, Philip Guenther wrote:
>
> > On Sat, 28 Apr 2018, Otto Moerbeek wrote:
> > > On Fri, Apr 27, 2018 at 09:12:36PM -0400, Rupert Gallagher wrote:
> > >
> > > > The lack of information originates from "mountd -d". When it terminates because of an error, it should log the name of the last function and its parameters.
> > >
> > > Wrap your lines.
> > >
> > > Well, ktrace does that for system calls. What I'm seeing is that a
> > > socket is closed and mountd tries to write to it and then gets
> > > SIGPIPE because of that. But I do not know yet what's cause and effect.
> >
> > Well, since it's handling TCP connections mountd should be ignoring
> > SIGPIPE, no?
>
> Yes, I was pondering that already.
>
> Worth a try (I cannot reproduce, not having Windows 10 Pro),

Realizing my daughter has an unused license to it, I upgraded the only
windows machine in this house to Windows 10 Education (which is almost
equivalent to Pro). I can now mount NFS volumes on Windows, but no
crashes so far, even when exercising the browse button in the dialog
muiltiple times.

So stil not able to reproduce. The windows machine has all the patches
up until today.

        -Otto

>
> >
> > Index: mountd.c
> > ===================================================================
> > RCS file: /data/src/openbsd/src/sbin/mountd/mountd.c,v
> > retrieving revision 1.85
> > diff -u -p -r1.85 mountd.c
> > --- mountd.c 23 Dec 2015 21:32:52 -0000 1.85
> > +++ mountd.c 28 Apr 2018 08:39:46 -0000
> > @@ -295,6 +295,7 @@ main(int argc, char *argv[])
> >  
> >   signal(SIGCHLD, (void (*)(int)) check_child);
> >   signal(SIGHUP, (void (*)(int)) new_exportlist);
> > + signal(SIGPIPE, SIG_IGN);
> >  
> >   if (socketpair(AF_UNIX, SOCK_STREAM, PF_UNSPEC, socks) == -1) {
> >   syslog(LOG_ERR, "socketpair: %m");

Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Rupert Gallagher
On Sat, Apr 28, 2018 at 16:48, Otto Moerbeek <[hidden email]> wrote:

> The windows machine has all the patches up until today.

It crashes here on multiple W10Pro  without the following KB. A single station is sufficient to crash the whole network.

https://support.microsoft.com/en-us/help/4088776/windows-10-update-kb4088776
Reply | Threaded
Open this post in threaded view
|

Re: NFS keeps crashing

Rupert Gallagher
Some stations crash it when mounting, other when browsing deeper and deeper into folders, other when trying to copy a 700MB file.
12