squid 4.6 issue on OpenBSD 6.5

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

squid 4.6 issue on OpenBSD 6.5

Matthieu Herrb-3
Hi,

I recently upgraded the squid cache at work to OpenBSD 6.5 (and thus
squid 4.6). After this, squid started locking up (and not accepting
new connections) every few hours with the error below logged in
cache.log:

oldAccept  FD 17, [::] [ job1]: ignoring: (53) Software caused
connection abort

After digging a bit in the source code and reading about ECONNREFUSED
and accept(), I came out with the patch below, which fixes the issue
for me. The patch was done on 6.5, but the squid port is the same in
-current for now.

Does this look ok ?

Index: Makefile
===================================================================
RCS file: /cvs/ports/www/squid/Makefile,v
retrieving revision 1.213
diff -u -r1.213 Makefile
--- Makefile 20 Feb 2019 16:23:27 -0000 1.213
+++ Makefile 10 May 2019 09:07:33 -0000
@@ -10,6 +10,7 @@
 PKGNAME-main= squid-$V
 PKGNAME-ldap= squid-ldap-$V
 PKGNAME-ntlm= squid-ntlm-$V
+REVISION= 0
 
 CATEGORIES= www
 
Index: patches/patch-src_comm_TcpAcceptor_cc
===================================================================
RCS file: patches/patch-src_comm_TcpAcceptor_cc
diff -N patches/patch-src_comm_TcpAcceptor_cc
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ patches/patch-src_comm_TcpAcceptor_cc 10 May 2019 09:07:33 -0000
@@ -0,0 +1,16 @@
+$OpenBSD$
+
+Patch for locksup observed when accept() returns ECONNABORTED
+
+--- src/comm/TcpAcceptor.cc.orig Tue Feb 19 03:46:22 2019
++++ src/comm/TcpAcceptor.cc Fri May 10 11:01:05 2019
+@@ -369,6 +369,9 @@
+         if (ignoreErrno(errcode)) {
+             debugs(50, 5, status() << ": " << xstrerr(errcode));
+             return Comm::NOMESSAGE;
++ } else if (ECONNABORTED == errno) {
++    debugs(50, DBG_IMPORTANT, MYNAME << status() << ": ignoring: " << xstrerr(errcode));
++    return Comm::NOMESSAGE;
+         } else if (ENFILE == errno || EMFILE == errno) {
+             debugs(50, 3, status() << ": " << xstrerr(errcode));
+             return Comm::COMM_ERROR;

--
Matthieu Herrb

Reply | Threaded
Open this post in threaded view
|

Re: squid 4.6 issue on OpenBSD 6.5

Stuart Henderson-6
On 2019/05/10 11:45, Matthieu Herrb wrote:

> Hi,
>
> I recently upgraded the squid cache at work to OpenBSD 6.5 (and thus
> squid 4.6). After this, squid started locking up (and not accepting
> new connections) every few hours with the error below logged in
> cache.log:
>
> oldAccept  FD 17, [::] [ job1]: ignoring: (53) Software caused
> connection abort
>
> After digging a bit in the source code and reading about ECONNREFUSED
> and accept(), I came out with the patch below, which fixes the issue
> for me. The patch was done on 6.5, but the squid port is the same in
> -current for now.
>
> Does this look ok ?

Thanks for digging - OK. Would you mind either doing a PR
(https://wiki.squid-cache.org/MergeProcedure) or bringing it up on
squid-dev please?

> Index: Makefile
> ===================================================================
> RCS file: /cvs/ports/www/squid/Makefile,v
> retrieving revision 1.213
> diff -u -r1.213 Makefile
> --- Makefile 20 Feb 2019 16:23:27 -0000 1.213
> +++ Makefile 10 May 2019 09:07:33 -0000
> @@ -10,6 +10,7 @@
>  PKGNAME-main= squid-$V
>  PKGNAME-ldap= squid-ldap-$V
>  PKGNAME-ntlm= squid-ntlm-$V
> +REVISION= 0

REVISION-main=0 would be enough, but ok either way.

>  CATEGORIES= www
>  
> Index: patches/patch-src_comm_TcpAcceptor_cc
> ===================================================================
> RCS file: patches/patch-src_comm_TcpAcceptor_cc
> diff -N patches/patch-src_comm_TcpAcceptor_cc
> --- /dev/null 1 Jan 1970 00:00:00 -0000
> +++ patches/patch-src_comm_TcpAcceptor_cc 10 May 2019 09:07:33 -0000
> @@ -0,0 +1,16 @@
> +$OpenBSD$
> +
> +Patch for locksup observed when accept() returns ECONNABORTED
> +
> +--- src/comm/TcpAcceptor.cc.orig Tue Feb 19 03:46:22 2019
> ++++ src/comm/TcpAcceptor.cc Fri May 10 11:01:05 2019
> +@@ -369,6 +369,9 @@
> +         if (ignoreErrno(errcode)) {
> +             debugs(50, 5, status() << ": " << xstrerr(errcode));
> +             return Comm::NOMESSAGE;
> ++ } else if (ECONNABORTED == errno) {
> ++    debugs(50, DBG_IMPORTANT, MYNAME << status() << ": ignoring: " << xstrerr(errcode));
> ++    return Comm::NOMESSAGE;
> +         } else if (ENFILE == errno || EMFILE == errno) {
> +             debugs(50, 3, status() << ": " << xstrerr(errcode));
> +             return Comm::COMM_ERROR;
>
> --
> Matthieu Herrb

Reply | Threaded
Open this post in threaded view
|

Re: squid 4.6 issue on OpenBSD 6.5

Matthieu Herrb-3
On Fri, May 10, 2019 at 01:38:19PM +0100, Stuart Henderson wrote:

> On 2019/05/10 11:45, Matthieu Herrb wrote:
> > Hi,
> >
> > I recently upgraded the squid cache at work to OpenBSD 6.5 (and thus
> > squid 4.6). After this, squid started locking up (and not accepting
> > new connections) every few hours with the error below logged in
> > cache.log:
> >
> > oldAccept  FD 17, [::] [ job1]: ignoring: (53) Software caused
> > connection abort
> >
> > After digging a bit in the source code and reading about ECONNREFUSED
> > and accept(), I came out with the patch below, which fixes the issue
> > for me. The patch was done on 6.5, but the squid port is the same in
> > -current for now.
> >
> > Does this look ok ?
>
> Thanks for digging - OK. Would you mind either doing a PR
> (https://wiki.squid-cache.org/MergeProcedure) or bringing it up on
> squid-dev please?

Done. https://github.com/squid-cache/squid/pull/404

--
Matthieu Herrb

Reply | Threaded
Open this post in threaded view
|

Re: squid 4.6 issue on OpenBSD 6.5

Matthieu Herrb-3
In reply to this post by Stuart Henderson-6
On Fri, May 10, 2019 at 01:38:19PM +0100, Stuart Henderson wrote:

> On 2019/05/10 11:45, Matthieu Herrb wrote:
> > Hi,
> >
> > I recently upgraded the squid cache at work to OpenBSD 6.5 (and thus
> > squid 4.6). After this, squid started locking up (and not accepting
> > new connections) every few hours with the error below logged in
> > cache.log:
> >
> > oldAccept  FD 17, [::] [ job1]: ignoring: (53) Software caused
> > connection abort
> >
> > After digging a bit in the source code and reading about ECONNREFUSED
> > and accept(), I came out with the patch below, which fixes the issue
> > for me. The patch was done on 6.5, but the squid port is the same in
> > -current for now.
> >
> > Does this look ok ?
New version based on your comments and the ones from upstreams.

Index: Makefile
===================================================================
RCS file: /cvs/OpenBSD/ports/www/squid/Makefile,v
retrieving revision 1.213
diff -u -p -u -r1.213 Makefile
--- Makefile 20 Feb 2019 16:23:27 -0000 1.213
+++ Makefile 10 May 2019 19:37:14 -0000
@@ -10,6 +10,7 @@ EXTRACT_SUFX= .tar.xz
 PKGNAME-main= squid-$V
 PKGNAME-ldap= squid-ldap-$V
 PKGNAME-ntlm= squid-ntlm-$V
+REVISION-main= 0
 
 CATEGORIES= www
 
Index: patches/patch-src_comm_TcpAcceptor_cc
===================================================================
RCS file: patches/patch-src_comm_TcpAcceptor_cc
diff -N patches/patch-src_comm_TcpAcceptor_cc
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ patches/patch-src_comm_TcpAcceptor_cc 10 May 2019 19:35:40 -0000
@@ -0,0 +1,16 @@
+$OpenBSD$
+
+Patch for locksup observed when accept() returns ECONNABORTED
+
+Index: src/comm/TcpAcceptor.cc
+--- src/comm/TcpAcceptor.cc.orig
++++ src/comm/TcpAcceptor.cc
+@@ -361,7 +361,7 @@ Comm::TcpAcceptor::oldAccept(Comm::ConnectionPointer &details)
+
+         PROF_stop(comm_accept);
+
+-        if (ignoreErrno(errcode)) {
++        if (ignoreErrno(errcode) || ECONNABORTED == errcode) {
+             debugs(50, 5, status() << ": " << xstrerr(errcode));
+             return Comm::NOMESSAGE;
+        } else if (ENFILE == errno || EMFILE == errno) {

--
Matthieu Herrb

Reply | Threaded
Open this post in threaded view
|

Re: squid 4.6 issue on OpenBSD 6.5

Jeremie Courreges-Anglas-2
On Fri, May 10 2019, Matthieu Herrb <[hidden email]> wrote:

> On Fri, May 10, 2019 at 01:38:19PM +0100, Stuart Henderson wrote:
>> On 2019/05/10 11:45, Matthieu Herrb wrote:
>> > Hi,
>> >
>> > I recently upgraded the squid cache at work to OpenBSD 6.5 (and thus
>> > squid 4.6). After this, squid started locking up (and not accepting
>> > new connections) every few hours with the error below logged in
>> > cache.log:
>> >
>> > oldAccept  FD 17, [::] [ job1]: ignoring: (53) Software caused
>> > connection abort
>> >
>> > After digging a bit in the source code and reading about ECONNREFUSED
>> > and accept(), I came out with the patch below, which fixes the issue
>> > for me. The patch was done on 6.5, but the squid port is the same in
>> > -current for now.
>> >
>> > Does this look ok ?
> New version based on your comments and the ones from upstreams.

Makes more sense to me, DBG_IMPORTANT looked too noisy for ECONNABORTED.
ok jca@

> Index: Makefile
> ===================================================================
> RCS file: /cvs/OpenBSD/ports/www/squid/Makefile,v
> retrieving revision 1.213
> diff -u -p -u -r1.213 Makefile
> --- Makefile 20 Feb 2019 16:23:27 -0000 1.213
> +++ Makefile 10 May 2019 19:37:14 -0000
> @@ -10,6 +10,7 @@ EXTRACT_SUFX= .tar.xz
>  PKGNAME-main= squid-$V
>  PKGNAME-ldap= squid-ldap-$V
>  PKGNAME-ntlm= squid-ntlm-$V
> +REVISION-main= 0
>  
>  CATEGORIES= www
>  
> Index: patches/patch-src_comm_TcpAcceptor_cc
> ===================================================================
> RCS file: patches/patch-src_comm_TcpAcceptor_cc
> diff -N patches/patch-src_comm_TcpAcceptor_cc
> --- /dev/null 1 Jan 1970 00:00:00 -0000
> +++ patches/patch-src_comm_TcpAcceptor_cc 10 May 2019 19:35:40 -0000
> @@ -0,0 +1,16 @@
> +$OpenBSD$
> +
> +Patch for locksup observed when accept() returns ECONNABORTED
> +
> +Index: src/comm/TcpAcceptor.cc
> +--- src/comm/TcpAcceptor.cc.orig
> ++++ src/comm/TcpAcceptor.cc
> +@@ -361,7 +361,7 @@ Comm::TcpAcceptor::oldAccept(Comm::ConnectionPointer &details)
> +
> +         PROF_stop(comm_accept);
> +
> +-        if (ignoreErrno(errcode)) {
> ++        if (ignoreErrno(errcode) || ECONNABORTED == errcode) {
> +             debugs(50, 5, status() << ": " << xstrerr(errcode));
> +             return Comm::NOMESSAGE;
> +        } else if (ENFILE == errno || EMFILE == errno) {

--
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE