uip_socket.c: issues when using sendmsg() with small send buffers and the new 6.1 control message (IP_SENDSRCADDR)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

uip_socket.c: issues when using sendmsg() with small send buffers and the new 6.1 control message (IP_SENDSRCADDR)

Markert, Alexander
Hi,

the length of the data to be sent and in addition the length of the associated control message (e.g. the source address: IP_SENDSRCADDR) musn't exceed the maximum size of the socket's send buffer (SO_SNDBUF).
Otherwise EMSGSIZE is returned. That is ok.

However, if the data to be sent fits into the send buffer, but exceeds in combination with the control message:
- either EWOULDBLOCK is returned when using non-blocking IO or
- send operation blocks in sbwait when using blocking IO

It seems that this situation will never dissolve, i.e. the sending application will be blocked forever.

In our opinion either EMSGSIZE should be returned instead in this case (like e.g. FreeBSD 11.0 does) or OpenBSD should reserve some space (comparable to MSG_OOB) in addition to the maximum size of the socket's send buffer for the control messages in order to avoid such problems.

What do you think about that behavior and the described alternatives?

Best,
Alex

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: uip_socket.c: issues when using sendmsg() with small send buffers and the new 6.1 control message (IP_SENDSRCADDR)

Jeremie Courreges-Anglas-2
"Markert, Alexander" <[hidden email]> writes:

> Hi,

Hi,

> the length of the data to be sent and in addition the length of the associated control message (e.g. the source address: IP_SENDSRCADDR) musn't exceed the maximum size of the socket's send buffer (SO_SNDBUF).
> Otherwise EMSGSIZE is returned. That is ok.
>
> However, if the data to be sent fits into the send buffer, but exceeds in combination with the control message:
> - either EWOULDBLOCK is returned when using non-blocking IO or
> - send operation blocks in sbwait when using blocking IO
>
> It seems that this situation will never dissolve, i.e. the sending application will be blocked forever.
>
> In our opinion either EMSGSIZE should be returned instead in this case
> (like e.g. FreeBSD 11.0 does) or OpenBSD should reserve some space
> (comparable to MSG_OOB) in addition to the maximum size of the socket's
> send buffer for the control messages in order to avoid such problems.
>
> What do you think about that behavior and the described alternatives?

Control messages are tricky.  One port (net/dnsmasq) is currently
failing, and appears to hang like you describe.  The issue is related to
control messages, but it's not a lack of send buffer space.

So how can one reproduce the bug you're talking about?  Do you have
a test case?

--
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: uip_socket.c: issues when using sendmsg() with small send buffers and the new 6.1 control message (IP_SENDSRCADDR)

Markert, Alexander
Hi,

actually you are right that this issue is related to control messages and not to the send buffer length. But the length of the control message is checked in combination with the data to be sent in uip_socket.c:

Let's assume we are sending data with the maximum send buffer size (e.g. 1024).
Since clen is NOT evaluated in the first if statement, EMSGSIZE is NOT returned.
But the second if statement becomes true, since clen is considered here. Hence either EWOULDBLOCK (snderr) is returned or sbwait() is called.

                if ((atomic && resid > so->so_snd.sb_hiwat) ||
                    (so->so_proto->pr_domain->dom_family != AF_LOCAL &&
                    clen > so->so_snd.sb_hiwat))
                        snderr(EMSGSIZE);
                if (space < clen ||
                    (space - clen < resid &&
                    (atomic || space < so->so_snd.sb_lowat))) {
                        if ((so->so_state & SS_NBIO) || (flags & MSG_DONTWAIT))
                                snderr(EWOULDBLOCK);
                        sbunlock(&so->so_snd);
                        error = sbwait(&so->so_snd);

I attached a small example, which
- sets the sockets maximum buffer length
- enables/disables non-blocking IO and
- calls sendmsg with the maximum buffer size including a control message (IP_SENDSRCADDR).


-----Original Message-----
From: Jeremie Courreges-Anglas [mailto:[hidden email]]
Sent: Dienstag, 25. April 2017 15:00
To: Markert, Alexander (DF FA AS DH FTH 2)
Cc: [hidden email]
Subject: Re: uip_socket.c: issues when using sendmsg() with small send buffers and the new 6.1 control message (IP_SENDSRCADDR)

"Markert, Alexander" <[hidden email]> writes:

> Hi,

Hi,

> the length of the data to be sent and in addition the length of the associated control message (e.g. the source address: IP_SENDSRCADDR) musn't exceed the maximum size of the socket's send buffer (SO_SNDBUF).
> Otherwise EMSGSIZE is returned. That is ok.
>
> However, if the data to be sent fits into the send buffer, but exceeds in combination with the control message:
> - either EWOULDBLOCK is returned when using non-blocking IO or
> - send operation blocks in sbwait when using blocking IO
>
> It seems that this situation will never dissolve, i.e. the sending application will be blocked forever.
>
> In our opinion either EMSGSIZE should be returned instead in this case
> (like e.g. FreeBSD 11.0 does) or OpenBSD should reserve some space
> (comparable to MSG_OOB) in addition to the maximum size of the
> socket's send buffer for the control messages in order to avoid such problems.
>
> What do you think about that behavior and the described alternatives?
Control messages are tricky.  One port (net/dnsmasq) is currently failing, and appears to hang like you describe.  The issue is related to control messages, but it's not a lack of send buffer space.

So how can one reproduce the bug you're talking about?  Do you have a test case?

--
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

main.cpp (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: uip_socket.c: issues when using sendmsg() with small send buffers and the new 6.1 control message (IP_SENDSRCADDR)

Martin Pieuchot
On 26/04/17(Wed) 12:29, Markert, Alexander wrote:

> Hi,
>
> actually you are right that this issue is related to control messages and not to the send buffer length. But the length of the control message is checked in combination with the data to be sent in uip_socket.c:
>
> Let's assume we are sending data with the maximum send buffer size (e.g. 1024).
> Since clen is NOT evaluated in the first if statement, EMSGSIZE is NOT returned.
> But the second if statement becomes true, since clen is considered here. Hence either EWOULDBLOCK (snderr) is returned or sbwait() is called.
>
> if ((atomic && resid > so->so_snd.sb_hiwat) ||
>    (so->so_proto->pr_domain->dom_family != AF_LOCAL &&
>    clen > so->so_snd.sb_hiwat))
> snderr(EMSGSIZE);
> if (space < clen ||
>    (space - clen < resid &&
>    (atomic || space < so->so_snd.sb_lowat))) {
> if ((so->so_state & SS_NBIO) || (flags & MSG_DONTWAIT))
> snderr(EWOULDBLOCK);
> sbunlock(&so->so_snd);
> error = sbwait(&so->so_snd);
>
> I attached a small example, which
> - sets the sockets maximum buffer length
> - enables/disables non-blocking IO and
> - calls sendmsg with the maximum buffer size including a control message (IP_SENDSRCADDR).

I KNF'ed & converted the test case to our framework.  I didn't include
it in the existing regress/sys/netinet/sendscraddr test because I don't
grok it.

Alexander I used our standard license template for your test, is it ok?

Vincent, Jeremie could you look at this?

Index: ip_sendsrcaddr/Makefile
===================================================================
RCS file: ip_sendsrcaddr/Makefile
diff -N ip_sendsrcaddr/Makefile
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ ip_sendsrcaddr/Makefile 8 May 2017 13:41:33 -0000
@@ -0,0 +1,5 @@
+# $OpenBSD: Makefile,v 1.1 2017/04/30 09:03:58 mpi Exp $
+
+PROG= ip_sendsrcaddr
+
+.include <bsd.regress.mk>
Index: ip_sendsrcaddr/ip_sendsrcaddr.c
===================================================================
RCS file: ip_sendsrcaddr/ip_sendsrcaddr.c
diff -N ip_sendsrcaddr/ip_sendsrcaddr.c
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ ip_sendsrcaddr/ip_sendsrcaddr.c 8 May 2017 13:39:54 -0000
@@ -0,0 +1,100 @@
+/*
+ * Copyright (c) 2017 Alexander Markert <[hidden email]>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <arpa/inet.h>
+#include <netinet/in.h>
+
+#include <assert.h>
+#include <err.h>
+#include <errno.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+#define CFG_SOURCE_ADDRESS "192.168.88.135"
+#define CFG_DESTINATION_ADDRESS "192.168.57.44"
+#define CFG_PORT 5000
+#define CFG_SO_MAX_SEND_BUFFER 1024
+
+int blocking = 0;
+
+int main(int argc, char* argv[])
+{
+ char cmsgbuf[CMSG_SPACE(sizeof(struct in_addr))];;
+ struct sockaddr_in to;
+ char data[CFG_SO_MAX_SEND_BUFFER] = {0};
+ unsigned int size = CFG_SO_MAX_SEND_BUFFER;
+ struct in_addr *source_address;
+ struct msghdr msg;
+ struct cmsghdr *cmsg;
+ struct iovec iov;
+ int so, bytes;
+
+ so = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
+ if (so < 0)
+ err(-1, "opening socket failed");
+
+ if (setsockopt(so, SOL_SOCKET, SO_SNDBUF, &size, sizeof(size)) < 0)
+ err(-1, "adjusting socket send buffer failed");
+
+ if (!blocking) {
+ unsigned long on = 1;
+
+ if (ioctl(so, FIONBIO, &on) < 0)
+ err(-1, "enabling non-blocking IO failed");
+ }
+
+ /* setup remote address */
+ memset(&to, 0, sizeof(to));
+ to.sin_family = AF_INET;
+ to.sin_addr.s_addr = inet_addr(CFG_DESTINATION_ADDRESS);
+ to.sin_port = htons(CFG_PORT);
+
+ /* setup buffer to be sent */
+ msg.msg_name = &to;
+ msg.msg_namelen = sizeof(to);
+ iov.iov_base = data;
+ iov.iov_len = sizeof(data);
+ msg.msg_iovlen = 1;
+ msg.msg_iov = &iov;
+
+ /* setup configuration for source address */
+ memset(cmsgbuf, 0, sizeof(cmsgbuf));
+ msg.msg_control = cmsgbuf;
+ msg.msg_controllen = sizeof(cmsgbuf);
+ cmsg = CMSG_FIRSTHDR(&msg);
+ cmsg->cmsg_level = IPPROTO_IP;
+ cmsg->cmsg_type = IP_SENDSRCADDR;
+ cmsg->cmsg_len = CMSG_LEN(sizeof(struct in_addr));
+ source_address = (struct in_addr *)(CMSG_DATA(cmsg));
+ source_address->s_addr = inet_addr(CFG_SOURCE_ADDRESS);
+
+ if ((bytes = sendmsg(so, &msg, 0)) < 0) {
+ if (!blocking)
+ if (errno != EMSGSIZE)
+ err(-1, "incorrect errno");
+ else
+ err(-1, "sendmsg failed");
+ } else {
+ printf("%d bytes sent\n", bytes);
+ }
+
+ close(so);
+ return 0;
+}

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: uip_socket.c: issues when using sendmsg() with small send buffers and the new 6.1 control message (IP_SENDSRCADDR)

Alexander Bluhm
In reply to this post by Markert, Alexander
On Fri, Apr 21, 2017 at 08:43:11AM +0000, Markert, Alexander wrote:
> In our opinion either EMSGSIZE should be returned instead in this case (like e.g. FreeBSD 11.0 does) or OpenBSD should reserve some space (comparable to MSG_OOB) in addition to the maximum size of the socket's send buffer for the control messages in order to avoid such problems.
>
> What do you think about that behavior and the described alternatives?

I think EMSGSIZE if send space is too small, would be the way to
go.  On the reveive path also some bytes of space must be reseverd
by the user program, if control messages are used.  This behavior
should be symmetric for the send path.

bluhm

Loading...