openrsync out of memory

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

openrsync out of memory

Jan Stary
This is 6.5-current on an old ALIX (dmesg bellow).
I am syncing its backups to a remote machine with openrsync.

It works on all files except one that is big:

196M /backup/gw.stare.cz/dump.home.0
32.0K /backup/gw.stare.cz/dump.home.1
32.0K /backup/gw.stare.cz/dump.home.2
32.0K /backup/gw.stare.cz/dump.home.3
64.0K /backup/gw.stare.cz/dump.home.4
75.3M /backup/gw.stare.cz/dump.root.0
64.0K /backup/gw.stare.cz/dump.root.1
64.0K /backup/gw.stare.cz/dump.root.2
64.0K /backup/gw.stare.cz/dump.root.3
64.0K /backup/gw.stare.cz/dump.root.4
10.2M /backup/gw.stare.cz/dump.var.0
2.1M /backup/gw.stare.cz/dump.var.1
2.1M /backup/gw.stare.cz/dump.var.2
2.1M /backup/gw.stare.cz/dump.var.3
2.2M /backup/gw.stare.cz/dump.var.4
1.6G /backup/gw.stare.cz/dump.var.log.0

$ openrsync -av --del /backup/gw.stare.cz/ remote:/backup/gw.stare.cz
(sits there for a long time once it gets to the big file)
/usr/src/usr.bin/rsync/sender.c:550: error: /backup/gw.stare.cz/dump.var.log.0: mmap: Cannot allocate memory
/usr/src/usr.bin/rsync/client.c:86: error: rsync_sender

According to ktrace,

66400 openrsync STRU  struct stat { dev=9, ino=16224, mode=-rw------- , nlink=1
, uid=1000<"hans">, gid=0<"wheel">, rdev=115088, atime=1562283014<"Jul  5 01:30:
14 2019">.166197875, mtime=1565912129<"Aug 16 01:35:29 2019">.689322159, ctime=1
565912129<"Aug 16 01:35:29 2019">.689322159, size=1692231680, blocks=3305600, bl
ksize=65536, flags=0x0, gen=0x31cdf9ee }
 66400 openrsync RET   fstat 0
 66400 openrsync CALL  kbind(0xcf7d08b0,12,0xd5a85aa662cb95c1)
 66400 openrsync RET   kbind 0
 66400 openrsync CALL  mmap(0,0x64dd6800,0x1<PROT_READ>,0x1<MAP_SHARED>,4,0)
 66400 openrsync RET   mmap -1 errno 12 Cannot allocate memory

Does that mean openrsync tries to mmap() the entire file?
The machine only has 256MB of memory, but it does transfer
a test file of 300MB, so that can't be it.

        Jan



OpenBSD 6.5-current (GENERIC) #6: Sat May  4 19:43:01 MDT 2019
    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC
real mem  = 267931648 (255MB)
avail mem = 247394304 (235MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: date 11/05/08, BIOS32 rev. 0 @ 0xfd088
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: pcibios_get_intr_routing - function not supported
pcibios0: PCI IRQ Routing information unavailable.
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xe0000/0xa800
cpu0 at mainbus0: (uniprocessor)
cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD" 586-class) 499 MHz, 05-0a-02
cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW
mtrr: K6-family MTRR support (2 registers)
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 1 function 0 "AMD Geode LX" rev 0x33
glxsb0 at pci0 dev 1 function 2 "AMD Geode LX Crypto" rev 0x00: RNG AES
vr0 at pci0 dev 9 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 10, address 00:0d:b9:1a:a4:10
ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI 0x004063, model 0x0034
vr1 at pci0 dev 10 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 11, address 00:0d:b9:1a:a4:11
ukphy1 at vr1 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI 0x004063, model 0x0034
vr2 at pci0 dev 11 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 15, address 00:0d:b9:1a:a4:12
ukphy2 at vr2 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI 0x004063, model 0x0034
athn0 at pci0 dev 12 function 0 "Atheros AR9280" rev 0x01: irq 9
athn0: AR9280 rev 2 (2T2R), ROM rev 22, address 04:f0:21:01:d6:86
glxpcib0 at pci0 dev 15 function 0 "AMD CS5536 ISA" rev 0x03: rev 3, 32-bit 3579545Hz timer, watchdog, gpio, i2c
gpio0 at glxpcib0: 32 pins
iic0 at glxpcib0
maxtmp0 at iic0 addr 0x4c: lm86
pciide0 at pci0 dev 15 function 2 "AMD CS5536 IDE" rev 0x01: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <SDCFHS-016G>
wd0: 1-sector PIO, LBA48, 15279MB, 31293360 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
ohci0 at pci0 dev 15 function 4 "AMD CS5536 USB" rev 0x02: irq 12, version 1.0, legacy support
ehci0 at pci0 dev 15 function 5 "AMD CS5536 USB" rev 0x02: irq 12
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
isa0 at glxpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
nvram: invalid checksum
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on wd0a (9cd0e5ba033bd225.a) swap on wd0b dump on wd0b
clock: unknown CMOS layout

Reply | Threaded
Open this post in threaded view
|

Re: openrsync out of memory

Joe Davis
By the looks of it, openrsync does attempt to map the entire file, from
usr.bin/rsync/uploader.c:

    mapsz = st.st_size;
    map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);

The likely reason for your out of memory error is the default datasize
in login.conf. IIRC on some arches it's set to 768MB by default, which
would allow your 300MB file to transfer, but would cause mmap to fail
upon attempting to map the 1.6GB one.

Increasing the default limits in /etc/login.conf should fix the problem.

Note that rsync (not openrsync), doesn't use mmap for other reasons,
from rsync-3.1.3/fileio.c:

/* This provides functionality somewhat similar to mmap() but using read().
 * It gives sliding window access to a file.  mmap() is not used because of
 * the possibility of another program (such as a mailer) truncating the
 * file thus giving us a SIGBUS. */

Cheers,
Joe

Reply | Threaded
Open this post in threaded view
|

Re: openrsync out of memory

Theo de Raadt-2
Joe Davis <[hidden email]> wrote:

> By the looks of it, openrsync does attempt to map the entire file, from
> usr.bin/rsync/uploader.c:
>
>     mapsz = st.st_size;
>     map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);
>
> The likely reason for your out of memory error is the default datasize
> in login.conf. IIRC on some arches it's set to 768MB by default, which
> would allow your 300MB file to transfer, but would cause mmap to fail
> upon attempting to map the 1.6GB one.
>
> Increasing the default limits in /etc/login.conf should fix the problem.
>
> Note that rsync (not openrsync), doesn't use mmap for other reasons,
> from rsync-3.1.3/fileio.c:
>
> /* This provides functionality somewhat similar to mmap() but using read().
>  * It gives sliding window access to a file.  mmap() is not used because of
>  * the possibility of another program (such as a mailer) truncating the
>  * file thus giving us a SIGBUS. */

It should be rewritten to not use mmap.  mmap is not needed in this case,
since it is a network speaker.  Meanwhile, fallback code isn't present.
Pretty ridiculous..

Reply | Threaded
Open this post in threaded view
|

Re: openrsync out of memory

Jan Stary
In reply to this post by Jan Stary
On Aug 16 10:43:41, [hidden email] wrote:
> Does that mean openrsync tries to mmap() the entire file?
> The machine only has 256MB of memory, but it does transfer
> a test file of 300MB, so that can't be it.

I forgot about 1GB swap, so that's why it works
for files up to around 1.2G, but not larger.

  Jan

Reply | Threaded
Open this post in threaded view
|

Re: openrsync out of memory

Christian Weisgerber
On 2019-08-16, Jan Stary <[hidden email]> wrote:

>> Does that mean openrsync tries to mmap() the entire file?
>> The machine only has 256MB of memory, but it does transfer
>> a test file of 300MB, so that can't be it.
>
> I forgot about 1GB swap, so that's why it works
> for files up to around 1.2G, but not larger.

Why would the size of physical memory + swap matter?
mmap() doesn't copy a file into memory, it maps it into the address
space.

--
Christian "naddy" Weisgerber                          [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: openrsync out of memory

Theo de Raadt-2
Christian Weisgerber <[hidden email]> wrote:

> On 2019-08-16, Jan Stary <[hidden email]> wrote:
>
> >> Does that mean openrsync tries to mmap() the entire file?
> >> The machine only has 256MB of memory, but it does transfer
> >> a test file of 300MB, so that can't be it.
> >
> > I forgot about 1GB swap, so that's why it works
> > for files up to around 1.2G, but not larger.
>
> Why would the size of physical memory + swap matter?
> mmap() doesn't copy a file into memory, it maps it into the address
> space.

Yes, but when rsync calculates the hash, before exchanging the hash
to decide if it should ship the file, it touches every page of the
file, which pulls it into the process.........

Reply | Threaded
Open this post in threaded view
|

Re: openrsync out of memory

Sebastian Benoit
In reply to this post by Joe Davis
Joe Davis([hidden email]) on 2019.08.16 12:26:36 +0100:

> By the looks of it, openrsync does attempt to map the entire file, from
> usr.bin/rsync/uploader.c:
>
>     mapsz = st.st_size;
>     map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);
>
> The likely reason for your out of memory error is the default datasize
> in login.conf. IIRC on some arches it's set to 768MB by default, which
> would allow your 300MB file to transfer, but would cause mmap to fail
> upon attempting to map the 1.6GB one.
>
> Increasing the default limits in /etc/login.conf should fix the problem.
>
> Note that rsync (not openrsync), doesn't use mmap for other reasons,
> from rsync-3.1.3/fileio.c:
>
> /* This provides functionality somewhat similar to mmap() but using read().
>  * It gives sliding window access to a file.  mmap() is not used because of
>  * the possibility of another program (such as a mailer) truncating the
>  * file thus giving us a SIGBUS. */
>
> Cheers,
> Joe

Hi,

this replaces the mmap() with pread(), please try it out.

I dont much like the error handling here, but its a start.

ok?


diff --git usr.bin/rsync/uploader.c usr.bin/rsync/uploader.c
index fd07b22caeb..cce8b47a4c9 100644
--- usr.bin/rsync/uploader.c
+++ usr.bin/rsync/uploader.c
@@ -158,8 +158,8 @@ init_blk(struct blk *p, const struct blkset *set, off_t offs,
  p->len = idx < set->blksz - 1 ? set->len : set->rem;
  p->offs = offs;
 
- p->chksum_short = hash_fast(map + offs, p->len);
- hash_slow(map + offs, p->len, p->chksum_long, sess);
+ p->chksum_short = hash_fast(map, p->len);
+ hash_slow(map, p->len, p->chksum_long, sess);
 }
 
 /*
@@ -741,8 +741,9 @@ rsync_uploader(struct upload *u, int *fileinfd,
 {
  struct blkset    blk;
  struct stat    st;
- void   *map, *bufp;
- size_t    i, mapsz, pos, sz;
+ void   *mbuf, *bufp;
+ ssize_t    msz;
+ size_t    i, pos, sz;
  off_t    offs;
  int    c;
  const struct flist *f;
@@ -909,35 +910,46 @@ rsync_uploader(struct upload *u, int *fileinfd,
  blk.csum = u->csumlen;
 
  if (*fileinfd != -1 && st.st_size > 0) {
- mapsz = st.st_size;
- map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);
- if (map == MAP_FAILED) {
- ERR("%s: mmap", u->fl[u->idx].path);
- close(*fileinfd);
- *fileinfd = -1;
- return -1;
- }
-
  init_blkset(&blk, st.st_size);
  assert(blk.blksz);
 
  blk.blks = calloc(blk.blksz, sizeof(struct blk));
  if (blk.blks == NULL) {
  ERR("calloc");
- munmap(map, mapsz);
+ close(*fileinfd);
+ *fileinfd = -1;
+ return -1;
+ }
+
+ if ((mbuf = calloc(1, blk.len)) == NULL) {
+ ERR("calloc");
  close(*fileinfd);
  *fileinfd = -1;
  return -1;
  }
 
  offs = 0;
- for (i = 0; i < blk.blksz; i++) {
- init_blk(&blk.blks[i],
- &blk, offs, i, map, sess);
+ i = 0;
+ do {
+ msz = pread(*fileinfd, mbuf, blk.len, offs);
+ if (msz < 0) {
+ ERR("pread");
+ close(*fileinfd);
+ *fileinfd = -1;
+ return -1;
+ }
+ if ((size_t)msz != blk.len && (size_t)msz != blk.rem) {
+ /* short read, try again */
+ continue;
+ }
+ init_blk(&blk.blks[i], &blk, offs, i, mbuf, sess);
  offs += blk.len;
- }
+ LOG3(
+    "i=%ld, offs=%lld, msz=%ld, blk.len=%lu, blk.rem=%lu",
+    i, offs, msz, blk.len, blk.rem);
+ i++;
+ } while (i < blk.blksz);
 
- munmap(map, mapsz);
  close(*fileinfd);
  *fileinfd = -1;
  LOG3("%s: mapped %jd B with %zu blocks",

Reply | Threaded
Open this post in threaded view
|

Re: openrsync out of memory

Olivier ANTOINE-2
Hi,
On i386: before patch:
$ dd if=/dev/urandom  of=in bs=1M count=2k
$ openrsync --rsync-path=/usr/bin/openrsync  -av in localhost:out
Transfer starting: 1 files
sender.c:551: error: in: mmap: Cannot allocate memory
client.c:85: error: rsync_sender
receiver.c:345: error: poll: hangup
server.c:145: error: rsync_receiver

With your patch:
$ patch -p0 < /tmp/1
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git usr.bin/rsync/uploader.c usr.bin/rsync/uploader.c
|index fd07b22caeb..cce8b47a4c9 100644
|--- usr.bin/rsync/uploader.c
|+++ usr.bin/rsync/uploader.c
--------------------------
Patching file usr.bin/rsync/uploader.c using Plan A...
Hunk #1 succeeded at 158.
Hunk #2 succeeded at 741.
Hunk #3 succeeded at 910.
Hmm...  Ignoring the trailing garbage.
done

$ cd usr.bin/rsync/ && make -j3 && doas make install


$ openrsync --rsync-path=/usr/bin/openrsync  -av in localhost:out
Transfer starting: 1 files
sender.c:551: error: in: mmap: Cannot allocate memory
client.c:85: error: rsync_sender
receiver.c:345: error: poll: hangup
server.c:145: error: rsync_receiver

From what I see, the mmap problem is on sender.c

Cheers,



On Sat, Aug 17, 2019 at 4:53 PM Sebastian Benoit <[hidden email]> wrote:

>
> Joe Davis([hidden email]) on 2019.08.16 12:26:36 +0100:
> > By the looks of it, openrsync does attempt to map the entire file, from
> > usr.bin/rsync/uploader.c:
> >
> >     mapsz = st.st_size;
> >     map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);
> >
> > The likely reason for your out of memory error is the default datasize
> > in login.conf. IIRC on some arches it's set to 768MB by default, which
> > would allow your 300MB file to transfer, but would cause mmap to fail
> > upon attempting to map the 1.6GB one.
> >
> > Increasing the default limits in /etc/login.conf should fix the problem.
> >
> > Note that rsync (not openrsync), doesn't use mmap for other reasons,
> > from rsync-3.1.3/fileio.c:
> >
> > /* This provides functionality somewhat similar to mmap() but using read().
> >  * It gives sliding window access to a file.  mmap() is not used because of
> >  * the possibility of another program (such as a mailer) truncating the
> >  * file thus giving us a SIGBUS. */
> >
> > Cheers,
> > Joe
>
> Hi,
>
> this replaces the mmap() with pread(), please try it out.
>
> I dont much like the error handling here, but its a start.
>
> ok?
>
>
> diff --git usr.bin/rsync/uploader.c usr.bin/rsync/uploader.c
> index fd07b22caeb..cce8b47a4c9 100644
> --- usr.bin/rsync/uploader.c
> +++ usr.bin/rsync/uploader.c
> @@ -158,8 +158,8 @@ init_blk(struct blk *p, const struct blkset *set, off_t offs,
>         p->len = idx < set->blksz - 1 ? set->len : set->rem;
>         p->offs = offs;
>
> -       p->chksum_short = hash_fast(map + offs, p->len);
> -       hash_slow(map + offs, p->len, p->chksum_long, sess);
> +       p->chksum_short = hash_fast(map, p->len);
> +       hash_slow(map, p->len, p->chksum_long, sess);
>  }
>
>  /*
> @@ -741,8 +741,9 @@ rsync_uploader(struct upload *u, int *fileinfd,
>  {
>         struct blkset       blk;
>         struct stat         st;
> -       void               *map, *bufp;
> -       size_t              i, mapsz, pos, sz;
> +       void               *mbuf, *bufp;
> +       ssize_t             msz;
> +       size_t              i, pos, sz;
>         off_t               offs;
>         int                 c;
>         const struct flist *f;
> @@ -909,35 +910,46 @@ rsync_uploader(struct upload *u, int *fileinfd,
>         blk.csum = u->csumlen;
>
>         if (*fileinfd != -1 && st.st_size > 0) {
> -               mapsz = st.st_size;
> -               map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);
> -               if (map == MAP_FAILED) {
> -                       ERR("%s: mmap", u->fl[u->idx].path);
> -                       close(*fileinfd);
> -                       *fileinfd = -1;
> -                       return -1;
> -               }
> -
>                 init_blkset(&blk, st.st_size);
>                 assert(blk.blksz);
>
>                 blk.blks = calloc(blk.blksz, sizeof(struct blk));
>                 if (blk.blks == NULL) {
>                         ERR("calloc");
> -                       munmap(map, mapsz);
> +                       close(*fileinfd);
> +                       *fileinfd = -1;
> +                       return -1;
> +               }
> +
> +               if ((mbuf = calloc(1, blk.len)) == NULL) {
> +                       ERR("calloc");
>                         close(*fileinfd);
>                         *fileinfd = -1;
>                         return -1;
>                 }
>
>                 offs = 0;
> -               for (i = 0; i < blk.blksz; i++) {
> -                       init_blk(&blk.blks[i],
> -                               &blk, offs, i, map, sess);
> +               i = 0;
> +               do {
> +                       msz = pread(*fileinfd, mbuf, blk.len, offs);
> +                       if (msz < 0) {
> +                               ERR("pread");
> +                               close(*fileinfd);
> +                               *fileinfd = -1;
> +                               return -1;
> +                       }
> +                       if ((size_t)msz != blk.len && (size_t)msz != blk.rem) {
> +                               /* short read, try again */
> +                               continue;
> +                       }
> +                       init_blk(&blk.blks[i], &blk, offs, i, mbuf, sess);
>                         offs += blk.len;
> -               }
> +                       LOG3(
> +                           "i=%ld, offs=%lld, msz=%ld, blk.len=%lu, blk.rem=%lu",
> +                           i, offs, msz, blk.len, blk.rem);
> +                       i++;
> +               } while (i < blk.blksz);
>
> -               munmap(map, mapsz);
>                 close(*fileinfd);
>                 *fileinfd = -1;
>                 LOG3("%s: mapped %jd B with %zu blocks",
>

Reply | Threaded
Open this post in threaded view
|

Re: openrsync out of memory

Sebastian Benoit
Olivier Antoine([hidden email]) on 2019.08.19 11:34:06 +0200:

> Hi,
> On i386: before patch:
> $ dd if=/dev/urandom  of=in bs=1M count=2k
> $ openrsync --rsync-path=/usr/bin/openrsync  -av in localhost:out
> Transfer starting: 1 files
> sender.c:551: error: in: mmap: Cannot allocate memory
> client.c:85: error: rsync_sender
> receiver.c:345: error: poll: hangup
> server.c:145: error: rsync_receiver
>
> With your patch:
> $ patch -p0 < /tmp/1
> Hmm...  Looks like a unified diff to me...
> The text leading up to this was:
> --------------------------
> |diff --git usr.bin/rsync/uploader.c usr.bin/rsync/uploader.c
> |index fd07b22caeb..cce8b47a4c9 100644
> |--- usr.bin/rsync/uploader.c
> |+++ usr.bin/rsync/uploader.c
> --------------------------
> Patching file usr.bin/rsync/uploader.c using Plan A...
> Hunk #1 succeeded at 158.
> Hunk #2 succeeded at 741.
> Hunk #3 succeeded at 910.
> Hmm...  Ignoring the trailing garbage.
> done
>
> $ cd usr.bin/rsync/ && make -j3 && doas make install
> ???
>
> $ openrsync --rsync-path=/usr/bin/openrsync  -av in localhost:out
> Transfer starting: 1 files
> sender.c:551: error: in: mmap: Cannot allocate memory
> client.c:85: error: rsync_sender
> receiver.c:345: error: poll: hangup
> server.c:145: error: rsync_receiver
>
> From what I see, the mmap problem is on sender.c

Thanks,

i'll work on that one too.

/Benno