Python 3.8 os.listdir EINVAL on large directories

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Python 3.8 os.listdir EINVAL on large directories

Aaron Miller
Hi all,

I am getting a stacktrace from the borg command in the borgbackup
package while checking a backup (see bottom of email for full
output, since it's verbose). The relevant part is this:

    filenames = os.listdir(os.path.join(data_path, dir))
  OSError: [Errno 22] Invalid argument:
'/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'

This is same error is reproducible with a test Python 3.8 program:

 #!/usr/bin/env python

 import os
 os.listdir('/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/')

Running ktrace & kdump reveals the error is from calling
getdents(2):

 76903 python3.8
CALL  open(0x1ec7f06de3b0,0x30000<O_RDONLY|O_CLOEXEC|O_DIRECTORY>)
 76903 python3.8
NAMI  "/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/"
 76903 python3.8 RET   open 3
 [...]
 76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
 76903 python3.8 RET   getdents 16384/0x4000
 [...]
 76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
 76903 python3.8 RET   getdents 16384/0x4000
 [...]
 76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
 76903 python3.8 RET   getdents 16384/0x4000
 [...]
 76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
 76903 python3.8 RET   getdents -1 errno 22 Invalid argument

Looking at the man page for getdents(2), I found it interesting
that it says this call "is not a portable interface and should not
be used directly by applications" and it recommends using
readdir(3) instead.

To give you a rough idea of the number of files and filename sizes
in this directory:

  $ ls /mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/ | wc
      1534    1534   10738

Where does the problem lie -- the upstream Python code, the
OpenBSD-specific patches in its port definition, or somewhere
else? And in case it matters, this is a -current amd64 system,
with "sysupgrade -s" executed on 7/15.

Thank you,
Aaron Miller

--
Exception ignored in: <function Repository.__del__ at
0x1e17e13fd310>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-
packages/borg/repository.py", line 180, in __del__
    assert False, "cleanup happened in Repository.__del__"
AssertionError: cleanup happened in Repository.__del__
Local Exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
line 4565, in main
    exit_code = archiver.run(args)
  File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
line 4497, in run
    return set_ec(func(args))
  File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
line 161, in wrapper
    with repository:
  File "/usr/local/lib/python3.8/site-
packages/borg/repository.py", line 190, in __enter__
    self.open(self.path, bool(self.exclusive),
lock_wait=self.lock_wait, lock=self.do_lock)
  File "/usr/local/lib/python3.8/site-
packages/borg/repository.py", line 450, in open
    segment = self.io.get_latest_segment()
  File "/usr/local/lib/python3.8/site-
packages/borg/repository.py", line 1253, in get_latest_segment
    for segment, filename in self.segment_iterator(reverse=True):
  File "/usr/local/lib/python3.8/site-
packages/borg/repository.py", line 1241, in segment_iterator
    filenames = os.listdir(os.path.join(data_path, dir))
OSError: [Errno 22] Invalid argument:
'/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'

Platform: OpenBSD millipede.iforgotmy.name 6.7 GENERIC.MP#348
amd64
Borg: 1.1.13  Python: CPython 3.8.3 msgpack: 0.5.6
PID: 31745  CWD: /mnt/thinkpad_void_obsd_borg
sys.argv: ['/usr/local/bin/borg', 'check', 'thinkpad.borg']
SSH_ORIGINAL_COMMAND: None

Reply | Threaded
Open this post in threaded view
|

Re: Python 3.8 os.listdir EINVAL on large directories

Alceu Rodrigues de Freitas Junior
I'm guessing that the usage of getdents was intentional: I've used this
system call in the same situation (thousands of entries in a directory,
inside a NFS exported directory) to avoid calling stat() in each entry
returned by getdents.

It is indead a "low level" system call, and usually readdir is the
preferred method, but might be too slow for your usage case (in my case
it was, it took hours to read the directory while with getdents took
minutes).

It doesn't seems a problem with OpenBSD, but with Python code. Looks
like some additional logic will be required to address the differences
betweens the UNIX-like OSs and their implementation of getdents.

On the other hand, by the error message the Python code is passing a
wrong argument, maybe it is a bug instead of a portability issue?

Em 25/07/2020 20:50, Aaron Miller escreveu:

> Hi all,
>
> I am getting a stacktrace from the borg command in the borgbackup
> package while checking a backup (see bottom of email for full
> output, since it's verbose). The relevant part is this:
>
>     filenames = os.listdir(os.path.join(data_path, dir))
>   OSError: [Errno 22] Invalid argument:
> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>
> This is same error is reproducible with a test Python 3.8 program:
>
>  #!/usr/bin/env python
>
>  import os
>  os.listdir('/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/')
>
> Running ktrace & kdump reveals the error is from calling
> getdents(2):
>
>  76903 python3.8
> CALL  open(0x1ec7f06de3b0,0x30000<O_RDONLY|O_CLOEXEC|O_DIRECTORY>)
>  76903 python3.8
> NAMI  "/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/"
>  76903 python3.8 RET   open 3
>  [...]
>  76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
>  76903 python3.8 RET   getdents 16384/0x4000
>  [...]
>  76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
>  76903 python3.8 RET   getdents 16384/0x4000
>  [...]
>  76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
>  76903 python3.8 RET   getdents 16384/0x4000
>  [...]
>  76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
>  76903 python3.8 RET   getdents -1 errno 22 Invalid argument
>
> Looking at the man page for getdents(2), I found it interesting
> that it says this call "is not a portable interface and should not
> be used directly by applications" and it recommends using
> readdir(3) instead.
>
> To give you a rough idea of the number of files and filename sizes
> in this directory:
>
>   $ ls /mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/ | wc
>       1534    1534   10738
>
> Where does the problem lie -- the upstream Python code, the
> OpenBSD-specific patches in its port definition, or somewhere
> else? And in case it matters, this is a -current amd64 system,
> with "sysupgrade -s" executed on 7/15.
>
> Thank you,
> Aaron Miller
>
> --
> Exception ignored in: <function Repository.__del__ at
> 0x1e17e13fd310>
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 180, in __del__
>     assert False, "cleanup happened in Repository.__del__"
> AssertionError: cleanup happened in Repository.__del__
> Local Exception
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
> line 4565, in main
>     exit_code = archiver.run(args)
>   File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
> line 4497, in run
>     return set_ec(func(args))
>   File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
> line 161, in wrapper
>     with repository:
>   File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 190, in __enter__
>     self.open(self.path, bool(self.exclusive),
> lock_wait=self.lock_wait, lock=self.do_lock)
>   File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 450, in open
>     segment = self.io.get_latest_segment()
>   File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 1253, in get_latest_segment
>     for segment, filename in self.segment_iterator(reverse=True):
>   File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 1241, in segment_iterator
>     filenames = os.listdir(os.path.join(data_path, dir))
> OSError: [Errno 22] Invalid argument:
> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>
> Platform: OpenBSD millipede.iforgotmy.name 6.7 GENERIC.MP#348
> amd64
> Borg: 1.1.13  Python: CPython 3.8.3 msgpack: 0.5.6
> PID: 31745  CWD: /mnt/thinkpad_void_obsd_borg
> sys.argv: ['/usr/local/bin/borg', 'check', 'thinkpad.borg']
> SSH_ORIGINAL_COMMAND: None
>

Reply | Threaded
Open this post in threaded view
|

Re: Python 3.8 os.listdir EINVAL on large directories

Aaron Miller
Hi Alceu,

Note that there are additional messages about this in tech@.

--Aaron


On July 26, 2020 8:53:00 AM PDT, Alceu Rodrigues de Freitas Junior <[hidden email]> wrote:

>I'm guessing that the usage of getdents was intentional: I've used this
>system call in the same situation (thousands of entries in a directory,
>inside a NFS exported directory) to avoid calling stat() in each entry
>returned by getdents.
>
>It is indead a "low level" system call, and usually readdir is the
>preferred method, but might be too slow for your usage case (in my case
>it was, it took hours to read the directory while with getdents took
>minutes).
>
>It doesn't seems a problem with OpenBSD, but with Python code. Looks
>like some additional logic will be required to address the differences
>betweens the UNIX-like OSs and their implementation of getdents.
>
>On the other hand, by the error message the Python code is passing a
>wrong argument, maybe it is a bug instead of a portability issue?
>
>Em 25/07/2020 20:50, Aaron Miller escreveu:
>> Hi all,
>>
>> I am getting a stacktrace from the borg command in the borgbackup
>> package while checking a backup (see bottom of email for full
>> output, since it's verbose). The relevant part is this:
>>
>>     filenames = os.listdir(os.path.join(data_path, dir))
>>   OSError: [Errno 22] Invalid argument:
>> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>>
>> This is same error is reproducible with a test Python 3.8 program:
>>
>>  #!/usr/bin/env python
>>
>>  import os
>>  os.listdir('/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/')
>>
>> Running ktrace & kdump reveals the error is from calling
>> getdents(2):
>>
>>  76903 python3.8
>> CALL  open(0x1ec7f06de3b0,0x30000<O_RDONLY|O_CLOEXEC|O_DIRECTORY>)
>>  76903 python3.8
>> NAMI  "/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/"
>>  76903 python3.8 RET   open 3
>>  [...]
>>  76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
>>  76903 python3.8 RET   getdents 16384/0x4000
>>  [...]
>>  76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
>>  76903 python3.8 RET   getdents 16384/0x4000
>>  [...]
>>  76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
>>  76903 python3.8 RET   getdents 16384/0x4000
>>  [...]
>>  76903 python3.8 CALL  getdents(3,0x1ec7c9257000,0x4000)
>>  76903 python3.8 RET   getdents -1 errno 22 Invalid argument
>>
>> Looking at the man page for getdents(2), I found it interesting
>> that it says this call "is not a portable interface and should not
>> be used directly by applications" and it recommends using
>> readdir(3) instead.
>>
>> To give you a rough idea of the number of files and filename sizes
>> in this directory:
>>
>>   $ ls /mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/ | wc
>>       1534    1534   10738
>>
>> Where does the problem lie -- the upstream Python code, the
>> OpenBSD-specific patches in its port definition, or somewhere
>> else? And in case it matters, this is a -current amd64 system,
>> with "sysupgrade -s" executed on 7/15.
>>
>> Thank you,
>> Aaron Miller
>>
>> --
>> Exception ignored in: <function Repository.__del__ at
>> 0x1e17e13fd310>
>> Traceback (most recent call last):
>>   File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 180, in __del__
>>     assert False, "cleanup happened in Repository.__del__"
>> AssertionError: cleanup happened in Repository.__del__
>> Local Exception
>> Traceback (most recent call last):
>>   File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
>> line 4565, in main
>>     exit_code = archiver.run(args)
>>   File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
>> line 4497, in run
>>     return set_ec(func(args))
>>   File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
>> line 161, in wrapper
>>     with repository:
>>   File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 190, in __enter__
>>     self.open(self.path, bool(self.exclusive),
>> lock_wait=self.lock_wait, lock=self.do_lock)
>>   File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 450, in open
>>     segment = self.io.get_latest_segment()
>>   File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 1253, in get_latest_segment
>>     for segment, filename in self.segment_iterator(reverse=True):
>>   File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 1241, in segment_iterator
>>     filenames = os.listdir(os.path.join(data_path, dir))
>> OSError: [Errno 22] Invalid argument:
>> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>>
>> Platform: OpenBSD millipede.iforgotmy.name 6.7 GENERIC.MP#348
>> amd64
>> Borg: 1.1.13  Python: CPython 3.8.3 msgpack: 0.5.6
>> PID: 31745  CWD: /mnt/thinkpad_void_obsd_borg
>> sys.argv: ['/usr/local/bin/borg', 'check', 'thinkpad.borg']
>> SSH_ORIGINAL_COMMAND: None
>>

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.