Hi Alceu,
Note that there are additional messages about this in tech@.
--Aaron
On July 26, 2020 8:53:00 AM PDT, Alceu Rodrigues de Freitas Junior
<glasswal...@yahoo.com.br> wrote:
>I'm guessing that the usage of getdents was intentional: I've used this
>system call in the same situation (thousands of entries in a directory,
>inside a NFS exported directory) to avoid calling stat() in each entry
>returned by getdents.
>
>It is indead a "low level" system call, and usually readdir is the
>preferred method, but might be too slow for your usage case (in my case
>it was, it took hours to read the directory while with getdents took
>minutes).
>
>It doesn't seems a problem with OpenBSD, but with Python code. Looks
>like some additional logic will be required to address the differences
>betweens the UNIX-like OSs and their implementation of getdents.
>
>On the other hand, by the error message the Python code is passing a
>wrong argument, maybe it is a bug instead of a portability issue?
>
>Em 25/07/2020 20:50, Aaron Miller escreveu:
>> Hi all,
>>
>> I am getting a stacktrace from the borg command in the borgbackup
>> package while checking a backup (see bottom of email for full
>> output, since it's verbose). The relevant part is this:
>>
>> filenames = os.listdir(os.path.join(data_path, dir))
>> OSError: [Errno 22] Invalid argument:
>> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>>
>> This is same error is reproducible with a test Python 3.8 program:
>>
>> #!/usr/bin/env python
>>
>> import os
>> os.listdir('/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/')
>>
>> Running ktrace & kdump reveals the error is from calling
>> getdents(2):
>>
>> 76903 python3.8
>> CALL open(0x1ec7f06de3b0,0x30000<O_RDONLY|O_CLOEXEC|O_DIRECTORY>)
>> 76903 python3.8
>> NAMI "/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/"
>> 76903 python3.8 RET open 3
>> [...]
>> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
>> 76903 python3.8 RET getdents 16384/0x4000
>> [...]
>> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
>> 76903 python3.8 RET getdents 16384/0x4000
>> [...]
>> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
>> 76903 python3.8 RET getdents 16384/0x4000
>> [...]
>> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
>> 76903 python3.8 RET getdents -1 errno 22 Invalid argument
>>
>> Looking at the man page for getdents(2), I found it interesting
>> that it says this call "is not a portable interface and should not
>> be used directly by applications" and it recommends using
>> readdir(3) instead.
>>
>> To give you a rough idea of the number of files and filename sizes
>> in this directory:
>>
>> $ ls /mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/ | wc
>> 1534 1534 10738
>>
>> Where does the problem lie -- the upstream Python code, the
>> OpenBSD-specific patches in its port definition, or somewhere
>> else? And in case it matters, this is a -current amd64 system,
>> with "sysupgrade -s" executed on 7/15.
>>
>> Thank you,
>> Aaron Miller
>>
>> --
>> Exception ignored in: <function Repository.__del__ at
>> 0x1e17e13fd310>
>> Traceback (most recent call last):
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 180, in __del__
>> assert False, "cleanup happened in Repository.__del__"
>> AssertionError: cleanup happened in Repository.__del__
>> Local Exception
>> Traceback (most recent call last):
>> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
>> line 4565, in main
>> exit_code = archiver.run(args)
>> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
>> line 4497, in run
>> return set_ec(func(args))
>> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
>> line 161, in wrapper
>> with repository:
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 190, in __enter__
>> self.open(self.path, bool(self.exclusive),
>> lock_wait=self.lock_wait, lock=self.do_lock)
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 450, in open
>> segment = self.io.get_latest_segment()
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 1253, in get_latest_segment
>> for segment, filename in self.segment_iterator(reverse=True):
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 1241, in segment_iterator
>> filenames = os.listdir(os.path.join(data_path, dir))
>> OSError: [Errno 22] Invalid argument:
>> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>>
>> Platform: OpenBSD millipede.iforgotmy.name 6.7 GENERIC.MP#348
>> amd64
>> Borg: 1.1.13 Python: CPython 3.8.3 msgpack: 0.5.6
>> PID: 31745 CWD: /mnt/thinkpad_void_obsd_borg
>> sys.argv: ['/usr/local/bin/borg', 'check', 'thinkpad.borg']
>> SSH_ORIGINAL_COMMAND: None
>>
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
