Scott, I think you are confused.
When I refer to *_librsync.so* it is really *_librsync.cpython-39-darwin.so <http://librsync.cpython-39-darwin.so>* that I'm referring to. The middle part will change depending on which python and its version it's compiled against. And no, _librsyncmodule.o is not needed at all once the compile and link have finished. The build directory is strictly transitory. *librsync* is a requirement for duplicity. See *README.md*. So yes, testing would fail, but setup would fail before that. *_librsyncmodule.c* is the C interface to librsync.so in the system libraries. It is compiled and linked against the system *librsync.so* producing *duplicity/_librsync.so* as a shim to the library. On most systems, both /usr/lib and /usr/local/lib are in the path for linker. On the Mac and some other systems, only /usr/lib is in the path for the linker, thus we add *LIBRSYNC_DIR* during setup to account for that. This is passed to setup.py to add extra dirs to link against and include in compiles. ...Ken On Mon, Oct 17, 2022 at 9:15 AM Scott Hannahs <s...@p-hall.net> wrote: > Ken, > > Just as a followup, the only reference I can find to _librsync.cpython is > in the build _librsync library that is built and in the build folder. > > It seems that your use of LIBRSYN_DIR is using an installed possibly > previous version of librsync rather than the one built in the distribution > folder? Should the testing have LIBRSYNC_DIR set to some folder in the > build directory structure rather than in the file system in general? > > Would the testing fail if this was an install in a clean system where > duplicity (and librsync) had not already been installed? > > *-Scott* > > > On Oct 10, 2022, at 11:42 AM, Kenneth Loafman <kenn...@loafman.com> wrote: > > Hi Scott, > > I would say that all your errors are from not being able to build > *_librsync.so* properly. There is one thing that may help in lines 65-76 > of *setup.py*. You can supply *LIBRSYNC_DIR* as an env variable, or on > the command line as *--librsync-dir=*. I have > *export LIBRSYNC_DIR=/usr/local* in my .bash_profile. That came from > *homebrew* installing into /usr/local instead of /usr. Seems none of > these installers care much for standards. > > Try exporting *LIBRSYNC_DIR* and let me know how it goes after that. > > ...Ken > > > On Sun, Oct 9, 2022 at 4:46 PM Scott Hannahs <s...@p-hall.net> wrote: > >> So there are two major failures in testing. One is four failures in the >> tahoe backend, (which I don’t use). >> >> testing/unit/test_backend_instance.py ..............................FF.. >> [ 10%] >> FF........................ssssssssss >> [ 18%] >> >> >> This shows a “Key Error” so is something wrong with my GPG version? >> >> self = <duplicity.backends.tahoebackend.TAHOEBackend object at >> 0x10f58deb0> >> commandline = 'tahoe cp >> /var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/local >> var:folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output/file-a' >> >> def subprocess_popen(self, commandline): >> u""" >> Execute the given command line with error check. >> Returns int Exitcode, string StdOut, string StdErr >> >> Raise a BackendException on failure. >> """ >> import shlex >> >> if isinstance(commandline, (list, tuple)): >> logstr = u' '.join(commandline) >> args = commandline >> else: >> logstr = commandline >> args = shlex.split(commandline) >> >> logstr = self.munge_password(logstr) >> log.Info(_(u"Reading results of '%s'") % logstr) >> >> result, stdout, stderr = self.__subprocess_popen(args) >> if result != 0: >> try: >> > ignores = self.popen_breaks[args[0]] >> E KeyError: >> '/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/overrides/bin/tahoe' >> >> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py:522: >> KeyError >> >> During handling of the above exception, another exception occurred: >> >> The second testing error is more general, so many of the tests pass and >> then starting at the “bad upload” test it all goes south. >> >> testing/functional/test_badupload.py F >> [ 70%] >> testing/functional/test_cleanup.py FFF >> [ 71%] >> testing/functional/test_final.py FFFFFFFFFFFFFFFFFFFFFFFFFFF >> [ 77%] >> testing/functional/test_log.py F >> [ 77%] >> testing/functional/test_rdiffdir.py F >> [ 77%] >> testing/functional/test_replicate.py FFF >> [ 78%] >> testing/functional/test_restart.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF >> [ 86%] >> testing/functional/test_selection.py .FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF >> [ 94%] >> FFFFssFFFFFFFssss >> [ 98%] >> >> >> This test has a long back trace, But I think that the basic problem is >> that duplicity cannot import “librsynch”??? The end of the traceback has >> the text "b"ImportError: cannot import name '_librsync' from 'duplicity' >> (/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/__init__.py)" >> ...return_val: 1" >> >> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py:527: >> BackendException >> _______________________ BadUploadTest.test_missing_file >> ________________________ >> >> self = <testing.functional.test_badupload.BadUploadTest >> testMethod=test_missing_file> >> >> @pytest.mark.slow >> def test_missing_file(self): >> u""" >> Test basic lost file >> """ >> try: >> > self.backup(u"full", >> u"{0}/testfiles/dir1".format(_runtest_dir), options=[u"--skip-volume=1"]) >> >> >> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/test_badupload.py:44: >> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> _ _ _ >> >> self = <testing.functional.test_badupload.BadUploadTest >> testMethod=test_missing_file> >> type = 'full' >> input_dir = >> '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1' >> options = ['full', >> '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1', ' >> file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output', >> '--volsize', '1', '--skip-volume=1'] >> kwargs = {}, before_files = set(), now = 1664937609.016365 >> >> def backup(self, type, input_dir, options=[], **kwargs): # pylint: >> disable=redefined-builtin >> u"""Run duplicity backup to default directory""" >> options = [type, input_dir, self.backend_url, u"--volsize", u"1"] >> + options >> before_files = self.get_backend_files() >> >> # If a chain ends with time X and the next full chain begins at >> time X, >> # we may trigger an assert in dup_collections.py. If needed, >> sleep to >> # avoid such problems >> now = time.time() >> if self.last_backup == int(now): >> time.sleep(1) >> >> > self.run_duplicity(options=options, **kwargs) >> >> >> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/__init__.py:189: >> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> _ _ _ >> >> self = <testing.functional.test_badupload.BadUploadTest >> testMethod=test_missing_file> >> options = ['full', >> '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1', ' >> file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output', >> '--volsize', '1', '--skip-volume=1'] >> current_time = None, fail = None, passphrase_input = [] >> >> def run_duplicity(self, options=[], current_time=None, fail=None, >> passphrase_input=[]): >> u""" >> Run duplicity binary with given arguments and options >> """ >> # We run under setsid and take input from /dev/null (below) >> because >> # this way we force a failure if duplicity tries to read from the >> # console unexpectedly (like for gpg password or such). >> >> # Check all string inputs are unicode -- we will convert to >> system encoding before running the command >> for item in options: >> if sys.version_info.major == 2: >> assert not isinstance(item, str), u"item " + >> unicode(item) + u" in options is not unicode" >> >> for item in passphrase_input: >> assert isinstance(item, u"".__class__), u"item " + >> unicode(item) + u" in passphrase_input is not unicode" >> >> if platform.platform().startswith(u'Linux'): >> cmd_list = [u'setsid'] >> if self._setsid_w: >> cmd_list.extend([u"-w"]) >> else: >> cmd_list = [] >> basepython = os.environ.get(u'TOXPYTHON', None) >> if basepython is not None: >> cmd_list.extend([basepython]) >> run_coverage = os.environ.get(u'RUN_COVERAGE', None) >> if run_coverage is not None: >> cmd_list.extend([u"-m", u"coverage", u"run", >> u"--source=duplicity", u"-p"]) >> cmd_list.extend([u"{0}/bin/duplicity".format(_top_dir)]) >> cmd_list.extend(options) >> cmd_list.extend([u"-v0"]) >> cmd_list.extend([u"--no-print-statistics"]) >> cmd_list.extend([u"--allow-source-mismatch"]) >> >> cmd_list.extend([u"--archive-dir={0}/testfiles/cache".format(_runtest_dir)]) >> if current_time: >> cmd_list.extend([u"--current-time", current_time]) >> cmd_list.extend(self.class_args) >> if fail: >> cmd_list.extend([u"--fail", u"".__class__(fail)]) >> cmdline = u" ".join([u'"%s"' % x for x in cmd_list]) >> >> cmdline = u" ".join([u'"%s"' % x for x in cmd_list]) >> >> if not passphrase_input: >> cmdline += u" < /dev/null" >> >> # The immediately following block is the nicer way to execute >> pexpect with >> # unicode strings, but we need to have the pre-4.0 version for >> some time yet, >> # so for now this is commented out so tests execute the same way >> on all systems. >> >> # if parse_version(pexpect.__version__) >= parse_version("4.0"): >> # # pexpect.spawn only supports unicode from version 4.0 >> # # there was a separate pexpect.spawnu in 3.x, but it has an >> error on readline >> # child = pexpect.spawn(u'/bin/sh', [u'-c', cmdline], >> timeout=None, encoding=sys.getfilesystemencoding()) >> # >> # for passphrase in passphrase_input: >> # child.expect(u'passphrase.*:') >> # child.sendline(passphrase) >> # else: >> >> # Manually encode to filesystem encoding and send to spawn as >> bytes >> # ToDo: Remove this once we no longer have to support systems >> with pexpect < 4.0 >> if sys.version_info.major > 2: >> child = pexpect.spawn(u'/bin/sh', [u'-c', cmdline], >> timeout=None) >> else: >> child = pexpect.spawn(b'/bin/sh', [b'-c', >> cmdline.encode(sys.getfilesystemencoding(), >> >> u'replace')], timeout=None) >> >> for passphrase in passphrase_input: >> child.expect(b'passphrase.*:') >> child.sendline(passphrase) >> >> # if the command fails, we need to clear its output >> # so it will terminate cleanly. >> child.expect_exact(pexpect.EOF) >> lines = child.before.splitlines() >> child.wait() >> child.ptyproc.delayafterclose = 0.0 >> return_val = child.exitstatus >> >> if fail: >> self.assertEqual(30, return_val) >> elif return_val: >> print(u"\n...command:", cmdline, file=sys.stderr) >> print(u"...cwd:", os.getcwd(), file=sys.stderr) >> print(u"...output:", file=sys.stderr) >> for line in lines: >> line = line.rstrip() >> if line: >> print(line, file=sys.stderr) >> print(u"...return_val:", return_val, file=sys.stderr) >> >> > raise CmdError(return_val) >> E testing.functional.CmdError: 1 >> >> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/__init__.py:175: >> CmdError >> >> During handling of the above exception, another exception occurred: >> >> self = <testing.functional.test_badupload.BadUploadTest >> testMethod=test_missing_file> >> >> @pytest.mark.slow >> def test_missing_file(self): >> u""" >> Test basic lost file >> """ >> try: >> self.backup(u"full", >> u"{0}/testfiles/dir1".format(_runtest_dir), options=[u"--skip-volume=1"]) >> self.fail() >> except CmdError as e: >> > self.assertEqual(e.exit_status, 44, str(e)) >> E AssertionError: 1 != 44 : 1 >> >> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/test_badupload.py:47: >> AssertionError >> ----------------------------- Captured stderr call >> ----------------------------- >> >> ...command: >> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/bin/duplicity" >> "full" "/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1" " >> file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output" >> "--volsize" "1" "--skip-volume=1" "-v0" "--no-print-statistics" >> "--allow-source-mismatch" >> "--archive-dir=/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/cache" >> < /dev/null >> ...cwd: /private/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T >> ...output: >> b'Traceback (most recent call last):' >> b' File >> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/bin/duplicity", >> line 37, in <module>' >> b' from duplicity.dup_main import main' >> b' File >> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/dup_main.py", >> line 50, in <module>' >> b' from duplicity import commandline' >> b' File >> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/commandline.py", >> line 43, in <module>' >> b' from duplicity import backend' >> b' File >> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py", >> line 44, in <module>' >> b' from duplicity import dup_temp' >> b' File >> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/dup_temp.py", >> line 34, in <module>' >> b' from duplicity import path' >> b' File >> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/path.py", >> line 49, in <module>' >> b' from duplicity import librsync' >> b' File >> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/librsync.py", >> line 36, in <module>' >> b' from . import _librsync' >> b"ImportError: cannot import name '_librsync' from 'duplicity' >> (/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/__init__.py)" >> ...return_val: 1 >> >> >> On Oct 9, 2022, at 11:26 AM, Kenneth Loafman <kenn...@loafman.com> wrote: >> >> Ok, sounds good. Any progress? >> >> ...Ken >> >> >> On Thu, Oct 6, 2022 at 3:15 PM Scott Hannahs <s...@p-hall.net> wrote: >> >>> I will try to reinstall Xcode and the command line tools. I had been >>> using py38 which is installed in both the standard and an updated version >>> in the non-standard location. >>> >>> On Oct 6, 2022, at 14:57, Kenneth Loafman <kenn...@loafman.com> wrote: >>> >>> OK, >>> >>> I could not remember what version of Python 3 you were running, so py3x >>> meant for you to fill it in. I'm guessing either py38 or py39 for Python >>> 3.8 and Python 3.9 respectively. Others follow the same model, >>> py27,py35,...py310. >>> >>> I am not sure why it's trying to build for anything but x86_64 if you >>> have Intel silicone and not Apple silicone. >>> >>> Here's what mine looks like for py38: >>> >>> ken@dione:~/workspace/duplicity-origin$ python3.8 ./setup.py build_ext >>> Unable to get SCM version: defaulting to 1.0.1 >>> running build_ext >>> Building extension for librsync... >>> building 'duplicity._librsync' extension >>> creating build/temp.macosx-12-x86_64-cpython-38 >>> creating build/temp.macosx-12-x86_64-cpython-38/duplicity >>> clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common >>> -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot >>> /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/usr/local/include >>> -I/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/include/python3.8 >>> -c duplicity/_librsyncmodule.c -o >>> build/temp.macosx-12-x86_64-cpython-38/duplicity/_librsyncmodule.o >>> creating build/lib.macosx-12-x86_64-cpython-38 >>> creating build/lib.macosx-12-x86_64-cpython-38/duplicity >>> clang -bundle -undefined dynamic_lookup -isysroot >>> /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk >>> build/temp.macosx-12-x86_64-cpython-38/duplicity/_librsyncmodule.o >>> -L/usr/local/lib -lrsync -o >>> build/lib.macosx-12-x86_64-cpython-38/duplicity/_ >>> librsync.cpython-38-darwin.so >>> ld: warning: -undefined dynamic_lookup may not work with chained fixups >>> copying build/lib.macosx-12-x86_64-cpython-38/duplicity/_ >>> librsync.cpython-38-darwin.so -> duplicity >>> >>> >>> It looks like your xcode libs are out of date, and it also looks like >>> you're not using the standard locations for software. That's gonna >>> complicate matters. >>> >>> I'm on Monterey 12.6 with the current Xcode from Apple, all in standard >>> places. >>> >>> ...Ken >>> >>> >>> >>> >>> On Thu, Oct 6, 2022 at 1:13 PM Scott Hannahs <s...@p-hall.net> wrote: >>> >>>> Ken, >>>> >>>> I will leave fink out, but it should be doing exactly the same thing! >>>> :-) >>>> >>>> the following doesn’t even build duplicity correctly, so I don’t get to >>>> the tox command. >>>> >>>> % tar xf duplicity-1.0.1.tar >>>> % cd duplicity-1.0.1 >>>> % % C_INCLUDE_PATH=/opt/sw/include LIBRARY_PATH=/opt/sw/lib python3 >>>> ./setup.py build_ext > build.out.txt 2>&1 >>>> >>>> >>>> This gives the following output. >>>> >>>> Unable to get SCM version: defaulting to 1.0.1 >>>> running build_ext >>>> Building extension for librsync... >>>> building 'duplicity._librsync' extension >>>> clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common >>>> -dynamic -DNDEBUG -g -fwrapv -O3 -Wall >>>> -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders >>>> -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers >>>> -arch arm64 -arch x86_64 -Werror=implicit-function-declaration >>>> -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/include/python3.9 >>>> -c duplicity/_librsyncmodule.c -o >>>> build/temp.macosx-10.9-universal2-3.9/duplicity/_librsyncmodule.o >>>> clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 >>>> -Wl,-headerpad,0x1000 >>>> build/temp.macosx-10.9-universal2-3.9/duplicity/_librsyncmodule.o -lrsync >>>> -o build/lib.macosx-10.9-universal2-3.9/duplicity/_ >>>> librsync.cpython-39-darwin.so >>>> ld: warning: ignoring file /opt/sw/lib/librsync.dylib, building for >>>> macOS-arm64 but attempting to link with file built for macOS-x86_64 >>>> ld: warning: dylib (/opt/sw/lib/librsync.dylib) was built for newer >>>> macOS version (12.4) than being linked (10.9) >>>> copying build/lib.macosx-10.9-universal2-3.9/duplicity/_ >>>> librsync.cpython-39-darwin.so -> duplicity >>>> >>>> >>>> I am building on an X86 machine and my libraries are x86 and I am not >>>> trying to build a universal arm 64 system. But with warnings it tries to >>>> build. >>>> >>>> tox does not like the py3x value >>>> >>>> % tox -epy3 >>>> py3 create: /Users/sth/Downloads/duplicity-1.0.1/.tox/py3 >>>> py3 installdeps: -rrequirements.txt >>>> >>>> and I ran out of time this AM waiting for more output. >>>> >>>> -Scott >>>> >>>> >>>> On Oct 5, 2022, at 11:54 AM, Kenneth Loafman <kenn...@loafman.com> >>>> wrote: >>>> >>>> Could you try just building and running the tests in a 'normal' user >>>> environment? Leave Fink out of the picture. >>>> >>>> do this: >>>> >>>> 1. $ tar xf duplicity-1.01.tar.gz >>>> 2. $ ./setup.py build_ext >>>> 3. $ tox -e py3x >>>> >>>> ...Thanks, >>>> ...Ken >>>> >>>> >>>> >>> >> >
_______________________________________________ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
