For the sake of pointing out the AWS creds which I think is what you are
hinting at with "check permission keys".
As I was in my original bash script, and for testing I am exporting in my shell:
AWS_ACCESS_KEY_ID=$akey
AWS_SECRET_ACCESS_KEY=$sakey
PASSPHRASE=$passphrase
s3cmd seemed to like this without intervention, but I manually made
~/.s3cfg and verified that working as well.
I also created an .aws/credentials file with
[default]
aws_access_key_id = $akey
aws_secret_access_key = $sakey
I then verified that boto3 was seeing this with
ubuntu@duplicity-f:~$ python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> client = boto3.client('s3')
>>> client.list_buckets()
{'ResponseMetadata': {'RequestId': '$responseid', 'HostId': '$hostid',
'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': '$amz-id-2',
'x-amz-request-id': 'amz-request-id',[snip],}}
And this properly returns the buckets in this IAM account, so I assume boto3 is
properly able to hit the s3 credentials file. I sanitized some data, hopefully
which doesn't nullify the utility.
Attaching the full sanitized response as boto3_demo_working.log
My python is very rusty, I'm more or less bashing my way through
verifying all of this so if something would be more effective please let
me know.
That said, I am still unable to get past the 403 in the duplicity run.
Attaching full 2>&1 output as duplicity_demo_nonworking.log.
Again, sanitized as much as possible, while still not obfuscating the important
parts (hopefully).
And just to provide some clean and clear repro steps:
I am using a fresh focal image in multipass for consistent clean testing.
# apt update; apt dist-upgrade; apt install duplicity python3-boto3
ubuntu@duplicity-f:~$ lsb_release -a ; uname -a ; dpkg -l | grep duplicity ;
dpkg -l | grep -i boto
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
Linux duplicity-f 5.4.0-54-generic #60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
ii duplicity 0.8.11.1612-1 amd64
encrypted bandwidth-efficient backup
ii python3-boto3 1.9.253-1 all
Python interface to Amazon's Web Services - Python 3.x
ii python3-botocore 1.16.19+repack-1ubuntu0.20.04.1 all
Low-level, data-driven core of boto 3 (Python 3)
Please let me know what else I may be able to provide.
** Attachment added: "duplicity -v9 output with command"
https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1904622/+attachment/5436047/+files/duplicity_demo_nonworking.log
--
You received this bug notification because you are a member of Desktop
Packages, which is subscribed to duplicity in Ubuntu.
https://bugs.launchpad.net/bugs/1904622
Title:
boto3+s3 bucket fails
Status in Duplicity:
Incomplete
Status in duplicity package in Ubuntu:
New
Bug description:
I have a duplicity job that runs on multiple hosts, and am now having
failures on my 20.04 hosts.
The syntax for the job (as I am currently testing to debug) is:
duplicity --dry-run list-current-files --name $name --s3-use-ia
--archive-dir $archiveDir --tempdir $tmpDir --s3-multipart-chunk-size 256
--s3-use-multiprocessing --volsize 1024 --verbosity 8
boto3+s3://s3.amazonaws.com/bucket.name
Using the duplicity packages from focal/main
duplicity=0.8.11.1612-1
And I installed the boto packages from focal/universe
python3-boto3=1.9.253-1
python3-botocore=1.14.14+repack-1
This runs into the error:
Using temporary directory $tmpDir/duplicity-iyhq3se4-tempdir
Traceback (innermost last):
File "/usr/bin/duplicity", line 106, in <module>
with_tempdir(main)
File "/usr/bin/duplicity", line 92, in with_tempdir
fn()
File "/usr/lib/python3/dist-packages/duplicity/dup_main.py", line 1525, in
main
action = commandline.ProcessCommandLine(sys.argv[1:])
File "/usr/lib/python3/dist-packages/duplicity/commandline.py", line 1175,
in ProcessCommandLine
globals.backend = backend.get_backend(args[0])
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 225, in
get_backend
obj = get_backend_object(url_string)
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 211, in
get_backend_object
return factory(pu)
File
"/usr/lib/python3/dist-packages/duplicity/backends/s3_boto3_backend.py", line
85, in __init__
self.reset_connection()
File
"/usr/lib/python3/dist-packages/duplicity/backends/s3_boto3_backend.py", line
96, in reset_connection
self.s3.meta.client.head_bucket(Bucket=self.bucket_name)
File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py",
line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py",
line 676, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the
HeadBucket operation: Forbidden
I then tried the duplicity snap from the snap store, both the stable and edge
builds, with the same issue:
Main action: list-current
================================================================================
duplicity 0.8.17
Args: /snap/duplicity/138/bin/duplicity --dry-run --ignore-errors
list-current-files --name $name --s3-use-ia --archive-dir $archiveDir --tempdir
$tmpDir --s3-multipart-chunk-size 256 --s3-use-multiprocessing --volsize 1024
--verbosity 8 boto3+s3://s3.amazonaws.com/bucket.name
Linux duplicity 5.4.0-53-generic #59-Ubuntu SMP Wed Oct 21 09:38:44 UTC 2020
x86_64 x86_64
/snap/duplicity/141/usr/bin/python3 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0]
================================================================================
Attempt 1 failed. ClientError: An error occurred (403) when calling the
HeadBucket operation: Forbidden
Main action: list-current
================================================================================
duplicity 0.8.18.dev11
Args: /snap/duplicity/141/bin/duplicity --dry-run --ignore-errors
list-current-files --name $name --s3-use-ia --archive-dir $archiveDir --tempdir
$tmpDir --s3-multipart-chunk-size 256 --s3-use-multiprocessing --volsize 1024
--verbosity 8 boto3+s3://s3.amazonaws.com/bucket.name
Linux duplicity 5.4.0-53-generic #59-Ubuntu SMP Wed Oct 21 09:38:44 UTC 2020
x86_64 x86_64
/snap/duplicity/141/usr/bin/python3 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0]
================================================================================
Attempt 1 failed. ClientError: An error occurred (403) when calling the
HeadBucket operation: Forbidden
I also tried to update the boto3/botocore packages by way of pip3:
pip3 freeze | grep boto
boto3==1.16.19
botocore==1.19.19
And lastly as a last ditch, I tried to pull the packages from the
groovy/universe repo, bringing them to:
python3-boto3 1.13.14-1
python3-botocore 1.17.22+repack-1_all
I have also tried in a 20.10 system as well to see if any changes
there, but same results.
Hoping for something I can look at to get this back working as it was
in 18.04 since 14.04.
For good measure, I verified in a bionic build that using the same arguments
as above with duplicity=0.7.17
python-boto=2.44.0-1ubuntu2.18.04.1
I then verified by uninstalling the deb and installing the snap
(0.8.17) on bionic that the behavior stays the same.
And just to satisfy the 1-4 guidelines:
1. The main system I am having issues with:
$ lsb_release -rd
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Also tested against
$ lsb_release -rd
Description: Ubuntu 18.04.5 LTS
Release: 18.04
$ lsb_release -rd
Description: Ubuntu 20.10
Release: 20.10
2. $ apt-cache policy duplicity
duplicity:
Installed: 0.8.11.1612-1
Candidate: 0.8.11.1612-1
Version table:
*** 0.8.11.1612-1 500
500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages
100 /var/lib/dpkg/status
3. Expect duplicity to function against my s3 bucket.
4. botocore.exceptions.ClientError: An error occurred (403) when calling the
HeadBucket operation: Forbidden
To manage notifications about this bug go to:
https://bugs.launchpad.net/duplicity/+bug/1904622/+subscriptions
--
Mailing list: https://launchpad.net/~desktop-packages
Post to : desktop-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~desktop-packages
More help : https://help.launchpad.net/ListHelp
