Hello All,
Kern report is accurate.
Oracle, Google Storage, Azure and Swift based have their own storage standards. The Enterprise Bacula version has specific plugins for each one of these.
S3 is compatible with AWS, Huaweii Cloud, CEPH based (using S3 API) and Backblaze. The last one, using an adapter - mini.io.
Regards,
Sent from TypeApp
On May 26, 2019, at 7:17 AM, Kern Sibbald <k...@sibbald.com> wrote:
Hello Mike,
Sorry, but I am 100% sure that Oracle cloud is not compatible with Amazon S3. Bacula was explicitly tested on Oracle (not by me) and it doesn't work. It requires a special Oracle driver that the community does not have at this time.
Best regards,
Kern
On 5/26/19 12:08 PM, Mike wrote:
Morning Kern,
Thanks for the quick response.
1) Oracle Cloud
2) I have not tried a restore but I repeatedly see this so have doubt in the consistency of the backups. I have verified the sd service is not restarting during the upload. I only copy backups from local stores to remote stores once a week but see it every time. This has happened repeatedly over the past three weeks.
It feels like a timing issue.
I'm happy to enable any debugging you suggest, as you imply the files are being uploaded but the driver does not appear to recognize all uploads (hence the low VolParts).
Appreciate the help,
Mike
On Sun, May 26, 2019, at 19:37, Kern Sibbald wrote:
Hello,
I have two questions:
1. What cloud server are you using. We support Amazon or 100% Amazon compatible. All S3 implementations are not the same or compatible with Amazon S3.
2. Have you tried doing a restore? Most of those errors look like they *may* be harmless in that
they indicate that the Volume has more parts than what the Catalog reports. This can happen when
some of the parts are not uploaded to the cloud (i.e. they are still in the cache). It could occur if you shutdown the SD while uploads are still in progress.
Best regards,
Kern
On 5/26/19 3:39 AM, Mike wrote:
Morning Folks,
I was keen to use the new cloud storage feature so I've configured things up to use it as an offsite repository of some backup jobs.
I use the 'Next Pool' Pool parameter and then a Copy type job on PoolUncopiedJobs
I start a job and it seems to run OK but I'm seeing dozens of errors similar to the following and the data from 'list volumes' is very different to what is actually present on the S3 volume.
---
10-May 07:06 ghost-sd JobId 12751: Warning: cloud_dev.c:1748 For Volume "Offsite-0183":
The Parts do not match! Metadata Volume=16915 Catalog=16889.
Correcting Catalog
---
Sometimes a single job will get multiple
---
07-May 11:04 ghost-sd JobId 12608: Warning: cloud_dev.c:1748 For Volume "Offsite-0182":
The Parts do not match! Metadata Volume=6985 Catalog=6711.
The Cloud Parts do not match! Metadata Volume=1817 Catalog=1574.
Correcting Catalog
07-May 11:47 ghost-sd JobId 12608: Warning: cloud_dev.c:1748 For Volume "Offsite-0182":
The Parts do not match! Metadata Volume=10080 Catalog=9986.
The Cloud Parts do not match! Metadata Volume=1953 Catalog=1574.
Correcting Catalog
07-May 11:54 ghost-sd JobId 12608: Warning: cloud_dev.c:1748 For Volume "Offsite-0182":
The Parts do not match! Metadata Volume=10521 Catalog=10480.
The Cloud Parts do not match! Metadata Volume=1976 Catalog=1574.
Correcting Catalog
----
Director: 9.4.2 on Solaris
Storage: 9.4.2 on Solaris (disk based), 9.4.2 on Linux (cloud based)
Clients: 9.4.2 on Solaris, Windows and Linux
Linux sd (ghost) is using Bacula community RPMs
bacula-libs-9.4.2-1.el7.x86_64
bacula-cloud-storage-9.4.2-1.el7.x86_64
bacula-mysql-9.4.2-1.el7.x86_64
I expect it's a bug in the new plugin but thought I'd see if anyone had suggestions on how to proceed: it could be my misconfiguration.
Note I do have Truncate Cache = AfterUpload since I already have a local copy of the job and restoring from cloud would be last resort.
Thanks for the help
* list volumes
<SNIP>
Pool: Offsite
+---------+--------------+-----------+---------+---------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+------------+
| MediaId | VolumeName | VolStatus | Enabled | VolBytes | VolFiles | VolRetention | Recycle | Slot | InChanger | MediaType | VolType | VolParts | LastWritten | ExpiresIn |
+---------+--------------+-----------+---------+---------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+------------+
| 181 | Offsite-1 | Full | 1 | 262,087,761 | 0 | 15,552,000 | 1 | 0 | 0 | CloudType | 14 | 2,866 | 2019-05-05 11:43:59 | 15,011,184 |
| 182 | Offsite-0182 | Full | 1 | 1,073,696,800 | 0 | 15,552,000 | 1 | 0 | 0 | CloudType | 14 | 13,922 | 2019-05-07 12:44:02 | 15,187,587 |
| 183 | Offsite-0183 | Append | 1 | 630,979,806 | 0 | 15,552,000 | 1 | 0 | 0 | CloudType | 14 | 17,827 | 2019-05-11 16:59:43 | 15,548,528 |
| 184 | Offsite-0184 | Append | 1 | 0 | 0 | 15,552,000 | 1 | 0 | 0 | CloudType | 14 | 0 | NULL | NULL |
| 185 | Offsite-0185 | Append | 1 | 0 | 0 | 15,552,000 | 1 | 0 | 0 | CloudType | 14 | 0 | NULL | NULL |
| 186 | Offsite-0186 | Append | 1 | 0 | 0 | 15,552,000 | 1 | 0 | 0 | CloudType | 14 | 0 | NULL | NULL |
| 187 | Offsite-0187 | Full | 1 | 1,073,729,432 | 0 | 15,552,000 | 1 | 0 | 0 | CloudType | 14 | 14,859 | 2019-05-06 18:51:26 | 15,123,231 |
+---------+--------------+-----------+---------+---------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+------------+
Firstly the VolBytes is very wrong, on the S3 side it's 439.633 GBytes (472052843913 Bytes) and what I'd expect based on the Copy jobs.
Secondly there are different counts of VolParts:
13916 Offsite-0182
17722 Offsite-0183
14855 Offsite-0187
724 Offsite-1
* cloud list
<SNIP>
+--------------------+-----------+----------------------+----------------------+---------------+
| Volume Name | Status | Media Type | Pool | VolBytes |
+--------------------+-----------+----------------------+----------------------+---------------+
| Offsite-0182 | Full | CloudType | Offsite | 1.073 GB |
| Offsite-0183 | Append | CloudType | Offsite | 630.9 MB |
| Offsite-0187 | Full | CloudType | Offsite | 1.073 GB |
| Offsite-1 | Full | CloudType | Offsite | 262.0 MB |
+--------------------+-----------+----------------------+----------------------+---------------+
---sd --
Device {
Name = CloudStorage
Device Type = Cloud
Cloud = RealS3
Archive Device = /opt/bacula/cloud_backups
Maximum Part Size = 10 MB
Media Type = CloudType
LabelMedia = yes
Random Access = Yes;
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = no
}
Cloud {
Name = RealS3
Driver = "S3"
HostName = <SNIP>
BucketName = <SNIP>
AccessKey = <SNIP>
SecretKey = <SNIP>
Protocol = HTTPS
UriStyle = Path
Truncate Cache = AfterUpload
Upload = EachPart
Region = "eu-frankfurt-1"
MaximumUploadBandwidth = 35MB/s
}
--- dir ---
Job {
Name = job.copyjob.full
Type = Copy
Pool = Full-Pool
Selection Type = PoolUncopiedJobs
Messages = Standard
Client = client.fake
FileSet="none"
Maximum Concurrent Jobs = 2
}
Pool {
Name = Offsite
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Storage = ghost-changer
Maximum Volume Jobs = 1
AutoPrune = yes
Volume Retention = 6 months
Maximum Volumes = 80
Label Format = Offsite-
}
Pool {
Name = Full-Pool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 3 months
Maximum Volume Jobs = 1
Label Format = Full-
Maximum Volumes = 60
Next Pool = "Offsite"
Storage = File1
}
Autochanger {
Name = ghost-changer
Address = ghost.<SNIP>
SDPort = 9103
Password = <SNIP>
Device = CloudStorage
Media Type = CloudType
Autochanger = ghost-changer
Maximum Concurrent Jobs = 10
TLS Enable = yes
TLS Require = no
TLS CA Certificate File = /etc/bacula/certs/cacert.pem
TLS Certificate = "<SNIP>"
TLS Key = "<SNIP>"
}
Autochanger {
Name = File1
Address = otherhost.<SNIP>
SDPort = 9103
Password = "<SNIP>"
Device = FileChgr1
Media Type = File1
Maximum Concurrent Jobs = 10
Autochanger = File1
TLS Enable = yes
TLS Require = yes
TLS CA Certificate File = /etc/bacula/certs/cacert.pem
TLS Certificate = "<SNIP>"
TLS Key = "<SNIP>"
}
_______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
_______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
