Hello users, I have started using gluster just a few weeks ago and I am rocking a Replicated-Distributed setup with arbiters (A) and SATA Volumes (V). I have around 6 volumes and 3 arbiters in this setup: V+V+A | V+V+A | V+V+A
All these volumes are spread across 3 different nodes, all of them being 1Gbit. Due to hardware limitations, SSD or 10Gbit network is not available. But even then, testing via iperf and normal rsync of files between servers, I am easily able to achieve 700Mbps~ [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 49.9 MBytes 419 Mbits/sec 21 132 KBytes [ 4] 1.00-2.00 sec 80.0 MBytes 671 Mbits/sec 0 214 KBytes [ 4] 2.00-3.00 sec 87.0 MBytes 730 Mbits/sec 3 228 KBytes [ 4] 3.00-4.00 sec 91.6 MBytes 769 Mbits/sec 15 215 KBytes But when rsyncing data from same server to another node with mounted glusterVolume, I am getting measly 50Mbps (7MBps). All servers have 64GB Ram and their memory usage is around 50% and CPU usage less than 10%. All bricks are zfs volumes, no Raid setup or anything. All volumes are direct hard disks formatted as ZFS (JBOD setup). My Gluster Vol Info gluster vol info Volume Name: glusterStore Type: Distributed-Replicate Volume ID: c7ac8094-f379-45fc-8cfd-f2937355e03d Status: Started Snapshot Count: 0 Number of Bricks: 3 x (2 + 1) = 9 Transport-type: tcp Bricks: Brick1: 62.0.0.1:/zpool1/proxmox Brick2: 5.0.0.1:/zpool1/proxmox Brick3: 62.0.0.1 :/home/glusterArbiter (arbiter) Brick4: 62.0.0.1 :/zpool2/proxmox Brick5: 5.0.0.1 :/zpool2/proxmox Brick6: 62.0.0.2:/home/glusterArbiter2 (arbiter) Brick7: 62.0.0.2:/zpool/proxmox Brick8: 5.0.0.1 :/zpool3/proxmox Brick9: 62.0.0.2:/home/glusterArbiter (arbiter) Options Reconfigured: performance.readdir-ahead: enable cluster.rsync-hash-regex: none client.event-threads: 16 server.event-threads: 16 network.ping-timeout: 5 performance.normal-prio-threads: 64 performance.high-prio-threads: 64 performance.io-thread-count: 64 performance.cache-size: 1GB performance.read-ahead: off performance.io-cache: off performance.flush-behind: off performance.quick-read: on network.frame-timeout: 60 storage.batch-fsync-delay-usec: 0 server.allow-insecure: on performance.stat-prefetch: off cluster.lookup-optimize: on performance.write-behind: on cluster.granular-entry-heal: on storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on performance.client-io-threads: off Regards
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
