Hi! I'm somewhat surprised: Maybe it's all about latency, because with FC-SAN we typically see a performance _decrease_ if large sequential requests are being transmitted. So actually we did limit the default amount of max_sectors_kb. Most "intelligent" SAN systems break down large requests to small internal requests (like 128kB) that are being handled, and the initiator gets an acknowledge once all such requests internal have finised.
Regards, Ulrich >>> Mike Christie <[email protected]> schrieb am 11.09.2018 um 18:30 in Nachricht <[email protected]>: > Hey, > > Cc [email protected], or I will not see these messages until I check > the list maybe once a week. > > On 09/05/2018 10:36 PM, [email protected] wrote: >> What lio fabric driver are you using? iSCSI? What kernel version >> and >> what version of tcmu-runner? >> >> io fabric driver : iscsi >> >> iscsid version: 2.0-873 >> >> OS version: CentOS Linux release 7.5.1804 (Core) >> >> kernel version: 3.10.0-862.el7.x86_64 >> >> tcmu-runner version: 1.4.0-rc1 >> >> >> Target Build: >> >> targetcli /iscsi create iqn.2018-09.com.test:target1 >> >> targetcli /backstores/user:rbd create name=my_replicated_test >> size=1000G cfgstring=rbd_pool/replicated_image1 hw_max_sectors=8192 > > That kernel has a default data area (kernel buffer used to pass scsi > command data) of 8MB, so with a command size of 8192 sectors you could > only 2 commands at once. > > When you create the backstore pass it a new max_data_area_mb value like > this (make sure you have the newest rtslib-fb, configshell-fb and > targetcli-fb from the upstream github repos): > > targetcli /backstores/user:rbd create name=my_replicated_test > size=1000G cfgstring=rbd_pool/replicated_image1 hw_max_sectors=8192 > control=max_data_area_mb=32 > > This would increase the buffer to 32 MB. > > Or for an existing setup add the control line in the saveconfig.json > between the config and hw_max_sectors line like this: > > "config": "rbd/rbd_pool/replicated_image" > "control": "max_data_area_mb=32", > "hw_max_sectors": 8192, > > Note that this will prealloocate 32 MBs of memory for the device. > >> >> targetcli /iscsi/iqn.2018-09.com.test:target1/tpg1/luns create >> /backstores/user:rbd/my_replicated_test >> >> targetcli /iscsi/iqn.2018-09.com.test:target1/tpg1/portals create >> 10.0.1.111 >> >> targetcli /iscsi/iqn.2018-09.com.test:target1/tpg1 set auth >> userid=****** password=****** >> >> targetcli /iscsi/iqn.2018-09.com.test:target1/tpg1 set attribute >> authentication=1 demo_mode_write_protect=0 generate_node_acls=1 >> >> >> Target Setting: >> >> >> >> 屏幕快照 2018-09-06 上午10.43.11.png >> >> >> >> >> >> >> >> >> >> >> >> > Could you give me some suggestions about improving the >> performance of >> > /backstore/user:rbd/ device. >> > >> > Thanks very much! >> > >> > 在 2018年9月3日星期一 UTC+8上午10:31:48，[email protected]写道： >> > >> > Hello Mike, >> > >> > Thank you for your informative response. >> > >> > >> > 在 2018年8月28日星期二 UTC+8上午8:49:46，Mike Christie写道： >> > >> > On 08/21/2018 08:52 PM, [email protected] wrote: >> > > Hi folks, >> > > >> > > I am newbie to open-iscsi. >> > > My case is I export ceph rbd by open-iscsi. >> > > >> > > I found the max_sectors_kb is 64, the value is so >> small, and >> > 4M sequence >> > > write is only about 10MB/s. >> > > I can not increase max_sectors_kb, if I do it will >> return >> > "bash: echo: >> > > write error: Invalid argument"(But I can change the >> value to a >> > small one >> > > < 64, max_hw_sectors_kb is 32767) >> > > >> > >> > In new version of the linux kernel the initiator will >> use the >> > optimal >> > value reported by the target and then uses the max >> reported as >> > the limit >> > that the user can set. It sounds like you are using >> rbd/ceph with >> > tcmu-runner which has a default limits of 64K. >> > >> > Yes, I am using tcmu-runner and gwcli. >> > >> > >> > If you are using targetcli/lio directly then you can set >> > hw_max_sectors >> > through targcli when you create the device or in the >> > saveconfig.json file. >> > >> > If you are using the ceph-iscsi tools then I am >> actually working >> > on a >> > gwcli command to configure this right now. >> > >> > Yes, I am using ceph-iscsi tools.How can I change the >> limit by >> > ceph-iscsi tools? >> > >> > Thanks. >> > >> > -- >> > You received this message because you are subscribed to the >> Google >> > Groups "open-iscsi" group. >> > To unsubscribe from this group and stop receiving emails from >> it, send >> > an email to [email protected] >> > <mailto:[email protected]>. >> > To post to this group, send email to [email protected] >> > <mailto:[email protected]>. >> > Visit this group at https://groups.google.com/group/open-iscsi >> <https://groups.google.com/group/open-iscsi>. >> > For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> -- >> You received this message because you are subscribed to the Google >> Groups "open-iscsi" group. >> To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/open-iscsi. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "open-iscsi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/open-iscsi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
