Hello I am having very similar issue.
Scenario 1 ) When I mount a rbd volume directly using rbd map command to a linux kernel. I am getting good performance. when I do dd on the above volume, I am getting 380 - 430 MB's. -- I am good with this. Scenario 2 ) I am using iscsi-gateway. iscsi1 and iscs2 as targets and presenting LUNS ro all my initiators. When I do dd on the volumes presented via iscsi gateway, my performance is 10 to 25 MB's. which is very poor. I am gwcli commandline and connot use the belwo command "create images image=test1 size=100g max_data_area_mb=128 Unexpected keyword parameter 'max_data_area_mb' I also cannot use targetcli on this server. How can I increase the max_data_area_mb && hw_max_sectors sizes in my cluster ? Thank you in advance On Friday, September 21, 2018 at 7:16:58 AM UTC-7, [email protected] wrote: > > Thank you for your opinion of max_sectors_kb, Ulrich. > It provided me some inspiration for understanding the whole thing. > > Thank you for your advice, Mike. > I will tune some parameters as you have mentioned below. > > I will share if I could make achievements in performance tuning. > > > 在 2018年9月21日星期五 UTC+8上午1:43:14，Mike Christie写道： >> >> On 09/12/2018 09:26 PM, [email protected] wrote: >> > Thank you for your reply, Mike. >> > Now my iscsi disk performance can be around 300MB/s in 4M sequence >> > write(TCMU+LIO) >> > It increase from 20MB/s to 300MB/s, after I can change max_data_area_mb >> > from 8 to 256 && hw_max_sectors from 128 to 8192. >> > To my cluster, after a lot of tests I found that I should keep >> > "max_data_area_mb>128 && hw_max_sectors>=4096" in order to get a good >> > performance. >> > Does my setting can cause some side effects? >> >> It depends on the kernel. For the RHEL/Centos kernel you are using the >> kernel will preallocate max_data_area_mb of memory for each device. For >> upstream, we no longer preallocate, but once it is allocated we do not >> free the memory unless global_max_data_area_mb is hit or the device is >> removed. >> >> With a high hw_max_sectors latency will increase due to sending really >> large commands, so it depends on your workload and what you need. >> >> We used to set hw_max_sectors to the rbd object size (4MB by default), >> but in our testing we would see throughput go down around 512k - 1MB. >> >> > Are there any other parameters can improve the performance quite >> obvious? >> >> The normal networking ones like using jumbo frames, >> net.core.*/net.ipv4.*, etc. Check your nic's documentation for the best >> settings. >> >> There are some iscsi ones like the cmdsn/cmds_max I mentioned and then >> also the segment related ones like MaxRecvDataSegmentLength, >> MaxXmitDataSegmentLength, MaxBurstLength, FirstBurstLength and have >> ImmediateData on. >> >> >> > Why the default value of max_data_area_mb and hw_max_sectors is so >> > small, and bad performance? >> >> I do not know. It was just what the original author had used initially. >> >> > Could you talk something about this? >> > At least, max_data_area_mb>128 && hw_max_sectors>=4096, I can get a >> > better performance seems to be acceptable. >> > If my settings can give other users some help, I will be happy. >> > >> > 在 2018年9月12日星期三 UTC+8上午12:39:16，Mike Christie写道： >> > >> > On 09/11/2018 11:30 AM, Mike Christie wrote: >> > > Hey, >> > > >> > > Cc [email protected] <javascript:>, or I will not see these >> > messages until I check >> > > the list maybe once a week. >> > > >> > > On 09/05/2018 10:36 PM, [email protected] <javascript:> wrote: >> > >> What lio fabric driver are you using? iSCSI? What kernel >> > version >> > >> and >> > >> what version of tcmu-runner? >> > >> >> > >> io fabric driver : iscsi >> > >> >> > >> iscsid version: 2.0-873 >> > >> >> > >> OS version: CentOS Linux release 7.5.1804 >> > (Core) >> > >> >> > >> kernel version: 3.10.0-862.el7.x86_64 >> > >> >> > >> tcmu-runner version: 1.4.0-rc1 >> > >> >> > >> >> > >> > There is also a perf bug in that initiator if the >> node.session.cmds_max >> > is greater than the LIO default_cmdsn_depth and your IO test tries >> to >> > send cmds > node.session.cmds_max. >> > >> > I have known the bug before, because I had google a lot. >> > It increase from 320MB/s to 340MB/s (4M seq write), seems like a >> stable >> > promotion. >> > >> > Settings Before: 320MB/s >> > node.session.cmds_max = 2048 >> > default_cmdsn_depth = 64 >> > >> > Settings After: 340MB/s >> > node.session.cmds_max = 64 >> > default_cmdsn_depth = 64 >> > >> > So set the node.session.cmds_max and default_cmdsn_depth to the >> same >> > value. You can set the default_cmdsn_depth in saveconfig.json, and >> set >> > cmds_max in the iscsiadm node db (after you set it make sure you >> logout >> > and login the session again). >> > >> > Now I set set the node.session.cmds_max and default_cmdsn_depth both >> to >> > be 64. >> > >> > Thank you very much! >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "open-iscsi" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an email to [email protected] >> > <mailto:[email protected]>. >> > To post to this group, send email to [email protected] >> > <mailto:[email protected]>. >> > Visit this group at https://groups.google.com/group/open-iscsi. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
