I have a rather odd use case where I would like to have several thousand iSCSI sessions (one connection per session) on a single initiator, but I'm having trouble reaching beyond 6000-8000 before things start breaking.
My hardware has 72GB memory, 12 Xeon Cores at 2.5GHz, and 2 10GB ethernet ports. For software I'm running on Ubuntu 12.04 (With Linux kernel 3.8) with open-iscsi 2.0-873. First, I make several ifaces using the same hardware so I can present different iqns to the target. iscsiadm -m iface -I testiface.0 -o new iscsiadm -m iface -I testiface.0 -o update -n iface.initiatorname -v iqn.something.0000 iscsiadm -m iface -I testiface.0 -o update -n iface.iface_num -v 0 iscsiadm -m iface -I testiface.1 -o new iscsiadm -m iface -I testiface.1 -o update -n iface.initiatorname -v iqn.something.0001 iscsiadm -m iface -I testiface.1 -o update -n iface.iface_num -v 1 Then I perform discovery for each iface. The discovery may report up to 2000 targets per initiator iface that I'm using. iscsiadm -m discovery -t sendtargets -p some.ip.address -I testiface.0 -o new iscsiadm -m discovery -t sendtargets -p some.ip.address -I testiface.1 -o new Finally I try to log in to everything: iscsiadm -m node -L all After iscsid crunches on it for a while (seemingly single threaded from watching top), and eventually all of the iSCSI sessions are created (I can see the sessions are established on the targets). After the sessions are created, iscsid then iscsiadm begins printing: Login to [iface: testiface.1, target: iqn.something.else, portal: some.ip.address,3260] successful. However, after quite a few successes, I finally start getting: iscsiadm: Could not login to [iface: testiface.5, target: iqn.something.else, portal: some.ip.address,3260]. iscsiadm: initiator reported error (9 - internal error) I did an strace of iscsid during this, and it seems to be getting ENOMEM back on clone system calls. Looking at ps after I start geting failures, there are quite a few defunct iscsid processes in the list that I guess the parent process has not attended to yet. What seems strange to me is that I seem to have quite a lot of free memory looking at vmstat/free (at least right up to the point where things start going crazy). I added a 300GB SSD as swap space, but that did not seem to affect the problem, nor did tweaking Linux's overcommit settings. Eventually the issues get so bad that bash complains about not being able to fork processes due to a lack of memory. Would 8000 sessions really take 72GB of memory? strace snippet: clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fc24aa7a9d0) = -1 ENOMEM (Cannot allocate memory) If I try to log in for each iface individually it is FAR slower. That is to say the first 2000 sessions are established rather quickly, but logging in things after that is extremely slow, i.e. using the commands iscsiadm -m node -l -I testiface.0 iscsiadm -m node -l -I testiface.1 The first command goes fairly quickly, but the second command takes an extreme amount of time. Is what I'm doing even advisable or feasible? Is there a reasonable maximum number of sessions that can exist on a single initiator before things start going bad? -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
