I'm trying to build a high-speed iSCSI server. I have two Chelsio T6 cards providing 4x 25GbE ports. I have a requirement for high-availability networking, and I also need multiple ports' worth of bandwidth. What's the best way to use them?
First I tried LACP, of course. That works. But it doesn't work in combination with cxgbei iSCSI offload. The clients can't connect. This makes sense, because the offload engine probably requires all packets from a single iSCSI session to enter and leave through the same network port. With LACP, that won't be the case. Next I tried ALUA. I disabled LACP and assigned a different IP address to each network port. On the clients I connected to each target via all four ports. Then I used gmultipath for each target in Active/Active mode. This did not work, even with cxgbei disabled. iSCSI requests were well-distributed, but the server sent all responses out of cc0. It seems that ctld doesn't do any kind of source-routing. Using cxgbei makes the situation even worse. Without source-routing, the clients almost immediately get disconnected. No doubt that's because requests and responses are going through different network ports again. Is there any way to use cxgbei iSCSI offload with multiple NICs? My best idea, which I haven't yet tried, is to assign each NIC to a different VLAN, then use ALUA over those. That should ensure that requests and responses for each iSCSI session go through the same port. However, gmultipath would still round-robin requests for each iSCSI target through different sessions. I don't know if that would be a problem. -Alan
