Hello Pawel, Is the maximum number of process you speak of defined somewhere (and can it be changed)? We've monitored these two numbers for a couple of days now in 5 minute intervals and observed that a) The number of workers is always higher or equal to the number of tasks; and b) the numbers seem to top out around 200.
We have not had the opportunity to observe the numbers in a failure scenario yet though. W > On June 24, 2014 at 3:00 PM Pawel Stefanski <pejo...@gmail.com> wrote: > > > hello > > You should monitor number of running stmf tasks versus number of stmf > (comstar) processes. We had such issues - once we hited max number of > processes and second time - there was a bug regarding stmf process pool > scaling (but I think it's Nexenta specific, not Illumos). > > keys to observe: > echo stmf_cur_ntasks/D | mdb -k # number of currently serving tasks > > echo stmf_nworkers_cur/D | mdb -k # number of running workers > > best regards! > -- > pawel > > > > On Tue, Jun 24, 2014 at 8:21 PM, w...@vandenberge.us <w...@vandenberge.us> > wrote: > > > Hello, > > > > I have three OpenIndiana (151A8) servers used as iSCSI targets. All > > servers have > > two 10Gbe interfaces to separate Dell 8024F switches running the latest > > firmware. These servers provide storage for a bank of 16 Windows 2012R2 > > virtualization servers, each running 16 virtual machines (Windows 7x64). > > Each > > virtualization server also connected to both 10Gbe switches. iSCSI is > > configured > > to use round-robin. The interfaces and switches are dedicated to iSCSI, all > > other traffic is routed over a separate admin/client network. The > > virtualization > > servers and the iSCSI servers do not share an admin network (the only paths > > between then are the iSCSI networks which are flat class C without a > > gateway. > > > > Everything works fine. When the systems are at their busiest we see a very > > nicely balanced load of 25MB/sec on each initiator's iSCSI interfaces > > with the > > occasionally quick peak close to 100MB on individual machines. load on the > > iSCSI > > servers hovers around 3 and network utilization on each of the six target > > interface sit slightly about 130MB/sec. > > > > However, every week or so, one of the systems will, without warning or log > > that > > I can find, start dropping iSCSI connections. The virtualization servers > > will > > report a loss of the storage volume. Over a period of 30 minutes or so all > > remaining iSCSI connections to that storage server will die and the only > > way to > > get them back is to restart the machine or disable the > > /network/iscsi/target > > service, wait about 2 minutes and then enable it (a simple restart will > > not work > > with a log entry that the service is still running when trying to restart. > > > > This problem occurs on all three servers randomly, sometimes within days, > > sometimes only after a couple of weeks. Servers are good, but commodity > > hardware > > (SuperMicrso, LSI, Seagate, Intel) and configured similarly but not > > identically > > (slightly different motherboards, processors (dual 1.8GHz quadcore Xeon > > min) > > and memory configurations (none less than 128GB) > > > > My problem is that nothing appears in the logs on the OpenIndiana servers. > > Spying on the network shows that requests are getting to the Open Indiana > > servers but essentially fall in a black hole. I've ruled out problems with > > individual disks, cables and controllers. > > > > Has anyone ever seen this before? Any ideas for something I could look at > > besides the obvious logs? > > > > thanks in advance, > > > > Wim > > _______________________________________________ > > openindiana-discuss mailing list > > openindiana-discuss@openindiana.org > > http://openindiana.org/mailman/listinfo/openindiana-discuss > > > _______________________________________________ > openindiana-discuss mailing list > openindiana-discuss@openindiana.org > http://openindiana.org/mailman/listinfo/openindiana-discuss _______________________________________________ openindiana-discuss mailing list openindiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
