Hello! The config looks correct. I'll check why ConnectorConfiguration doesn't have proper string representation, hence: https://issues.apache.org/jira/browse/IGNITE-8935
Regards, -- Ilya Kasnacheev 2018-07-05 9:21 GMT+03:00 胡海麟 <[email protected]>: > Hi, > > I added the config above. But I don't know whether it is applied. > > connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@ > 7d8704ef > > There is such a piece of relevant info but no detail. Is there any way > to confirm the details? > > Thanks. > > On Wed, Jul 4, 2018 at 7:25 PM, Ilya Kasnacheev > <[email protected]> wrote: > > Hello! > > > > You should add it to config/default-config.xml in the form of > > > > <property name="connectorConfiguration"> > > <bean > > class="org.apache.ignite.configuration.ConnectorConfiguration"> > > <property name="threadPoolSize" value="64"/> > > </bean> > > </property> > > > > inside IgniteConfiguration bean. > > > > Come to think of it, maybe your slowdown is caused not by thread > starvation > > but by genuine locking due to repeated usage of the same keys. Are you > sure > > that you are using a lot of different keys and not just a couple which > you > > over-write over and over? > > > > Makes sense to try both. > > > > Regards, > > > > -- > > Ilya Kasnacheev > > > > 2018-07-04 13:19 GMT+03:00 胡海麟 <[email protected]>: > >> > >> Hi, > >> > >> After googling for 1 hour to get nothing, I'm back for some basic > >> knowledges. > >> > >> igniteConfiguration.setConnectorConfiguration(new > >> ConnectorConfiguration().setThreadPoolSize(64)) > >> > >> I suppose this is a piece of java code but I don't know how and where > >> to execute it. > >> I started the cluster with `bin/ignite.sh config/default-config.xml` > >> and use golang written program as a client. > >> > >> Am I missing something? > >> > >> Thanks. > >> > >> On Wed, Jul 4, 2018 at 6:22 PM, Ilya Kasnacheev > >> <[email protected]> wrote: > >> > Hello! > >> > > >> > You can try increasing number of threads in REST thread pool by > setting > >> > igniteConfiguration.setConnectorConfiguration(new > >> > ConnectorConfiguration().setThreadPoolSize(64)) > >> > Or the corresponding Spring XML. > >> > > >> > This is as per https://apacheignite.readme.io/docs/rest-api > >> > > >> > Regards, > >> > > >> > -- > >> > Ilya Kasnacheev > >> > > >> > 2018-07-04 12:04 GMT+03:00 胡海麟 <[email protected]>: > >> >> > >> >> Hi, > >> >> > >> >> Here is the thread dump. > >> >> > >> >> Thank you. > >> >> > >> >> On Wed, Jul 4, 2018 at 5:52 PM, Ilya Kasnacheev > >> >> <[email protected]> wrote: > >> >> > Hello! > >> >> > > >> >> > Can you provide the thread dump collected when the system is under > >> >> > peak > >> >> > load? > >> >> > > >> >> > I think it's some other thread pool, such as client pool or > >> >> > management > >> >> > pool, > >> >> > but have to take a look at the thread dump to be sure. > >> >> > > >> >> > Regards, > >> >> > > >> >> > -- > >> >> > Ilya Kasnacheev > >> >> > > >> >> > 2018-07-04 11:33 GMT+03:00 胡海麟 <[email protected]>: > >> >> >> > >> >> >> Hi, > >> >> >> > >> >> >> We use ignite as a redis server. > >> >> >> > >> >> >> The use case is > >> >> >> a. Write timeout is 15ms on the client side. > >> >> >> b. 2 server nodes. each is an EC2 r4.4xlarge instance. > >> >> >> c. Write req/s is about 120,000. In another word, 60,000 for each > >> >> >> node. > >> >> >> > >> >> >> The problem is that timeout happens frequently, several ones per > >> >> >> second. > >> >> >> A lower write req/s results less timeout. I guest we have > bottleneck > >> >> >> somewhere. > >> >> >> > >> >> >> ========== > >> >> >> $ tail -f > >> >> >> /opt/apache-ignite-fabric-2.5.0-bin/work/log/ignite- > ee4f25ed.0.log > >> >> >> | grep pool > >> >> >> ^-- Public thread pool [active=0, idle=0, qSize=0] > >> >> >> ^-- System thread pool [active=0, idle=16, qSize=0] > >> >> >> ========== > >> >> >> system thread pool seems not busy at all. > >> >> >> > >> >> >> ========== > >> >> >> $ tail -f > >> >> >> /opt/apache-ignite-fabric-2.5.0-bin/work/log/ignite- > ee4f25ed.0.log > >> >> >> | grep "CPU " > >> >> >> ^-- CPU [cur=14.77%, avg=6.21%, GC=0%] > >> >> >> ^-- CPU [cur=13.43%, avg=6.23%, GC=0%] > >> >> >> ========== > >> >> >> CPU is not busy, either. > >> >> >> > >> >> >> We expected milli second level performance and we have too many > >> >> >> timeout > >> >> >> now. > >> >> >> Any idea for optimizing the performance? > >> >> >> > >> >> >> Thanks. > >> >> > > >> >> > > >> > > >> > > > > > >
