Actually, no... On haproxy I have allowed all traffic with input interface matching the internal interface (eth0), rest of nodes no firewall whatsoever (which will be changed)..
On 8 June 2015 at 15:17, Rafael Fonseca <rsafons...@gmail.com> wrote: > Do you have a firewall in between that might be closing long or idle > sessions? > > On Mon, Jun 8, 2015 at 3:14 PM, Andrija Panic <andrija.pa...@gmail.com> > wrote: > > > Will try Simon. > > > > I expect no latency issues, since these 3 VMs are all on same > > bridge/physical host. Other 2 Galera nodes are remote, but still low > > latency - and zero seconds backend downtime... > > > > Not sure where to dig further... > > > > Thx > > > > > > > > On 8 June 2015 at 15:05, Simon Weller <swel...@ena.com> wrote: > > > > > Can you point the mgmt server directly at a single Galera node > (removing > > > haproxy), and see if the problem persists? > > > > > > Is your Galera cluster healthy? Could you be seeing latency on commit? > > > > > > > > > ________________________________________ > > > From: Andrija Panic <andrija.pa...@gmail.com> > > > Sent: Monday, June 8, 2015 7:42 AM > > > To: us...@cloudstack.apache.org; dev@cloudstack.apache.org > > > Subject: [HELP needed] haproxy vs mysql client timeout > > > > > > Hi, > > > > > > I'm running into intermittent problems with ACS 4.5.1 mgmt server - it > > > seems it's loosing connection to the haproxy/mysql, since we are using > > > haproxy VIP to in db.properties, and this haproxy is using 1 galera > node > > in > > > the backend section...(2 others are backup etc). > > > > > > > > > haproxy timeout is 100sec. > > > mysql timeouts are all defaults, except innodb_lock_wait_timeout. > > > > > > +-----------------------------+----------+ > > > | Variable_name | Value | > > > +-----------------------------+----------+ > > > | connect_timeout | 10 | > > > | delayed_insert_timeout | 300 | > > > | have_statement_timeout | YES | > > > | innodb_flush_log_at_timeout | 1 | > > > | innodb_lock_wait_timeout | 600 | > > > | innodb_rollback_on_timeout | ON | > > > | interactive_timeout | 28800 | > > > | net_read_timeout | 30 | > > > | net_write_timeout | 60 | > > > | thread_pool_idle_timeout | 60 | > > > | wait_timeout | 28800 | > > > +-----------------------------+----------+ > > > > > > To make things more interesting, all 3 servers (acs mgmt, haproxy, > > > galera-node1) are on the same physical host, so I rule out network > > > issues... and I dont see anything interesting in the haproxy logs, also > > no > > > backend downtimes etc... > > > > > > > > > The errors are like folowing, from mgmt logs, but after that mgmt > server > > > continues to work fine... > > > > > > Any suggestions on tuning timeouts ? > > > > > > > > > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > > > Communications link failure > > > > > > The last packet successfully received from the server was 137,744 > > > milliseconds ago. The last packet sent successfully to the server was > 1 > > > milliseconds ago. > > > at > > sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown > > > Source) > > > at > > > > > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > > at > > java.lang.reflect.Constructor.newInstance(Constructor.java:526) > > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > > at > > > > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129) > > > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720) > > > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609) > > > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160) > > > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617) > > > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778) > > > at > > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825) > > > at > > > > > > > > > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156) > > > at > > > > > > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313) > > > at > > > > > > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) > > > at > > > > > > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) > > > at > > > com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009) > > > ... 61 more > > > Caused by: java.io.EOFException: Can not read response from server. > > > Expected to read 4 bytes, read 0 bytes before connection was > unexpectedly > > > lost. > > > ... 76 more > > > > > > Thanks, > > > > > > -- > > > > > > Andrija Panić > > > > > > > > > > > -- > > > > Andrija Panić > > > -- Andrija Panić
