For completeness here the routing table, and ip listing for both test1 and test2. Doubt this will help much: http://paste2.org/p/3117125
On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach <[email protected]>wrote: > #1 - No change > #2 - All of grants are in the ip/mask form such as: 'nova'@' > 10.21.0.0/255.255.0.0' I have also tried adding 'nova'@'test1' and > 'nova'@'10.21.0.1'. No change. > #3 - I changed the SQL connection string over to IP instead of hostname. > No change. I didn't restart nova-api because it isn't running. If I > understand correctly nova-manage communicated directly with the db, > bypassing nova-api. This would appear true seeing "nova-manage service > list" works correctly on test2. > > > :( > > Thanks for the help! > Sam > > > On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza < > [email protected]> wrote: > >> When looking at MySQL 5.1 refman ( >> http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would >> suggest to follow the procedure : >> 1. 'mysqladmin flush-hosts' >> 2. replace DNS entries in mysql.user table by IP addresses instead >> 3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and >> restart nova-api !) >> >> I wouldn't bet on it, but I would say this is due to some name resolution >> which is incorrect. >> >> -Sylvain >> >> >> Le 11/03/2013 17:00, Sylvain Bauza a écrit : >> >> Ok, lemme try to summarize. >> You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to >> Pacemaker. >> This setup is relying on two hosts, test1 (10.21.0.1) and test2 >> (10.21.0.2). >> Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP. >> >> Are you sure your my.cnf is actually the same in between both DRBD nodes >> ? (I would recommend to symlink it to a physical file hosted on the DRBD >> device). >> >> One thing is hurting me : you told me that nova is also pacemake'd. If >> so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to >> nova-ha (assuming 10.21.2.4 as per /etc/hosts). >> >> Also, as per my understanding of Pacemaker, DRBD partition is setup by >> default on test2, correct ? >> >> >> Sorry, as per my first reading, I can't see anything obvious. That said, >> I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct >> MySQL connection attempt. But maybe I'm wrong ? >> >> Anyway, are you sure you only have *one* MySQL engine running (either on >> test1 or test2) and nova-manage trying to access this right one ? >> >> Perms look good to me. As it a test setup, you could try to unleash the >> grants by deleting them and allowing nova@'%' to see if it's a basic dns >> mapping issue. >> >> -Sylvain >> >> >> >> Le 11/03/2013 16:09, Samuel Winchenbach a écrit : >> >> I >> enabled general_log in /etc/mysql/my.cnf Here are the results of >> connecting from "test1", "test2" and using the client: >> http://paste2.org/p/3115525 >> I purposefully used the real password in case there is a problem with >> it. >> I changed before submitting post. >> >> here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an >> attempted "nova-manage service list" from test1: >> https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump >> >> I looked at it with wireshark and couldn't see anything that jumped out >> at me as incorrect. I have not yet tried to recreate the salted password. >> >> >> Here is my pacemaker configuration for mysql. I stripped out >> openstack services, rabbitmq and others for clarity. All resources are >> currently disabled (other than MySQL): >> http://paste2.org/p/3115685 >> >> >> Please don't yell at me for having STONITH disabled :P This is a >> testing cluster and I am working on getting routed to the IPMI interface. >> >> /etc/hosts: >> http://paste2.org/p/3115713 >> /etc/nova/nova.conf: >> http://paste2.org/p/3115739 >> >> >> If there is anything else I can provide you, please let me know! I >> have pulled out most of my hair at this point! >> >> Sam >> >> >> >> >> On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza < >> [email protected]> wrote: >> >>> So as to reproduce the nova-manage SQL command, I would recommand to >>> tcpdump -A port 3306 on the host and get the SQL trace on what's failing. >>> >>> Could you please explain further what is your HA config ? Are you using >>> pacemaker/heartbeat or any VIP ? >>> >>> -Sylvain >>> >>> Le 11/03/2013 14:23, Samuel Winchenbach a écrit : >>> >>> Does anyone think this could be an openstack bug? I just want to >>> check before submitting a bug report. >>> >>> Sam >>> >>> >>> On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <[email protected]> wrote: >>> >>>> Sorry, I really can't think of anything :( >>>> >>>> On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: >>>> > I dropped those users and no change. >>>> > >>>> > I also set up general logging in mysql but it really doesn't provide >>>> any >>>> > additional information. Any idea for a next step I could take? >>>> > >>>> > I am almost at the point of taking a tcpdump and trying to recreate >>>> the >>>> > salted password. :/ >>>> > >>>> > Thanks for the help >>>> > >>>> > Sam >>>> > >>>> > >>>> > >>>> > >>>> > On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes <[email protected] >>>> > <mailto:[email protected]>> wrote: >>>> > >>>> > I'm stumped :( Looks like everything is set up correctly to me. >>>> What is >>>> > interested is that your nova user access works from test2, but >>>> there is >>>> > no nova@test2 user in the mysql.user table. What about doing a >>>> DROP USER >>>> > nova@test1; FLUSH PRIVILEGES; and then see if that fixes >>>> things... since >>>> > the [email protected]/255.255.0.0 < >>>> http://[email protected]/255.255.0.0> >>>> > user is clearly working for the access >>>> > from test2. >>>> > >>>> > Also, I'd recommend highly removing the nova@% user. >>>> > >>>> > Best, >>>> > -jay >>>> > >>>> > On 03/08/2013 03:09 PM, Samuel Winchenbach wrote: >>>> > > >>>> > > http://paste2.org/p/3085807 >>>> > > >>>> > > >>>> > > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes <[email protected] >>>> > <mailto:[email protected]> >>>> > > <mailto:[email protected] <mailto:[email protected]>>> >>>> wrote: >>>> > > >>>> > > Please paste the results of SELECT User, Host, Password >>>> FROM >>>> > mysql.user >>>> > > when running as root... >>>> > > >>>> > > Thanks! >>>> > > -jay >>>> > > >>>> > > On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: >>>> > > > Here are my grants. I don't know if this helps, but I did >>>> > verify that >>>> > > > the password was identical for each grant: >>>> > > http://paste2.org/p/3085361 >>>> > > > >>>> > > > >>>> > > > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach >>>> > > <[email protected] <mailto:[email protected]> >>>> > <mailto:[email protected] <mailto:[email protected]>> >>>> > > > <mailto:[email protected] <mailto:[email protected]> >>>> > <mailto:[email protected] <mailto:[email protected]>>>> wrote: >>>> > > > >>>> > > > root@test1:/var/log# mysql -hmysql-ha -unova >>>> > > > -p******************************** -e"SELECT User, >>>> Host, >>>> > Password >>>> > > > FROM mysql.user;" >>>> > > > ERROR 1142 (42000) at line 1: SELECT command denied >>>> to user >>>> > > > 'nova'@'test1' for table 'user' >>>> > > > >>>> > > > >>>> > > > On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes >>>> > <[email protected] <mailto:[email protected]> >>>> > > <mailto:[email protected] <mailto:[email protected]>> >>>> > > > <mailto:[email protected] <mailto: >>>> [email protected]> >>>> > <mailto:[email protected] <mailto:[email protected]>>>> wrote: >>>> > > > >>>> > > > What does this show? >>>> > > > >>>> > > > mysql -hmysql-ha -unova -p<PASS> -e"SELECT User, >>>> Host, >>>> > > Password FROM >>>> > > > mysql.user" >>>> > > > >>>> > > > -jay >>>> > > > >>>> > > > On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: >>>> > > > > Sorry, that must have been a copy and paste >>>> error. >>>> > Here >>>> > > is what I >>>> > > > > actually ran: >>>> > > > > >>>> > > > > http://paste2.org/p/3084996 >>>> > > > > >>>> > > > > >>>> > > > > On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes >>>> > > <[email protected] <mailto:[email protected]> >>>> > <mailto:[email protected] <mailto:[email protected]>> >>>> > > > <mailto:[email protected] >>>> > <mailto:[email protected]> <mailto:[email protected] >>>> > <mailto:[email protected]>>> >>>> > > > > <mailto:[email protected] >>>> > <mailto:[email protected]> <mailto:[email protected] >>>> > <mailto:[email protected]>> >>>> > > <mailto:[email protected] <mailto:[email protected]> >>>> > <mailto:[email protected] <mailto:[email protected]>>>>> wrote: >>>> > > > > >>>> > > > > On 03/08/2013 12:19 PM, Samuel Winchenbach >>>> wrote: >>>> > > > > > Hi All, >>>> > > > > > >>>> > > > > > I have two nodes (test1 and test2) that I >>>> am >>>> > trying to >>>> > > > set up in a >>>> > > > > > highly available configuration. >>>> > > > > > >>>> > > > > > During the setup process I tried running >>>> > "nova-manage >>>> > > > service list" on >>>> > > > > > both nodes. It worked fine on test2, but >>>> > fails on >>>> > > > test1 even >>>> > > > > though I >>>> > > > > > can connect to the database with the mysql >>>> > client from >>>> > > > test1. >>>> > > > > > >>>> > > > > > Here is a screen capture that shows the >>>> setup on >>>> > > the two >>>> > > > nodes are >>>> > > > > > basically identical: >>>> > http://paste2.org/p/3084223 >>>> > > > > >>>> > > > > In the above paste you are doing: >>>> > > > > >>>> > > > > mysql -unova - hmysql-ha -u root >>>> nova >>>> > > > > -p******************************** >>>> > > > > >>>> > > > > Note you are supplying 2 -u arguments, and >>>> mysql >>>> > > will take >>>> > > > the second >>>> > > > > (root). >>>> > > > > >>>> > > > > -jay >>>> > > > > >>>> > > > > >>>> _______________________________________________ >>>> > > > > Mailing list: >>>> https://launchpad.net/~openstack >>>> > > > > Post to : [email protected] >>>> > <mailto:[email protected]> >>>> > > <mailto:[email protected] >>>> > <mailto:[email protected]>> >>>> > > > <mailto:[email protected] >>>> > <mailto:[email protected]> >>>> > > <mailto:[email protected] >>>> > <mailto:[email protected]>>> >>>> > > > > <mailto:[email protected] >>>> > <mailto:[email protected]> >>>> > > <mailto:[email protected] >>>> > <mailto:[email protected]>> >>>> > > > <mailto:[email protected] >>>> > <mailto:[email protected]> >>>> > > <mailto:[email protected] >>>> > <mailto:[email protected]>>>> >>>> > > > > Unsubscribe : >>>> https://launchpad.net/~openstack >>>> > > > > More help : >>>> https://help.launchpad.net/ListHelp >>>> > > > > >>>> > > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > >>>> > > >>>> > >>>> > >>>> >>> >>> >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~openstack >>> Post to : [email protected] >>> Unsubscribe : https://launchpad.net/~openstack >>> More help : https://help.launchpad.net/ListHelp >>> >>> >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~openstack >>> Post to : [email protected] >>> Unsubscribe : https://launchpad.net/~openstack >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> >> >> >
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
