OK!!!! Someone on the IRC channel got me closer, but we have no idea why this would happen:
this works: "root@test1:~# nova-manage --config-file=/etc/nova/nova.conf service list" Why would I have to specify the config file though? It is in the standard place. Thanks, Sam On Mon, Mar 11, 2013 at 2:01 PM, Samuel Winchenbach <[email protected]>wrote: > For completeness here the routing table, and ip listing for both test1 and > test2. Doubt this will help much: http://paste2.org/p/3117125 > > > On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach <[email protected]>wrote: > >> #1 - No change >> #2 - All of grants are in the ip/mask form such as: 'nova'@' >> 10.21.0.0/255.255.0.0' I have also tried adding 'nova'@'test1' and >> 'nova'@'10.21.0.1'. No change. >> #3 - I changed the SQL connection string over to IP instead of hostname. >> No change. I didn't restart nova-api because it isn't running. If I >> understand correctly nova-manage communicated directly with the db, >> bypassing nova-api. This would appear true seeing "nova-manage service >> list" works correctly on test2. >> >> >> :( >> >> Thanks for the help! >> Sam >> >> >> On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza < >> [email protected]> wrote: >> >>> When looking at MySQL 5.1 refman ( >>> http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would >>> suggest to follow the procedure : >>> 1. 'mysqladmin flush-hosts' >>> 2. replace DNS entries in mysql.user table by IP addresses instead >>> 3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and >>> restart nova-api !) >>> >>> I wouldn't bet on it, but I would say this is due to some name >>> resolution which is incorrect. >>> >>> -Sylvain >>> >>> >>> Le 11/03/2013 17:00, Sylvain Bauza a écrit : >>> >>> Ok, lemme try to summarize. >>> You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to >>> Pacemaker. >>> This setup is relying on two hosts, test1 (10.21.0.1) and test2 >>> (10.21.0.2). >>> Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP. >>> >>> Are you sure your my.cnf is actually the same in between both DRBD nodes >>> ? (I would recommend to symlink it to a physical file hosted on the DRBD >>> device). >>> >>> One thing is hurting me : you told me that nova is also pacemake'd. If >>> so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to >>> nova-ha (assuming 10.21.2.4 as per /etc/hosts). >>> >>> Also, as per my understanding of Pacemaker, DRBD partition is setup by >>> default on test2, correct ? >>> >>> >>> Sorry, as per my first reading, I can't see anything obvious. That said, >>> I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct >>> MySQL connection attempt. But maybe I'm wrong ? >>> >>> Anyway, are you sure you only have *one* MySQL engine running (either on >>> test1 or test2) and nova-manage trying to access this right one ? >>> >>> Perms look good to me. As it a test setup, you could try to unleash the >>> grants by deleting them and allowing nova@'%' to see if it's a basic >>> dns mapping issue. >>> >>> -Sylvain >>> >>> >>> >>> Le 11/03/2013 16:09, Samuel Winchenbach a écrit : >>> >>> I >>> enabled general_log in /etc/mysql/my.cnf Here are the results of >>> connecting from "test1", "test2" and using the client: >>> http://paste2.org/p/3115525 >>> I purposefully used the real password in case there is a problem with >>> it. >>> I changed before submitting post. >>> >>> here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an >>> attempted "nova-manage service list" from test1: >>> https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump >>> >>> I looked at it with wireshark and couldn't see anything that jumped >>> out at me as incorrect. I have not yet tried to recreate the salted >>> password. >>> >>> >>> Here is my pacemaker configuration for mysql. I stripped out >>> openstack services, rabbitmq and others for clarity. All resources are >>> currently disabled (other than MySQL): >>> http://paste2.org/p/3115685 >>> >>> >>> Please don't yell at me for having STONITH disabled :P This is a >>> testing cluster and I am working on getting routed to the IPMI interface. >>> >>> /etc/hosts: >>> http://paste2.org/p/3115713 >>> /etc/nova/nova.conf: >>> http://paste2.org/p/3115739 >>> >>> >>> If there is anything else I can provide you, please let me know! I >>> have pulled out most of my hair at this point! >>> >>> Sam >>> >>> >>> >>> >>> On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza < >>> [email protected]> wrote: >>> >>>> So as to reproduce the nova-manage SQL command, I would recommand to >>>> tcpdump -A port 3306 on the host and get the SQL trace on what's failing. >>>> >>>> Could you please explain further what is your HA config ? Are you using >>>> pacemaker/heartbeat or any VIP ? >>>> >>>> -Sylvain >>>> >>>> Le 11/03/2013 14:23, Samuel Winchenbach a écrit : >>>> >>>> Does anyone think this could be an openstack bug? I just want to >>>> check before submitting a bug report. >>>> >>>> Sam >>>> >>>> >>>> On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <[email protected]> wrote: >>>> >>>>> Sorry, I really can't think of anything :( >>>>> >>>>> On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: >>>>> > I dropped those users and no change. >>>>> > >>>>> > I also set up general logging in mysql but it really doesn't provide >>>>> any >>>>> > additional information. Any idea for a next step I could take? >>>>> > >>>>> > I am almost at the point of taking a tcpdump and trying to recreate >>>>> the >>>>> > salted password. :/ >>>>> > >>>>> > Thanks for the help >>>>> > >>>>> > Sam >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes <[email protected] >>>>> > <mailto:[email protected]>> wrote: >>>>> > >>>>> > I'm stumped :( Looks like everything is set up correctly to me. >>>>> What is >>>>> > interested is that your nova user access works from test2, but >>>>> there is >>>>> > no nova@test2 user in the mysql.user table. What about doing a >>>>> DROP USER >>>>> > nova@test1; FLUSH PRIVILEGES; and then see if that fixes >>>>> things... since >>>>> > the [email protected]/255.255.0.0 < >>>>> http://[email protected]/255.255.0.0> >>>>> > user is clearly working for the access >>>>> > from test2. >>>>> > >>>>> > Also, I'd recommend highly removing the nova@% user. >>>>> > >>>>> > Best, >>>>> > -jay >>>>> > >>>>> > On 03/08/2013 03:09 PM, Samuel Winchenbach wrote: >>>>> > > >>>>> > > http://paste2.org/p/3085807 >>>>> > > >>>>> > > >>>>> > > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes <[email protected] >>>>> > <mailto:[email protected]> >>>>> > > <mailto:[email protected] <mailto:[email protected]>>> >>>>> wrote: >>>>> > > >>>>> > > Please paste the results of SELECT User, Host, Password >>>>> FROM >>>>> > mysql.user >>>>> > > when running as root... >>>>> > > >>>>> > > Thanks! >>>>> > > -jay >>>>> > > >>>>> > > On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: >>>>> > > > Here are my grants. I don't know if this helps, but I >>>>> did >>>>> > verify that >>>>> > > > the password was identical for each grant: >>>>> > > http://paste2.org/p/3085361 >>>>> > > > >>>>> > > > >>>>> > > > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach >>>>> > > <[email protected] <mailto:[email protected]> >>>>> > <mailto:[email protected] <mailto:[email protected]>> >>>>> > > > <mailto:[email protected] <mailto:[email protected]> >>>>> > <mailto:[email protected] <mailto:[email protected]>>>> wrote: >>>>> > > > >>>>> > > > root@test1:/var/log# mysql -hmysql-ha -unova >>>>> > > > -p******************************** -e"SELECT User, >>>>> Host, >>>>> > Password >>>>> > > > FROM mysql.user;" >>>>> > > > ERROR 1142 (42000) at line 1: SELECT command denied >>>>> to user >>>>> > > > 'nova'@'test1' for table 'user' >>>>> > > > >>>>> > > > >>>>> > > > On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes >>>>> > <[email protected] <mailto:[email protected]> >>>>> > > <mailto:[email protected] <mailto:[email protected]>> >>>>> > > > <mailto:[email protected] <mailto: >>>>> [email protected]> >>>>> > <mailto:[email protected] <mailto:[email protected]>>>> wrote: >>>>> > > > >>>>> > > > What does this show? >>>>> > > > >>>>> > > > mysql -hmysql-ha -unova -p<PASS> -e"SELECT User, >>>>> Host, >>>>> > > Password FROM >>>>> > > > mysql.user" >>>>> > > > >>>>> > > > -jay >>>>> > > > >>>>> > > > On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: >>>>> > > > > Sorry, that must have been a copy and paste >>>>> error. >>>>> > Here >>>>> > > is what I >>>>> > > > > actually ran: >>>>> > > > > >>>>> > > > > http://paste2.org/p/3084996 >>>>> > > > > >>>>> > > > > >>>>> > > > > On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes >>>>> > > <[email protected] <mailto:[email protected]> >>>>> > <mailto:[email protected] <mailto:[email protected]>> >>>>> > > > <mailto:[email protected] >>>>> > <mailto:[email protected]> <mailto:[email protected] >>>>> > <mailto:[email protected]>>> >>>>> > > > > <mailto:[email protected] >>>>> > <mailto:[email protected]> <mailto:[email protected] >>>>> > <mailto:[email protected]>> >>>>> > > <mailto:[email protected] <mailto:[email protected]> >>>>> > <mailto:[email protected] <mailto:[email protected]>>>>> >>>>> wrote: >>>>> > > > > >>>>> > > > > On 03/08/2013 12:19 PM, Samuel Winchenbach >>>>> wrote: >>>>> > > > > > Hi All, >>>>> > > > > > >>>>> > > > > > I have two nodes (test1 and test2) that >>>>> I am >>>>> > trying to >>>>> > > > set up in a >>>>> > > > > > highly available configuration. >>>>> > > > > > >>>>> > > > > > During the setup process I tried running >>>>> > "nova-manage >>>>> > > > service list" on >>>>> > > > > > both nodes. It worked fine on test2, >>>>> but >>>>> > fails on >>>>> > > > test1 even >>>>> > > > > though I >>>>> > > > > > can connect to the database with the >>>>> mysql >>>>> > client from >>>>> > > > test1. >>>>> > > > > > >>>>> > > > > > Here is a screen capture that shows the >>>>> setup on >>>>> > > the two >>>>> > > > nodes are >>>>> > > > > > basically identical: >>>>> > http://paste2.org/p/3084223 >>>>> > > > > >>>>> > > > > In the above paste you are doing: >>>>> > > > > >>>>> > > > > mysql -unova - hmysql-ha -u root >>>>> nova >>>>> > > > > -p******************************** >>>>> > > > > >>>>> > > > > Note you are supplying 2 -u arguments, and >>>>> mysql >>>>> > > will take >>>>> > > > the second >>>>> > > > > (root). >>>>> > > > > >>>>> > > > > -jay >>>>> > > > > >>>>> > > > > >>>>> _______________________________________________ >>>>> > > > > Mailing list: >>>>> https://launchpad.net/~openstack >>>>> > > > > Post to : >>>>> [email protected] >>>>> > <mailto:[email protected]> >>>>> > > <mailto:[email protected] >>>>> > <mailto:[email protected]>> >>>>> > > > <mailto:[email protected] >>>>> > <mailto:[email protected]> >>>>> > > <mailto:[email protected] >>>>> > <mailto:[email protected]>>> >>>>> > > > > <mailto:[email protected] >>>>> > <mailto:[email protected]> >>>>> > > <mailto:[email protected] >>>>> > <mailto:[email protected]>> >>>>> > > > <mailto:[email protected] >>>>> > <mailto:[email protected]> >>>>> > > <mailto:[email protected] >>>>> > <mailto:[email protected]>>>> >>>>> > > > > Unsubscribe : >>>>> https://launchpad.net/~openstack >>>>> > > > > More help : >>>>> https://help.launchpad.net/ListHelp >>>>> > > > > >>>>> > > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > >>>>> > > >>>>> > >>>>> > >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~openstack >>>> Post to : [email protected] >>>> Unsubscribe : https://launchpad.net/~openstack >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~openstack >>>> Post to : [email protected] >>>> Unsubscribe : https://launchpad.net/~openstack >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >>> >>> >>> >> >
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
