Tried once again, this time its a different error: *Error* Deployment has failed. All nodes are finished. Failed tasks: Task[top-role-compute/3], Task[top-role-compute/2], Task[top-role-compute/5], Task[top-role-compute/4] Stopping the deployment process!
I don't know if I should continue doing this in the hope that it will complete eventually. Am I missing something? Are all these known bugs and I had to apply some patch at some stage? A guiding note will be highly appreciated. Many Thanks, Raja. On 25 October 2017 at 13:06, Raja T Nair <[email protected]> wrote: > Hello All, > > I tried deploying again, without making any change. > It went past the 42% mark where it was stuck last time, and after around 2 > hours, errored out again, with the following message: > > *Error* > Deployment has failed. All nodes are finished. Failed tasks: > Task[primary-keystone/1] Stopping the deployment process! > > keystone-all.log in the errored node - controller - shows only these lines > marked as error: > > <43>Oct 25 07:24:35 node-1 keystone_wsgi_admin_error [Wed Oct 25 > 07:24:29.560129 2017] [wsgi:error] [pid 193507:tid 139913962108672] Option > "verbose" from group "DEFAULT" is deprecated for removal. Its value may be > silently ignored in the future. > > The above line had repeated 23 times. > I can attach the complete log file if needed. > > Doing this for the first time, I do not know where to look for possible > issues. Please help me get past this. > > Many Thanks, > Raja. > > > > On 25 October 2017 at 07:14, Raja T Nair <[email protected]> wrote: > >> Thanks Evgeniy, for the support. >> >> I ran the command on shell, and it immediately came out, leaving a line >> in the logfile which said that all required images are present. >> After this I re-ran the deployment. It went past the point, but errored >> out, with the following info on the dashboard: >> >> Error >> Deployment has failed. All nodes are finished. Failed tasks: >> Task[fuel_pkgs/3], Task[fuel_pkgs/2], Task[fuel_pkgs/4] Stopping the >> deployment process! >> >> 3 of the nodes show `Error Deployment' and 2 of them show `Stopped' >> Log tab gives the following lines for nodes: >> >> <> >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/bin/puppet:8:in `<main>' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) >> in run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in >> `run_command' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in >> main' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in >> `apply_catalog' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in >> apply_catalog' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in >> `as_logging_destination' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in >> apply' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in >> `traverse' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in >> evaluate' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) >> in evaluate' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:221:in `eval_resource' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:19:in >> `process_events' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in >> `queued_events' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in >> `each' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:93:in >> `block in queued_events' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:20:in >> `block in process_events' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:101:in >> `process_callback' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:583:in `refresh' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `sync' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `times' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:130:in `block in sync' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:45:in `run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:29:in `run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:29:in `chdir' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:64:in `block in run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:68:in `block (2 >> levels) in run' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:186:in `execute' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> /usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:186:in `waitpid2' >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> Command exceeded timeout >> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) >> Failed to call refresh: Command exceeded timeout >> </> >> >> Inside individual nodes, puppet.log has these lines, which shows of some >> unmet dependency: >> >> 2017-10-24 17:14:00 +0000 /Package[fuel-misc] (info): Starting to >> evaluate the resource >> 2017-10-24 17:14:00 +0000 Puppet (debug): Executing '/usr/bin/dpkg-query >> -W --showformat '${Status} ${P >> ackage} ${Version}\n' fuel-misc' >> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug): >> Call: install >> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug): >> Call: wait_for_lock >> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug): >> Call: locked? >> 2017-10-24 17:14:00 +0000 Puppet (debug): Executing '/usr/bin/apt-get -q >> -y -o DPkg::Options::=--force- >> confold -o APT::Get::AllowUnauthenticated=1 install fuel-misc' >> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) >> (warning): Attempt '1' of '3' has faile >> d: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold >> -o APT::Get::AllowUnauthenti >> cated=1 install fuel-misc' returned 100: Reading package lists... >> Building dependency tree... >> Reading state information... >> Some packages could not be installed. This may mean that you have >> requested an impossible situation or if you are using the unstable >> distribution that some required packages have not yet been created >> or been moved out of Incoming. >> The following information may help to resolve the situation: >> >> The following packages have unmet dependencies: >> fuel-misc : Depends: socat but it is not installable >> E: Unable to correct problems, you have held broken packages. >> >> But dpkg shows that both are installed: >> >> root@node-1:/var/log# dpkg -l fuel-misc socat >> Desired=Unknown/Install/Remove/Purge/Hold >> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig >> -aWait/Trig-pend >> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) >> ||/ Name Version Architecture Description >> +++-===================-==============-==============-====== >> ====================================== >> ii fuel-misc 9.0.0-1~u14.04 all Misc Fuel library >> scripts >> ii socat 1.7.2.3-1 amd64 multipurpose relay >> for bidirectional data tr >> root@node-1:/var/log# >> >> >> Please help me to get this deployment completed. >> >> Many Thanks, >> >> Raja. >> >> On 24 October 2017 at 21:13, Evgeniy L <[email protected]> wrote: >> >>> Hi, >>> >>> Yes, execution expired, means that it could not build an image within >>> required period of time. >>> The message you see can be used to build an image (copy paste into >>> console "command"), also there is a file with logs >>> /var/log/fuel-agent-env-1.log, which you can use to try to identify >>> where the problem is. >>> >>> Thanks, >>> >>> On Tue, Oct 24, 2017 at 7:48 AM, Raja T Nair <[email protected]> wrote: >>> >>>> Hello All, >>>> >>>> We are trying to deploy one environment using Fuel 9.2, and it failed >>>> at around 23%. >>>> >>>> The error says: >>>> >>>> <> >>>> Error >>>> Provision has failed. Failed to execute hook 'shell' >>>> >>>> command: cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp >>>> --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image >>>> --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": " >>>> http://172.22.66.2:8080/targetimages/env_1_ubuntu_1404_amd6 >>>> 4-boot.img.gz", "format": "ext2"}, "/": {"container": "gzip", "uri": " >>>> http://172.22.66.2:8080/targetimages/env_1_ubuntu_1404_amd64.img.gz";, >>>> "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", >>>> "repos": [{"name": "ubuntu", "section": "main universe multiverse", "uri": >>>> " >>>> http://archive.ubuntu.com/ubuntu/";, "priority": null, "suite": >>>> "trusty", "type": "deb"}, {"name": "ubuntu-updates", "section": "main >>>> universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/";, >>>> "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name": >>>> "ubuntu-security", "section": "main universe multiverse", "uri": " >>>> http://archive.ubuntu.com/ubuntu/";, "priority": null, "suite": >>>> "trusty-security", "type": "deb"}, {"name": "mos", "section": "main >>>> restricted", "uri": "http://172.22.66.2:8080/mitaka-9.0/ubuntu/x86_64";, >>>> "priority": 1050, "suite": "mos9.0", "type": "deb"}, {"name": >>>> "mos-updates", "section": "main restricted", "uri": " >>>> http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/";, "priority": 1050, >>>> "suite": "mos9.0-updates", "type": "deb"}, {"name": "mos-security", >>>> "section": "main restricted", "uri": "http://mirror.fuel-infra.org/ >>>> mos-repos/ubuntu/9.0/", "priority": 1050, "suite": "mos9.0-security", >>>> "type": "deb"}, {"name": "mos9.2-updates", "section": "main restricted", >>>> "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2";, >>>> "priority": 1050, "suite": "mos9.0-updates", "type": "deb"}, {"name": >>>> "mos9.2-security", "section": "main restricted", "uri": " >>>> http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2";, "priority": 1050, >>>> "suite": "mos9.0-security", "type": "deb"}, {"name": "Auxiliary", >>>> "section": "main restricted", "uri": "http://172.22.66.2:8080/mitak >>>> a-9.0/ubuntu/auxiliary", "priority": 1150, "suite": "auxiliary", >>>> "type": "deb"}, {"name": "mos9.2-holdback", "section": "main restricted", >>>> "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2";, >>>> "priority": 1100, "suite": "mos9.0-holdback", "type": "deb"}], "packages": >>>> ["acl", "anacron", "bash-completion", "bridge-utils", "bsdmainutils", >>>> "build-essential", "cloud-init", "curl", "daemonize", "debconf-utils", >>>> "gdisk", "grub-pc", "hwloc", "i40e-dkms", "linux-firmware", >>>> "linux-firmware-nonfree", "linux-headers-generic-lts-xenial", >>>> "linux-image-generic-lts-xenial", "lvm2", "mcollective", "mdadm", >>>> "multipath-tools", "multipath-tools-boot", "nailgun-agent", >>>> "nailgun-mcagents", "network-checker", "ntp", "openssh-client", >>>> "openssh-server", "puppet", "python-amqp", "ruby-augeas", "ruby-ipaddress", >>>> "ruby-json", "ruby-netaddr", "ruby-openstack", "ruby-shadow", "ruby-stomp", >>>> "telnet", "ubuntu-minimal", "ubuntu-standard", "uuid-runtime", "vim", >>>> "virt-what", "vlan"], "codename": "trusty"}' >>>> >>>> Task: a84a3af7-2ed4-4d5e-9911-a12dfc0f05e9: overall timeout error: >>>> execution expired >>>> Task timeout: 3600, Retries: 1 >>>> >>>> </> >>>> >>>> I understand it was a timeout, probably because of a slow Internet >>>> connection, while building a required image - please correct me if I'm >>>> wrong here. >>>> >>>> Might be stupid question - but is there a way for me to build such >>>> images before clicking the deploy button, thereby avoiding the timeout >>>> issue? >>>> >>>> Can anyone guide me or point me to a document to understand what is >>>> going wrong here? >>>> >>>> Many thanks in advance. >>>> >>>> Regards, >>>> Raja. >>>> >>>> -- >>>> :^) >>>> >>>> _______________________________________________ >>>> Mailing list: http://lists.openstack.org/cgi >>>> -bin/mailman/listinfo/openstack >>>> Post to : [email protected] >>>> Unsubscribe : http://lists.openstack.org/cgi >>>> -bin/mailman/listinfo/openstack >>>> >>>> >>> >> >> >> -- >> :^) >> > > > > -- > :^) > -- :^)
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
