Hello All, I tried deploying again, without making any change. It went past the 42% mark where it was stuck last time, and after around 2 hours, errored out again, with the following message:
*Error* Deployment has failed. All nodes are finished. Failed tasks: Task[primary-keystone/1] Stopping the deployment process! keystone-all.log in the errored node - controller - shows only these lines marked as error: <43>Oct 25 07:24:35 node-1 keystone_wsgi_admin_error [Wed Oct 25 07:24:29.560129 2017] [wsgi:error] [pid 193507:tid 139913962108672] Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. The above line had repeated 23 times. I can attach the complete log file if needed. Doing this for the first time, I do not know where to look for possible issues. Please help me get past this. Many Thanks, Raja. On 25 October 2017 at 07:14, Raja T Nair <[email protected]> wrote: > Thanks Evgeniy, for the support. > > I ran the command on shell, and it immediately came out, leaving a line in > the logfile which said that all required images are present. > After this I re-ran the deployment. It went past the point, but errored > out, with the following info on the dashboard: > > Error > Deployment has failed. All nodes are finished. Failed tasks: > Task[fuel_pkgs/3], Task[fuel_pkgs/2], Task[fuel_pkgs/4] Stopping the > deployment process! > > 3 of the nodes show `Error Deployment' and 2 of them show `Stopped' > Log tab gives the following lines for nodes: > > <> > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/bin/puppet:8:in `<main>' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) > in run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in > main' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in > `apply_catalog' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in > apply_catalog' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in > `as_logging_destination' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in > apply' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in > `traverse' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) > in evaluate' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:221:in `eval_resource' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:19:in > `process_events' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in > `queued_events' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in `each' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:93:in > `block in queued_events' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:20:in > `block in process_events' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:101:in > `process_callback' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:583:in `refresh' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `sync' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `times' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:130:in `block in sync' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:45:in `run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:29:in `run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:29:in `chdir' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:64:in `block in run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:68:in `block (2 levels) > in run' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:186:in `execute' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > /usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:186:in `waitpid2' > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > Command exceeded timeout > 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update]) > Failed to call refresh: Command exceeded timeout > </> > > Inside individual nodes, puppet.log has these lines, which shows of some > unmet dependency: > > 2017-10-24 17:14:00 +0000 /Package[fuel-misc] (info): Starting to evaluate > the resource > 2017-10-24 17:14:00 +0000 Puppet (debug): Executing '/usr/bin/dpkg-query > -W --showformat '${Status} ${P > ackage} ${Version}\n' fuel-misc' > 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug): > Call: install > 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug): > Call: wait_for_lock > 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug): > Call: locked? > 2017-10-24 17:14:00 +0000 Puppet (debug): Executing '/usr/bin/apt-get -q > -y -o DPkg::Options::=--force- > confold -o APT::Get::AllowUnauthenticated=1 install fuel-misc' > 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) > (warning): Attempt '1' of '3' has faile > d: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold > -o APT::Get::AllowUnauthenti > cated=1 install fuel-misc' returned 100: Reading package lists... > Building dependency tree... > Reading state information... > Some packages could not be installed. This may mean that you have > requested an impossible situation or if you are using the unstable > distribution that some required packages have not yet been created > or been moved out of Incoming. > The following information may help to resolve the situation: > > The following packages have unmet dependencies: > fuel-misc : Depends: socat but it is not installable > E: Unable to correct problems, you have held broken packages. > > But dpkg shows that both are installed: > > root@node-1:/var/log# dpkg -l fuel-misc socat > Desired=Unknown/Install/Remove/Purge/Hold > | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/ > trig-aWait/Trig-pend > |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) > ||/ Name Version Architecture Description > +++-===================-==============-==============-====== > ====================================== > ii fuel-misc 9.0.0-1~u14.04 all Misc Fuel library > scripts > ii socat 1.7.2.3-1 amd64 multipurpose relay > for bidirectional data tr > root@node-1:/var/log# > > > Please help me to get this deployment completed. > > Many Thanks, > > Raja. > > On 24 October 2017 at 21:13, Evgeniy L <[email protected]> wrote: > >> Hi, >> >> Yes, execution expired, means that it could not build an image within >> required period of time. >> The message you see can be used to build an image (copy paste into >> console "command"), also there is a file with logs >> /var/log/fuel-agent-env-1.log, which you can use to try to identify >> where the problem is. >> >> Thanks, >> >> On Tue, Oct 24, 2017 at 7:48 AM, Raja T Nair <[email protected]> wrote: >> >>> Hello All, >>> >>> We are trying to deploy one environment using Fuel 9.2, and it failed at >>> around 23%. >>> >>> The error says: >>> >>> <> >>> Error >>> Provision has failed. Failed to execute hook 'shell' >>> >>> command: cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp >>> --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image >>> --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": " >>> http://172.22.66.2:8080/targetimages/env_1_ubuntu_1404_amd64-boot.img.gz";, >>> "format": "ext2"}, "/": {"container": "gzip", "uri": " >>> http://172.22.66.2:8080/targetimages/env_1_ubuntu_1404_amd64.img.gz";, >>> "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", "repos": >>> [{"name": "ubuntu", "section": "main universe multiverse", "uri": " >>> http://archive.ubuntu.com/ubuntu/";, "priority": null, "suite": >>> "trusty", "type": "deb"}, {"name": "ubuntu-updates", "section": "main >>> universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/";, >>> "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name": >>> "ubuntu-security", "section": "main universe multiverse", "uri": " >>> http://archive.ubuntu.com/ubuntu/";, "priority": null, "suite": >>> "trusty-security", "type": "deb"}, {"name": "mos", "section": "main >>> restricted", "uri": "http://172.22.66.2:8080/mitaka-9.0/ubuntu/x86_64";, >>> "priority": 1050, "suite": "mos9.0", "type": "deb"}, {"name": >>> "mos-updates", "section": "main restricted", "uri": " >>> http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/";, "priority": 1050, >>> "suite": "mos9.0-updates", "type": "deb"}, {"name": "mos-security", >>> "section": "main restricted", "uri": "http://mirror.fuel-infra.org/ >>> mos-repos/ubuntu/9.0/", "priority": 1050, "suite": "mos9.0-security", >>> "type": "deb"}, {"name": "mos9.2-updates", "section": "main restricted", >>> "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2";, "priority": >>> 1050, "suite": "mos9.0-updates", "type": "deb"}, {"name": >>> "mos9.2-security", "section": "main restricted", "uri": " >>> http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2";, "priority": 1050, >>> "suite": "mos9.0-security", "type": "deb"}, {"name": "Auxiliary", >>> "section": "main restricted", "uri": "http://172.22.66.2:8080/mitak >>> a-9.0/ubuntu/auxiliary", "priority": 1150, "suite": "auxiliary", >>> "type": "deb"}, {"name": "mos9.2-holdback", "section": "main restricted", >>> "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2";, "priority": >>> 1100, "suite": "mos9.0-holdback", "type": "deb"}], "packages": ["acl", >>> "anacron", "bash-completion", "bridge-utils", "bsdmainutils", >>> "build-essential", "cloud-init", "curl", "daemonize", "debconf-utils", >>> "gdisk", "grub-pc", "hwloc", "i40e-dkms", "linux-firmware", >>> "linux-firmware-nonfree", "linux-headers-generic-lts-xenial", >>> "linux-image-generic-lts-xenial", "lvm2", "mcollective", "mdadm", >>> "multipath-tools", "multipath-tools-boot", "nailgun-agent", >>> "nailgun-mcagents", "network-checker", "ntp", "openssh-client", >>> "openssh-server", "puppet", "python-amqp", "ruby-augeas", "ruby-ipaddress", >>> "ruby-json", "ruby-netaddr", "ruby-openstack", "ruby-shadow", "ruby-stomp", >>> "telnet", "ubuntu-minimal", "ubuntu-standard", "uuid-runtime", "vim", >>> "virt-what", "vlan"], "codename": "trusty"}' >>> >>> Task: a84a3af7-2ed4-4d5e-9911-a12dfc0f05e9: overall timeout error: >>> execution expired >>> Task timeout: 3600, Retries: 1 >>> >>> </> >>> >>> I understand it was a timeout, probably because of a slow Internet >>> connection, while building a required image - please correct me if I'm >>> wrong here. >>> >>> Might be stupid question - but is there a way for me to build such >>> images before clicking the deploy button, thereby avoiding the timeout >>> issue? >>> >>> Can anyone guide me or point me to a document to understand what is >>> going wrong here? >>> >>> Many thanks in advance. >>> >>> Regards, >>> Raja. >>> >>> -- >>> :^) >>> >>> _______________________________________________ >>> Mailing list: http://lists.openstack.org/cgi >>> -bin/mailman/listinfo/openstack >>> Post to : [email protected] >>> Unsubscribe : http://lists.openstack.org/cgi >>> -bin/mailman/listinfo/openstack >>> >>> >> > > > -- > :^) > -- :^)
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
