There might be a preexisting bug in the test [<twitter.common.testing.clock.ThreadedClock object at 0x101d50210>] Time now: 0.0
File "/private/var/folders/rd/_tjz8zts3g14md1kmf38z6w80000gn/T/tmp5K5MuG/.deps/twitter.common.exceptions-0.3.3-py2-none-any.whl/twitter/common/exceptions/__init__.py", line 126, in _excepting_run self.__real_run(*args, **kw) File "/var/folders/rd/_tjz8zts3g14md1kmf38z6w80000gn/T/tmp5K5MuG/apache/aurora/executor/common/health_checker.py", line 112, in run is_healthy, reason = self._perform_check_if_not_disabled() File "/var/folders/rd/_tjz8zts3g14md1kmf38z6w80000gn/T/tmp5K5MuG/apache/aurora/executor/common/health_checker.py", line 92, in _perform_check_if_not_disabled return self.checker() File "/var/folders/rd/_tjz8zts3g14md1kmf38z6w80000gn/T/tmp5K5MuG/apache/aurora/executor/common/health_checker.py", line 168, in wrapper success, failure_reason = closure(*args, **kw) File "/private/var/folders/rd/_tjz8zts3g14md1kmf38z6w80000gn/T/tmp5K5MuG/.deps/mock-1.0.1-py2-none-any.whl/mock.py", line 955, in __call__ return _mock_self._mock_call(*args, **kwargs) File "/private/var/folders/rd/_tjz8zts3g14md1kmf38z6w80000gn/T/tmp5K5MuG/.deps/mock-1.0.1-py2-none-any.whl/mock.py", line 1018, in _mock_call ret_val = effect(*args, **kwargs) File "/Users/wickman/clients/aurora/src/test/python/apache/aurora/executor/common/test_health_checker.py", line 46, in mock_health_check return self.fake_health_checks.pop(0) IndexError: pop from empty list WARNING:root:Health check failure: reason WARNING:root:Health check failure: reason On Fri, Feb 20, 2015 at 4:42 PM, Brian Wickman <wick...@apache.org> wrote: > Looking. > > On Fri, Feb 20, 2015 at 4:17 PM, Zameer Manji <zma...@apache.org> wrote: > >> The health checker test seems flaky or incorrect. Wickman, can you please >> take a look? >> >> On Fri, Feb 20, 2015 at 4:15 PM, Apache Jenkins Server < >> jenk...@builds.apache.org> wrote: >> >>> See <https://builds.apache.org/job/Aurora/889/changes> >>> >>> Changes: >>> >>> [zmanji] Remove single caller methods from AuroraCommandContext >>> >>> [wickman] Instrument the HealthChecker to export stats. >>> >>> ------------------------------------------ >>> [...truncated 4179 lines...] >>> >>> >>> src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_disconnected >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_error >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_frameworkMessage >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_log >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_registered >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_reregistered >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_send_update >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_status_is_terminal >>> [32mPASSED [0m >>> >>> generated xml file: < >>> https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.executor_base.xml >>> > >>> [32m [1m============ 8 passed in 0.18 seconds >>> ============ [0m >>> [1m============== test session starts >>> =============== [0m >>> platform linux2 -- Python 2.7.6 -- py-1.4.26 -- >>> pytest-2.6.4 -- < >>> https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 >>> > >>> plugins: cov, timeout >>> [1m >>> collecting 0 items [0m [1m >>> collecting 2 items [0m [1m >>> collected 2 items >>> [0m >>> >>> src/test/python/apache/aurora/executor/test_executor_vars.py::test_init >>> [32mPASSED [0m >>> >>> src/test/python/apache/aurora/executor/test_executor_vars.py::test_sample >>> [32mPASSED [0m >>> >>> generated xml file: < >>> https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.executor_vars.xml >>> > >>> [32m [1m============ 2 passed in 0.11 seconds >>> ============ [0m >>> [1m============== test session starts >>> =============== [0m >>> platform linux2 -- Python 2.7.6 -- py-1.4.26 -- >>> pytest-2.6.4 -- < >>> https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 >>> > >>> plugins: cov, timeout >>> [1m >>> collecting 0 items [0m [1m >>> collecting 1 items [0m [1m >>> collecting 1 items [0m [1m >>> collecting 1 items [0m [1m >>> collected 1 items >>> [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_status_manager.py::TestStatusManager::test_run >>> [32mPASSED [0m >>> >>> generated xml file: < >>> https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.status_manager.xml >>> > >>> [32m [1m============ 1 passed in 0.18 seconds >>> ============ [0m >>> [1m============== test session starts >>> =============== [0m >>> platform linux2 -- Python 2.7.6 -- py-1.4.26 -- >>> pytest-2.6.4 -- < >>> https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 >>> > >>> plugins: cov, timeout >>> [1m >>> collecting 0 items [0m [1m >>> collecting 8 items [0m [1m >>> collecting 8 items [0m [1m >>> collecting 8 items [0m [1m >>> collected 8 items >>> [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_success >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_failed >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_stop >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_lose >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_quitquitquit >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_thermos_normal_exit_status >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_thermos_abnormal_exit_statuses >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_thermos_runner_killed >>> [32mPASSED [0m >>> >>> generated xml file: < >>> https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.thermos_task_runner.xml >>> > >>> [32m [1m=========== 8 passed in 76.21 seconds >>> ============ [0m >>> [1m============== test session starts >>> =============== [0m >>> platform linux2 -- Python 2.7.6 -- py-1.4.26 -- >>> pytest-2.6.4 -- < >>> https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 >>> > >>> plugins: cov, timeout >>> [1m >>> collecting 0 items [0m [1m >>> collecting 8 items [0m [1m >>> collected 8 items >>> [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_announcer.py::test_serverset_join_thread >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_announcer.py::test_announcer_under_normal_circumstances >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_announcer.py::test_announcer_on_expiration >>> [33mSKIPPED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_announcer.py::test_announcer_under_abnormal_circumstances >>> [33mSKIPPED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_announcer.py::test_make_empty_endpoints >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_announcer.py::test_announcer_provider_with_timeout >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_announcer.py::test_default_announcer_provider >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_announcer.py::test_default_announcer_provider_without_announce >>> [32mPASSED [0m >>> >>> generated xml file: < >>> https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.announcer.xml >>> > >>> [32m [1m====== 6 passed, 2 skipped in 0.67 seconds >>> ======= [0m >>> [1m============== test session starts >>> =============== [0m >>> platform linux2 -- Python 2.7.6 -- py-1.4.26 -- >>> pytest-2.6.4 -- < >>> https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 >>> > >>> plugins: cov, timeout >>> [1m >>> collecting 0 items [0m [1m >>> collecting 5 items [0m [1m >>> collected 5 items >>> [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_directory_sandbox >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_create >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_user_does_not_exist >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_create_ioerror >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_destroy_ioerror >>> [32mPASSED [0m >>> >>> generated xml file: < >>> https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.directory_sandbox.xml >>> > >>> [32m [1m============ 5 passed in 0.07 seconds >>> ============ [0m >>> [1m============== test session starts >>> =============== [0m >>> platform linux2 -- Python 2.7.6 -- py-1.4.26 -- >>> pytest-2.6.4 -- < >>> https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 >>> > >>> plugins: cov, timeout >>> [1m >>> collecting 0 items [0m [1m >>> collecting 4 items [0m [1m >>> collected 4 items >>> [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_executor_detector.py::test_find_root >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_executor_detector.py::test_match_inverse >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_executor_detector.py::test_bad_match >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_executor_detector.py::test_integration >>> [32mPASSED [0m >>> >>> generated xml file: < >>> https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.executor_detector.xml >>> > >>> [32m [1m============ 4 passed in 0.04 seconds >>> ============ [0m >>> [1m============== test session starts >>> =============== [0m >>> platform linux2 -- Python 2.7.6 -- py-1.4.26 -- >>> pytest-2.6.4 -- < >>> https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 >>> > >>> plugins: cov, timeout >>> [1m >>> collecting 0 items [0m [1m >>> collecting 2 items [0m [1m >>> collecting 2 items [0m [1m >>> collecting 2 items [0m [1m >>> collected 2 items >>> [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_executor_timeout.py::TestExecutorTimeout::test_run >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_executor_timeout.py::TestExecutorTimeout::test_run_timeout >>> [32mPASSED [0m >>> >>> generated xml file: < >>> https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.executor_timeout.xml >>> > >>> [32m [1m============ 2 passed in 0.07 seconds >>> ============ [0m >>> [1m============== test session starts >>> =============== [0m >>> platform linux2 -- Python 2.7.6 -- py-1.4.26 -- >>> pytest-2.6.4 -- < >>> https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 >>> > >>> plugins: cov, timeout >>> [1m >>> collecting 0 items [0m [1m >>> collecting 4 items [0m [1m >>> collecting 5 items [0m [1m >>> collecting 12 items [0m [1m >>> collecting 12 items [0m [1m >>> collected 12 items >>> [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthChecker::test_consecutive_failures >>> [31mFAILED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthChecker::test_health_checker_metrics >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthChecker::test_initial_interval_2x >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthChecker::test_initial_interval_whatev >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthCheckerProvider::test_from_assigned_task >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_maybe_update_failure_count >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_perform_check_if_not_disabled_no_snooze_file >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_perform_check_if_not_disabled_snooze_file_exists >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_perform_check_if_not_disabled_snooze_file_is_none >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_run >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_start >>> [32mPASSED [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_stop >>> [32mPASSED [0m >>> >>> ==================== FAILURES ==================== >>> __ TestHealthChecker.test_consecutive_failures ___ >>> >>> self = >>> <common.test_health_checker.TestHealthChecker >>> testMethod=test_consecutive_failures> >>> >>> [1m def test_consecutive_failures(self): [0m >>> [1m '''Verify that a task is unhealthy only >>> after max_consecutive_failures is exceeded''' [0m >>> [1m initial_interval_secs = 2 [0m >>> [1m interval_secs = 1 [0m >>> [1m self.append_health_checks(False, >>> num_calls=2) [0m >>> [1m self.append_health_checks(True) [0m >>> [1m self.append_health_checks(False, >>> num_calls=3) [0m >>> [1m hct = HealthChecker( [0m >>> [1m self._checker.health, [0m >>> [1m interval_secs=interval_secs, [0m >>> [1m >>> initial_interval_secs=initial_interval_secs, [0m >>> [1m max_consecutive_failures=2, [0m >>> [1m clock=self._clock) [0m >>> [1m hct.start() [0m >>> [1m [0m >>> [1m # 2 consecutive health check failures >>> followed by a successful health check. [0m >>> [1m self._clock.tick(initial_interval_secs) >>> [0m >>> [1m assert >>> self._clock.converge(threads=[hct.threaded_health_checker], timeout=1) [0m >>> [1m assert hct.status is None [0m >>> [1m> assert >>> hct.metrics.sample()['consecutive_failures'] == 1 [0m >>> [1m [31mE AssertionError: assert 0 == 1 [0m >>> >>> >>> src/test/python/apache/aurora/executor/common/test_health_checker.py:108: >>> AssertionError >>> -------------- Captured stderr call -------------- >>> [<twitter.common.testing.clock.ThreadedClock object >>> at 0x7fc87bd87f10>] Time now: 0.0 >>> [<twitter.common.testing.clock.ThreadedClock object >>> at 0x7fc87bd87f10>] Time now: 2.0 >>> generated xml file: < >>> https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.health_checker.xml >>> > >>> [1m [31m====== 1 failed, 11 passed in 0.77 seconds >>> ======= [0m >>> src.test.python.apache.aurora.admin.admin >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.admin.host_maintenance >>> ..... SUCCESS >>> src.test.python.apache.aurora.admin.maintenance >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.api.api >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.client.api.instance_watcher >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.client.api.job_monitor >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.api.mux >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.client.api.quota_check >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.api.restarter >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.client.api.scheduler_client >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.api.sla >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.api.task_util >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.api.updater >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.client.api.updater_util >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.base >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.client.binding_helper >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.api >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.client >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.client.cli.command_hooks >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.config >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.context >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.cron >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.inspect >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.job >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.plugins >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.quota >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.sla >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.supdate >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.task >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.update >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.cli.version >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.config >>> ..... SUCCESS >>> src.test.python.apache.aurora.client.factory >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.client.hooks.hooked_api >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.client.hooks.non_hooked_api >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.common.test_aurora_job_key >>> ..... SUCCESS >>> src.test.python.apache.aurora.common.test_cluster >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.common.test_cluster_option >>> ..... SUCCESS >>> src.test.python.apache.aurora.common.test_clusters >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.common.test_http_signaler >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.common.test_pex_version >>> ..... SUCCESS >>> src.test.python.apache.aurora.common.test_shellify >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.common.test_transport >>> ..... SUCCESS >>> src.test.python.apache.aurora.config.test_base >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.config.test_constraint_parsing >>> ..... SUCCESS >>> src.test.python.apache.aurora.config.test_loader >>> ..... SUCCESS >>> src.test.python.apache.aurora.config.test_thrift >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.common.announcer >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.common.directory_sandbox >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.common.executor_detector >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.common.executor_timeout >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.common.health_checker >>> ..... FAILURE >>> >>> src.test.python.apache.aurora.executor.common.path_detector >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.common.task_info >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.executor_base >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.executor_vars >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.status_manager >>> ..... SUCCESS >>> >>> src.test.python.apache.aurora.executor.thermos_task_runner >>> ..... SUCCESS >>> >>> src.test.python.apache.thermos.common.test_pathspec >>> ..... SUCCESS >>> >>> src.test.python.apache.thermos.core.test_runner_integration >>> ..... SUCCESS >>> >>> src.test.python.apache.thermos.monitoring.test_disk >>> ..... SUCCESS >>> >>> FAILURE >>> >>> [31m >>> FAILURE [0m >>> Build step 'Execute shell' marked build as failure >>> Recording test results >>> >> >> >> >> -- >> Zameer Manji >> > >