The health checker test seems flaky or incorrect. Wickman, can you please take a look?
On Fri, Feb 20, 2015 at 4:15 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See <https://builds.apache.org/job/Aurora/889/changes> > > Changes: > > [zmanji] Remove single caller methods from AuroraCommandContext > > [wickman] Instrument the HealthChecker to export stats. > > ------------------------------------------ > [...truncated 4179 lines...] > > > src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_disconnected > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_error > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_frameworkMessage > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_log > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_registered > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_reregistered > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_send_update > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_executor_base.py::TestExecutorBase::test_status_is_terminal > [32mPASSED [0m > > generated xml file: < > https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.executor_base.xml > > > [32m [1m============ 8 passed in 0.18 seconds > ============ [0m > [1m============== test session starts > =============== [0m > platform linux2 -- Python 2.7.6 -- py-1.4.26 -- > pytest-2.6.4 -- < > https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 > > > plugins: cov, timeout > [1m > collecting 0 items [0m [1m > collecting 2 items [0m [1m > collected 2 items > [0m > > src/test/python/apache/aurora/executor/test_executor_vars.py::test_init > [32mPASSED [0m > > src/test/python/apache/aurora/executor/test_executor_vars.py::test_sample > [32mPASSED [0m > > generated xml file: < > https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.executor_vars.xml > > > [32m [1m============ 2 passed in 0.11 seconds > ============ [0m > [1m============== test session starts > =============== [0m > platform linux2 -- Python 2.7.6 -- py-1.4.26 -- > pytest-2.6.4 -- < > https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 > > > plugins: cov, timeout > [1m > collecting 0 items [0m [1m > collecting 1 items [0m [1m > collecting 1 items [0m [1m > collecting 1 items [0m [1m > collected 1 items > [0m > > > src/test/python/apache/aurora/executor/test_status_manager.py::TestStatusManager::test_run > [32mPASSED [0m > > generated xml file: < > https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.status_manager.xml > > > [32m [1m============ 1 passed in 0.18 seconds > ============ [0m > [1m============== test session starts > =============== [0m > platform linux2 -- Python 2.7.6 -- py-1.4.26 -- > pytest-2.6.4 -- < > https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 > > > plugins: cov, timeout > [1m > collecting 0 items [0m [1m > collecting 8 items [0m [1m > collecting 8 items [0m [1m > collecting 8 items [0m [1m > collected 8 items > [0m > > > src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_success > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_failed > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_stop > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_lose > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_quitquitquit > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_thermos_normal_exit_status > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_thermos_abnormal_exit_statuses > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_thermos_runner_killed > [32mPASSED [0m > > generated xml file: < > https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.thermos_task_runner.xml > > > [32m [1m=========== 8 passed in 76.21 seconds > ============ [0m > [1m============== test session starts > =============== [0m > platform linux2 -- Python 2.7.6 -- py-1.4.26 -- > pytest-2.6.4 -- < > https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 > > > plugins: cov, timeout > [1m > collecting 0 items [0m [1m > collecting 8 items [0m [1m > collected 8 items > [0m > > > src/test/python/apache/aurora/executor/common/test_announcer.py::test_serverset_join_thread > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_announcer.py::test_announcer_under_normal_circumstances > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_announcer.py::test_announcer_on_expiration > [33mSKIPPED [0m > > > src/test/python/apache/aurora/executor/common/test_announcer.py::test_announcer_under_abnormal_circumstances > [33mSKIPPED [0m > > > src/test/python/apache/aurora/executor/common/test_announcer.py::test_make_empty_endpoints > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_announcer.py::test_announcer_provider_with_timeout > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_announcer.py::test_default_announcer_provider > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_announcer.py::test_default_announcer_provider_without_announce > [32mPASSED [0m > > generated xml file: < > https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.announcer.xml > > > [32m [1m====== 6 passed, 2 skipped in 0.67 seconds > ======= [0m > [1m============== test session starts > =============== [0m > platform linux2 -- Python 2.7.6 -- py-1.4.26 -- > pytest-2.6.4 -- < > https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 > > > plugins: cov, timeout > [1m > collecting 0 items [0m [1m > collecting 5 items [0m [1m > collected 5 items > [0m > > > src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_directory_sandbox > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_create > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_user_does_not_exist > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_create_ioerror > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_directory_sandbox.py::test_destroy_ioerror > [32mPASSED [0m > > generated xml file: < > https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.directory_sandbox.xml > > > [32m [1m============ 5 passed in 0.07 seconds > ============ [0m > [1m============== test session starts > =============== [0m > platform linux2 -- Python 2.7.6 -- py-1.4.26 -- > pytest-2.6.4 -- < > https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 > > > plugins: cov, timeout > [1m > collecting 0 items [0m [1m > collecting 4 items [0m [1m > collected 4 items > [0m > > > src/test/python/apache/aurora/executor/common/test_executor_detector.py::test_find_root > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_executor_detector.py::test_match_inverse > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_executor_detector.py::test_bad_match > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_executor_detector.py::test_integration > [32mPASSED [0m > > generated xml file: < > https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.executor_detector.xml > > > [32m [1m============ 4 passed in 0.04 seconds > ============ [0m > [1m============== test session starts > =============== [0m > platform linux2 -- Python 2.7.6 -- py-1.4.26 -- > pytest-2.6.4 -- < > https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 > > > plugins: cov, timeout > [1m > collecting 0 items [0m [1m > collecting 2 items [0m [1m > collecting 2 items [0m [1m > collecting 2 items [0m [1m > collected 2 items > [0m > > > src/test/python/apache/aurora/executor/common/test_executor_timeout.py::TestExecutorTimeout::test_run > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_executor_timeout.py::TestExecutorTimeout::test_run_timeout > [32mPASSED [0m > > generated xml file: < > https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.executor_timeout.xml > > > [32m [1m============ 2 passed in 0.07 seconds > ============ [0m > [1m============== test session starts > =============== [0m > platform linux2 -- Python 2.7.6 -- py-1.4.26 -- > pytest-2.6.4 -- < > https://builds.apache.org/job/Aurora/ws/build-support/pants.venv/bin/python2.7 > > > plugins: cov, timeout > [1m > collecting 0 items [0m [1m > collecting 4 items [0m [1m > collecting 5 items [0m [1m > collecting 12 items [0m [1m > collecting 12 items [0m [1m > collected 12 items > [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthChecker::test_consecutive_failures > [31mFAILED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthChecker::test_health_checker_metrics > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthChecker::test_initial_interval_2x > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthChecker::test_initial_interval_whatev > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestHealthCheckerProvider::test_from_assigned_task > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_maybe_update_failure_count > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_perform_check_if_not_disabled_no_snooze_file > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_perform_check_if_not_disabled_snooze_file_exists > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_perform_check_if_not_disabled_snooze_file_is_none > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_run > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_start > [32mPASSED [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py::TestThreadedHealthChecker::test_stop > [32mPASSED [0m > > ==================== FAILURES ==================== > __ TestHealthChecker.test_consecutive_failures ___ > > self = <common.test_health_checker.TestHealthChecker > testMethod=test_consecutive_failures> > > [1m def test_consecutive_failures(self): [0m > [1m '''Verify that a task is unhealthy only > after max_consecutive_failures is exceeded''' [0m > [1m initial_interval_secs = 2 [0m > [1m interval_secs = 1 [0m > [1m self.append_health_checks(False, > num_calls=2) [0m > [1m self.append_health_checks(True) [0m > [1m self.append_health_checks(False, > num_calls=3) [0m > [1m hct = HealthChecker( [0m > [1m self._checker.health, [0m > [1m interval_secs=interval_secs, [0m > [1m > initial_interval_secs=initial_interval_secs, [0m > [1m max_consecutive_failures=2, [0m > [1m clock=self._clock) [0m > [1m hct.start() [0m > [1m [0m > [1m # 2 consecutive health check failures > followed by a successful health check. [0m > [1m self._clock.tick(initial_interval_secs) [0m > [1m assert > self._clock.converge(threads=[hct.threaded_health_checker], timeout=1) [0m > [1m assert hct.status is None [0m > [1m> assert > hct.metrics.sample()['consecutive_failures'] == 1 [0m > [1m [31mE AssertionError: assert 0 == 1 [0m > > > src/test/python/apache/aurora/executor/common/test_health_checker.py:108: > AssertionError > -------------- Captured stderr call -------------- > [<twitter.common.testing.clock.ThreadedClock object > at 0x7fc87bd87f10>] Time now: 0.0 > [<twitter.common.testing.clock.ThreadedClock object > at 0x7fc87bd87f10>] Time now: 2.0 > generated xml file: < > https://builds.apache.org/job/Aurora/ws/dist/test-results/src.test.python.apache.aurora.executor.common.health_checker.xml > > > [1m [31m====== 1 failed, 11 passed in 0.77 seconds > ======= [0m > src.test.python.apache.aurora.admin.admin > ..... SUCCESS > src.test.python.apache.aurora.admin.host_maintenance > ..... SUCCESS > src.test.python.apache.aurora.admin.maintenance > ..... SUCCESS > src.test.python.apache.aurora.client.api.api > ..... SUCCESS > > src.test.python.apache.aurora.client.api.instance_watcher > ..... SUCCESS > src.test.python.apache.aurora.client.api.job_monitor > ..... SUCCESS > src.test.python.apache.aurora.client.api.mux > ..... SUCCESS > src.test.python.apache.aurora.client.api.quota_check > ..... SUCCESS > src.test.python.apache.aurora.client.api.restarter > ..... SUCCESS > > src.test.python.apache.aurora.client.api.scheduler_client > ..... SUCCESS > src.test.python.apache.aurora.client.api.sla > ..... SUCCESS > src.test.python.apache.aurora.client.api.task_util > ..... SUCCESS > src.test.python.apache.aurora.client.api.updater > ..... SUCCESS > > src.test.python.apache.aurora.client.api.updater_util > ..... SUCCESS > src.test.python.apache.aurora.client.base > ..... SUCCESS > src.test.python.apache.aurora.client.binding_helper > ..... SUCCESS > src.test.python.apache.aurora.client.cli.api > ..... SUCCESS > src.test.python.apache.aurora.client.cli.client > ..... SUCCESS > > src.test.python.apache.aurora.client.cli.command_hooks > ..... SUCCESS > src.test.python.apache.aurora.client.cli.config > ..... SUCCESS > src.test.python.apache.aurora.client.cli.context > ..... SUCCESS > src.test.python.apache.aurora.client.cli.cron > ..... SUCCESS > src.test.python.apache.aurora.client.cli.inspect > ..... SUCCESS > src.test.python.apache.aurora.client.cli.job > ..... SUCCESS > src.test.python.apache.aurora.client.cli.plugins > ..... SUCCESS > src.test.python.apache.aurora.client.cli.quota > ..... SUCCESS > src.test.python.apache.aurora.client.cli.sla > ..... SUCCESS > src.test.python.apache.aurora.client.cli.supdate > ..... SUCCESS > src.test.python.apache.aurora.client.cli.task > ..... SUCCESS > src.test.python.apache.aurora.client.cli.update > ..... SUCCESS > src.test.python.apache.aurora.client.cli.version > ..... SUCCESS > src.test.python.apache.aurora.client.config > ..... SUCCESS > src.test.python.apache.aurora.client.factory > ..... SUCCESS > > src.test.python.apache.aurora.client.hooks.hooked_api > ..... SUCCESS > > src.test.python.apache.aurora.client.hooks.non_hooked_api > ..... SUCCESS > > src.test.python.apache.aurora.common.test_aurora_job_key > ..... SUCCESS > src.test.python.apache.aurora.common.test_cluster > ..... SUCCESS > > src.test.python.apache.aurora.common.test_cluster_option > ..... SUCCESS > src.test.python.apache.aurora.common.test_clusters > ..... SUCCESS > > src.test.python.apache.aurora.common.test_http_signaler > ..... SUCCESS > > src.test.python.apache.aurora.common.test_pex_version > ..... SUCCESS > src.test.python.apache.aurora.common.test_shellify > ..... SUCCESS > src.test.python.apache.aurora.common.test_transport > ..... SUCCESS > src.test.python.apache.aurora.config.test_base > ..... SUCCESS > > src.test.python.apache.aurora.config.test_constraint_parsing > ..... SUCCESS > src.test.python.apache.aurora.config.test_loader > ..... SUCCESS > src.test.python.apache.aurora.config.test_thrift > ..... SUCCESS > > src.test.python.apache.aurora.executor.common.announcer > ..... SUCCESS > > src.test.python.apache.aurora.executor.common.directory_sandbox > ..... SUCCESS > > src.test.python.apache.aurora.executor.common.executor_detector > ..... SUCCESS > > src.test.python.apache.aurora.executor.common.executor_timeout > ..... SUCCESS > > src.test.python.apache.aurora.executor.common.health_checker > ..... FAILURE > > src.test.python.apache.aurora.executor.common.path_detector > ..... SUCCESS > > src.test.python.apache.aurora.executor.common.task_info > ..... SUCCESS > src.test.python.apache.aurora.executor.executor_base > ..... SUCCESS > src.test.python.apache.aurora.executor.executor_vars > ..... SUCCESS > > src.test.python.apache.aurora.executor.status_manager > ..... SUCCESS > > src.test.python.apache.aurora.executor.thermos_task_runner > ..... SUCCESS > src.test.python.apache.thermos.common.test_pathspec > ..... SUCCESS > > src.test.python.apache.thermos.core.test_runner_integration > ..... SUCCESS > src.test.python.apache.thermos.monitoring.test_disk > ..... SUCCESS > > FAILURE > > [31m > FAILURE [0m > Build step 'Execute shell' marked build as failure > Recording test results > -- Zameer Manji
