Yingbo,
Since your job was defined as service but was exiting right away, it was
penalized by the scheduler for its a "flapping" behavior. The penalty is set by
the exponential back-off with a max cut off at 5 minutes. The "No matching
hosts" displayed in the UI is a bit misleading here and will be improved soon
when the Throttled task state is implemented.
Thanks,
Maxim
On Dec 20, 2013, at 12:01 PM, Yingbo Wang <ybw...@gmail.com> wrote:
> It is a good catch! Thanks Brian.
>
> The task indeed restarts every 5 minutes. I guess the job just doesn't
> can't find the running service that's why it says "Not matching hosts"?
> I will try it to see if it works.
>
>
>
>
> On Fri, Dec 20, 2013 at 9:40 AM, Brian Wickman
> <wick...@twopensource.com>wrote:
>
>> There is an executor_id in the state there, so it looks like it is managing
>> to launch the tasks. Since you've listed your job as a Service(), it means
>> that every task will be rescheduled after it exits, and since it looks like
>> your task just immediately returns, it's probably launching dozens of them
>> per minute if not more. You can change this behavior by changing Service
>> to Job. (Service is just an alias for Job(service=True).)
>>
>> ~brian
>>
>>
>>
>> On Fri, Dec 20, 2013 at 9:20 AM, Yingbo Wang <ybw...@gmail.com> wrote:
>>
>>> Thanks Maxim.
>>>
>>> did you mean hello.aurora?
>>>
>>>
>>> hello = Process(
>>> name = 'hello',
>>> cmdline = """
>>> echo "ls file " ;
>>> whoami;
>>> hostname;
>>> """)
>>>
>>> task = SequentialTask(
>>> processes = [hello],
>>> resources = Resources(cpu = 1.0, ram = 128*MB, disk = 128*MB))
>>>
>>> jobs = [Service(
>>> task = task, cluster = 'lychee', role = 'lychee', environment = 'prod',
>>> name = 'hello')]
>>>
>>> The slave machine has enough resource to satisfy this request.
>>>
>>> I did encounter an issue saying "Constraint Not Satisfied, host". After I
>>> change my hostname of both master and slave machines, this problem seems
>> to
>>> be solved, and now the problem becomes "No matching hosts"
>>>
>>>
>>>
>>> Here's the offers that scheduler received:
>>>
>>> [
>>>
>>> -
>>> {
>>> - id: "201312200632-302033088-5050-15916-396",
>>> - framework_id: "201312110654-302033088-5050-31688-0000",
>>> - slave_id: "201312170824-302033088-5050-25318-0",
>>> - hostname: "192.168.0.11",
>>> - resources:
>>> [
>>> -
>>> {
>>> - name: "cpus",
>>> - scalar: 7.75
>>> },
>>> -
>>> {
>>> - name: "disk",
>>> - scalar: 917136
>>> },
>>> -
>>> {
>>> - name: "mem",
>>> - scalar: 14814
>>> },
>>> -
>>> {
>>> - name: "ports",
>>> - ranges:
>>> [
>>> - "31000-32000"
>>> ]
>>> }
>>> ],
>>> - attributes:
>>> [
>>> -
>>> {
>>> - name: "host",
>>> - text: "192.168.0.11"
>>> },
>>> -
>>> {
>>> - name: "rack",
>>> - text: "a"
>>> }
>>> ],
>>> - executor_ids:
>>> [
>>> -
>>>
>>>
>> "thermos-1387559509785-lychee-prod-hello-0-ea46f1c4-4619-49c1-94b5-7d09e1238b7d"
>>> ]
>>> }
>>>
>>> ]
>>>
>>>
>>> and the /vars are:
>>>
>>>
>>> async_tasks_completed 393
>>> both_instance_ids_set 0
>>> build_date null
>>> build_git_branchname null
>>> build_git_revision null
>>> build_git_revision_number null
>>> build_git_tag null
>>> build_machine null
>>> build_path null
>>> build_time null
>>> build_timestamp null
>>> build_user_name null
>>> cron_job_launch_failures 0
>>> cron_jobs_triggered 0
>>> cron_num_pending_runs 0
>>> empty_slots_large 0
>>> empty_slots_medium 1
>>> empty_slots_small 7
>>> empty_slots_xlarge 0
>>> framework_registered 1
>>> http_200_responses_events 22
>>> http_200_responses_events_per_sec 0.0
>>> http_200_responses_nanos_per_event 0.0
>>> http_200_responses_nanos_total 981542679
>>> http_200_responses_nanos_total_per_sec 0.0
>>> instance_ids_inconsistent 0
>>> jvm_available_processors 4
>>> jvm_class_loaded_count 6658
>>> jvm_class_total_loaded_count 6658
>>> jvm_class_unloaded_count 0
>>> jvm_gc_PS_MarkSweep_collection_count 0
>>> jvm_gc_PS_MarkSweep_collection_time_ms 0
>>> jvm_gc_PS_Scavenge_collection_count 7
>>> jvm_gc_PS_Scavenge_collection_time_ms 534
>>> jvm_gc_collection_count 7
>>> jvm_gc_collection_time_ms 534
>>> jvm_input_arguments [-Xmx1g, -Xms1g, -Djava.library.path=/usr/local/lib]
>>> jvm_memory_free_mb 688
>>> jvm_memory_heap_mb_committed 968
>>> jvm_memory_heap_mb_max 968
>>> jvm_memory_heap_mb_used 279
>>> jvm_memory_max_mb 968
>>> jvm_memory_mb_total 968
>>> jvm_memory_non_heap_mb_committed 39
>>> jvm_memory_non_heap_mb_max 130
>>> jvm_memory_non_heap_mb_used 38
>>> jvm_prop_awt_toolkit sun.awt.X11.XToolkit
>>> jvm_prop_file_encoding UTF-8
>>> jvm_prop_file_encoding_pkg sun.io
>>> jvm_prop_file_separator /
>>> jvm_prop_java_awt_graphicsenv sun.awt.X11GraphicsEnvironment
>>> jvm_prop_java_awt_printerjob sun.print.PSPrinterJob
>>> jvm_prop_java_class_path
>>>
>>>
>> /usr/local/aurora-scheduler/lib/incubator-aurora.jar:/usr/local/aurora-scheduler/lib/aopalliance-1.0.jar:/usr/local/aurora-scheduler/lib/jsr305-1.3.9.jar:/usr/local/aurora-scheduler/lib/guava-14.0.1.jar:/usr/local/aurora-scheduler/lib/guice-3.0.jar:/usr/local/aurora-scheduler/lib/protobuf-java-2.4.1.jar:/usr/local/aurora-scheduler/lib/jersey-core-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-json-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-server-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-servlet-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-guice-1.12.jar:/usr/local/aurora-scheduler/lib/commons-lang-2.5.jar:/usr/local/aurora-scheduler/lib/javax.inject-1.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5.jar:/usr/local/aurora-scheduler/lib/log4j-1.2.17.jar:/usr/local/aurora-scheduler/lib/stringtemplate-3.2.1.jar:/usr/local/aurora-scheduler/lib/mesos-0.15.0-rc3.jar:/usr/local/aurora-scheduler/lib/libthrift-0.9.1.jar:/usr/local/aurora-scheduler/lib/zookeeper-3.3.4.jar:/usr/local/aurora-scheduler/lib/slf4j-api-1.6.1.jar:/usr/local/aurora-scheduler/lib/slf4j-jdk14-1.6.1.jar:/usr/local/aurora-scheduler/lib/log4j-0.0.3.jar:/usr/local/aurora-scheduler/lib/bootstrap-0.0.1.jar:/usr/local/aurora-scheduler/lib/jquery-0.0.1.jar:/usr/local/aurora-scheduler/lib/client-flagged-0.0.2.jar:/usr/local/aurora-scheduler/lib/client-0.0.2.jar:/usr/local/aurora-scheduler/lib/candidate-0.0.48.jar:/usr/local/aurora-scheduler/lib/client-0.0.40.jar:/usr/local/aurora-scheduler/lib/group-0.0.51.jar:/usr/local/aurora-scheduler/lib/server-set-1.0.52.jar:/usr/local/aurora-scheduler/lib/singleton-service-0.0.65.jar:/usr/local/aurora-scheduler/lib/application-http-0.0.49.jar:/usr/local/aurora-scheduler/lib/application-module-http-0.0.46.jar:/usr/local/aurora-scheduler/lib/application-module-lifecycle-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-module-log-0.0.47.jar:/usr/local/aurora-scheduler/lib/application-module-stats-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-0.0.65.jar:/usr/local/aurora-scheduler/lib/args-0.1.71.jar:/usr/local/aurora-scheduler/lib/base-0.0.72.jar:/usr/local/aurora-scheduler/lib/collections-0.0.59.jar:/usr/local/aurora-scheduler/lib/dynamic-host-set-0.0.35.jar:/usr/local/aurora-scheduler/lib/inject-timed-0.0.8.jar:/usr/local/aurora-scheduler/lib/inject-0.0.29.jar:/usr/local/aurora-scheduler/lib/logging-0.0.49.jar:/usr/local/aurora-scheduler/lib/net-http-filters-0.0.3.jar:/usr/local/aurora-scheduler/lib/net-util-0.0.65.jar:/usr/local/aurora-scheduler/lib/quantity-0.0.56.jar:/usr/local/aurora-scheduler/lib/stats-0.0.81.jar:/usr/local/aurora-scheduler/lib/testing-easymock-0.0.1.jar:/usr/local/aurora-scheduler/lib/thrift-0.0.61.jar:/usr/local/aurora-scheduler/lib/util-system-mocks-0.0.55.jar:/usr/local/aurora-scheduler/lib/util-templating-0.0.17.jar:/usr/local/aurora-scheduler/lib/util-testing-0.0.6.jar:/usr/local/aurora-scheduler/lib/util-0.0.83.jar:/usr/local/aurora-scheduler/lib/zookeeper-testing-0.0.37.jar:/usr/local/aurora-scheduler/lib/cglib-2.2.1-v20090111.jar:/usr/local/aurora-scheduler/lib/jettison-1.1.jar:/usr/local/aurora-scheduler/lib/jaxb-impl-2.2.3-1.jar:/usr/local/aurora-scheduler/lib/jackson-core-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-jaxrs-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-xc-1.9.2.jar:/usr/local/aurora-scheduler/lib/asm-3.1.jar:/usr/local/aurora-scheduler/lib/guice-servlet-3.0.jar:/usr/local/aurora-scheduler/lib/antlr-2.7.7.jar:/usr/local/aurora-scheduler/lib/commons-lang3-3.1.jar:/usr/local/aurora-scheduler/lib/httpclient-4.2.5.jar:/usr/local/aurora-scheduler/lib/httpcore-4.2.4.jar:/usr/local/aurora-scheduler/lib/jline-0.9.94.jar:/usr/local/aurora-scheduler/lib/lock-0.0.25.jar:/usr/local/aurora-scheduler/lib/map-0.0.34.jar:/usr/local/aurora-scheduler/lib/node-0.0.34.jar:/usr/local/aurora-scheduler/lib/partitioner-0.0.34.jar:/usr/local/aurora-scheduler/lib/application-action-0.0.60.jar:/usr/local/aurora-scheduler/lib/net-http-0.0.37.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-0.0.61.jar:/usr/local/aurora-scheduler/lib/guice-multibindings-3.0.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-time-series-0.0.42.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-pprof-0.0.18.jar:/usr/local/aurora-scheduler/lib/stats-time-series-0.0.47.jar:/usr/local/aurora-scheduler/lib/stats-jvm-0.0.41.jar:/usr/local/aurora-scheduler/lib/commons-io-2.1.jar:/usr/local/aurora-scheduler/lib/commons-codec-1.6.jar:/usr/local/aurora-scheduler/lib/easymock-3.1.jar:/usr/local/aurora-scheduler/lib/junit-dep-4.10.jar:/usr/local/aurora-scheduler/lib/hamcrest-core-1.2.jar:/usr/local/aurora-scheduler/lib/test-libraries-for-java-1.1.1.jar:/usr/local/aurora-scheduler/lib/net-pool-0.0.53.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0.1.jar:/usr/local/aurora-scheduler/lib/jaxb-api-2.2.2.jar:/usr/local/aurora-scheduler/lib/junit-3.8.1.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-text-0.0.27.jar:/usr/local/aurora-scheduler/lib/jetty-6.1.25.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-params-0.0.10.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-string-template-0.0.42.jar:/usr/local/aurora-scheduler/lib/gson-2.2.2.jar:/usr/local/aurora-scheduler/lib/util-core-6.3.8.jar:/usr/local/aurora-scheduler/lib/util-jvm-6.3.8.jar:/usr/local/aurora-scheduler/lib/cglib-nodep-2.2.2.jar:/usr/local/aurora-scheduler/lib/objenesis-1.2.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0-2.jar:/usr/local/aurora-scheduler/lib/activation-1.1.jar:/usr/local/aurora-scheduler/lib/jetty-util-6.1.25.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5-20081211.jar:/usr/local/aurora-scheduler/lib/scala-library-2.9.2.jar:/usr/local/aurora-scheduler/lib/joda-time-2.1.jar:/usr/local/aurora-scheduler/lib/joda-convert-1.2.jar:/usr/local/aurora-scheduler/lib/args-apt-0.0.45.jar:/usr/local/aurora-scheduler/lib/args-core-0.0.31.jar:/usr/local/aurora-scheduler/lib/io-json-0.0.34.jar:/usr/local/aurora-scheduler/lib/io-thrift-0.0.42.jar:/usr/local/aurora-scheduler/lib/stat-registry-0.0.21.jar:/usr/local/aurora-scheduler/lib/stat-0.0.23.jar:/usr/local/aurora-scheduler/lib/stats-provider-0.0.48.jar:/usr/local/aurora-scheduler/lib/util-sampler-0.0.45.jar:/usr/local/aurora-scheduler/lib/commons-logging-1.1.1.jar:/usr/local/aurora-scheduler/lib/io-0.0.44.jar:/usr/local/aurora-scheduler/lib/service-thrift-1.0.25.jar:/usr/local/aurora-scheduler/lib/application-module-applauncher-0.0.44.jar:/usr/local/aurora-scheduler/lib/util-executor-service-shutdown-0.0.41.jar:/usr/local/aurora-scheduler/lib/jdk-logging-0.0.36.jar
>>> jvm_prop_java_class_version 51.0
>>> jvm_prop_java_endorsed_dirs /usr/lib/jvm/java-7-oracle/jre/lib/endorsed
>>> jvm_prop_java_ext_dirs
>>> /usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext
>>> jvm_prop_java_home /usr/lib/jvm/java-7-oracle/jre
>>> jvm_prop_java_io_tmpdir /tmp
>>> jvm_prop_java_library_path /usr/local/lib
>>> jvm_prop_java_runtime_name Java(TM) SE Runtime Environment
>>> jvm_prop_java_runtime_version 1.7.0_45-b18
>>> jvm_prop_java_specification_name Java Platform API Specification
>>> jvm_prop_java_specification_vendor Oracle Corporation
>>> jvm_prop_java_specification_version 1.7
>>> jvm_prop_java_vendor Oracle Corporation
>>> jvm_prop_java_vendor_url http://java.oracle.com/
>>> jvm_prop_java_vendor_url_bug http://bugreport.sun.com/bugreport/
>>> jvm_prop_java_version 1.7.0_45
>>> jvm_prop_java_vm_info mixed mode
>>> jvm_prop_java_vm_name Java HotSpot(TM) 64-Bit Server VM
>>> jvm_prop_java_vm_specification_name Java Virtual Machine Specification
>>> jvm_prop_java_vm_specification_vendor Oracle Corporation
>>> jvm_prop_java_vm_specification_version 1.7
>>> jvm_prop_java_vm_vendor Oracle Corporation
>>> jvm_prop_java_vm_version 24.45-b08
>>> jvm_prop_line_separator
>>>
>>> jvm_prop_os_arch amd64
>>> jvm_prop_os_name Linux
>>> jvm_prop_os_version 3.5.0-44-generic
>>> jvm_prop_path_separator :
>>> jvm_prop_sun_arch_data_model 64
>>> jvm_prop_sun_boot_class_path
>>>
>>>
>> /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes
>>> jvm_prop_sun_boot_library_path /usr/lib/jvm/java-7-oracle/jre/lib/amd64
>>> jvm_prop_sun_cpu_endian little
>>> jvm_prop_sun_cpu_isalist
>>> jvm_prop_sun_io_unicode_encoding UnicodeLittle
>>> jvm_prop_sun_java_command
>>> com.twitter.aurora.scheduler.app.SchedulerMain -cluster_name=lychee
>>> -http_port=8081 -thrift_port=8082 -native_log_quorum_size=1
>>> -zk_endpoints=192.168.0.13:2181
>>> -mesos_master_address=zk://192.168.0.13:2181/mesos/master
>>> -serverset_path=/aurora/scheduler
>>> -native_log_zk_group_path=/aurora/replicated-log
>>> -native_log_file_path=/usr/local/aurora-scheduler/db
>>> -backup_dir=/usr/local/aurora-scheduler/backups
>>> -thermos_executor_path=/usr/local/bin/thermos_executor
>>> -gc_executor_path=/usr/local/bin/gc_executor -vlog=INFO -logtostderr
>>> jvm_prop_sun_java_launcher SUN_STANDARD
>>> jvm_prop_sun_jnu_encoding UTF-8
>>> jvm_prop_sun_management_compiler HotSpot 64-Bit Tiered Compilers
>>> jvm_prop_sun_os_patch_level unknown
>>> jvm_prop_user_country US
>>> jvm_prop_user_dir /home/lychee
>>> jvm_prop_user_home /root
>>> jvm_prop_user_language en
>>> jvm_prop_user_name root
>>> jvm_prop_user_timezone Etc/UTC
>>> jvm_threads_active 21
>>> jvm_threads_daemon 12
>>> jvm_threads_peak 25
>>> jvm_threads_started 867
>>> jvm_time_ms 1387559947214
>>> jvm_uptime_secs 38781
>>> log_compressed_entry_bytes_saved 2537635
>>> logging_dir /root
>>> mem_storage_consistent_read_operation_events 619
>>> mem_storage_consistent_read_operation_events_per_sec 0.0
>>> mem_storage_consistent_read_operation_nanos_per_event 0.0
>>> mem_storage_consistent_read_operation_nanos_total 272565131
>>> mem_storage_consistent_read_operation_nanos_total_per_sec 0.0
>>> mem_storage_delete_all_tasks_events 1
>>> mem_storage_delete_all_tasks_events_per_sec 0.0
>>> mem_storage_delete_all_tasks_nanos_per_event 0.0
>>> mem_storage_delete_all_tasks_nanos_total 2323293
>>> mem_storage_delete_all_tasks_nanos_total_per_sec 0.0
>>> mem_storage_delete_tasks_events 179
>>> mem_storage_delete_tasks_events_per_sec 0.0
>>> mem_storage_delete_tasks_nanos_per_event 0.0
>>> mem_storage_delete_tasks_nanos_total 8638495
>>> mem_storage_delete_tasks_nanos_total_per_sec 0.0
>>> mem_storage_fetch_tasks_events 3219
>>> mem_storage_fetch_tasks_events_per_sec 0.0
>>> mem_storage_fetch_tasks_nanos_per_event 0.0
>>> mem_storage_fetch_tasks_nanos_total 193613049
>>> mem_storage_fetch_tasks_nanos_total_per_sec 0.0
>>> mem_storage_mutate_tasks_events 677
>>> mem_storage_mutate_tasks_events_per_sec 0.0
>>> mem_storage_mutate_tasks_nanos_per_event 0.0
>>> mem_storage_mutate_tasks_nanos_total 149943165
>>> mem_storage_mutate_tasks_nanos_total_per_sec 0.0
>>> mem_storage_save_tasks_events 306
>>> mem_storage_save_tasks_events_per_sec 0.0
>>> mem_storage_save_tasks_nanos_per_event 0.0
>>> mem_storage_save_tasks_nanos_total 165988968
>>> mem_storage_save_tasks_nanos_total_per_sec 0.0
>>> mem_storage_weakly_consistent_read_operation_events 1538
>>> mem_storage_weakly_consistent_read_operation_events_per_sec 0.0
>>> mem_storage_weakly_consistent_read_operation_nanos_per_event 0.0
>>> mem_storage_weakly_consistent_read_operation_nanos_total 138584844
>>> mem_storage_weakly_consistent_read_operation_nanos_total_per_sec 0.0
>>> mem_storage_write_operation_events 2932
>>> mem_storage_write_operation_events_per_sec 0.0
>>> mem_storage_write_operation_nanos_per_event 0.0
>>> mem_storage_write_operation_nanos_total 28077154879
>>> mem_storage_write_operation_nanos_total_per_sec 0.0
>>> new_instance_id_set 0
>>> offer_accept_races 0
>>> old_instance_id_set 0
>>> outstanding_offers 1
>>> preemptor_attempts 0
>>> preemptor_failed_preemptions 0
>>> preemptor_no_slots_found 0
>>> preemptor_tasks_preempted 0
>>> process_cpu_cores_utilized 0.0
>>> process_cpu_time_nanos 73230000000
>>> process_max_fd_count 4096
>>> process_open_fd_count 157
>>> read_lock_wait_nanos 545041
>>> resources_allocated_quota_cpu 0
>>> resources_allocated_quota_disk_gb 0
>>> resources_allocated_quota_ram_gb 0
>>> resources_dedicated_consumed_cpu 0
>>> resources_dedicated_consumed_disk_gb 0
>>> resources_dedicated_consumed_ram_gb 0
>>> resources_free_pool_consumed_cpu 1
>>> resources_free_pool_consumed_disk_gb 0
>>> resources_free_pool_consumed_ram_gb 0
>>> resources_quota_consumed_cpu 0
>>> resources_quota_consumed_disk_gb 0
>>> resources_quota_consumed_ram_gb 0
>>> resources_total_consumed_cpu 1
>>> resources_total_consumed_disk_gb 0
>>> resources_total_consumed_ram_gb 0
>>> schedule_attempts_failed 0
>>> schedule_attempts_fired 135
>>> schedule_queue_size 1
>>> scheduler_backup_failed 0
>>> scheduler_backup_success 10
>>> scheduler_driver_kill_failures 0
>>> scheduler_failed_offers 0
>>> scheduler_framework_disconnects 0
>>> scheduler_framework_reregisters 0
>>> scheduler_illegal_task_state_transitions 200
>>> scheduler_log_bad_frames_read 0
>>> scheduler_log_bytes_read 292813
>>> scheduler_log_bytes_written 1080061
>>> scheduler_log_deflated_entries_read 1
>>> scheduler_log_entries_read 175
>>> scheduler_log_entries_written 555
>>> scheduler_log_native_append_events 556
>>> scheduler_log_native_append_events_per_sec 0.0
>>> scheduler_log_native_append_failures 0
>>> scheduler_log_native_append_nanos_per_event 0.0
>>> scheduler_log_native_append_nanos_total 25082628348
>>> scheduler_log_native_append_nanos_total_per_sec 0.0
>>> scheduler_log_native_append_timeouts 0
>>> scheduler_log_native_native_entries_skipped 1
>>> scheduler_log_native_read_events 176
>>> scheduler_log_native_read_events_per_sec 0.0
>>> scheduler_log_native_read_failures 0
>>> scheduler_log_native_read_nanos_per_event 0.0
>>> scheduler_log_native_read_nanos_total 73255842
>>> scheduler_log_native_read_nanos_total_per_sec 0.0
>>> scheduler_log_native_read_timeouts 0
>>> scheduler_log_native_truncate_events 10
>>> scheduler_log_native_truncate_events_per_sec 0.0
>>> scheduler_log_native_truncate_failures 0
>>> scheduler_log_native_truncate_nanos_per_event 0.0
>>> scheduler_log_native_truncate_nanos_total 276600750
>>> scheduler_log_native_truncate_nanos_total_per_sec 0.0
>>> scheduler_log_native_truncate_timeouts 0
>>> scheduler_log_recover_events 1
>>> scheduler_log_recover_events_per_sec 0.0
>>> scheduler_log_recover_nanos_per_event 0.0
>>> scheduler_log_recover_nanos_total 565333612
>>> scheduler_log_recover_nanos_total_per_sec 0.0
>>> scheduler_log_save_framework_id_events 2
>>> scheduler_log_save_framework_id_events_per_sec 0.0
>>> scheduler_log_save_framework_id_nanos_per_event 0.0
>>> scheduler_log_save_framework_id_nanos_total 437292
>>> scheduler_log_save_framework_id_nanos_total_per_sec 0.0
>>> scheduler_log_snapshot_events 10
>>> scheduler_log_snapshot_events_per_sec 0.0
>>> scheduler_log_snapshot_nanos_per_event 0.0
>>> scheduler_log_snapshot_nanos_total 961465418
>>> scheduler_log_snapshot_nanos_total_per_sec 0.0
>>> scheduler_log_snapshot_persist_events 10
>>> scheduler_log_snapshot_persist_events_per_sec 0.0
>>> scheduler_log_snapshot_persist_nanos_per_event 0.0
>>> scheduler_log_snapshot_persist_nanos_total 840444765
>>> scheduler_log_snapshot_persist_nanos_total_per_sec 0.0
>>> scheduler_log_snapshots 10
>>> scheduler_log_tasks_mutate_events 677
>>> scheduler_log_tasks_mutate_events_per_sec 0.0
>>> scheduler_log_tasks_mutate_nanos_per_event 0.0
>>> scheduler_log_tasks_mutate_nanos_total 182346727
>>> scheduler_log_tasks_mutate_nanos_total_per_sec 0.0
>>> scheduler_log_tasks_remove_events 179
>>> scheduler_log_tasks_remove_events_per_sec 0.0
>>> scheduler_log_tasks_remove_nanos_per_event 0.0
>>> scheduler_log_tasks_remove_nanos_total 13588163
>>> scheduler_log_tasks_remove_nanos_total_per_sec 0.0
>>> scheduler_log_tasks_save_events 305
>>> scheduler_log_tasks_save_events_per_sec 0.0
>>> scheduler_log_tasks_save_nanos_per_event 0.0
>>> scheduler_log_tasks_save_nanos_total 114479852
>>> scheduler_log_tasks_save_nanos_total_per_sec 0.0
>>> scheduler_log_un_snapshotted_transactions 36
>>> scheduler_lost_executors 0
>>> scheduler_max_ASSIGNED_waiting_ms 0
>>> scheduler_max_KILLING_waiting_ms 0
>>> scheduler_max_PREEMPTING_waiting_ms 0
>>> scheduler_max_RESTARTING_waiting_ms 0
>>> scheduler_resource_offers 397
>>> scheduler_resource_offers_events 397
>>> scheduler_resource_offers_events_per_sec 0.0
>>> scheduler_resource_offers_nanos_per_event 0.0
>>> scheduler_resource_offers_nanos_total 305282357
>>> scheduler_resource_offers_nanos_total_per_sec 0.0
>>> scheduler_save_host_attribute_events 397
>>> scheduler_save_host_attribute_events_per_sec 0.0
>>> scheduler_save_host_attribute_nanos_per_event 0.0
>>> scheduler_save_host_attribute_nanos_total 12466197
>>> scheduler_save_host_attribute_nanos_total_per_sec 0.0
>>> scheduler_status_update_events 426
>>> scheduler_status_update_events_per_sec 0.0
>>> scheduler_status_update_nanos_per_event 0.0
>>> scheduler_status_update_nanos_total 19334004116
>>> scheduler_status_update_nanos_total_per_sec 0.0
>>> scheduler_status_updates 0
>>> scheduler_thrift_createJob_events 2
>>> scheduler_thrift_createJob_events_per_sec 0.0
>>> scheduler_thrift_createJob_nanos_per_event 0.0
>>> scheduler_thrift_createJob_nanos_total 54102009
>>> scheduler_thrift_createJob_nanos_total_per_sec 0.0
>>> scheduler_thrift_getTasksStatus_events 2
>>> scheduler_thrift_getTasksStatus_events_per_sec 0.0
>>> scheduler_thrift_getTasksStatus_nanos_per_event 0.0
>>> scheduler_thrift_getTasksStatus_nanos_total 5684762
>>> scheduler_thrift_getTasksStatus_nanos_total_per_sec 0.0
>>> scheduler_thrift_getVersion_events 4
>>> scheduler_thrift_getVersion_events_per_sec 0.0
>>> scheduler_thrift_getVersion_nanos_per_event 0.0
>>> scheduler_thrift_getVersion_nanos_total 13854714
>>> scheduler_thrift_getVersion_nanos_total_per_sec 0.0
>>> scheduler_thrift_killTasks_events 2
>>> scheduler_thrift_killTasks_events_per_sec 0.0
>>> scheduler_thrift_killTasks_nanos_per_event 0.0
>>> scheduler_thrift_killTasks_nanos_total 186407312
>>> scheduler_thrift_killTasks_nanos_total_per_sec 0.0
>>> shard_sanity_check_failures 0
>>> snapshot_apply_events 1
>>> snapshot_apply_events_per_sec 0.0
>>> snapshot_apply_nanos_per_event 0.0
>>> snapshot_apply_nanos_total 93051831
>>> snapshot_apply_nanos_total_per_sec 0.0
>>> snapshot_create_events 10
>>> snapshot_create_events_per_sec 0.0
>>> snapshot_create_nanos_per_event 0.0
>>> snapshot_create_nanos_total 37066472
>>> snapshot_create_nanos_total_per_sec 0.0
>>> system_env_GLOG_v 0
>>> system_env_HOME /home/lychee
>>> system_env_JAVA_OPTS -server -Xmx1g -Xms1g
>>> -Djava.library.path=/usr/local/lib
>>> system_env_LANG en_US.UTF-8
>>> system_env_LIBPROCESS_IP 192.168.0.18
>>> system_env_LIBPROCESS_PORT 8083
>>> system_env_LOGNAME root
>>> system_env_MAIL /var/mail/root
>>> system_env_NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
>>> system_env_OLDPWD /usr/local/aurora-scheduler
>>> system_env_PATH
>>> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>> system_env_PWD /home/lychee
>>> system_env_SHELL /bin/bash
>>> system_env_SHLVL 0
>>> system_env_SUDO_COMMAND /etc/rc.aurora_scheduler.local
>>> system_env_SUDO_GID 1002
>>> system_env_SUDO_UID 1002
>>> system_env_SUDO_USER lychee
>>> system_env_TERM xterm
>>> system_env_USER root
>>> system_env_USERNAME root
>>> system_env_XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
>>> system_free_physical_memory_mb 148
>>> system_free_swap_mb 8063
>>> system_load_avg 2.22
>>> task_queries_all 45
>>> task_queries_by_id 3668
>>> task_queries_by_job 183
>>> task_schedule_attempt_events 135
>>> task_schedule_attempt_events_per_sec 0.0
>>> task_schedule_attempt_nanos_per_event 0.0
>>> task_schedule_attempt_nanos_total 6289005821
>>> task_schedule_attempt_nanos_total_per_sec 0.0
>>> task_store_ASSIGNED 0
>>> task_store_FAILED 0
>>> task_store_FINISHED 200
>>> task_store_INIT 0
>>> task_store_KILLED 0
>>> task_store_KILLING 0
>>> task_store_LOST 0
>>> task_store_PENDING 1
>>> task_store_PREEMPTING 0
>>> task_store_RESTARTING 0
>>> task_store_RUNNING 0
>>> task_store_STARTING 0
>>> task_store_THROTTLED 0
>>> task_store_UNKNOWN 0
>>> timed_out_tasks 0
>>> timeout_queue_size 338
>>> transient_states 0
>>> uncaught_exceptions 0
>>> variable_scrape_events 38778
>>> variable_scrape_events_per_sec 1.0000103021061324
>>> variable_scrape_micros_per_event 455.0001369564521
>>> variable_scrape_micros_total 18809481
>>> variable_scrape_micros_total_per_sec 455.00482441615327
>>> work_queue_depth 0
>>> write_lock_wait_nanos 13138744
>>>
>>>
>>>
>>>
>>> On Fri, Dec 20, 2013 at 9:07 AM, Maxim Khutornenko <max...@twitter.com
>>>> wrote:
>>>
>>>> Hi Yingbo,
>>>>
>>>> Do you mind sharing your .aurora file? Perhaps your are requesting
>>>> Resources or Constraints that are infeasible for your local setup to
>>>> satisfy?
>>>>
>>>> Also, you might want to check that your scheduler is receiving offers
>>> from
>>>> mesos master. You could check your scheduler log file to see if any
>>> offers
>>>> are coming. Alternatively, you might want to look at
>>>> "scheduler_resource_offers" stat exposed by the scheduler /vars
>> endpoint
>>>> (http://<your_local_scheduler_ip:port>/vars).
>>>>
>>>> Thanks,
>>>> Maxim
>>>>
>>>>
>>>>
>>>> On Dec 19, 2013, at 11:20 PM, Yingbo Wang <ybw...@gmail.com> wrote:
>>>>
>>>>> hi,
>>>>> Hope you guys have a fun night at city hall.
>>>>> While you might be drunk, can anyone help me a little bit on
>> setting
>>>> up
>>>>> the local dev environment to debug aurora ?
>>>>> My current set up.
>>>>> Machine A: 64bit ubuntu server running mesos-master,
>>>>> Machine B: 64bit ubuntu server running mesos-slave, thermo
>>>>> observer server;
>>>>> Machine C: 32bit ubuntu server running zookeeper.
>>>>> Machine D: Mac air for developing aurora.
>>>>>
>>>>> I tried to run aurora-scheduler on Machine A and also created a
>>>> hello
>>>>> job from machine A. the job was submitted successfully. however the
>> job
>>>>> keeps staying at "PENDING" status. i checked thermo observer from
>> port
>>>>> 1336, it says "No matching hosts".
>>>>>
>>>>> I spent a couple of hours and still can't figure out the
>> reason
>>> so
>>>>> I decided to settup the dev environment on my Mac so that I can trace
>>> the
>>>>> error in IntelliJ. Can someone give me some hint on that? I am not
>>> quite
>>>>> familiar with IntelliJ and java.
>>>>>
>>>>>
>>>>>
>>>>> best
>>>>> Yingbo
>>>>
>>>>
>>>
>>
