Guest User

Untitled

a guest
Apr 26th, 2017
55
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 70.43 KB | None | 0 0
  1. [ RUN ] SlaveRecoveryTest/0.RecoverUnregisteredExecutor
  2. I0425 21:50:13.245477 22006 cluster.cpp:162] Creating default 'local' authorizer
  3. I0425 21:50:13.247051 22025 master.cpp:438] Master a292a7d1-5a93-4c03-a3f4-bd175ba8de15 (ip-10-51-168-165) started on 10.51.168.165:33073
  4. I0425 21:50:13.247071 22025 master.cpp:440] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/hh17aJ/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="100secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/hh17aJ/master" --zk_session_timeout="10secs"
  5. I0425 21:50:13.247159 22025 master.cpp:490] Master only allowing authenticated frameworks to register
  6. I0425 21:50:13.247165 22025 master.cpp:504] Master only allowing authenticated agents to register
  7. I0425 21:50:13.247169 22025 master.cpp:517] Master only allowing authenticated HTTP frameworks to register
  8. I0425 21:50:13.247174 22025 credentials.hpp:37] Loading credentials for authentication from '/tmp/hh17aJ/credentials'
  9. I0425 21:50:13.247233 22025 master.cpp:562] Using default 'crammd5' authenticator
  10. I0425 21:50:13.247277 22025 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readonly'
  11. I0425 21:50:13.247318 22025 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readwrite'
  12. I0425 21:50:13.247382 22025 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-scheduler'
  13. I0425 21:50:13.247428 22025 master.cpp:642] Authorization enabled
  14. I0425 21:50:13.247490 22028 hierarchical.cpp:159] Initialized hierarchical allocator process
  15. I0425 21:50:13.247504 22027 whitelist_watcher.cpp:77] No whitelist given
  16. I0425 21:50:13.248033 22027 master.cpp:2163] Elected as the leading master!
  17. I0425 21:50:13.248050 22027 master.cpp:1702] Recovering from registrar
  18. I0425 21:50:13.248107 22026 registrar.cpp:345] Recovering registrar
  19. I0425 21:50:13.248383 22032 registrar.cpp:389] Successfully fetched the registry (0B) in 239872ns
  20. I0425 21:50:13.248416 22032 registrar.cpp:493] Applied 1 operations in 2992ns; attempting to update the registry
  21. I0425 21:50:13.248589 22027 registrar.cpp:550] Successfully updated the registry in 158976ns
  22. I0425 21:50:13.248628 22027 registrar.cpp:422] Successfully recovered registrar
  23. I0425 21:50:13.248795 22025 master.cpp:1801] Recovered 0 agents from the registry (146B); allowing 10mins for agents to re-register
  24. I0425 21:50:13.248811 22026 hierarchical.cpp:186] Skipping recovery of hierarchical allocator: nothing to recover
  25. I0425 21:50:13.249630 22006 containerizer.cpp:221] Using isolation: cgroups/cpu,cgroups/mem,filesystem/posix,network/cni
  26. I0425 21:50:13.253229 22006 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
  27. I0425 21:50:13.253561 22006 provisioner.cpp:249] Using default backend 'overlay'
  28. I0425 21:50:13.266118 22006 cluster.cpp:448] Creating default 'local' authorizer
  29. I0425 21:50:13.266548 22028 slave.cpp:225] Mesos agent started on (448)@10.51.168.165:33073
  30. I0425 21:50:13.266566 22028 slave.cpp:226] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_secret_key="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/executor_secret_key" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" --launcher="linux" --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.10/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV"
  31. I0425 21:50:13.266707 22028 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/credential'
  32. I0425 21:50:13.266751 22028 slave.cpp:258] Agent using credential for: test-principal
  33. I0425 21:50:13.266765 22028 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/http_credentials'
  34. I0425 21:50:13.266875 22028 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor'
  35. I0425 21:50:13.266913 22028 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor'
  36. I0425 21:50:13.267017 22028 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
  37. I0425 21:50:13.267058 22028 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly'
  38. I0425 21:50:13.267117 22028 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
  39. I0425 21:50:13.267149 22028 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite'
  40. I0425 21:50:13.268548 22028 slave.cpp:525] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
  41. I0425 21:50:13.268584 22028 slave.cpp:533] Agent attributes: [ ]
  42. I0425 21:50:13.268590 22028 slave.cpp:538] Agent hostname: ip-10-51-168-165
  43. I0425 21:50:13.268666 22029 status_update_manager.cpp:177] Pausing sending status updates
  44. I0425 21:50:13.268946 22030 state.cpp:62] Recovering state from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta'
  45. I0425 21:50:13.269053 22025 status_update_manager.cpp:203] Recovering status update manager
  46. I0425 21:50:13.269139 22027 containerizer.cpp:608] Recovering containerizer
  47. I0425 21:50:13.269400 22006 sched.cpp:232] Version: 1.3.0
  48. I0425 21:50:13.269526 22028 sched.cpp:336] New master detected at [email protected]:33073
  49. I0425 21:50:13.269563 22028 sched.cpp:407] Authenticating with master [email protected]:33073
  50. I0425 21:50:13.269572 22028 sched.cpp:414] Using default CRAM-MD5 authenticatee
  51. I0425 21:50:13.269654 22031 authenticatee.cpp:121] Creating new client SASL connection
  52. I0425 21:50:13.270473 22031 master.cpp:7257] Authenticating [email protected]:33073
  53. I0425 21:50:13.270547 22031 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(943)@10.51.168.165:33073
  54. I0425 21:50:13.270620 22031 authenticator.cpp:98] Creating new server SASL connection
  55. I0425 21:50:13.270849 22028 provisioner.cpp:410] Provisioner recovery complete
  56. I0425 21:50:13.270952 22026 slave.cpp:5963] Finished recovery
  57. I0425 21:50:13.271172 22026 slave.cpp:6145] Querying resource estimator for oversubscribable resources
  58. I0425 21:50:13.271265 22026 slave.cpp:918] New master detected at [email protected]:33073
  59. I0425 21:50:13.271281 22028 status_update_manager.cpp:177] Pausing sending status updates
  60. I0425 21:50:13.271298 22026 slave.cpp:953] Detecting new master
  61. I0425 21:50:13.271353 22026 slave.cpp:6159] Received oversubscribable resources {} from the resource estimator
  62. I0425 21:50:13.271450 22031 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
  63. I0425 21:50:13.271477 22031 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
  64. I0425 21:50:13.271523 22031 authenticator.cpp:204] Received SASL authentication start
  65. I0425 21:50:13.271561 22031 authenticator.cpp:326] Authentication requires more steps
  66. I0425 21:50:13.271598 22031 authenticatee.cpp:259] Received SASL authentication step
  67. I0425 21:50:13.271667 22027 authenticator.cpp:232] Received SASL authentication step
  68. I0425 21:50:13.271692 22027 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
  69. I0425 21:50:13.271697 22027 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
  70. I0425 21:50:13.271705 22027 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
  71. I0425 21:50:13.271711 22027 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
  72. I0425 21:50:13.271716 22027 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
  73. I0425 21:50:13.271720 22027 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
  74. I0425 21:50:13.271728 22027 authenticator.cpp:318] Authentication success
  75. I0425 21:50:13.271782 22025 authenticatee.cpp:299] Authentication success
  76. I0425 21:50:13.271797 22027 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(943)@10.51.168.165:33073
  77. I0425 21:50:13.271813 22031 master.cpp:7287] Successfully authenticated principal 'test-principal' at [email protected]:33073
  78. I0425 21:50:13.271973 22025 sched.cpp:513] Successfully authenticated with master [email protected]:33073
  79. I0425 21:50:13.271993 22025 sched.cpp:836] Sending SUBSCRIBE call to [email protected]:33073
  80. I0425 21:50:13.272037 22025 sched.cpp:869] Will retry registration in 653.278647ms if necessary
  81. I0425 21:50:13.272073 22029 master.cpp:2815] Received SUBSCRIBE call for framework 'default' at [email protected]:33073
  82. I0425 21:50:13.272102 22029 master.cpp:2199] Authorizing framework principal 'test-principal' to receive offers for roles '{ * }'
  83. I0425 21:50:13.272222 22027 master.cpp:2892] Subscribing framework default with checkpointing enabled and capabilities [ ]
  84. I0425 21:50:13.272379 22025 hierarchical.cpp:271] Added framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  85. I0425 21:50:13.272380 22029 sched.cpp:759] Framework registered with a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  86. I0425 21:50:13.272436 22029 sched.cpp:773] Scheduler::registered took 15077ns
  87. I0425 21:50:13.272452 22025 hierarchical.cpp:1862] No allocations performed
  88. I0425 21:50:13.272464 22025 hierarchical.cpp:1952] No inverse offers to send out!
  89. I0425 21:50:13.272470 22025 hierarchical.cpp:1446] Performed allocation for 0 agents in 24114ns
  90. I0425 21:50:13.277460 22031 slave.cpp:980] Authenticating with master [email protected]:33073
  91. I0425 21:50:13.277493 22031 slave.cpp:991] Using default CRAM-MD5 authenticatee
  92. I0425 21:50:13.277559 22029 authenticatee.cpp:121] Creating new client SASL connection
  93. I0425 21:50:13.278302 22029 master.cpp:7257] Authenticating slave(448)@10.51.168.165:33073
  94. I0425 21:50:13.278363 22026 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(944)@10.51.168.165:33073
  95. I0425 21:50:13.278448 22032 authenticator.cpp:98] Creating new server SASL connection
  96. I0425 21:50:13.279145 22032 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
  97. I0425 21:50:13.279165 22032 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
  98. I0425 21:50:13.279223 22028 authenticator.cpp:204] Received SASL authentication start
  99. I0425 21:50:13.279264 22028 authenticator.cpp:326] Authentication requires more steps
  100. I0425 21:50:13.279309 22028 authenticatee.cpp:259] Received SASL authentication step
  101. I0425 21:50:13.279389 22027 authenticator.cpp:232] Received SASL authentication step
  102. I0425 21:50:13.279414 22027 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
  103. I0425 21:50:13.279422 22027 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
  104. I0425 21:50:13.279428 22027 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
  105. I0425 21:50:13.279433 22027 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
  106. I0425 21:50:13.279438 22027 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
  107. I0425 21:50:13.279443 22027 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
  108. I0425 21:50:13.279450 22027 authenticator.cpp:318] Authentication success
  109. I0425 21:50:13.279505 22026 authenticatee.cpp:299] Authentication success
  110. I0425 21:50:13.279505 22028 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(944)@10.51.168.165:33073
  111. I0425 21:50:13.279521 22027 master.cpp:7287] Successfully authenticated principal 'test-principal' at slave(448)@10.51.168.165:33073
  112. I0425 21:50:13.279656 22026 slave.cpp:1075] Successfully authenticated with master [email protected]:33073
  113. I0425 21:50:13.279711 22026 slave.cpp:1503] Will retry registration in 6.518332ms if necessary
  114. I0425 21:50:13.279758 22029 master.cpp:5447] Registering agent at slave(448)@10.51.168.165:33073 (ip-10-51-168-165) with id a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0
  115. I0425 21:50:13.279930 22032 registrar.cpp:493] Applied 1 operations in 16215ns; attempting to update the registry
  116. I0425 21:50:13.280089 22032 registrar.cpp:550] Successfully updated the registry in 126976ns
  117. I0425 21:50:13.280272 22032 slave.cpp:4745] Received ping from slave-observer(450)@10.51.168.165:33073
  118. I0425 21:50:13.280293 22029 master.cpp:5521] Registered agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(448)@10.51.168.165:33073 (ip-10-51-168-165) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
  119. I0425 21:50:13.280335 22032 slave.cpp:1121] Registered with master [email protected]:33073; given agent ID a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0
  120. I0425 21:50:13.280354 22032 fetcher.cpp:94] Clearing fetcher cache
  121. I0425 21:50:13.280376 22028 hierarchical.cpp:527] Added agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 (ip-10-51-168-165) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: {})
  122. I0425 21:50:13.280422 22026 status_update_manager.cpp:184] Resuming sending status updates
  123. I0425 21:50:13.280539 22032 slave.cpp:1149] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/slave.info'
  124. I0425 21:50:13.280688 22028 hierarchical.cpp:1952] No inverse offers to send out!
  125. I0425 21:50:13.280699 22032 slave.cpp:1187] Forwarding total oversubscribed resources {}
  126. I0425 21:50:13.280719 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 281240ns
  127. I0425 21:50:13.280771 22029 master.cpp:7087] Sending 1 offers to framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 (default) at [email protected]:33073
  128. I0425 21:50:13.280850 22029 master.cpp:6106] Received update of agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(448)@10.51.168.165:33073 (ip-10-51-168-165) with total oversubscribed resources {}
  129. I0425 21:50:13.280971 22027 sched.cpp:933] Scheduler::resourceOffers took 54848ns
  130. I0425 21:50:13.281440 22027 master.cpp:3853] Processing ACCEPT call for offers: [ a292a7d1-5a93-4c03-a3f4-bd175ba8de15-O0 ] on agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(448)@10.51.168.165:33073 (ip-10-51-168-165) for framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 (default) at [email protected]:33073
  131. I0425 21:50:13.281474 22027 master.cpp:3429] Authorizing framework principal 'test-principal' to launch task 302016c4-76e3-4dba-bc96-22c661d7d0f6
  132. I0425 21:50:13.281931 22026 master.cpp:9102] Adding task 302016c4-76e3-4dba-bc96-22c661d7d0f6 with resources cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] on agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(448)@10.51.168.165:33073 (ip-10-51-168-165)
  133. I0425 21:50:13.282002 22026 master.cpp:4503] Launching task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 (default) at [email protected]:33073 with resources cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] on agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(448)@10.51.168.165:33073 (ip-10-51-168-165)
  134. I0425 21:50:13.282235 22032 slave.cpp:1613] Got assigned task '302016c4-76e3-4dba-bc96-22c661d7d0f6' for framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  135. I0425 21:50:13.282316 22032 slave.cpp:6836] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/framework.info'
  136. I0425 21:50:13.282531 22032 slave.cpp:6847] Checkpointing framework pid '[email protected]:33073' to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/framework.pid'
  137. I0425 21:50:13.282651 22030 hierarchical.cpp:852] Updated allocation of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 on agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 from cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] to cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000]
  138. I0425 21:50:13.282835 22032 slave.cpp:1894] Authorizing task '302016c4-76e3-4dba-bc96-22c661d7d0f6' for framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  139. I0425 21:50:13.282862 22032 slave.cpp:6582] Authorizing framework principal 'test-principal' to launch task 302016c4-76e3-4dba-bc96-22c661d7d0f6
  140. I0425 21:50:13.283092 22031 slave.cpp:2081] Launching task '302016c4-76e3-4dba-bc96-22c661d7d0f6' for framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  141. I0425 21:50:13.283331 22031 paths.cpp:556] Trying to chown '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/runs/6677fa46-1865-41e3-8aa3-d7c16b7943e7' to user 'root'
  142. I0425 21:50:13.283448 22031 slave.cpp:7289] Checkpointing ExecutorInfo to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/executor.info'
  143. I0425 21:50:13.283692 22031 slave.cpp:6926] Launching executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 with resources cpus(*)(allocated: *):0.1; mem(*)(allocated: *):32 in work directory '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/runs/6677fa46-1865-41e3-8aa3-d7c16b7943e7'
  144. I0425 21:50:13.283926 22031 slave.cpp:7317] Checkpointing TaskInfo to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/runs/6677fa46-1865-41e3-8aa3-d7c16b7943e7/tasks/302016c4-76e3-4dba-bc96-22c661d7d0f6/task.info'
  145. I0425 21:50:13.284093 22031 slave.cpp:2310] Queued task '302016c4-76e3-4dba-bc96-22c661d7d0f6' for executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  146. I0425 21:50:13.284123 22031 slave.cpp:871] Successfully attached file '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/runs/6677fa46-1865-41e3-8aa3-d7c16b7943e7'
  147. I0425 21:50:13.284279 22029 containerizer.cpp:1021] Starting container 6677fa46-1865-41e3-8aa3-d7c16b7943e7 for executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  148. I0425 21:50:13.284749 22031 cgroups.cpp:410] Creating cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7' for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  149. I0425 21:50:13.328438 22031 cgroups.cpp:473] Chown the cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7' to user 'root' for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  150. I0425 21:50:13.328514 22031 cgroups.cpp:410] Creating cgroup at '/sys/fs/cgroup/memory/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7' for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  151. I0425 21:50:13.329522 22031 memory.cpp:479] Started listening for OOM events for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  152. I0425 21:50:13.329856 22031 memory.cpp:590] Started listening on 'low' memory pressure events for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  153. I0425 21:50:13.330181 22031 memory.cpp:590] Started listening on 'medium' memory pressure events for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  154. I0425 21:50:13.330503 22031 memory.cpp:590] Started listening on 'critical' memory pressure events for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  155. I0425 21:50:13.330526 22031 cgroups.cpp:473] Chown the cgroup at '/sys/fs/cgroup/memory/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7' to user 'root' for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  156. I0425 21:50:13.331136 22031 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 1056MB for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  157. I0425 21:50:13.331717 22031 memory.cpp:228] Updated 'memory.limit_in_bytes' to 1056MB for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  158. I0425 21:50:13.332052 22031 cpu.cpp:101] Updated 'cpu.shares' to 2150 (cpus 2.1) for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  159. I0425 21:50:13.332823 22032 containerizer.cpp:1541] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"command":{"arguments":["mesos-executor","--launcher_dir=\/home\/ubuntu\/workspace\/mesos\/Mesos_CI-build\/FLAG\/SSL\/label\/mesos-ec2-ubuntu-16.10\/mesos\/build\/src"],"shell":false,"value":"\/home\/ubuntu\/workspace\/mesos\/Mesos_CI-build\/FLAG\/SSL\/label\/mesos-ec2-ubuntu-16.10\/mesos\/build\/src\/mesos-executor"},"environment":{"variables":[{"name":"LIBPROCESS_PORT","type":"VALUE","value":"0"},{"name":"MESOS_AGENT_ENDPOINT","type":"VALUE","value":"10.51.168.165:33073"},{"name":"MESOS_CHECKPOINT","type":"VALUE","value":"1"},{"name":"MESOS_DIRECTORY","type":"VALUE","value":"\/tmp\/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV\/slaves\/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0\/frameworks\/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000\/executors\/302016c4-76e3-4dba-bc96-22c661d7d0f6\/runs\/6677fa46-1865-41e3-8aa3-d7c16b7943e7"},{"name":"MESOS_EXECUTOR_AUTHENTICATION_TOKEN","type":"VALUE","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjaWQiOiI2Njc3ZmE0Ni0xODY1LTQxZTMtOGFhMy1kN2MxNmI3OTQzZTciLCJlaWQiOiIzMDIwMTZjNC03NmUzLTRkYmEtYmM5Ni0yMmM2NjFkN2QwZjYiLCJmaWQiOiJhMjkyYTdkMS01YTkzLTRjMDMtYTNmNC1iZDE3NWJhOGRlMTUtMDAwMCJ9.7hh21b2FHKTd2gZon9riiq7JIrF-rREmETA-xwh7bZQ"},{"name":"MESOS_EXECUTOR_ID","type":"VALUE","value":"302016c4-76e3-4dba-bc96-22c661d7d0f6"},{"name":"MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD","type":"VALUE","value":"5secs"},{"name":"MESOS_FRAMEWORK_ID","type":"VALUE","value":"a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000"},{"name":"MESOS_HTTP_COMMAND_EXECUTOR","type":"VALUE","value":"0"},{"name":"MESOS_RECOVERY_TIMEOUT","type":"VALUE","value":"15mins"},{"name":"MESOS_SLAVE_ID","type":"VALUE","value":"a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0"},{"name":"MESOS_SLAVE_PID","type":"VALUE","value":"slave(448)@10.51.168.165:33073"},{"name":"MESOS_SUBSCRIPTION_BACKOFF_MAX","type":"VALUE","value":"2secs"},{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/tmp\/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV\/slaves\/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0\/frameworks\/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000\/executors\/302016c4-76e3-4dba-bc96-22c661d7d0f6\/runs\/6677fa46-1865-41e3-8aa3-d7c16b7943e7"}]},"user":"root","working_directory":"\/tmp\/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV\/slaves\/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0\/frameworks\/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000\/executors\/302016c4-76e3-4dba-bc96-22c661d7d0f6\/runs\/6677fa46-1865-41e3-8aa3-d7c16b7943e7"}" --pipe_read="32" --pipe_write="34" --runtime_directory="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/containers/6677fa46-1865-41e3-8aa3-d7c16b7943e7" --unshare_namespace_mnt="false"'
  160. I0425 21:50:13.333106 22027 linux_launcher.cpp:429] Launching container 6677fa46-1865-41e3-8aa3-d7c16b7943e7 and cloning with namespaces
  161. I0425 21:50:13.353049 22032 containerizer.cpp:1640] Checkpointing container's forked pid 1754 to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/runs/6677fa46-1865-41e3-8aa3-d7c16b7943e7/pids/forked.pid'
  162. I0425 21:50:13.354950 22030 fetcher.cpp:353] Starting to fetch URIs for container: 6677fa46-1865-41e3-8aa3-d7c16b7943e7, directory: /tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/runs/6677fa46-1865-41e3-8aa3-d7c16b7943e7
  163. I0425 21:50:13.426503 1779 exec.cpp:162] Version: 1.3.0
  164. I0425 21:50:13.429482 22025 slave.cpp:790] Agent terminating
  165. I0425 21:50:13.429587 22031 master.cpp:1315] Agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(448)@10.51.168.165:33073 (ip-10-51-168-165) disconnected
  166. I0425 21:50:13.429612 22031 master.cpp:3199] Disconnecting agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(448)@10.51.168.165:33073 (ip-10-51-168-165)
  167. I0425 21:50:13.429635 22031 master.cpp:3218] Deactivating agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(448)@10.51.168.165:33073 (ip-10-51-168-165)
  168. I0425 21:50:13.429677 22025 hierarchical.cpp:655] Agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 deactivated
  169. I0425 21:50:13.429769 22006 containerizer.cpp:221] Using isolation: cgroups/cpu,cgroups/mem,filesystem/posix,network/cni
  170. I0425 21:50:13.432929 22006 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
  171. I0425 21:50:13.433243 22006 provisioner.cpp:249] Using default backend 'overlay'
  172. I0425 21:50:13.445747 22006 cluster.cpp:448] Creating default 'local' authorizer
  173. I0425 21:50:13.446148 22031 slave.cpp:225] Mesos agent started on (449)@10.51.168.165:33073
  174. I0425 21:50:13.446174 22031 slave.cpp:226] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_secret_key="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/executor_secret_key" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" --launcher="linux" --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.10/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV"
  175. I0425 21:50:13.446375 22031 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/credential'
  176. I0425 21:50:13.446435 22031 slave.cpp:258] Agent using credential for: test-principal
  177. I0425 21:50:13.446496 22031 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_2zz5r8/http_credentials'
  178. I0425 21:50:13.446590 22031 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor'
  179. I0425 21:50:13.446749 22031 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor'
  180. I0425 21:50:13.446854 22031 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
  181. I0425 21:50:13.446914 22031 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly'
  182. I0425 21:50:13.447037 22031 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
  183. I0425 21:50:13.447085 22031 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite'
  184. I0425 21:50:13.448305 22031 slave.cpp:525] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
  185. I0425 21:50:13.448348 22031 slave.cpp:533] Agent attributes: [ ]
  186. I0425 21:50:13.448361 22031 slave.cpp:538] Agent hostname: ip-10-51-168-165
  187. I0425 21:50:13.448480 22027 status_update_manager.cpp:177] Pausing sending status updates
  188. I0425 21:50:13.448709 22025 state.cpp:62] Recovering state from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta'
  189. I0425 21:50:13.448747 22025 state.cpp:710] No committed checkpointed resources found at '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/resources/resources.info'
  190. W0425 21:50:13.449177 22025 state.cpp:617] Failed to find status updates file '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/runs/6677fa46-1865-41e3-8aa3-d7c16b7943e7/tasks/302016c4-76e3-4dba-bc96-22c661d7d0f6/task.updates'
  191. W0425 21:50:13.449292 22025 state.cpp:552] Failed to find 'libprocess.pid' or 'http.marker' for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7 of executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  192. I0425 21:50:13.449640 22031 fetcher.cpp:94] Clearing fetcher cache
  193. I0425 21:50:13.449669 22031 slave.cpp:6054] Recovering framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  194. I0425 21:50:13.449684 22031 slave.cpp:7034] Recovering executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  195. I0425 21:50:13.449926 22027 status_update_manager.cpp:203] Recovering status update manager
  196. I0425 21:50:13.449949 22027 status_update_manager.cpp:211] Recovering executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  197. W0425 21:50:13.449975 22027 status_update_manager.cpp:250] No updates found for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  198. I0425 21:50:13.450217 22027 containerizer.cpp:608] Recovering containerizer
  199. I0425 21:50:13.450249 22027 containerizer.cpp:664] Recovering container 6677fa46-1865-41e3-8aa3-d7c16b7943e7 for executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  200. I0425 21:50:13.450877 22026 linux_launcher.cpp:291] Recovered container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  201. I0425 21:50:13.452065 22025 memory.cpp:479] Started listening for OOM events for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  202. I0425 21:50:13.452405 22025 memory.cpp:590] Started listening on 'low' memory pressure events for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  203. I0425 21:50:13.452728 22025 memory.cpp:590] Started listening on 'medium' memory pressure events for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  204. I0425 21:50:13.453104 22025 memory.cpp:590] Started listening on 'critical' memory pressure events for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  205. I0425 21:50:13.454558 22026 provisioner.cpp:410] Provisioner recovery complete
  206. I0425 21:50:13.454896 22032 slave.cpp:5913] Unable to reconnect to executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 because no pid or http checkpoint file was found
  207. I0425 21:50:13.455222 22030 hierarchical.cpp:1862] No allocations performed
  208. I0425 21:50:13.455288 22030 hierarchical.cpp:1952] No inverse offers to send out!
  209. I0425 21:50:13.455304 22029 slave.cpp:4145] Cleaning up un-reregistered executors
  210. I0425 21:50:13.455329 22029 slave.cpp:4163] Killing un-reregistered executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  211. I0425 21:50:13.455308 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 120098ns
  212. I0425 21:50:13.455394 22030 containerizer.cpp:2118] Destroying container 6677fa46-1865-41e3-8aa3-d7c16b7943e7 in RUNNING state
  213. I0425 21:50:13.455399 22029 slave.cpp:5963] Finished recovery
  214. I0425 21:50:13.455516 22030 linux_launcher.cpp:505] Asked to destroy container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  215. I0425 21:50:13.455656 22029 slave.cpp:6145] Querying resource estimator for oversubscribable resources
  216. I0425 21:50:13.455765 22026 slave.cpp:918] New master detected at [email protected]:33073
  217. I0425 21:50:13.455780 22029 status_update_manager.cpp:177] Pausing sending status updates
  218. I0425 21:50:13.455790 22026 slave.cpp:953] Detecting new master
  219. I0425 21:50:13.455864 22026 slave.cpp:6159] Received oversubscribable resources {} from the resource estimator
  220. I0425 21:50:13.455998 22030 linux_launcher.cpp:548] Using freezer to destroy cgroup mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7
  221. I0425 21:50:13.456708 22031 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7
  222. I0425 21:50:13.457685 22029 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7 after 0ns
  223. I0425 21:50:13.458756 22026 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7
  224. I0425 21:50:13.459765 22028 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7 after 0ns
  225. I0425 21:50:13.460053 22026 slave.cpp:980] Authenticating with master [email protected]:33073
  226. I0425 21:50:13.460122 22026 slave.cpp:991] Using default CRAM-MD5 authenticatee
  227. I0425 21:50:13.460144 22027 hierarchical.cpp:1862] No allocations performed
  228. I0425 21:50:13.460170 22027 hierarchical.cpp:1952] No inverse offers to send out!
  229. I0425 21:50:13.460182 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 63081ns
  230. I0425 21:50:13.460209 22026 authenticatee.cpp:121] Creating new client SASL connection
  231. I0425 21:50:13.461156 22029 master.cpp:7257] Authenticating slave(449)@10.51.168.165:33073
  232. I0425 21:50:13.461239 22029 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(945)@10.51.168.165:33073
  233. I0425 21:50:13.461364 22026 authenticator.cpp:98] Creating new server SASL connection
  234. I0425 21:50:13.462193 22026 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
  235. I0425 21:50:13.462221 22026 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
  236. I0425 21:50:13.462280 22026 authenticator.cpp:204] Received SASL authentication start
  237. I0425 21:50:13.462313 22026 authenticator.cpp:326] Authentication requires more steps
  238. I0425 21:50:13.462364 22026 authenticatee.cpp:259] Received SASL authentication step
  239. I0425 21:50:13.462424 22026 authenticator.cpp:232] Received SASL authentication step
  240. I0425 21:50:13.462443 22026 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
  241. I0425 21:50:13.462450 22026 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
  242. I0425 21:50:13.462457 22026 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
  243. I0425 21:50:13.462463 22026 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
  244. I0425 21:50:13.462468 22026 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
  245. I0425 21:50:13.462472 22026 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
  246. I0425 21:50:13.462481 22026 authenticator.cpp:318] Authentication success
  247. I0425 21:50:13.462533 22028 authenticatee.cpp:299] Authentication success
  248. I0425 21:50:13.462541 22029 master.cpp:7287] Successfully authenticated principal 'test-principal' at slave(449)@10.51.168.165:33073
  249. I0425 21:50:13.462622 22025 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(945)@10.51.168.165:33073
  250. I0425 21:50:13.462721 22029 slave.cpp:1075] Successfully authenticated with master [email protected]:33073
  251. I0425 21:50:13.462828 22029 slave.cpp:1503] Will retry registration in 14.678761ms if necessary
  252. I0425 21:50:13.462905 22026 master.cpp:5598] Re-registering agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(448)@10.51.168.165:33073 (ip-10-51-168-165)
  253. I0425 21:50:13.463027 22028 slave.cpp:1229] Re-registered with master [email protected]:33073
  254. I0425 21:50:13.463073 22027 hierarchical.cpp:643] Agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 reactivated
  255. I0425 21:50:13.463068 22026 master.cpp:6015] Sending updated checkpointed resources {} to agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(449)@10.51.168.165:33073 (ip-10-51-168-165)
  256. I0425 21:50:13.463124 22028 slave.cpp:1266] Forwarding total oversubscribed resources {}
  257. I0425 21:50:13.463142 22026 status_update_manager.cpp:184] Resuming sending status updates
  258. I0425 21:50:13.463279 22028 slave.cpp:3228] Updating info for framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 with pid updated to [email protected]:33073
  259. I0425 21:50:13.463289 22025 master.cpp:6106] Received update of agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(449)@10.51.168.165:33073 (ip-10-51-168-165) with total oversubscribed resources {}
  260. I0425 21:50:13.463327 22028 slave.cpp:6836] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/framework.info'
  261. I0425 21:50:13.463513 22028 slave.cpp:6847] Checkpointing framework pid '[email protected]:33073' to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/framework.pid'
  262. I0425 21:50:13.463667 22028 slave.cpp:3289] Ignoring new checkpointed resources identical to the current version: {}
  263. I0425 21:50:13.463688 22031 status_update_manager.cpp:184] Resuming sending status updates
  264. I0425 21:50:13.463928 22027 hierarchical.cpp:1862] No allocations performed
  265. I0425 21:50:13.463955 22027 hierarchical.cpp:1952] No inverse offers to send out!
  266. I0425 21:50:13.463973 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 56840ns
  267. I0425 21:50:13.464012 22030 containerizer.cpp:2524] Container 6677fa46-1865-41e3-8aa3-d7c16b7943e7 has exited
  268. I0425 21:50:13.464155 22028 hierarchical.cpp:1862] No allocations performed
  269. I0425 21:50:13.464174 22028 hierarchical.cpp:1952] No inverse offers to send out!
  270. I0425 21:50:13.464184 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 40362ns
  271. I0425 21:50:13.464313 22025 hierarchical.cpp:1862] No allocations performed
  272. I0425 21:50:13.464334 22025 hierarchical.cpp:1952] No inverse offers to send out!
  273. I0425 21:50:13.464342 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 42399ns
  274. I0425 21:50:13.464529 22027 hierarchical.cpp:1862] No allocations performed
  275. I0425 21:50:13.464547 22027 hierarchical.cpp:1952] No inverse offers to send out!
  276. I0425 21:50:13.464555 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 40151ns
  277. I0425 21:50:13.464880 22025 hierarchical.cpp:1862] No allocations performed
  278. I0425 21:50:13.464901 22025 hierarchical.cpp:1952] No inverse offers to send out!
  279. I0425 21:50:13.464910 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 50033ns
  280. I0425 21:50:13.465092 22028 hierarchical.cpp:1862] No allocations performed
  281. I0425 21:50:13.465112 22028 hierarchical.cpp:1952] No inverse offers to send out!
  282. I0425 21:50:13.465121 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 42356ns
  283. I0425 21:50:13.465343 22030 hierarchical.cpp:1862] No allocations performed
  284. I0425 21:50:13.465363 22030 hierarchical.cpp:1952] No inverse offers to send out!
  285. I0425 21:50:13.465373 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 46887ns
  286. I0425 21:50:13.465591 22031 hierarchical.cpp:1862] No allocations performed
  287. I0425 21:50:13.465615 22031 hierarchical.cpp:1952] No inverse offers to send out!
  288. I0425 21:50:13.465631 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 51663ns
  289. I0425 21:50:13.465770 22029 hierarchical.cpp:1862] No allocations performed
  290. I0425 21:50:13.465795 22029 hierarchical.cpp:1952] No inverse offers to send out!
  291. I0425 21:50:13.465806 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 51298ns
  292. I0425 21:50:13.465935 22032 hierarchical.cpp:1862] No allocations performed
  293. I0425 21:50:13.465955 22032 hierarchical.cpp:1952] No inverse offers to send out!
  294. I0425 21:50:13.465963 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 48196ns
  295. I0425 21:50:13.466142 22026 hierarchical.cpp:1862] No allocations performed
  296. I0425 21:50:13.466166 22026 hierarchical.cpp:1952] No inverse offers to send out!
  297. I0425 21:50:13.466177 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 55335ns
  298. I0425 21:50:13.466330 22025 hierarchical.cpp:1862] No allocations performed
  299. I0425 21:50:13.466348 22025 hierarchical.cpp:1952] No inverse offers to send out!
  300. I0425 21:50:13.466357 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 45341ns
  301. I0425 21:50:13.466500 22031 hierarchical.cpp:1862] No allocations performed
  302. I0425 21:50:13.466521 22031 hierarchical.cpp:1952] No inverse offers to send out!
  303. I0425 21:50:13.466537 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 48636ns
  304. I0425 21:50:13.466648 22029 slave.cpp:6145] Querying resource estimator for oversubscribable resources
  305. I0425 21:50:13.466683 22030 hierarchical.cpp:1862] No allocations performed
  306. I0425 21:50:13.466716 22030 hierarchical.cpp:1952] No inverse offers to send out!
  307. I0425 21:50:13.466725 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 53737ns
  308. I0425 21:50:13.466810 22027 slave.cpp:6159] Received oversubscribable resources {} from the resource estimator
  309. I0425 21:50:13.466964 22028 hierarchical.cpp:1862] No allocations performed
  310. I0425 21:50:13.466989 22028 hierarchical.cpp:1952] No inverse offers to send out!
  311. I0425 21:50:13.466998 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 50340ns
  312. I0425 21:50:13.467133 22030 hierarchical.cpp:1862] No allocations performed
  313. I0425 21:50:13.467154 22030 hierarchical.cpp:1952] No inverse offers to send out!
  314. I0425 21:50:13.467170 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 53459ns
  315. I0425 21:50:13.467310 22026 hierarchical.cpp:1862] No allocations performed
  316. I0425 21:50:13.467334 22026 hierarchical.cpp:1952] No inverse offers to send out!
  317. I0425 21:50:13.467344 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 46445ns
  318. I0425 21:50:13.467470 22029 hierarchical.cpp:1862] No allocations performed
  319. I0425 21:50:13.467489 22029 hierarchical.cpp:1952] No inverse offers to send out!
  320. I0425 21:50:13.467497 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 38080ns
  321. I0425 21:50:13.467624 22032 hierarchical.cpp:1862] No allocations performed
  322. I0425 21:50:13.467644 22032 hierarchical.cpp:1952] No inverse offers to send out!
  323. I0425 21:50:13.467653 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 46688ns
  324. I0425 21:50:13.467787 22025 hierarchical.cpp:1862] No allocations performed
  325. I0425 21:50:13.467806 22025 hierarchical.cpp:1952] No inverse offers to send out!
  326. I0425 21:50:13.467814 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 40167ns
  327. I0425 21:50:13.467973 22027 hierarchical.cpp:1862] No allocations performed
  328. I0425 21:50:13.467999 22027 hierarchical.cpp:1952] No inverse offers to send out!
  329. I0425 21:50:13.468008 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 49812ns
  330. I0425 21:50:13.468122 22028 hierarchical.cpp:1862] No allocations performed
  331. I0425 21:50:13.468142 22028 hierarchical.cpp:1952] No inverse offers to send out!
  332. I0425 21:50:13.468150 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 39259ns
  333. I0425 21:50:13.468273 22031 hierarchical.cpp:1862] No allocations performed
  334. I0425 21:50:13.468294 22031 hierarchical.cpp:1952] No inverse offers to send out!
  335. I0425 21:50:13.468310 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 51356ns
  336. I0425 21:50:13.468451 22030 hierarchical.cpp:1862] No allocations performed
  337. I0425 21:50:13.468477 22030 hierarchical.cpp:1952] No inverse offers to send out!
  338. I0425 21:50:13.468488 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 49761ns
  339. I0425 21:50:13.468633 22032 hierarchical.cpp:1862] No allocations performed
  340. I0425 21:50:13.468659 22032 hierarchical.cpp:1952] No inverse offers to send out!
  341. I0425 21:50:13.468672 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 49789ns
  342. I0425 21:50:13.468811 22026 hierarchical.cpp:1862] No allocations performed
  343. I0425 21:50:13.468833 22026 hierarchical.cpp:1952] No inverse offers to send out!
  344. I0425 21:50:13.468844 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 49077ns
  345. I0425 21:50:13.468992 22030 hierarchical.cpp:1862] No allocations performed
  346. I0425 21:50:13.469010 22030 hierarchical.cpp:1952] No inverse offers to send out!
  347. I0425 21:50:13.469019 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 39054ns
  348. I0425 21:50:13.469166 22025 hierarchical.cpp:1862] No allocations performed
  349. I0425 21:50:13.469184 22025 hierarchical.cpp:1952] No inverse offers to send out!
  350. I0425 21:50:13.469193 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 39466ns
  351. I0425 21:50:13.469285 22031 slave.cpp:6145] Querying resource estimator for oversubscribable resources
  352. I0425 21:50:13.469318 22029 hierarchical.cpp:1862] No allocations performed
  353. I0425 21:50:13.469331 22029 hierarchical.cpp:1952] No inverse offers to send out!
  354. I0425 21:50:13.469338 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 31947ns
  355. I0425 21:50:13.469375 22026 slave.cpp:6159] Received oversubscribable resources {} from the resource estimator
  356. I0425 21:50:13.469516 22030 hierarchical.cpp:1862] No allocations performed
  357. I0425 21:50:13.469533 22030 hierarchical.cpp:1952] No inverse offers to send out!
  358. I0425 21:50:13.469542 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 39298ns
  359. I0425 21:50:13.469681 22025 hierarchical.cpp:1862] No allocations performed
  360. I0425 21:50:13.469704 22025 hierarchical.cpp:1952] No inverse offers to send out!
  361. I0425 21:50:13.469718 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 47784ns
  362. I0425 21:50:13.469862 22031 hierarchical.cpp:1862] No allocations performed
  363. I0425 21:50:13.469883 22031 hierarchical.cpp:1952] No inverse offers to send out!
  364. I0425 21:50:13.469900 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 48040ns
  365. I0425 21:50:13.470041 22027 hierarchical.cpp:1862] No allocations performed
  366. I0425 21:50:13.470065 22027 hierarchical.cpp:1952] No inverse offers to send out!
  367. I0425 21:50:13.470077 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 48370ns
  368. I0425 21:50:13.470202 22029 hierarchical.cpp:1862] No allocations performed
  369. I0425 21:50:13.470221 22029 hierarchical.cpp:1952] No inverse offers to send out!
  370. I0425 21:50:13.470229 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 38678ns
  371. I0425 21:50:13.470371 22026 hierarchical.cpp:1862] No allocations performed
  372. I0425 21:50:13.470393 22026 hierarchical.cpp:1952] No inverse offers to send out!
  373. I0425 21:50:13.470405 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 47262ns
  374. I0425 21:50:13.470553 22025 hierarchical.cpp:1862] No allocations performed
  375. I0425 21:50:13.470576 22025 hierarchical.cpp:1952] No inverse offers to send out!
  376. I0425 21:50:13.470592 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 47947ns
  377. I0425 21:50:13.470710 22028 hierarchical.cpp:1862] No allocations performed
  378. I0425 21:50:13.470728 22028 hierarchical.cpp:1952] No inverse offers to send out!
  379. I0425 21:50:13.470736 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 36760ns
  380. I0425 21:50:13.470855 22030 hierarchical.cpp:1862] No allocations performed
  381. I0425 21:50:13.470873 22030 hierarchical.cpp:1952] No inverse offers to send out!
  382. I0425 21:50:13.470882 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 39019ns
  383. I0425 21:50:13.471007 22032 hierarchical.cpp:1862] No allocations performed
  384. I0425 21:50:13.471026 22032 hierarchical.cpp:1952] No inverse offers to send out!
  385. I0425 21:50:13.471035 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 44571ns
  386. I0425 21:50:13.471199 22031 hierarchical.cpp:1862] No allocations performed
  387. I0425 21:50:13.471220 22031 hierarchical.cpp:1952] No inverse offers to send out!
  388. I0425 21:50:13.471236 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 52245ns
  389. I0425 21:50:13.471426 22027 hierarchical.cpp:1862] No allocations performed
  390. I0425 21:50:13.471444 22027 hierarchical.cpp:1952] No inverse offers to send out!
  391. I0425 21:50:13.471452 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 38430ns
  392. I0425 21:50:13.471592 22026 hierarchical.cpp:1862] No allocations performed
  393. I0425 21:50:13.471614 22026 hierarchical.cpp:1952] No inverse offers to send out!
  394. I0425 21:50:13.471626 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 47411ns
  395. I0425 21:50:13.471781 22030 hierarchical.cpp:1862] No allocations performed
  396. I0425 21:50:13.471798 22030 hierarchical.cpp:1952] No inverse offers to send out!
  397. I0425 21:50:13.471807 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 38616ns
  398. I0425 21:50:13.471933 22025 slave.cpp:6145] Querying resource estimator for oversubscribable resources
  399. I0425 21:50:13.471983 22031 hierarchical.cpp:1862] No allocations performed
  400. I0425 21:50:13.472008 22031 hierarchical.cpp:1952] No inverse offers to send out!
  401. I0425 21:50:13.472020 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 55014ns
  402. I0425 21:50:13.472059 22031 slave.cpp:6159] Received oversubscribable resources {} from the resource estimator
  403. I0425 21:50:13.472204 22029 hierarchical.cpp:1862] No allocations performed
  404. I0425 21:50:13.472229 22029 hierarchical.cpp:1952] No inverse offers to send out!
  405. I0425 21:50:13.472242 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 50929ns
  406. I0425 21:50:13.472373 22032 hierarchical.cpp:1862] No allocations performed
  407. I0425 21:50:13.472393 22032 hierarchical.cpp:1952] No inverse offers to send out!
  408. I0425 21:50:13.472403 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 41174ns
  409. I0425 21:50:13.472523 22030 hierarchical.cpp:1862] No allocations performed
  410. I0425 21:50:13.472546 22030 hierarchical.cpp:1952] No inverse offers to send out!
  411. I0425 21:50:13.472560 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 48259ns
  412. I0425 21:50:13.472702 22025 hierarchical.cpp:1862] No allocations performed
  413. I0425 21:50:13.472721 22025 hierarchical.cpp:1952] No inverse offers to send out!
  414. I0425 21:50:13.472730 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 39335ns
  415. I0425 21:50:13.472942 22028 hierarchical.cpp:1862] No allocations performed
  416. I0425 21:50:13.472961 22028 hierarchical.cpp:1952] No inverse offers to send out!
  417. I0425 21:50:13.472970 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 40684ns
  418. I0425 21:50:13.473122 22031 hierarchical.cpp:1862] No allocations performed
  419. I0425 21:50:13.473141 22031 hierarchical.cpp:1952] No inverse offers to send out!
  420. I0425 21:50:13.473150 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 42208ns
  421. I0425 21:50:13.473310 22032 hierarchical.cpp:1862] No allocations performed
  422. I0425 21:50:13.473330 22032 hierarchical.cpp:1952] No inverse offers to send out!
  423. I0425 21:50:13.473338 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 42781ns
  424. I0425 21:50:13.473480 22026 hierarchical.cpp:1862] No allocations performed
  425. I0425 21:50:13.473505 22026 hierarchical.cpp:1952] No inverse offers to send out!
  426. I0425 21:50:13.473516 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 49972ns
  427. I0425 21:50:13.473645 22029 hierarchical.cpp:1862] No allocations performed
  428. I0425 21:50:13.473664 22029 hierarchical.cpp:1952] No inverse offers to send out!
  429. I0425 21:50:13.473672 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 40122ns
  430. I0425 21:50:13.473811 22028 hierarchical.cpp:1862] No allocations performed
  431. I0425 21:50:13.473829 22028 hierarchical.cpp:1952] No inverse offers to send out!
  432. I0425 21:50:13.473839 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 39913ns
  433. I0425 21:50:13.473990 22031 hierarchical.cpp:1862] No allocations performed
  434. I0425 21:50:13.474012 22031 hierarchical.cpp:1952] No inverse offers to send out!
  435. I0425 21:50:13.474028 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 48497ns
  436. I0425 21:50:13.474195 22026 slave.cpp:5659] Current disk usage 13.64%. Max allowed age: 5.345250469458090days
  437. I0425 21:50:13.474230 22030 hierarchical.cpp:1862] No allocations performed
  438. I0425 21:50:13.474256 22030 hierarchical.cpp:1952] No inverse offers to send out!
  439. I0425 21:50:13.474272 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 55552ns
  440. I0425 21:50:13.474419 22026 hierarchical.cpp:1862] No allocations performed
  441. I0425 21:50:13.474442 22026 hierarchical.cpp:1952] No inverse offers to send out!
  442. I0425 21:50:13.474452 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 47504ns
  443. I0425 21:50:13.474589 22032 hierarchical.cpp:1862] No allocations performed
  444. I0425 21:50:13.474611 22032 hierarchical.cpp:1952] No inverse offers to send out!
  445. I0425 21:50:13.474622 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 50992ns
  446. I0425 21:50:13.474732 22029 slave.cpp:6145] Querying resource estimator for oversubscribable resources
  447. I0425 21:50:13.474800 22030 hierarchical.cpp:1862] No allocations performed
  448. I0425 21:50:13.474921 22030 hierarchical.cpp:1952] No inverse offers to send out!
  449. I0425 21:50:13.474949 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 167485ns
  450. E0425 21:50:13.474858 22029 slave.cpp:5139] Termination of executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 failed: Failed to kill all processes in the container: Timed out after 1mins
  451. I0425 21:50:13.475008 22029 slave.cpp:4215] Handling status update TASK_LOST (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 from @0.0.0.0:0
  452. I0425 21:50:13.475107 22029 slave.cpp:6159] Received oversubscribable resources {} from the resource estimator
  453. W0425 21:50:13.475406 22029 containerizer.cpp:2071] Skipping status for container 6677fa46-1865-41e3-8aa3-d7c16b7943e7 because: Container does not exist
  454. W0425 21:50:13.475648 22028 containerizer.cpp:1940] Ignoring update for currently being destroyed container 6677fa46-1865-41e3-8aa3-d7c16b7943e7
  455. I0425 21:50:13.475801 22031 status_update_manager.cpp:323] Received status update TASK_LOST (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  456. I0425 21:50:13.475847 22031 status_update_manager.cpp:500] Creating StatusUpdate stream for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  457. I0425 21:50:13.475988 22031 status_update_manager.cpp:832] Checkpointing UPDATE for status update TASK_LOST (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  458. I0425 21:50:13.476089 22031 status_update_manager.cpp:377] Forwarding update TASK_LOST (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 to the agent
  459. I0425 21:50:13.476205 22028 slave.cpp:4655] Forwarding the update TASK_LOST (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 to [email protected]:33073
  460. I0425 21:50:13.476279 22028 slave.cpp:4549] Status update manager successfully handled status update TASK_LOST (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  461. I0425 21:50:13.476336 22032 master.cpp:6251] Status update TASK_LOST (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 from agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(449)@10.51.168.165:33073 (ip-10-51-168-165)
  462. I0425 21:50:13.476361 22032 master.cpp:6319] Forwarding status update TASK_LOST (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  463. I0425 21:50:13.476410 22032 master.cpp:8350] Updating the state of task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 (latest state: TASK_LOST, status update state: TASK_LOST)
  464. I0425 21:50:13.476495 22031 sched.cpp:1041] Scheduler::statusUpdate took 33890ns
  465. I0425 21:50:13.476613 22032 master.cpp:5165] Processing ACKNOWLEDGE call 141535b3-067b-4682-ac6f-25b3d943a1b2 for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 (default) at [email protected]:33073 on agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0
  466. I0425 21:50:13.476642 22032 master.cpp:8444] Removing task 302016c4-76e3-4dba-bc96-22c661d7d0f6 with resources cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 on agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 at slave(449)@10.51.168.165:33073 (ip-10-51-168-165)
  467. I0425 21:50:13.476661 22030 hierarchical.cpp:1116] Recovered cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: {}) on agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 from framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  468. I0425 21:50:13.476809 22027 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  469. I0425 21:50:13.476860 22027 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_LOST (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  470. I0425 21:50:13.476897 22027 status_update_manager.cpp:531] Cleaning up status update stream for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  471. I0425 21:50:13.476976 22027 slave.cpp:3504] Status update manager successfully handled status update acknowledgement (UUID: 141535b3-067b-4682-ac6f-25b3d943a1b2) for task 302016c4-76e3-4dba-bc96-22c661d7d0f6 of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  472. I0425 21:50:13.476994 22027 slave.cpp:7270] Completing task 302016c4-76e3-4dba-bc96-22c661d7d0f6
  473. I0425 21:50:13.477002 22027 slave.cpp:5261] Cleaning up executor '302016c4-76e3-4dba-bc96-22c661d7d0f6' of framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  474. I0425 21:50:13.477133 22032 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/runs/6677fa46-1865-41e3-8aa3-d7c16b7943e7' for gc 6.99999482064889days in the future
  475. I0425 21:50:13.477185 22027 slave.cpp:5349] Cleaning up framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  476. I0425 21:50:13.477185 22032 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6' for gc 6.99999482064889days in the future
  477. I0425 21:50:13.477249 22032 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6/runs/6677fa46-1865-41e3-8aa3-d7c16b7943e7' for gc 6.99999482064889days in the future
  478. I0425 21:50:13.477265 22028 status_update_manager.cpp:285] Closing status update streams for framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  479. I0425 21:50:13.477288 22032 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000/executors/302016c4-76e3-4dba-bc96-22c661d7d0f6' for gc 6.99999482064889days in the future
  480. I0425 21:50:13.477308 22032 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000' for gc 6.99999482064889days in the future
  481. I0425 21:50:13.477324 22032 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_qxDBOV/meta/slaves/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0/frameworks/a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000' for gc 6.99999482064889days in the future
  482. I0425 21:50:13.477691 22026 hierarchical.cpp:1952] No inverse offers to send out!
  483. I0425 21:50:13.477718 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 247125ns
  484. I0425 21:50:13.477807 22031 master.cpp:7087] Sending 1 offers to framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 (default) at [email protected]:33073
  485. I0425 21:50:13.477941 22031 sched.cpp:933] Scheduler::resourceOffers took 21us
  486. I0425 21:50:13.478097 22006 sched.cpp:2021] Asked to stop the driver
  487. I0425 21:50:13.478160 22027 sched.cpp:1203] Stopping framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  488. I0425 21:50:13.478255 22032 master.cpp:7770] Processing TEARDOWN call for framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 (default) at [email protected]:33073
  489. I0425 21:50:13.478278 22032 master.cpp:7782] Removing framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 (default) at [email protected]:33073
  490. I0425 21:50:13.478286 22032 master.cpp:3162] Deactivating framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 (default) at [email protected]:33073
  491. I0425 21:50:13.478363 22031 hierarchical.cpp:376] Deactivated framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  492. I0425 21:50:13.478515 22030 slave.cpp:3057] Asked to shut down framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000 by [email protected]:33073
  493. I0425 21:50:13.478539 22030 slave.cpp:3072] Cannot shut down unknown framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  494. I0425 21:50:13.478595 22031 hierarchical.cpp:1116] Recovered cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: {}) on agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0 from framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  495. I0425 21:50:13.478693 22031 hierarchical.cpp:323] Removed framework a292a7d1-5a93-4c03-a3f4-bd175ba8de15-0000
  496. ../../src/tests/cluster.cpp:580: Failure
  497. Value of: containers->empty()
  498. Actual: false
  499. Expected: true
  500. Failed to destroy containers: { 6677fa46-1865-41e3-8aa3-d7c16b7943e7 }
  501. I0425 21:50:13.479107 22006 slave.cpp:790] Agent terminating
  502. I0425 21:50:13.481299 22006 master.cpp:1157] Master terminating
  503. I0425 21:50:13.481468 22030 hierarchical.cpp:560] Removed agent a292a7d1-5a93-4c03-a3f4-bd175ba8de15-S0
  504. I0425 21:50:13.499274 22026 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7
  505. I0425 21:50:13.500252 22028 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7 after 950016ns
  506. I0425 21:50:13.501188 22030 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7
  507. I0425 21:50:13.502099 22029 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026/6677fa46-1865-41e3-8aa3-d7c16b7943e7 after 877056ns
  508. I0425 21:50:13.503921 22029 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026
  509. I0425 21:50:13.605914 22025 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026 after 101.959936ms
  510. I0425 21:50:13.606812 22025 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026
  511. I0425 21:50:13.607703 22029 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_875fbfc7-bf0b-4a1e-a4c0-af86ba549026 after 862976ns
  512. [ FAILED ] SlaveRecoveryTest/0.RecoverUnregisteredExecutor, where TypeParam = mesos::internal::slave::MesosContainerizer (415 ms)
Advertisement
Add Comment
Please, Sign In to add comment