Guest User

Untitled

a guest
Apr 26th, 2017
74
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 73.05 KB | None | 0 0
  1. [ RUN ] SlaveRecoveryTest/0.CleanupExecutor
  2. I0425 21:50:17.062870 22006 cluster.cpp:162] Creating default 'local' authorizer
  3. I0425 21:50:17.064646 22027 master.cpp:438] Master 9a71c48e-7660-46f3-9ce6-cde41efba5e3 (ip-10-51-168-165) started on 10.51.168.165:33073
  4. I0425 21:50:17.064673 22027 master.cpp:440] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/LGCxfH/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="100secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/LGCxfH/master" --zk_session_timeout="10secs"
  5. I0425 21:50:17.064765 22027 master.cpp:490] Master only allowing authenticated frameworks to register
  6. I0425 21:50:17.064772 22027 master.cpp:504] Master only allowing authenticated agents to register
  7. I0425 21:50:17.064776 22027 master.cpp:517] Master only allowing authenticated HTTP frameworks to register
  8. I0425 21:50:17.064780 22027 credentials.hpp:37] Loading credentials for authentication from '/tmp/LGCxfH/credentials'
  9. I0425 21:50:17.064848 22027 master.cpp:562] Using default 'crammd5' authenticator
  10. I0425 21:50:17.064890 22027 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readonly'
  11. I0425 21:50:17.064941 22027 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readwrite'
  12. I0425 21:50:17.064999 22027 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-scheduler'
  13. I0425 21:50:17.065064 22027 master.cpp:642] Authorization enabled
  14. I0425 21:50:17.065135 22031 whitelist_watcher.cpp:77] No whitelist given
  15. I0425 21:50:17.065137 22026 hierarchical.cpp:159] Initialized hierarchical allocator process
  16. I0425 21:50:17.065593 22027 master.cpp:2163] Elected as the leading master!
  17. I0425 21:50:17.065614 22027 master.cpp:1702] Recovering from registrar
  18. I0425 21:50:17.065691 22028 registrar.cpp:345] Recovering registrar
  19. I0425 21:50:17.065945 22031 registrar.cpp:389] Successfully fetched the registry (0B) in 225024ns
  20. I0425 21:50:17.065979 22031 registrar.cpp:493] Applied 1 operations in 2687ns; attempting to update the registry
  21. I0425 21:50:17.066102 22026 registrar.cpp:550] Successfully updated the registry in 105984ns
  22. I0425 21:50:17.066151 22026 registrar.cpp:422] Successfully recovered registrar
  23. I0425 21:50:17.066274 22026 master.cpp:1801] Recovered 0 agents from the registry (146B); allowing 10mins for agents to re-register
  24. I0425 21:50:17.066319 22029 hierarchical.cpp:186] Skipping recovery of hierarchical allocator: nothing to recover
  25. I0425 21:50:17.067112 22006 containerizer.cpp:221] Using isolation: cgroups/cpu,cgroups/mem,filesystem/posix,network/cni
  26. I0425 21:50:17.070595 22006 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
  27. I0425 21:50:17.070876 22006 provisioner.cpp:249] Using default backend 'overlay'
  28. I0425 21:50:17.083324 22006 cluster.cpp:448] Creating default 'local' authorizer
  29. I0425 21:50:17.083753 22026 slave.cpp:225] Mesos agent started on (457)@10.51.168.165:33073
  30. I0425 21:50:17.083771 22026 slave.cpp:226] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_secret_key="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/executor_secret_key" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" --launcher="linux" --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.10/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5"
  31. I0425 21:50:17.083935 22026 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/credential'
  32. I0425 21:50:17.083995 22026 slave.cpp:258] Agent using credential for: test-principal
  33. I0425 21:50:17.084014 22026 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/http_credentials'
  34. I0425 21:50:17.084100 22026 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor'
  35. I0425 21:50:17.084133 22026 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor'
  36. I0425 21:50:17.084189 22026 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
  37. I0425 21:50:17.084233 22026 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly'
  38. I0425 21:50:17.084291 22026 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
  39. I0425 21:50:17.084333 22026 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite'
  40. I0425 21:50:17.085695 22026 slave.cpp:525] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
  41. I0425 21:50:17.085728 22026 slave.cpp:533] Agent attributes: [ ]
  42. I0425 21:50:17.085736 22026 slave.cpp:538] Agent hostname: ip-10-51-168-165
  43. I0425 21:50:17.085796 22030 status_update_manager.cpp:177] Pausing sending status updates
  44. I0425 21:50:17.086024 22026 state.cpp:62] Recovering state from '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta'
  45. I0425 21:50:17.086187 22032 status_update_manager.cpp:203] Recovering status update manager
  46. I0425 21:50:17.086287 22025 containerizer.cpp:608] Recovering containerizer
  47. I0425 21:50:17.086581 22006 sched.cpp:232] Version: 1.3.0
  48. I0425 21:50:17.086737 22027 sched.cpp:336] New master detected at [email protected]:33073
  49. I0425 21:50:17.086771 22027 sched.cpp:407] Authenticating with master [email protected]:33073
  50. I0425 21:50:17.086779 22027 sched.cpp:414] Using default CRAM-MD5 authenticatee
  51. I0425 21:50:17.086859 22028 authenticatee.cpp:121] Creating new client SASL connection
  52. I0425 21:50:17.087702 22027 master.cpp:7257] Authenticating [email protected]:33073
  53. I0425 21:50:17.087770 22028 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(958)@10.51.168.165:33073
  54. I0425 21:50:17.088017 22029 authenticator.cpp:98] Creating new server SASL connection
  55. I0425 21:50:17.088069 22025 provisioner.cpp:410] Provisioner recovery complete
  56. I0425 21:50:17.088194 22027 slave.cpp:5963] Finished recovery
  57. I0425 21:50:17.088392 22027 slave.cpp:6145] Querying resource estimator for oversubscribable resources
  58. I0425 21:50:17.088512 22026 slave.cpp:918] New master detected at [email protected]:33073
  59. I0425 21:50:17.088512 22025 status_update_manager.cpp:177] Pausing sending status updates
  60. I0425 21:50:17.088567 22026 slave.cpp:953] Detecting new master
  61. I0425 21:50:17.088660 22026 slave.cpp:6159] Received oversubscribable resources {} from the resource estimator
  62. I0425 21:50:17.088914 22029 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
  63. I0425 21:50:17.088937 22029 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
  64. I0425 21:50:17.088977 22029 authenticator.cpp:204] Received SASL authentication start
  65. I0425 21:50:17.089018 22029 authenticator.cpp:326] Authentication requires more steps
  66. I0425 21:50:17.089066 22029 authenticatee.cpp:259] Received SASL authentication step
  67. I0425 21:50:17.089141 22030 authenticator.cpp:232] Received SASL authentication step
  68. I0425 21:50:17.089169 22030 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
  69. I0425 21:50:17.089175 22030 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
  70. I0425 21:50:17.089182 22030 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
  71. I0425 21:50:17.089190 22030 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
  72. I0425 21:50:17.089193 22030 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
  73. I0425 21:50:17.089197 22030 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
  74. I0425 21:50:17.089206 22030 authenticator.cpp:318] Authentication success
  75. I0425 21:50:17.089259 22028 authenticatee.cpp:299] Authentication success
  76. I0425 21:50:17.089273 22030 master.cpp:7287] Successfully authenticated principal 'test-principal' at [email protected]:33073
  77. I0425 21:50:17.089298 22029 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(958)@10.51.168.165:33073
  78. I0425 21:50:17.089437 22031 sched.cpp:513] Successfully authenticated with master [email protected]:33073
  79. I0425 21:50:17.089457 22031 sched.cpp:836] Sending SUBSCRIBE call to [email protected]:33073
  80. I0425 21:50:17.089500 22031 sched.cpp:869] Will retry registration in 693.906546ms if necessary
  81. I0425 21:50:17.089540 22032 master.cpp:2815] Received SUBSCRIBE call for framework 'default' at [email protected]:33073
  82. I0425 21:50:17.089579 22032 master.cpp:2199] Authorizing framework principal 'test-principal' to receive offers for roles '{ * }'
  83. I0425 21:50:17.089728 22026 master.cpp:2892] Subscribing framework default with checkpointing enabled and capabilities [ ]
  84. I0425 21:50:17.089853 22026 sched.cpp:759] Framework registered with 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  85. I0425 21:50:17.089860 22032 hierarchical.cpp:271] Added framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  86. I0425 21:50:17.089880 22026 sched.cpp:773] Scheduler::registered took 13484ns
  87. I0425 21:50:17.089911 22032 hierarchical.cpp:1862] No allocations performed
  88. I0425 21:50:17.089931 22032 hierarchical.cpp:1952] No inverse offers to send out!
  89. I0425 21:50:17.089944 22032 hierarchical.cpp:1446] Performed allocation for 0 agents in 39929ns
  90. I0425 21:50:17.091742 22031 slave.cpp:980] Authenticating with master [email protected]:33073
  91. I0425 21:50:17.091768 22031 slave.cpp:991] Using default CRAM-MD5 authenticatee
  92. I0425 21:50:17.091825 22031 authenticatee.cpp:121] Creating new client SASL connection
  93. I0425 21:50:17.092597 22031 master.cpp:7257] Authenticating slave(457)@10.51.168.165:33073
  94. I0425 21:50:17.092646 22026 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(959)@10.51.168.165:33073
  95. I0425 21:50:17.092710 22026 authenticator.cpp:98] Creating new server SASL connection
  96. I0425 21:50:17.093487 22026 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
  97. I0425 21:50:17.093514 22026 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
  98. I0425 21:50:17.093560 22026 authenticator.cpp:204] Received SASL authentication start
  99. I0425 21:50:17.093595 22026 authenticator.cpp:326] Authentication requires more steps
  100. I0425 21:50:17.093657 22027 authenticatee.cpp:259] Received SASL authentication step
  101. I0425 21:50:17.093721 22027 authenticator.cpp:232] Received SASL authentication step
  102. I0425 21:50:17.093745 22027 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
  103. I0425 21:50:17.093757 22027 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
  104. I0425 21:50:17.093770 22027 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
  105. I0425 21:50:17.093782 22027 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-51-168-165.ec2.internal' server FQDN: 'ip-10-51-168-165.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
  106. I0425 21:50:17.093801 22027 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
  107. I0425 21:50:17.093811 22027 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
  108. I0425 21:50:17.093825 22027 authenticator.cpp:318] Authentication success
  109. I0425 21:50:17.093889 22026 authenticatee.cpp:299] Authentication success
  110. I0425 21:50:17.093911 22027 master.cpp:7287] Successfully authenticated principal 'test-principal' at slave(457)@10.51.168.165:33073
  111. I0425 21:50:17.093933 22030 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(959)@10.51.168.165:33073
  112. I0425 21:50:17.094058 22026 slave.cpp:1075] Successfully authenticated with master [email protected]:33073
  113. I0425 21:50:17.094112 22026 slave.cpp:1503] Will retry registration in 153790ns if necessary
  114. I0425 21:50:17.094159 22027 master.cpp:5447] Registering agent at slave(457)@10.51.168.165:33073 (ip-10-51-168-165) with id 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0
  115. I0425 21:50:17.094259 22025 registrar.cpp:493] Applied 1 operations in 12367ns; attempting to update the registry
  116. I0425 21:50:17.094449 22032 registrar.cpp:550] Successfully updated the registry in 167168ns
  117. I0425 21:50:17.094617 22032 slave.cpp:4745] Received ping from slave-observer(455)@10.51.168.165:33073
  118. I0425 21:50:17.094667 22032 slave.cpp:1121] Registered with master [email protected]:33073; given agent ID 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0
  119. I0425 21:50:17.094683 22032 fetcher.cpp:94] Clearing fetcher cache
  120. I0425 21:50:17.094651 22026 master.cpp:5521] Registered agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
  121. I0425 21:50:17.094722 22028 hierarchical.cpp:527] Added agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 (ip-10-51-168-165) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: {})
  122. I0425 21:50:17.094743 22029 status_update_manager.cpp:184] Resuming sending status updates
  123. I0425 21:50:17.094805 22032 slave.cpp:1149] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/slave.info'
  124. I0425 21:50:17.094921 22032 slave.cpp:1187] Forwarding total oversubscribed resources {}
  125. I0425 21:50:17.094955 22028 hierarchical.cpp:1952] No inverse offers to send out!
  126. I0425 21:50:17.094985 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 207us
  127. I0425 21:50:17.095031 22032 master.cpp:7087] Sending 1 offers to framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (default) at [email protected]:33073
  128. I0425 21:50:17.095095 22032 master.cpp:6106] Received update of agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165) with total oversubscribed resources {}
  129. I0425 21:50:17.095194 22027 sched.cpp:933] Scheduler::resourceOffers took 41367ns
  130. I0425 21:50:17.095595 22026 master.cpp:3853] Processing ACCEPT call for offers: [ 9a71c48e-7660-46f3-9ce6-cde41efba5e3-O0 ] on agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165) for framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (default) at [email protected]:33073
  131. I0425 21:50:17.095631 22026 master.cpp:3429] Authorizing framework principal 'test-principal' to launch task 6b25338c-478d-4a54-a191-8ea188d0ae8c
  132. I0425 21:50:17.096096 22026 master.cpp:9102] Adding task 6b25338c-478d-4a54-a191-8ea188d0ae8c with resources cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] on agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165)
  133. I0425 21:50:17.096177 22026 master.cpp:4503] Launching task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (default) at [email protected]:33073 with resources cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] on agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165)
  134. I0425 21:50:17.096395 22028 slave.cpp:1613] Got assigned task '6b25338c-478d-4a54-a191-8ea188d0ae8c' for framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  135. I0425 21:50:17.096456 22028 slave.cpp:6836] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/framework.info'
  136. I0425 21:50:17.096632 22028 slave.cpp:6847] Checkpointing framework pid '[email protected]:33073' to '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/framework.pid'
  137. I0425 21:50:17.096699 22025 hierarchical.cpp:852] Updated allocation of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 on agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 from cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] to cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000]
  138. I0425 21:50:17.096869 22028 slave.cpp:1894] Authorizing task '6b25338c-478d-4a54-a191-8ea188d0ae8c' for framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  139. I0425 21:50:17.096895 22028 slave.cpp:6582] Authorizing framework principal 'test-principal' to launch task 6b25338c-478d-4a54-a191-8ea188d0ae8c
  140. I0425 21:50:17.097120 22025 slave.cpp:2081] Launching task '6b25338c-478d-4a54-a191-8ea188d0ae8c' for framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  141. I0425 21:50:17.097362 22025 paths.cpp:556] Trying to chown '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/runs/256cb6d6-c74c-47b1-bc07-463f2db990ef' to user 'root'
  142. I0425 21:50:17.097486 22025 slave.cpp:7289] Checkpointing ExecutorInfo to '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/executor.info'
  143. I0425 21:50:17.097740 22025 slave.cpp:6926] Launching executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 with resources cpus(*)(allocated: *):0.1; mem(*)(allocated: *):32 in work directory '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/runs/256cb6d6-c74c-47b1-bc07-463f2db990ef'
  144. I0425 21:50:17.097970 22025 slave.cpp:7317] Checkpointing TaskInfo to '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/runs/256cb6d6-c74c-47b1-bc07-463f2db990ef/tasks/6b25338c-478d-4a54-a191-8ea188d0ae8c/task.info'
  145. I0425 21:50:17.098140 22025 slave.cpp:2310] Queued task '6b25338c-478d-4a54-a191-8ea188d0ae8c' for executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  146. I0425 21:50:17.098179 22025 slave.cpp:871] Successfully attached file '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/runs/256cb6d6-c74c-47b1-bc07-463f2db990ef'
  147. I0425 21:50:17.098369 22032 containerizer.cpp:1021] Starting container 256cb6d6-c74c-47b1-bc07-463f2db990ef for executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  148. I0425 21:50:17.098810 22028 cgroups.cpp:410] Creating cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef' for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  149. I0425 21:50:17.099593 22028 cgroups.cpp:473] Chown the cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef' to user 'root' for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  150. I0425 21:50:17.099663 22028 cgroups.cpp:410] Creating cgroup at '/sys/fs/cgroup/memory/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef' for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  151. I0425 21:50:17.100724 22028 memory.cpp:479] Started listening for OOM events for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  152. I0425 21:50:17.101073 22028 memory.cpp:590] Started listening on 'low' memory pressure events for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  153. I0425 21:50:17.101419 22028 memory.cpp:590] Started listening on 'medium' memory pressure events for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  154. I0425 21:50:17.101760 22028 memory.cpp:590] Started listening on 'critical' memory pressure events for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  155. I0425 21:50:17.101780 22028 cgroups.cpp:473] Chown the cgroup at '/sys/fs/cgroup/memory/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef' to user 'root' for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  156. I0425 21:50:17.102403 22025 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 1056MB for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  157. I0425 21:50:17.102995 22025 memory.cpp:228] Updated 'memory.limit_in_bytes' to 1056MB for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  158. I0425 21:50:17.103308 22025 cpu.cpp:101] Updated 'cpu.shares' to 2150 (cpus 2.1) for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  159. I0425 21:50:17.104152 22026 containerizer.cpp:1541] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"command":{"arguments":["mesos-executor","--launcher_dir=\/home\/ubuntu\/workspace\/mesos\/Mesos_CI-build\/FLAG\/SSL\/label\/mesos-ec2-ubuntu-16.10\/mesos\/build\/src"],"shell":false,"value":"\/home\/ubuntu\/workspace\/mesos\/Mesos_CI-build\/FLAG\/SSL\/label\/mesos-ec2-ubuntu-16.10\/mesos\/build\/src\/mesos-executor"},"environment":{"variables":[{"name":"LIBPROCESS_PORT","type":"VALUE","value":"0"},{"name":"MESOS_AGENT_ENDPOINT","type":"VALUE","value":"10.51.168.165:33073"},{"name":"MESOS_CHECKPOINT","type":"VALUE","value":"1"},{"name":"MESOS_DIRECTORY","type":"VALUE","value":"\/tmp\/SlaveRecoveryTest_0_CleanupExecutor_Om3py5\/slaves\/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0\/frameworks\/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000\/executors\/6b25338c-478d-4a54-a191-8ea188d0ae8c\/runs\/256cb6d6-c74c-47b1-bc07-463f2db990ef"},{"name":"MESOS_EXECUTOR_AUTHENTICATION_TOKEN","type":"VALUE","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjaWQiOiIyNTZjYjZkNi1jNzRjLTQ3YjEtYmMwNy00NjNmMmRiOTkwZWYiLCJlaWQiOiI2YjI1MzM4Yy00NzhkLTRhNTQtYTE5MS04ZWExODhkMGFlOGMiLCJmaWQiOiI5YTcxYzQ4ZS03NjYwLTQ2ZjMtOWNlNi1jZGU0MWVmYmE1ZTMtMDAwMCJ9.WUma0IiavR6kyXJH1sO6G3oIu1zf1lU8mohUffX_ZTY"},{"name":"MESOS_EXECUTOR_ID","type":"VALUE","value":"6b25338c-478d-4a54-a191-8ea188d0ae8c"},{"name":"MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD","type":"VALUE","value":"5secs"},{"name":"MESOS_FRAMEWORK_ID","type":"VALUE","value":"9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000"},{"name":"MESOS_HTTP_COMMAND_EXECUTOR","type":"VALUE","value":"0"},{"name":"MESOS_RECOVERY_TIMEOUT","type":"VALUE","value":"15mins"},{"name":"MESOS_SLAVE_ID","type":"VALUE","value":"9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0"},{"name":"MESOS_SLAVE_PID","type":"VALUE","value":"slave(457)@10.51.168.165:33073"},{"name":"MESOS_SUBSCRIPTION_BACKOFF_MAX","type":"VALUE","value":"2secs"},{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/tmp\/SlaveRecoveryTest_0_CleanupExecutor_Om3py5\/slaves\/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0\/frameworks\/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000\/executors\/6b25338c-478d-4a54-a191-8ea188d0ae8c\/runs\/256cb6d6-c74c-47b1-bc07-463f2db990ef"}]},"user":"root","working_directory":"\/tmp\/SlaveRecoveryTest_0_CleanupExecutor_Om3py5\/slaves\/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0\/frameworks\/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000\/executors\/6b25338c-478d-4a54-a191-8ea188d0ae8c\/runs\/256cb6d6-c74c-47b1-bc07-463f2db990ef"}" --pipe_read="32" --pipe_write="34" --runtime_directory="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/containers/256cb6d6-c74c-47b1-bc07-463f2db990ef" --unshare_namespace_mnt="false"'
  160. I0425 21:50:17.104426 22028 linux_launcher.cpp:429] Launching container 256cb6d6-c74c-47b1-bc07-463f2db990ef and cloning with namespaces
  161. I0425 21:50:17.160599 22026 containerizer.cpp:1640] Checkpointing container's forked pid 2096 to '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/runs/256cb6d6-c74c-47b1-bc07-463f2db990ef/pids/forked.pid'
  162. I0425 21:50:17.162922 22027 fetcher.cpp:353] Starting to fetch URIs for container: 256cb6d6-c74c-47b1-bc07-463f2db990ef, directory: /tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/runs/256cb6d6-c74c-47b1-bc07-463f2db990ef
  163. I0425 21:50:17.233806 2149 exec.cpp:162] Version: 1.3.0
  164. I0425 21:50:17.236673 22026 slave.cpp:3784] Got registration for executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 from executor(1)@10.51.168.165:36023
  165. I0425 21:50:17.236822 22026 slave.cpp:3870] Checkpointing executor pid 'executor(1)@10.51.168.165:36023' to '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/runs/256cb6d6-c74c-47b1-bc07-463f2db990ef/pids/libprocess.pid'
  166. I0425 21:50:17.237391 2145 exec.cpp:237] Executor registered on agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0
  167. I0425 21:50:17.237927 22026 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 1056MB for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  168. Received SUBSCRIBED event
  169. Subscribed executor on ip-10-51-168-165
  170. I0425 21:50:17.238571 22026 cpu.cpp:101] Updated 'cpu.shares' to 2150 (cpus 2.1) for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  171. I0425 21:50:17.238945 22025 slave.cpp:2523] Sending queued task '6b25338c-478d-4a54-a191-8ea188d0ae8c' to executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 at executor(1)@10.51.168.165:36023
  172. Received LAUNCH event
  173. Starting task 6b25338c-478d-4a54-a191-8ea188d0ae8c
  174. Running '/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.10/mesos/build/src/mesos-containerizer launch <POSSIBLY-SENSITIVE-DATA>'
  175. Forked command at 2153
  176. I0425 21:50:17.241330 22029 slave.cpp:4215] Handling status update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 from executor(1)@10.51.168.165:36023
  177. I0425 21:50:17.241925 22029 status_update_manager.cpp:323] Received status update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  178. I0425 21:50:17.241955 22029 status_update_manager.cpp:500] Creating StatusUpdate stream for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  179. I0425 21:50:17.242144 22029 status_update_manager.cpp:832] Checkpointing UPDATE for status update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  180. I0425 21:50:17.242249 22029 status_update_manager.cpp:377] Forwarding update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 to the agent
  181. I0425 21:50:17.242362 22030 slave.cpp:4655] Forwarding the update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 to [email protected]:33073
  182. I0425 21:50:17.242451 22030 slave.cpp:4549] Status update manager successfully handled status update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  183. I0425 21:50:17.242475 22030 slave.cpp:4565] Sending acknowledgement for status update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 to executor(1)@10.51.168.165:36023
  184. I0425 21:50:17.242544 22031 master.cpp:6251] Status update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 from agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165)
  185. I0425 21:50:17.242565 22031 master.cpp:6319] Forwarding status update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  186. I0425 21:50:17.242619 22031 master.cpp:8350] Updating the state of task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
  187. I0425 21:50:17.242770 22032 sched.cpp:1041] Scheduler::statusUpdate took 13605ns
  188. I0425 21:50:17.242866 22032 master.cpp:5165] Processing ACKNOWLEDGE call b42c98db-cbfb-48f9-a288-556c2cfff870 for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (default) at [email protected]:33073 on agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0
  189. I0425 21:50:17.243000 22025 status_update_manager.cpp:395] Received status update acknowledgement (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  190. I0425 21:50:17.243065 22025 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_RUNNING (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  191. I0425 21:50:17.243176 22025 slave.cpp:3504] Status update manager successfully handled status update acknowledgement (UUID: b42c98db-cbfb-48f9-a288-556c2cfff870) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  192. I0425 21:50:17.243327 22030 slave.cpp:790] Agent terminating
  193. I0425 21:50:17.243417 22029 master.cpp:1315] Agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165) disconnected
  194. I0425 21:50:17.243439 22029 master.cpp:3199] Disconnecting agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165)
  195. I0425 21:50:17.243463 22029 master.cpp:3218] Deactivating agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165)
  196. I0425 21:50:17.243525 22030 hierarchical.cpp:655] Agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 deactivated
  197. I0425 21:50:17.243582 22006 containerizer.cpp:221] Using isolation: cgroups/cpu,cgroups/mem,filesystem/posix,network/cni
  198. I0425 21:50:17.246712 22006 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
  199. I0425 21:50:17.246994 22006 provisioner.cpp:249] Using default backend 'overlay'
  200. I0425 21:50:17.259186 22006 cluster.cpp:448] Creating default 'local' authorizer
  201. I0425 21:50:17.259573 22025 slave.cpp:225] Mesos agent started on (458)@10.51.168.165:33073
  202. I0425 21:50:17.259593 22025 slave.cpp:226] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_secret_key="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/executor_secret_key" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" --launcher="linux" --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.10/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="cleanup" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5"
  203. I0425 21:50:17.259750 22025 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/credential'
  204. I0425 21:50:17.259812 22025 slave.cpp:258] Agent using credential for: test-principal
  205. I0425 21:50:17.259850 22025 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_CleanupExecutor_zJ5iRt/http_credentials'
  206. I0425 21:50:17.259951 22025 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor'
  207. I0425 21:50:17.260000 22025 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor'
  208. I0425 21:50:17.260056 22025 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
  209. I0425 21:50:17.260095 22025 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly'
  210. I0425 21:50:17.260200 22025 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
  211. I0425 21:50:17.260260 22025 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite'
  212. I0425 21:50:17.260679 22029 hierarchical.cpp:1862] No allocations performed
  213. I0425 21:50:17.260707 22029 hierarchical.cpp:1952] No inverse offers to send out!
  214. I0425 21:50:17.260723 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 79759ns
  215. I0425 21:50:17.261823 22025 slave.cpp:525] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
  216. I0425 21:50:17.261878 22025 slave.cpp:533] Agent attributes: [ ]
  217. I0425 21:50:17.261890 22025 slave.cpp:538] Agent hostname: ip-10-51-168-165
  218. I0425 21:50:17.262049 22026 status_update_manager.cpp:177] Pausing sending status updates
  219. I0425 21:50:17.262388 22032 state.cpp:62] Recovering state from '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta'
  220. I0425 21:50:17.262432 22032 state.cpp:710] No committed checkpointed resources found at '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/resources/resources.info'
  221. I0425 21:50:17.263365 22025 fetcher.cpp:94] Clearing fetcher cache
  222. I0425 21:50:17.263401 22025 slave.cpp:6054] Recovering framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  223. I0425 21:50:17.263424 22025 slave.cpp:7034] Recovering executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  224. I0425 21:50:17.263721 22032 status_update_manager.cpp:203] Recovering status update manager
  225. I0425 21:50:17.263741 22032 status_update_manager.cpp:211] Recovering executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  226. I0425 21:50:17.263763 22032 status_update_manager.cpp:500] Creating StatusUpdate stream for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  227. I0425 21:50:17.263913 22032 status_update_manager.cpp:808] Replaying status update stream for task 6b25338c-478d-4a54-a191-8ea188d0ae8c
  228. I0425 21:50:17.264166 22029 containerizer.cpp:608] Recovering containerizer
  229. I0425 21:50:17.264210 22029 containerizer.cpp:664] Recovering container 256cb6d6-c74c-47b1-bc07-463f2db990ef for executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  230. I0425 21:50:17.264854 22032 linux_launcher.cpp:291] Recovered container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  231. I0425 21:50:17.266034 22029 memory.cpp:479] Started listening for OOM events for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  232. I0425 21:50:17.266374 22029 memory.cpp:590] Started listening on 'low' memory pressure events for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  233. I0425 21:50:17.266703 22029 memory.cpp:590] Started listening on 'medium' memory pressure events for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  234. I0425 21:50:17.267035 22029 memory.cpp:590] Started listening on 'critical' memory pressure events for container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  235. I0425 21:50:17.268573 22030 provisioner.cpp:410] Provisioner recovery complete
  236. I0425 21:50:17.268950 22025 slave.cpp:5923] Sending shutdown to executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 at executor(1)@10.51.168.165:36023
  237. I0425 21:50:17.268975 22025 slave.cpp:5475] Shutting down executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 at executor(1)@10.51.168.165:36023
  238. I0425 21:50:17.269055 22025 slave.cpp:5963] Finished recovery
  239. I0425 21:50:17.269285 2142 exec.cpp:415] Executor asked to shutdown
  240. Received SHUTDOWN event
  241. Shutting down
  242. Sending SIGTERM to process tree at pid 2153
  243. I0425 21:50:17.269423 22026 hierarchical.cpp:1862] No allocations performed
  244. I0425 21:50:17.269438 22026 hierarchical.cpp:1952] No inverse offers to send out!
  245. I0425 21:50:17.269449 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 37959ns
  246. I0425 21:50:17.269665 22027 hierarchical.cpp:1862] No allocations performed
  247. I0425 21:50:17.269687 22027 hierarchical.cpp:1952] No inverse offers to send out!
  248. I0425 21:50:17.269703 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 58002ns
  249. I0425 21:50:17.269948 22029 hierarchical.cpp:1862] No allocations performed
  250. I0425 21:50:17.269968 22029 hierarchical.cpp:1952] No inverse offers to send out!
  251. I0425 21:50:17.269975 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 42261ns
  252. I0425 21:50:17.270134 22027 slave.cpp:5548] Killing executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 at executor(1)@10.51.168.165:36023
  253. I0425 21:50:17.270237 22028 hierarchical.cpp:1862] No allocations performed
  254. I0425 21:50:17.270259 22028 hierarchical.cpp:1952] No inverse offers to send out!
  255. I0425 21:50:17.270274 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 62914ns
  256. I0425 21:50:17.270270 22026 containerizer.cpp:2118] Destroying container 256cb6d6-c74c-47b1-bc07-463f2db990ef in RUNNING state
  257. I0425 21:50:17.270409 22026 linux_launcher.cpp:505] Asked to destroy container 256cb6d6-c74c-47b1-bc07-463f2db990ef
  258. I0425 21:50:17.270750 22026 linux_launcher.cpp:548] Using freezer to destroy cgroup mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef
  259. I0425 21:50:17.271479 22030 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef
  260. I0425 21:50:17.272534 22029 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef after 0ns
  261. I0425 21:50:17.274091 22030 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef
  262. I0425 21:50:17.276352 22031 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef after 0ns
  263. I0425 21:50:17.276713 22029 hierarchical.cpp:1862] No allocations performed
  264. I0425 21:50:17.276737 22029 hierarchical.cpp:1952] No inverse offers to send out!
  265. I0425 21:50:17.276753 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 61320ns
  266. I0425 21:50:17.276790 22030 containerizer.cpp:2524] Container 256cb6d6-c74c-47b1-bc07-463f2db990ef has exited
  267. I0425 21:50:17.276975 22032 hierarchical.cpp:1862] No allocations performed
  268. I0425 21:50:17.276995 22032 hierarchical.cpp:1952] No inverse offers to send out!
  269. I0425 21:50:17.277004 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 47405ns
  270. I0425 21:50:17.277168 22031 hierarchical.cpp:1862] No allocations performed
  271. I0425 21:50:17.277189 22031 hierarchical.cpp:1952] No inverse offers to send out!
  272. I0425 21:50:17.277197 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 41202ns
  273. I0425 21:50:17.277335 22025 hierarchical.cpp:1862] No allocations performed
  274. I0425 21:50:17.277359 22025 hierarchical.cpp:1952] No inverse offers to send out!
  275. I0425 21:50:17.277370 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 50047ns
  276. I0425 21:50:17.277518 22026 hierarchical.cpp:1862] No allocations performed
  277. I0425 21:50:17.277540 22026 hierarchical.cpp:1952] No inverse offers to send out!
  278. I0425 21:50:17.277555 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 51651ns
  279. I0425 21:50:17.277698 22028 hierarchical.cpp:1862] No allocations performed
  280. I0425 21:50:17.277719 22028 hierarchical.cpp:1952] No inverse offers to send out!
  281. I0425 21:50:17.277726 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 45579ns
  282. I0425 21:50:17.277853 22030 hierarchical.cpp:1862] No allocations performed
  283. I0425 21:50:17.277878 22030 hierarchical.cpp:1952] No inverse offers to send out!
  284. I0425 21:50:17.277889 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 48228ns
  285. I0425 21:50:17.278033 22032 hierarchical.cpp:1862] No allocations performed
  286. I0425 21:50:17.278056 22032 hierarchical.cpp:1952] No inverse offers to send out!
  287. I0425 21:50:17.278069 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 51359ns
  288. I0425 21:50:17.278218 22027 hierarchical.cpp:1862] No allocations performed
  289. I0425 21:50:17.278239 22027 hierarchical.cpp:1952] No inverse offers to send out!
  290. I0425 21:50:17.278247 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 45225ns
  291. I0425 21:50:17.278442 22026 hierarchical.cpp:1862] No allocations performed
  292. I0425 21:50:17.278461 22026 hierarchical.cpp:1952] No inverse offers to send out!
  293. I0425 21:50:17.278470 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 43553ns
  294. I0425 21:50:17.278619 22025 hierarchical.cpp:1862] No allocations performed
  295. I0425 21:50:17.278641 22025 hierarchical.cpp:1952] No inverse offers to send out!
  296. I0425 21:50:17.278652 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 45542ns
  297. I0425 21:50:17.278794 22029 hierarchical.cpp:1862] No allocations performed
  298. I0425 21:50:17.278810 22029 hierarchical.cpp:1952] No inverse offers to send out!
  299. I0425 21:50:17.278818 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 35265ns
  300. I0425 21:50:17.278959 22026 hierarchical.cpp:1862] No allocations performed
  301. I0425 21:50:17.278976 22026 hierarchical.cpp:1952] No inverse offers to send out!
  302. I0425 21:50:17.278985 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 34735ns
  303. I0425 21:50:17.279157 22029 hierarchical.cpp:1862] No allocations performed
  304. I0425 21:50:17.279181 22029 hierarchical.cpp:1952] No inverse offers to send out!
  305. I0425 21:50:17.279189 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 46008ns
  306. I0425 21:50:17.279326 22028 hierarchical.cpp:1862] No allocations performed
  307. I0425 21:50:17.279347 22028 hierarchical.cpp:1952] No inverse offers to send out!
  308. I0425 21:50:17.279362 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 43918ns
  309. I0425 21:50:17.279516 22031 hierarchical.cpp:1862] No allocations performed
  310. I0425 21:50:17.279534 22031 hierarchical.cpp:1952] No inverse offers to send out!
  311. I0425 21:50:17.279543 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 37804ns
  312. I0425 21:50:17.279672 22030 hierarchical.cpp:1862] No allocations performed
  313. I0425 21:50:17.279692 22030 hierarchical.cpp:1952] No inverse offers to send out!
  314. I0425 21:50:17.279700 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 39718ns
  315. I0425 21:50:17.279853 22027 hierarchical.cpp:1862] No allocations performed
  316. I0425 21:50:17.279875 22027 hierarchical.cpp:1952] No inverse offers to send out!
  317. I0425 21:50:17.279884 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 68314ns
  318. I0425 21:50:17.280016 22025 hierarchical.cpp:1862] No allocations performed
  319. I0425 21:50:17.280038 22025 hierarchical.cpp:1952] No inverse offers to send out!
  320. I0425 21:50:17.280050 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 45992ns
  321. I0425 21:50:17.280179 22026 hierarchical.cpp:1862] No allocations performed
  322. I0425 21:50:17.280200 22026 hierarchical.cpp:1952] No inverse offers to send out!
  323. I0425 21:50:17.280215 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 44652ns
  324. I0425 21:50:17.280385 22032 hierarchical.cpp:1862] No allocations performed
  325. I0425 21:50:17.280403 22032 hierarchical.cpp:1952] No inverse offers to send out!
  326. I0425 21:50:17.280411 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 42307ns
  327. I0425 21:50:17.280541 22031 hierarchical.cpp:1862] No allocations performed
  328. I0425 21:50:17.280560 22031 hierarchical.cpp:1952] No inverse offers to send out!
  329. I0425 21:50:17.280572 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 41876ns
  330. I0425 21:50:17.280745 22028 hierarchical.cpp:1862] No allocations performed
  331. I0425 21:50:17.280766 22028 hierarchical.cpp:1952] No inverse offers to send out!
  332. I0425 21:50:17.280779 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 43623ns
  333. I0425 21:50:17.280961 22025 hierarchical.cpp:1862] No allocations performed
  334. I0425 21:50:17.280982 22025 hierarchical.cpp:1952] No inverse offers to send out!
  335. I0425 21:50:17.280993 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 43204ns
  336. I0425 21:50:17.281139 22027 hierarchical.cpp:1862] No allocations performed
  337. I0425 21:50:17.281158 22027 hierarchical.cpp:1952] No inverse offers to send out!
  338. I0425 21:50:17.281167 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 37755ns
  339. I0425 21:50:17.281285 22026 hierarchical.cpp:1862] No allocations performed
  340. I0425 21:50:17.281304 22026 hierarchical.cpp:1952] No inverse offers to send out!
  341. I0425 21:50:17.281319 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 43570ns
  342. I0425 21:50:17.281458 22030 hierarchical.cpp:1862] No allocations performed
  343. I0425 21:50:17.281481 22030 hierarchical.cpp:1952] No inverse offers to send out!
  344. I0425 21:50:17.281493 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 46558ns
  345. I0425 21:50:17.281616 22029 hierarchical.cpp:1862] No allocations performed
  346. I0425 21:50:17.281635 22029 hierarchical.cpp:1952] No inverse offers to send out!
  347. I0425 21:50:17.281644 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 38196ns
  348. I0425 21:50:17.281769 22032 hierarchical.cpp:1862] No allocations performed
  349. I0425 21:50:17.281791 22032 hierarchical.cpp:1952] No inverse offers to send out!
  350. I0425 21:50:17.281802 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 44285ns
  351. I0425 21:50:17.281949 22031 hierarchical.cpp:1862] No allocations performed
  352. I0425 21:50:17.281968 22031 hierarchical.cpp:1952] No inverse offers to send out!
  353. I0425 21:50:17.281977 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 42893ns
  354. I0425 21:50:17.282109 22028 hierarchical.cpp:1862] No allocations performed
  355. I0425 21:50:17.282126 22028 hierarchical.cpp:1952] No inverse offers to send out!
  356. I0425 21:50:17.282135 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 36986ns
  357. I0425 21:50:17.282274 22026 hierarchical.cpp:1862] No allocations performed
  358. I0425 21:50:17.282292 22026 hierarchical.cpp:1952] No inverse offers to send out!
  359. I0425 21:50:17.282300 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 38525ns
  360. I0425 21:50:17.282459 22029 hierarchical.cpp:1862] No allocations performed
  361. I0425 21:50:17.282477 22029 hierarchical.cpp:1952] No inverse offers to send out!
  362. I0425 21:50:17.282486 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 38983ns
  363. I0425 21:50:17.282626 22025 hierarchical.cpp:1862] No allocations performed
  364. I0425 21:50:17.282647 22025 hierarchical.cpp:1952] No inverse offers to send out!
  365. I0425 21:50:17.282658 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 44371ns
  366. I0425 21:50:17.282793 22030 hierarchical.cpp:1862] No allocations performed
  367. I0425 21:50:17.282815 22030 hierarchical.cpp:1952] No inverse offers to send out!
  368. I0425 21:50:17.282825 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 47222ns
  369. I0425 21:50:17.282954 22032 hierarchical.cpp:1862] No allocations performed
  370. I0425 21:50:17.282973 22032 hierarchical.cpp:1952] No inverse offers to send out!
  371. I0425 21:50:17.282980 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 36078ns
  372. I0425 21:50:17.283120 22027 hierarchical.cpp:1862] No allocations performed
  373. I0425 21:50:17.283139 22027 hierarchical.cpp:1952] No inverse offers to send out!
  374. I0425 21:50:17.283148 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 38219ns
  375. I0425 21:50:17.283273 22031 hierarchical.cpp:1862] No allocations performed
  376. I0425 21:50:17.283298 22031 hierarchical.cpp:1952] No inverse offers to send out!
  377. I0425 21:50:17.283310 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 46799ns
  378. I0425 21:50:17.283438 22028 hierarchical.cpp:1862] No allocations performed
  379. I0425 21:50:17.283458 22028 hierarchical.cpp:1952] No inverse offers to send out!
  380. I0425 21:50:17.283466 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 42227ns
  381. I0425 21:50:17.283702 22026 hierarchical.cpp:1862] No allocations performed
  382. I0425 21:50:17.283720 22026 hierarchical.cpp:1952] No inverse offers to send out!
  383. I0425 21:50:17.283730 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 38830ns
  384. I0425 21:50:17.283874 22030 hierarchical.cpp:1862] No allocations performed
  385. I0425 21:50:17.283898 22030 hierarchical.cpp:1952] No inverse offers to send out!
  386. I0425 21:50:17.283908 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 47655ns
  387. I0425 21:50:17.284036 22029 hierarchical.cpp:1862] No allocations performed
  388. I0425 21:50:17.284056 22029 hierarchical.cpp:1952] No inverse offers to send out!
  389. I0425 21:50:17.284065 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 37573ns
  390. I0425 21:50:17.284198 22032 hierarchical.cpp:1862] No allocations performed
  391. I0425 21:50:17.284219 22032 hierarchical.cpp:1952] No inverse offers to send out!
  392. I0425 21:50:17.284235 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 45773ns
  393. I0425 21:50:17.284373 22027 hierarchical.cpp:1862] No allocations performed
  394. I0425 21:50:17.284396 22027 hierarchical.cpp:1952] No inverse offers to send out!
  395. I0425 21:50:17.284407 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 45710ns
  396. I0425 21:50:17.284538 22031 hierarchical.cpp:1862] No allocations performed
  397. I0425 21:50:17.284557 22031 hierarchical.cpp:1952] No inverse offers to send out!
  398. I0425 21:50:17.284565 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 37308ns
  399. I0425 21:50:17.284698 22028 hierarchical.cpp:1862] No allocations performed
  400. I0425 21:50:17.284718 22028 hierarchical.cpp:1952] No inverse offers to send out!
  401. I0425 21:50:17.284724 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 36537ns
  402. I0425 21:50:17.284847 22032 hierarchical.cpp:1862] No allocations performed
  403. I0425 21:50:17.284868 22032 hierarchical.cpp:1952] No inverse offers to send out!
  404. I0425 21:50:17.284883 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 42997ns
  405. I0425 21:50:17.285023 22027 hierarchical.cpp:1862] No allocations performed
  406. I0425 21:50:17.285042 22027 hierarchical.cpp:1952] No inverse offers to send out!
  407. I0425 21:50:17.285050 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 37080ns
  408. I0425 21:50:17.285187 22025 hierarchical.cpp:1862] No allocations performed
  409. I0425 21:50:17.285209 22025 hierarchical.cpp:1952] No inverse offers to send out!
  410. I0425 21:50:17.285220 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 45858ns
  411. I0425 21:50:17.285357 22031 hierarchical.cpp:1862] No allocations performed
  412. I0425 21:50:17.285377 22031 hierarchical.cpp:1952] No inverse offers to send out!
  413. I0425 21:50:17.285384 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 37924ns
  414. I0425 21:50:17.285522 22029 hierarchical.cpp:1862] No allocations performed
  415. I0425 21:50:17.285540 22029 hierarchical.cpp:1952] No inverse offers to send out!
  416. I0425 21:50:17.285548 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 36602ns
  417. I0425 21:50:17.285671 22026 hierarchical.cpp:1862] No allocations performed
  418. I0425 21:50:17.285689 22026 hierarchical.cpp:1952] No inverse offers to send out!
  419. I0425 21:50:17.285698 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 37499ns
  420. I0425 21:50:17.285827 22028 hierarchical.cpp:1862] No allocations performed
  421. I0425 21:50:17.285845 22028 hierarchical.cpp:1952] No inverse offers to send out!
  422. I0425 21:50:17.285854 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 37584ns
  423. I0425 21:50:17.285984 22032 hierarchical.cpp:1862] No allocations performed
  424. I0425 21:50:17.286005 22032 hierarchical.cpp:1952] No inverse offers to send out!
  425. I0425 21:50:17.286015 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 44738ns
  426. I0425 21:50:17.286154 22031 slave.cpp:5659] Current disk usage 13.64%. Max allowed age: 5.345249089930695days
  427. I0425 21:50:17.286171 22025 hierarchical.cpp:1862] No allocations performed
  428. I0425 21:50:17.286186 22025 hierarchical.cpp:1952] No inverse offers to send out!
  429. I0425 21:50:17.286218 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 58315ns
  430. I0425 21:50:17.286360 22031 hierarchical.cpp:1862] No allocations performed
  431. I0425 21:50:17.286379 22031 hierarchical.cpp:1952] No inverse offers to send out!
  432. I0425 21:50:17.286387 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 36798ns
  433. I0425 21:50:17.286531 22029 hierarchical.cpp:1862] No allocations performed
  434. I0425 21:50:17.286550 22029 hierarchical.cpp:1952] No inverse offers to send out!
  435. I0425 21:50:17.286558 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 43433ns
  436. I0425 21:50:17.286710 22032 hierarchical.cpp:1862] No allocations performed
  437. I0425 21:50:17.286731 22032 hierarchical.cpp:1952] No inverse offers to send out!
  438. I0425 21:50:17.286746 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 45865ns
  439. I0425 21:50:17.286876 22028 hierarchical.cpp:1862] No allocations performed
  440. I0425 21:50:17.286900 22028 hierarchical.cpp:1952] No inverse offers to send out!
  441. I0425 21:50:17.286911 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 45275ns
  442. I0425 21:50:17.287076 22030 hierarchical.cpp:1862] No allocations performed
  443. I0425 21:50:17.287099 22030 hierarchical.cpp:1952] No inverse offers to send out!
  444. I0425 21:50:17.287111 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 49611ns
  445. E0425 21:50:17.287184 22025 slave.cpp:5139] Termination of executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 failed: Failed to kill all processes in the container: Timed out after 1mins
  446. I0425 21:50:17.287220 22025 slave.cpp:4215] Handling status update TASK_FAILED (UUID: a689f3f1-2bc2-49b5-be2f-f536306315c3) for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 from @0.0.0.0:0
  447. I0425 21:50:17.287284 22025 slave.cpp:5261] Cleaning up executor '6b25338c-478d-4a54-a191-8ea188d0ae8c' of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 at executor(1)@10.51.168.165:36023
  448. I0425 21:50:17.287452 22027 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/runs/256cb6d6-c74c-47b1-bc07-463f2db990ef' for gc 6.9999969862963days in the future
  449. I0425 21:50:17.287542 22025 slave.cpp:5349] Cleaning up framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  450. I0425 21:50:17.287621 22030 status_update_manager.cpp:285] Closing status update streams for framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  451. I0425 21:50:17.287639 22030 status_update_manager.cpp:531] Cleaning up status update stream for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  452. I0425 21:50:17.287652 22027 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c' for gc 6.9999969862963days in the future
  453. W0425 21:50:17.287715 22031 containerizer.cpp:2071] Skipping status for container 256cb6d6-c74c-47b1-bc07-463f2db990ef because: Container does not exist
  454. I0425 21:50:17.287726 22025 slave.cpp:790] Agent terminating
  455. I0425 21:50:17.287735 22027 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c/runs/256cb6d6-c74c-47b1-bc07-463f2db990ef' for gc 6.9999969862963days in the future
  456. I0425 21:50:17.287760 22027 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000/executors/6b25338c-478d-4a54-a191-8ea188d0ae8c' for gc 6.9999969862963days in the future
  457. I0425 21:50:17.287786 22027 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000' for gc 6.9999969862963days in the future
  458. I0425 21:50:17.287809 22027 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_CleanupExecutor_Om3py5/meta/slaves/9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0/frameworks/9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000' for gc 6.9999969862963days in the future
  459. I0425 21:50:17.288036 22025 hierarchical.cpp:1862] No allocations performed
  460. I0425 21:50:17.288056 22025 hierarchical.cpp:1952] No inverse offers to send out!
  461. I0425 21:50:17.288064 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 39303ns
  462. I0425 21:50:17.288206 22032 hierarchical.cpp:1862] No allocations performed
  463. I0425 21:50:17.288226 22032 hierarchical.cpp:1952] No inverse offers to send out!
  464. I0425 21:50:17.288234 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 53261ns
  465. I0425 21:50:17.288352 22027 hierarchical.cpp:1862] No allocations performed
  466. I0425 21:50:17.288377 22027 hierarchical.cpp:1952] No inverse offers to send out!
  467. I0425 21:50:17.288390 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 48297ns
  468. I0425 21:50:17.288548 22031 hierarchical.cpp:1862] No allocations performed
  469. I0425 21:50:17.288571 22031 hierarchical.cpp:1952] No inverse offers to send out!
  470. I0425 21:50:17.288583 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 51567ns
  471. I0425 21:50:17.288740 22026 hierarchical.cpp:1862] No allocations performed
  472. I0425 21:50:17.288758 22026 hierarchical.cpp:1952] No inverse offers to send out!
  473. I0425 21:50:17.288767 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 45195ns
  474. I0425 21:50:17.288915 22030 hierarchical.cpp:1862] No allocations performed
  475. I0425 21:50:17.288934 22030 hierarchical.cpp:1952] No inverse offers to send out!
  476. I0425 21:50:17.288944 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 43429ns
  477. I0425 21:50:17.289079 22025 hierarchical.cpp:1862] No allocations performed
  478. I0425 21:50:17.289098 22025 hierarchical.cpp:1952] No inverse offers to send out!
  479. I0425 21:50:17.289106 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 37429ns
  480. I0425 21:50:17.289234 22032 hierarchical.cpp:1862] No allocations performed
  481. I0425 21:50:17.289258 22032 hierarchical.cpp:1952] No inverse offers to send out!
  482. I0425 21:50:17.289266 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 47300ns
  483. I0425 21:50:17.289386 22029 hierarchical.cpp:1862] No allocations performed
  484. I0425 21:50:17.289410 22029 hierarchical.cpp:1952] No inverse offers to send out!
  485. I0425 21:50:17.289420 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 51052ns
  486. I0425 21:50:17.289582 22031 hierarchical.cpp:1862] No allocations performed
  487. I0425 21:50:17.289603 22031 hierarchical.cpp:1952] No inverse offers to send out!
  488. I0425 21:50:17.289614 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 44157ns
  489. I0425 21:50:17.289753 22030 hierarchical.cpp:1862] No allocations performed
  490. I0425 21:50:17.289772 22030 hierarchical.cpp:1952] No inverse offers to send out!
  491. I0425 21:50:17.289780 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 38674ns
  492. I0425 21:50:17.289912 22026 hierarchical.cpp:1862] No allocations performed
  493. I0425 21:50:17.289932 22026 hierarchical.cpp:1952] No inverse offers to send out!
  494. I0425 21:50:17.289939 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 39274ns
  495. I0425 21:50:17.290072 22025 hierarchical.cpp:1862] No allocations performed
  496. I0425 21:50:17.290094 22025 hierarchical.cpp:1952] No inverse offers to send out!
  497. I0425 21:50:17.290109 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 51565ns
  498. I0425 21:50:17.290242 22027 hierarchical.cpp:1862] No allocations performed
  499. I0425 21:50:17.290263 22027 hierarchical.cpp:1952] No inverse offers to send out!
  500. I0425 21:50:17.290272 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 45375ns
  501. I0425 21:50:17.290395 22032 hierarchical.cpp:1862] No allocations performed
  502. I0425 21:50:17.290418 22032 hierarchical.cpp:1952] No inverse offers to send out!
  503. I0425 21:50:17.290428 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 48105ns
  504. I0425 21:50:17.290540 22028 hierarchical.cpp:1862] No allocations performed
  505. I0425 21:50:17.290565 22028 hierarchical.cpp:1952] No inverse offers to send out!
  506. I0425 21:50:17.290578 22028 hierarchical.cpp:1446] Performed allocation for 1 agents in 50130ns
  507. I0425 21:50:17.290705 22029 hierarchical.cpp:1862] No allocations performed
  508. I0425 21:50:17.290724 22029 hierarchical.cpp:1952] No inverse offers to send out!
  509. I0425 21:50:17.290732 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 38080ns
  510. I0425 21:50:17.290868 22025 hierarchical.cpp:1862] No allocations performed
  511. I0425 21:50:17.290886 22025 hierarchical.cpp:1952] No inverse offers to send out!
  512. I0425 21:50:17.290894 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 42088ns
  513. I0425 21:50:17.291020 22030 hierarchical.cpp:1862] No allocations performed
  514. I0425 21:50:17.291041 22030 hierarchical.cpp:1952] No inverse offers to send out!
  515. I0425 21:50:17.291056 22030 hierarchical.cpp:1446] Performed allocation for 1 agents in 45391ns
  516. I0425 21:50:17.291184 22026 hierarchical.cpp:1862] No allocations performed
  517. I0425 21:50:17.291205 22026 hierarchical.cpp:1952] No inverse offers to send out!
  518. I0425 21:50:17.291215 22026 hierarchical.cpp:1446] Performed allocation for 1 agents in 45927ns
  519. I0425 21:50:17.291335 22027 hierarchical.cpp:1862] No allocations performed
  520. I0425 21:50:17.291360 22027 hierarchical.cpp:1952] No inverse offers to send out!
  521. I0425 21:50:17.291373 22027 hierarchical.cpp:1446] Performed allocation for 1 agents in 47461ns
  522. I0425 21:50:17.291501 22032 hierarchical.cpp:1862] No allocations performed
  523. I0425 21:50:17.291520 22032 hierarchical.cpp:1952] No inverse offers to send out!
  524. I0425 21:50:17.291535 22032 hierarchical.cpp:1446] Performed allocation for 1 agents in 45777ns
  525. I0425 21:50:17.291682 22031 hierarchical.cpp:1862] No allocations performed
  526. I0425 21:50:17.291703 22031 hierarchical.cpp:1952] No inverse offers to send out!
  527. I0425 21:50:17.291714 22031 hierarchical.cpp:1446] Performed allocation for 1 agents in 44927ns
  528. I0425 21:50:17.291873 22025 hierarchical.cpp:1862] No allocations performed
  529. I0425 21:50:17.291891 22025 hierarchical.cpp:1952] No inverse offers to send out!
  530. I0425 21:50:17.291900 22025 hierarchical.cpp:1446] Performed allocation for 1 agents in 40353ns
  531. I0425 21:50:17.292070 22029 hierarchical.cpp:1862] No allocations performed
  532. I0425 21:50:17.292094 22029 hierarchical.cpp:1952] No inverse offers to send out!
  533. I0425 21:50:17.292093 22030 master.cpp:6478] Marking agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165) unreachable: health check timed out
  534. I0425 21:50:17.292126 22029 hierarchical.cpp:1446] Performed allocation for 1 agents in 69219ns
  535. I0425 21:50:17.292256 22030 registrar.cpp:493] Applied 1 operations in 20382ns; attempting to update the registry
  536. I0425 21:50:17.292551 22029 registrar.cpp:550] Successfully updated the registry in 0ns
  537. I0425 21:50:17.292624 22027 master.cpp:6526] Marked agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165) unreachable: health check timed out
  538. I0425 21:50:17.292680 22027 master.cpp:8350] Updating the state of task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (latest state: TASK_LOST, status update state: TASK_LOST)
  539. I0425 21:50:17.292775 22025 hierarchical.cpp:560] Removed agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0
  540. I0425 21:50:17.292826 22027 master.cpp:8444] Removing task 6b25338c-478d-4a54-a191-8ea188d0ae8c with resources cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 on agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 at slave(457)@10.51.168.165:33073 (ip-10-51-168-165)
  541. I0425 21:50:17.292912 22027 master.cpp:6314] Sending status update TASK_LOST for task 6b25338c-478d-4a54-a191-8ea188d0ae8c of framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 'Agent ip-10-51-168-165 is unreachable: health check timed out'
  542. I0425 21:50:17.293057 22029 sched.cpp:1041] Scheduler::statusUpdate took 17032ns
  543. I0425 21:50:17.293095 22027 master.cpp:2080] Notifying framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (default) at [email protected]:33073 of lost agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0 (ip-10-51-168-165)
  544. I0425 21:50:17.293195 22031 sched.cpp:1103] Lost agent 9a71c48e-7660-46f3-9ce6-cde41efba5e3-S0
  545. I0425 21:50:17.293228 22031 sched.cpp:1114] Scheduler::slaveLost took 11043ns
  546. I0425 21:50:17.293347 22006 sched.cpp:2021] Asked to stop the driver
  547. I0425 21:50:17.293411 22027 sched.cpp:1203] Stopping framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  548. I0425 21:50:17.293515 22032 master.cpp:7770] Processing TEARDOWN call for framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (default) at [email protected]:33073
  549. I0425 21:50:17.293541 22032 master.cpp:7782] Removing framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (default) at [email protected]:33073
  550. I0425 21:50:17.293557 22032 master.cpp:3162] Deactivating framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000 (default) at [email protected]:33073
  551. I0425 21:50:17.293644 22029 hierarchical.cpp:376] Deactivated framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  552. I0425 21:50:17.293732 22029 hierarchical.cpp:323] Removed framework 9a71c48e-7660-46f3-9ce6-cde41efba5e3-0000
  553. ../../src/tests/cluster.cpp:580: Failure
  554. Value of: containers->empty()
  555. Actual: false
  556. Expected: true
  557. Failed to destroy containers: { 256cb6d6-c74c-47b1-bc07-463f2db990ef }
  558. I0425 21:50:17.296525 22006 master.cpp:1157] Master terminating
  559. I0425 21:50:17.327363 22031 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef
  560. I0425 21:50:17.328301 22031 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef after 909056ns
  561. I0425 21:50:17.329176 22031 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef
  562. I0425 21:50:17.330118 22030 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d/256cb6d6-c74c-47b1-bc07-463f2db990ef after 914944ns
  563. I0425 21:50:17.331872 22032 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d
  564. I0425 21:50:17.433928 22031 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d after 102.022144ms
  565. I0425 21:50:17.434875 22027 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d
  566. I0425 21:50:17.435773 22031 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_f8c79038-d81c-41d6-9e43-059b4bdd974d after 869120ns
  567. [ FAILED ] SlaveRecoveryTest/0.CleanupExecutor, where TypeParam = mesos::internal::slave::MesosContainerizer (449 ms)
Advertisement
Add Comment
Please, Sign In to add comment