API tools faq
paste
Advertisement
Guest User

server1:/var/log/slurmctld.log

a guest
Jul 23rd, 2024
1,174
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Arduino 15.60 KB | None | 0 0
raw download clone embed print report
  1. [2024-07-22T14:47:32.282] debug:  slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet
  2. [2024-07-22T14:47:32.282] debug:  Log file re-opened
  3. [2024-07-22T14:47:32.284] slurmscriptd: debug:  slurmscriptd: Got ack from slurmctld, initialization successful
  4. [2024-07-22T14:47:32.284] slurmscriptd: debug:  _slurmscriptd_mainloop: started
  5. [2024-07-22T14:47:32.284] slurmscriptd: debug4: eio: handling events for 1 objects
  6. [2024-07-22T14:47:32.284] debug:  slurmctld: slurmscriptd fork()'d and initialized.
  7. [2024-07-22T14:47:32.284] slurmscriptd: debug3: Called _msg_readable
  8. [2024-07-22T14:47:32.284] debug:  _slurmctld_listener_thread: started listening to slurmscriptd
  9. [2024-07-22T14:47:32.284] debug4: eio: handling events for 1 objects
  10. [2024-07-22T14:47:32.284] debug3: Called _msg_readable
  11. [2024-07-22T14:47:32.284] slurmctld version 22.05.8 started on cluster dlabcluster
  12. [2024-07-22T14:47:32.284] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/cred_munge.so
  13. [2024-07-22T14:47:32.285] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x160508
  14. [2024-07-22T14:47:32.285] cred/munge: init: Munge credential signature plugin loaded
  15. [2024-07-22T14:47:32.285] debug3: Success.
  16. [2024-07-22T14:47:32.285] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/auth_munge.so
  17. [2024-07-22T14:47:32.285] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge authentication plugin type:auth/munge version:0x160508
  18. [2024-07-22T14:47:32.285] debug:  auth/munge: init: Munge authentication plugin loaded
  19. [2024-07-22T14:47:32.286] debug3: Success.
  20. [2024-07-22T14:47:32.286] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/select_linear.so
  21. [2024-07-22T14:47:32.286] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Linear node selection plugin type:select/linear version:0x160508
  22. [2024-07-22T14:47:32.286] debug3: Success.
  23. [2024-07-22T14:47:32.286] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/select_cons_res.so
  24. [2024-07-22T14:47:32.286] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Consumable Resources (CR) Node Selection plugin type:select/cons_res version:0x160508
  25. [2024-07-22T14:47:32.286] select/cons_res: common_init: select/cons_res loaded
  26. [2024-07-22T14:47:32.286] debug3: Success.
  27. [2024-07-22T14:47:32.286] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/select_cons_tres.so
  28. [2024-07-22T14:47:32.287] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Trackable RESources (TRES) Selection plugin type:select/cons_tres version:0x160508
  29. [2024-07-22T14:47:32.287] select/cons_tres: common_init: select/cons_tres loaded
  30. [2024-07-22T14:47:32.287] debug3: Success.
  31. [2024-07-22T14:47:32.287] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/select_cray_aries.so
  32. [2024-07-22T14:47:32.287] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Cray/Aries node selection plugin type:select/cray_aries version:0x160508
  33. [2024-07-22T14:47:32.287] select/cray_aries: init: Cray/Aries node selection plugin loaded
  34. [2024-07-22T14:47:32.287] debug3: Success.
  35. [2024-07-22T14:47:32.287] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/preempt_none.so
  36. [2024-07-22T14:47:32.287] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Preemption disabled type:preempt/none version:0x160508
  37. [2024-07-22T14:47:32.287] preempt/none: init: preempt/none loaded
  38. [2024-07-22T14:47:32.287] debug3: Success.
  39. [2024-07-22T14:47:32.287] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/acct_gather_energy_none.so
  40. [2024-07-22T14:47:32.287] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:AcctGatherEnergy NONE plugin type:acct_gather_energy/none version:0x160508
  41. [2024-07-22T14:47:32.287] debug:  acct_gather_energy/none: init: AcctGatherEnergy NONE plugin loaded
  42. [2024-07-22T14:47:32.287] debug3: Success.
  43. [2024-07-22T14:47:32.287] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/acct_gather_profile_none.so
  44. [2024-07-22T14:47:32.287] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:AcctGatherProfile NONE plugin type:acct_gather_profile/none version:0x160508
  45. [2024-07-22T14:47:32.287] debug:  acct_gather_profile/none: init: AcctGatherProfile NONE plugin loaded
  46. [2024-07-22T14:47:32.287] debug3: Success.
  47. [2024-07-22T14:47:32.287] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/acct_gather_interconnect_none.so
  48. [2024-07-22T14:47:32.288] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:AcctGatherInterconnect NONE plugin type:acct_gather_interconnect/none version:0x160508
  49. [2024-07-22T14:47:32.288] debug:  acct_gather_interconnect/none: init: AcctGatherInterconnect NONE plugin loaded
  50. [2024-07-22T14:47:32.288] debug3: Success.
  51. [2024-07-22T14:47:32.288] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/acct_gather_filesystem_none.so
  52. [2024-07-22T14:47:32.288] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:AcctGatherFilesystem NONE plugin type:acct_gather_filesystem/none version:0x160508
  53. [2024-07-22T14:47:32.288] debug:  acct_gather_filesystem/none: init: AcctGatherFilesystem NONE plugin loaded
  54. [2024-07-22T14:47:32.288] debug3: Success.
  55. [2024-07-22T14:47:32.288] debug2: No acct_gather.conf file (/etc/slurm/acct_gather.conf)
  56. [2024-07-22T14:47:32.288] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/jobacct_gather_none.so
  57. [2024-07-22T14:47:32.288] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Job accounting gather NOT_INVOKED plugin type:jobacct_gather/none version:0x160508
  58. [2024-07-22T14:47:32.288] debug:  jobacct_gather/none: init: Job accounting gather NOT_INVOKED plugin loaded
  59. [2024-07-22T14:47:32.288] debug3: Success.
  60. [2024-07-22T14:47:32.288] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/prep_script.so
  61. [2024-07-22T14:47:32.288] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Script PrEp plugin type:prep/script version:0x160508
  62. [2024-07-22T14:47:32.288] debug3: Success.
  63. [2024-07-22T14:47:32.288] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/ext_sensors_none.so
  64. [2024-07-22T14:47:32.288] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:ExtSensors NONE plugin type:ext_sensors/none version:0x160508
  65. [2024-07-22T14:47:32.289] ext_sensors/none: init: ExtSensors NONE plugin loaded
  66. [2024-07-22T14:47:32.289] debug3: Success.
  67. [2024-07-22T14:47:32.289] debug:  MPI: Loading all types
  68. [2024-07-22T14:47:32.289] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/mpi_pmix_v4.so
  69. [2024-07-22T14:47:32.289] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:PMIx plugin type:mpi/pmix_v4 version:0x160508
  70. [2024-07-22T14:47:32.290] debug:  mpi/pmix_v4: init: PMIx plugin loaded
  71. [2024-07-22T14:47:32.290] debug3: Success.
  72. [2024-07-22T14:47:32.290] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/mpi_pmi2.so
  73. [2024-07-22T14:47:32.290] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:mpi PMI2 plugin type:mpi/pmi2 version:0x160508
  74. [2024-07-22T14:47:32.290] debug3: Success.
  75. [2024-07-22T14:47:32.290] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/mpi_cray_shasta.so
  76. [2024-07-22T14:47:32.290] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:mpi Cray Shasta plugin type:mpi/cray_shasta version:0x160508
  77. [2024-07-22T14:47:32.290] debug3: Success.
  78. [2024-07-22T14:47:32.290] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/mpi_none.so
  79. [2024-07-22T14:47:32.290] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:mpi none plugin type:mpi/none version:0x160508
  80. [2024-07-22T14:47:32.290] debug3: Success.
  81. [2024-07-22T14:47:32.290] debug2: No mpi.conf file (/etc/slurm/mpi.conf)
  82. [2024-07-22T14:47:32.291] debug3: plugin_peek->_verify_syms: found Slurm plugin name:Serializer URL encoded plugin type:serializer/url-encoded version:0x160508
  83. [2024-07-22T14:47:32.292] debug3: plugin_peek->_verify_syms: found Slurm plugin name:Serializer JSON plugin type:serializer/json version:0x160508
  84. [2024-07-22T14:47:32.292] debug3: plugin_peek->_verify_syms: found Slurm plugin name:Serializer YAML plugin type:serializer/yaml version:0x160508
  85. [2024-07-22T14:47:32.293] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Serializer URL encoded plugin type:serializer/url-encoded version:0x160508
  86. [2024-07-22T14:47:32.293] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Serializer JSON plugin type:serializer/json version:0x160508
  87. [2024-07-22T14:47:32.293] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Serializer YAML plugin type:serializer/yaml version:0x160508
  88. [2024-07-22T14:47:32.293] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_none.so
  89. [2024-07-22T14:47:32.293] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Accounting storage NOT INVOKED plugin type:accounting_storage/none version:0x160508
  90. [2024-07-22T14:47:32.293] accounting_storage/none: init: Accounting storage NOT INVOKED plugin loaded
  91. [2024-07-22T14:47:32.293] debug3: Success.
  92. [2024-07-22T14:47:32.293] debug3: not enforcing associations and no list was given so we are giving a blank list
  93. [2024-07-22T14:47:32.293] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/assoc_usage` as buf_t
  94. [2024-07-22T14:47:32.293] debug3: Version in assoc_usage header is 9728
  95. [2024-07-22T14:47:32.294] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/switch_cray_aries.so
  96. [2024-07-22T14:47:32.294] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:switch Cray/Aries plugin type:switch/cray_aries version:0x160508
  97. [2024-07-22T14:47:32.294] debug:  switch Cray/Aries plugin loaded.
  98. [2024-07-22T14:47:32.294] debug3: Success.
  99. [2024-07-22T14:47:32.294] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/switch_none.so
  100. [2024-07-22T14:47:32.294] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:switch NONE plugin type:switch/none version:0x160508
  101. [2024-07-22T14:47:32.294] debug:  switch/none: init: switch NONE plugin loaded
  102. [2024-07-22T14:47:32.294] debug3: Success.
  103. [2024-07-22T14:47:32.294] debug:  Reading slurm.conf file: /etc/slurm/slurm.conf
  104. [2024-07-22T14:47:32.295] No memory enforcing mechanism configured.
  105. [2024-07-22T14:47:32.295] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/topology_none.so
  106. [2024-07-22T14:47:32.295] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:topology NONE plugin type:topology/none version:0x160508
  107. [2024-07-22T14:47:32.295] topology/none: init: topology NONE plugin loaded
  108. [2024-07-22T14:47:32.295] debug3: Success.
  109. [2024-07-22T14:47:32.295] debug:  No DownNodes
  110. [2024-07-22T14:47:32.295] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/last_config_lite` as buf_t
  111. [2024-07-22T14:47:32.295] debug3: Version in last_conf_lite header is 9728
  112. [2024-07-22T14:47:32.298] debug:  slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet
  113. [2024-07-22T14:47:32.298] debug:  Log file re-opened
  114. [2024-07-22T14:47:32.298] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/jobcomp_none.so
  115. [2024-07-22T14:47:32.298] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Job completion logging NONE plugin type:jobcomp/none version:0x160508
  116. [2024-07-22T14:47:32.298] debug3: Success.
  117. [2024-07-22T14:47:32.298] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/sched_backfill.so
  118. [2024-07-22T14:47:32.299] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Slurm Backfill Scheduler plugin type:sched/backfill version:0x160508
  119. [2024-07-22T14:47:32.299] sched: Backfill scheduler plugin loaded
  120. [2024-07-22T14:47:32.299] debug3: Success.
  121. [2024-07-22T14:47:32.299] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/route_default.so
  122. [2024-07-22T14:47:32.299] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:route default plugin type:route/default version:0x160508
  123. [2024-07-22T14:47:32.299] route/default: init: route default plugin loaded
  124. [2024-07-22T14:47:32.299] debug3: Success.
  125. [2024-07-22T14:47:32.299] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/node_state` as buf_t
  126. [2024-07-22T14:47:32.300] debug3: Version string in node_state header is PROTOCOL_VERSION
  127. [2024-07-22T14:47:32.300] Recovered state of 3 nodes
  128. [2024-07-22T14:47:32.300] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/job_state` as buf_t
  129. [2024-07-22T14:47:32.300] debug3: Version string in job_state header is PROTOCOL_VERSION
  130. [2024-07-22T14:47:32.300] debug3: Job id in job_state header is 255
  131. [2024-07-22T14:47:32.300] debug3: Set job_id_sequence to 255
  132. [2024-07-22T14:47:32.300] Recovered information about 0 jobs
  133. [2024-07-22T14:47:32.300] select/cons_tres: select_p_node_init: select/cons_tres SelectTypeParameters not specified, using default value: CR_Core_Memory
  134. [2024-07-22T14:47:32.300] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions
  135. [2024-07-22T14:47:32.301] debug:  Updating partition uid access list
  136. [2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/resv_state` as buf_t
  137. [2024-07-22T14:47:32.301] debug3: Version string in resv_state header is PROTOCOL_VERSION
  138. [2024-07-22T14:47:32.301] Recovered state of 0 reservations
  139. [2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/trigger_state` as buf_t
  140. [2024-07-22T14:47:32.301] State of 0 triggers recovered
  141. [2024-07-22T14:47:32.301] read_slurm_conf: backup_controller not specified
  142. [2024-07-22T14:47:32.301] select/cons_tres: select_p_reconfigure: select/cons_tres: reconfigure
  143. [2024-07-22T14:47:32.301] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions
  144. [2024-07-22T14:47:32.301] debug:  power_save module disabled, SuspendTime < 0
  145. [2024-07-22T14:47:32.301] Running as primary controller
  146. [2024-07-22T14:47:32.301] debug:  No backup controllers, not launching heartbeat.
  147. [2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/priority_basic.so
  148. [2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Priority BASIC plugin type:priority/basic version:0x160508
  149. [2024-07-22T14:47:32.301] debug:  priority/basic: init: Priority BASIC plugin loaded
  150. [2024-07-22T14:47:32.301] debug3: Success.
  151. [2024-07-22T14:47:32.301] No parameter for mcs plugin, default values set
  152. [2024-07-22T14:47:32.301] mcs: MCSParameters = (null). ondemand set.
  153. [2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/mcs_none.so
  154. [2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:mcs none plugin type:mcs/none version:0x160508
  155. [2024-07-22T14:47:32.301] debug:  mcs/none: init: mcs none plugin loaded
  156. [2024-07-22T14:47:32.301] debug3: Success.
  157. [2024-07-22T14:47:32.302] debug3: _slurmctld_rpc_mgr pid = 3159324
  158. [2024-07-22T14:47:32.302] debug3: _slurmctld_background pid = 3159324
  159. [2024-07-22T14:47:32.302] error: Error binding slurm stream socket: Address already in use
  160. [2024-07-22T14:47:32.302] fatal: slurm_init_msg_engine_port error Address already in use
  161. [2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _handle_close
  162. [2024-07-22T14:47:32.304] slurmscriptd: debug4: eio: handling events for 1 objects
  163. [2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _msg_readable
  164. [2024-07-22T14:47:32.304] slurmscriptd: debug:  _slurmscriptd_mainloop: finished
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!