API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Dec 17th, 2021
70
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 144.05 KB | None | 0 0
raw download clone embed print report
  1. Dec 17 08:50:32 foreman.com pulpcore-worker-2[19128]: sub_objs
  2. Dec 17 08:50:32 foreman.com pulpcore-worker-2[19128]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  3. Dec 17 08:50:33 foreman.com systemd[1]: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
  4. Dec 17 08:50:33 foreman.com systemd[1]: Unit pulpcore-worker@2.service entered failed state.
  5. Dec 17 08:50:33 foreman.com systemd[1]: pulpcore-worker@2.service failed.
  6. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  7. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: Traceback (most recent call last):
  8. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  9. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  10. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  11. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: return self.main(*args, **kwargs)
  12. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  13. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: rv = self.invoke(ctx)
  14. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  15. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: return ctx.invoke(self.callback, **ctx.params)
  16. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  17. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: return callback(*args, **kwargs)
  18. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  19. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: NewPulpWorker().run_forever()
  20. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  21. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: self.sleep()
  22. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  23. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: self.beat()
  24. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  25. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: self.worker_cleanup()
  26. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  27. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: worker.delete()
  28. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  29. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: value = super().delete(*args, **kwargs)
  30. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  31. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: collector.collect([self], keep_parents=keep_parents)
  32. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  33. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  34. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  35. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: source_attr=field.name, nullable=field.null)
  36. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  37. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  38. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  39. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: sub_objs
  40. Dec 17 08:50:35 foreman.com pulpcore-worker-3[21846]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  41. Dec 17 08:50:35 foreman.com systemd[1]: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
  42. Dec 17 08:50:35 foreman.com systemd[1]: Unit pulpcore-worker@3.service entered failed state.
  43. Dec 17 08:50:35 foreman.com systemd[1]: pulpcore-worker@3.service failed.
  44. Dec 17 08:50:36 foreman.com systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
  45. Dec 17 08:50:36 foreman.com systemd[1]: Stopped Pulp Worker.
  46. Dec 17 08:50:36 foreman.com systemd[1]: Started Pulp Worker.
  47. Dec 17 08:50:39 foreman.com systemd[1]: pulpcore-worker@3.service holdoff time over, scheduling restart.
  48. Dec 17 08:50:39 foreman.com systemd[1]: Stopped Pulp Worker.
  49. Dec 17 08:50:39 foreman.com systemd[1]: Started Pulp Worker.
  50. Dec 17 08:50:41 foreman.com pulpcore-worker-2[24922]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  51. Dec 17 08:50:41 foreman.com pulpcore-worker-2[24922]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '24922@foreman.com' discovered
  52. Dec 17 08:50:43 foreman.com pulpcore-worker-3[24928]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  53. Dec 17 08:50:43 foreman.com pulpcore-worker-3[24928]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '24928@foreman.com' discovered
  54. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 19128@foreman.com.
  55. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 21846@foreman.com.
  56. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  57. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: Traceback (most recent call last):
  58. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  59. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  60. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  61. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: return self.main(*args, **kwargs)
  62. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  63. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: rv = self.invoke(ctx)
  64. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  65. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: return ctx.invoke(self.callback, **ctx.params)
  66. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  67. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: return callback(*args, **kwargs)
  68. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  69. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: NewPulpWorker().run_forever()
  70. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  71. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: self.sleep()
  72. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  73. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: self.beat()
  74. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  75. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: self.worker_cleanup()
  76. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  77. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: worker.delete()
  78. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  79. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: value = super().delete(*args, **kwargs)
  80. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  81. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: collector.collect([self], keep_parents=keep_parents)
  82. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  83. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  84. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  85. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: source_attr=field.name, nullable=field.null)
  86. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  87. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  88. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  89. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: sub_objs
  90. Dec 17 08:54:01 foreman.com pulpcore-worker-1[23654]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  91. Dec 17 08:54:01 foreman.com systemd[1]: pulpcore-worker@1.service: main process exited, code=exited, status=1/FAILURE
  92. Dec 17 08:54:01 foreman.com systemd[1]: Unit pulpcore-worker@1.service entered failed state.
  93. Dec 17 08:54:01 foreman.com systemd[1]: pulpcore-worker@1.service failed.
  94. Dec 17 08:54:04 foreman.com systemd[1]: pulpcore-worker@1.service holdoff time over, scheduling restart.
  95. Dec 17 08:54:04 foreman.com systemd[1]: Stopped Pulp Worker.
  96. Dec 17 08:54:04 foreman.com systemd[1]: Started Pulp Worker.
  97. Dec 17 08:54:09 foreman.com pulpcore-worker-1[26202]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  98. Dec 17 08:54:09 foreman.com pulpcore-worker-1[26202]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '26202@foreman.com' discovered
  99. Dec 17 09:00:11 foreman.com pulpcore-worker-1[26202]: pulp [bb9c98d8-ac86-4ed9-aae7-20a347d18485]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 558300e4-03bf-46c5-8c89-31ea58033d1a
  100. Dec 17 09:00:11 foreman.com pulpcore-worker-1[26202]: pulp [bb9c98d8-ac86-4ed9-aae7-20a347d18485]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 558300e4-03bf-46c5-8c89-31ea58033d1a
  101. Dec 17 09:00:12 foreman.com pulpcore-worker-4[23079]: pulp [bb9c98d8-ac86-4ed9-aae7-20a347d18485]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 5c427f6c-0dd2-4889-94c2-1e310b3275c3
  102. Dec 17 09:00:12 foreman.com pulpcore-worker-4[23079]: pulp [bb9c98d8-ac86-4ed9-aae7-20a347d18485]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=Red_Hat_Enterprise_Linux_7_Workstation_-_Optional_RPMs_x86_64_7_9-23578862 remote=Red_Hat_Enterprise_Linux_7_Workstation_-_Optional_RPMs_x86_64_7_9-23577182
  103. Dec 17 09:01:12 foreman.com pulpcore-worker-4[23079]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 23654@foreman.com.
  104. Dec 17 09:01:13 foreman.com pulpcore-worker-4[23079]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  105. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 23079@foreman.com.
  106. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  107. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: Traceback (most recent call last):
  108. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  109. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  110. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  111. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: return self.main(*args, **kwargs)
  112. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  113. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: rv = self.invoke(ctx)
  114. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  115. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: return ctx.invoke(self.callback, **ctx.params)
  116. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  117. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: return callback(*args, **kwargs)
  118. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  119. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: NewPulpWorker().run_forever()
  120. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  121. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: self.sleep()
  122. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  123. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: self.beat()
  124. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  125. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: self.worker_cleanup()
  126. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  127. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: worker.delete()
  128. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  129. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: value = super().delete(*args, **kwargs)
  130. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  131. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: collector.collect([self], keep_parents=keep_parents)
  132. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  133. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  134. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  135. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: source_attr=field.name, nullable=field.null)
  136. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  137. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  138. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  139. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: sub_objs
  140. Dec 17 09:05:13 foreman.com pulpcore-worker-2[24922]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  141. Dec 17 09:05:13 foreman.com systemd[1]: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
  142. Dec 17 09:05:13 foreman.com systemd[1]: Unit pulpcore-worker@2.service entered failed state.
  143. Dec 17 09:05:13 foreman.com systemd[1]: pulpcore-worker@2.service failed.
  144. Dec 17 09:05:17 foreman.com systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
  145. Dec 17 09:05:17 foreman.com systemd[1]: Stopped Pulp Worker.
  146. Dec 17 09:05:17 foreman.com systemd[1]: Started Pulp Worker.
  147. Dec 17 09:05:22 foreman.com pulpcore-worker-2[31213]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  148. Dec 17 09:05:22 foreman.com pulpcore-worker-2[31213]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '31213@foreman.com' discovered
  149. Dec 17 09:05:22 foreman.com pulpcore-worker-2[31213]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Cleaning up task 5c427f6c-0dd2-4889-94c2-1e310b3275c3 and marking as failed. Reason: Worker has gone missing.
  150. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: Process Process-1:
  151. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: Traceback (most recent call last):
  152. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
  153. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: return self.connection.commit()
  154. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: psycopg2.errors.ForeignKeyViolation: insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
  155. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: DETAIL: Key (version_added_id)=(ea701cc9-6e33-48cc-932f-902daf4b3acf) is not present in table "core_repositoryversion".
  156. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: The above exception was the direct cause of the following exception:
  157. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: Traceback (most recent call last):
  158. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
  159. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: loop.run_until_complete(pipeline)
  160. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
  161. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: return future.result()
  162. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
  163. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: await asyncio.gather(*futures)
  164. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
  165. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: await self.run()
  166. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 284, in run
  167. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: self.new_version.add_content(Content.objects.filter(pk__in=to_add))
  168. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 764, in add_content
  169. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: RepositoryContent.objects.bulk_create(repo_content)
  170. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
  171. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: return getattr(self.get_queryset(), name)(*args, **kwargs)
  172. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 480, in bulk_create
  173. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: obj_without_pk._state.db = self.db
  174. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/transaction.py", line 240, in __exit__
  175. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: connection.commit()
  176. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 262, in commit
  177. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: self._commit()
  178. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
  179. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: return self.connection.commit()
  180. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
  181. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: raise dj_exc_value.with_traceback(traceback) from exc_value
  182. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
  183. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: return self.connection.commit()
  184. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: django.db.utils.IntegrityError: insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
  185. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: DETAIL: Key (version_added_id)=(ea701cc9-6e33-48cc-932f-902daf4b3acf) is not present in table "core_repositoryversion".
  186. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: During handling of the above exception, another exception occurred:
  187. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: Traceback (most recent call last):
  188. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
  189. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: result = func(*args, **kwargs)
  190. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 490, in synchronize
  191. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: version = dv.create()
  192. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
  193. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: loop.run_until_complete(pipeline)
  194. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib64/python3.6/tempfile.py", line 809, in __exit__
  195. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: self.cleanup()
  196. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib64/python3.6/tempfile.py", line 813, in cleanup
  197. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: _shutil.rmtree(self.name)
  198. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib64/python3.6/shutil.py", line 477, in rmtree
  199. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: onerror(os.lstat, path, sys.exc_info())
  200. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib64/python3.6/shutil.py", line 475, in rmtree
  201. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: orig_st = os.lstat(path)
  202. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: FileNotFoundError: [Errno 2] No such file or directory: './tmplarwxjfb'
  203. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: During handling of the above exception, another exception occurred:
  204. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: Traceback (most recent call last):
  205. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
  206. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: self.run()
  207. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
  208. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: self._target(*self._args, **self._kwargs)
  209. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 325, in _perform_task
  210. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: task.set_failed(exc, tb)
  211. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/task.py", line 389, in set_failed
  212. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: raise RuntimeError("Attempt to set a finished task to failed.")
  213. Dec 17 09:05:26 foreman.com pulpcore-worker-4[23079]: RuntimeError: Attempt to set a finished task to failed.
  214. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: Traceback (most recent call last):
  215. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  216. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  217. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  218. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: return self.main(*args, **kwargs)
  219. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  220. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: rv = self.invoke(ctx)
  221. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  222. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: return ctx.invoke(self.callback, **ctx.params)
  223. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  224. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: return callback(*args, **kwargs)
  225. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  226. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: NewPulpWorker().run_forever()
  227. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 273, in run_forever
  228. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: self.supervise_task(task)
  229. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 223, in supervise_task
  230. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: self.beat()
  231. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  232. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: self.worker_cleanup()
  233. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  234. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: worker.delete()
  235. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  236. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: value = super().delete(*args, **kwargs)
  237. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  238. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: collector.collect([self], keep_parents=keep_parents)
  239. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  240. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  241. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  242. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: source_attr=field.name, nullable=field.null)
  243. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  244. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  245. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  246. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: sub_objs
  247. Dec 17 09:05:27 foreman.com pulpcore-worker-4[23079]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  248. Dec 17 09:05:27 foreman.com systemd[1]: pulpcore-worker@4.service: main process exited, code=exited, status=1/FAILURE
  249. Dec 17 09:05:27 foreman.com systemd[1]: Unit pulpcore-worker@4.service entered failed state.
  250. Dec 17 09:05:27 foreman.com systemd[1]: pulpcore-worker@4.service failed.
  251. Dec 17 09:05:30 foreman.com systemd[1]: pulpcore-worker@4.service holdoff time over, scheduling restart.
  252. Dec 17 09:05:30 foreman.com systemd[1]: Stopped Pulp Worker.
  253. Dec 17 09:05:30 foreman.com systemd[1]: Started Pulp Worker.
  254. Dec 17 09:05:36 foreman.com pulpcore-worker-4[31304]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  255. Dec 17 09:05:36 foreman.com pulpcore-worker-4[31304]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '31304@foreman.com' discovered
  256. Dec 17 09:05:38 foreman.com pulpcore-worker-1[26202]: pulp [bb9c98d8-ac86-4ed9-aae7-20a347d18485]: pulpcore.tasking.pulpcore_worker:INFO: Starting task cbe7a11e-49ab-4f55-b568-20340a9b00e3
  257. Dec 17 09:05:38 foreman.com pulpcore-worker-1[26202]: pulp [bb9c98d8-ac86-4ed9-aae7-20a347d18485]: pulpcore.tasking.pulpcore_worker:INFO: Task completed cbe7a11e-49ab-4f55-b568-20340a9b00e3
  258. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 24922@foreman.com.
  259. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  260. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: Traceback (most recent call last):
  261. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  262. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  263. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  264. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: return self.main(*args, **kwargs)
  265. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  266. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: rv = self.invoke(ctx)
  267. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  268. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: return ctx.invoke(self.callback, **ctx.params)
  269. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  270. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: return callback(*args, **kwargs)
  271. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  272. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: NewPulpWorker().run_forever()
  273. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  274. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: self.sleep()
  275. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  276. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: self.beat()
  277. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  278. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: self.worker_cleanup()
  279. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  280. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: worker.delete()
  281. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  282. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: value = super().delete(*args, **kwargs)
  283. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  284. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: collector.collect([self], keep_parents=keep_parents)
  285. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  286. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  287. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  288. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: source_attr=field.name, nullable=field.null)
  289. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  290. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  291. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  292. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: sub_objs
  293. Dec 17 09:07:39 foreman.com pulpcore-worker-3[24928]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  294. Dec 17 09:07:39 foreman.com systemd[1]: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
  295. Dec 17 09:07:39 foreman.com systemd[1]: Unit pulpcore-worker@3.service entered failed state.
  296. Dec 17 09:07:39 foreman.com systemd[1]: pulpcore-worker@3.service failed.
  297. Dec 17 09:07:42 foreman.com systemd[1]: pulpcore-worker@3.service holdoff time over, scheduling restart.
  298. Dec 17 09:07:42 foreman.com systemd[1]: Stopped Pulp Worker.
  299. Dec 17 09:07:42 foreman.com systemd[1]: Started Pulp Worker.
  300. Dec 17 09:07:47 foreman.com pulpcore-worker-3[32179]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  301. Dec 17 09:07:47 foreman.com pulpcore-worker-3[32179]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '32179@foreman.com' discovered
  302. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 24928@foreman.com.
  303. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  304. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: Traceback (most recent call last):
  305. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  306. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  307. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  308. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: return self.main(*args, **kwargs)
  309. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  310. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: rv = self.invoke(ctx)
  311. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  312. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: return ctx.invoke(self.callback, **ctx.params)
  313. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  314. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: return callback(*args, **kwargs)
  315. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  316. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: NewPulpWorker().run_forever()
  317. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  318. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: self.sleep()
  319. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  320. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: self.beat()
  321. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  322. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: self.worker_cleanup()
  323. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  324. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: worker.delete()
  325. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  326. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: value = super().delete(*args, **kwargs)
  327. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  328. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: collector.collect([self], keep_parents=keep_parents)
  329. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  330. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  331. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  332. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: source_attr=field.name, nullable=field.null)
  333. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  334. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  335. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  336. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: sub_objs
  337. Dec 17 09:09:19 foreman.com pulpcore-worker-1[26202]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  338. Dec 17 09:09:19 foreman.com systemd[1]: pulpcore-worker@1.service: main process exited, code=exited, status=1/FAILURE
  339. Dec 17 09:09:19 foreman.com systemd[1]: Unit pulpcore-worker@1.service entered failed state.
  340. Dec 17 09:09:19 foreman.com systemd[1]: pulpcore-worker@1.service failed.
  341. Dec 17 09:09:23 foreman.com systemd[1]: pulpcore-worker@1.service holdoff time over, scheduling restart.
  342. Dec 17 09:09:23 foreman.com systemd[1]: Stopped Pulp Worker.
  343. Dec 17 09:09:23 foreman.com systemd[1]: Started Pulp Worker.
  344. Dec 17 09:09:27 foreman.com pulpcore-worker-1[32857]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  345. Dec 17 09:09:27 foreman.com pulpcore-worker-1[32857]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '32857@foreman.com' discovered
  346. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 26202@foreman.com.
  347. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  348. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: Traceback (most recent call last):
  349. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  350. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  351. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  352. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: return self.main(*args, **kwargs)
  353. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  354. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: rv = self.invoke(ctx)
  355. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  356. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: return ctx.invoke(self.callback, **ctx.params)
  357. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  358. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: return callback(*args, **kwargs)
  359. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  360. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: NewPulpWorker().run_forever()
  361. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  362. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: self.sleep()
  363. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  364. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: self.beat()
  365. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  366. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: self.worker_cleanup()
  367. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  368. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: worker.delete()
  369. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  370. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: value = super().delete(*args, **kwargs)
  371. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  372. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: collector.collect([self], keep_parents=keep_parents)
  373. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  374. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  375. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  376. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: source_attr=field.name, nullable=field.null)
  377. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  378. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  379. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  380. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: sub_objs
  381. Dec 17 09:10:39 foreman.com pulpcore-worker-2[31213]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  382. Dec 17 09:10:39 foreman.com systemd[1]: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
  383. Dec 17 09:10:39 foreman.com systemd[1]: Unit pulpcore-worker@2.service entered failed state.
  384. Dec 17 09:10:39 foreman.com systemd[1]: pulpcore-worker@2.service failed.
  385. Dec 17 09:10:43 foreman.com systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
  386. Dec 17 09:10:43 foreman.com systemd[1]: Stopped Pulp Worker.
  387. Dec 17 09:10:43 foreman.com systemd[1]: Started Pulp Worker.
  388. Dec 17 09:10:47 foreman.com pulpcore-worker-2[33539]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  389. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  390. Dec 17 09:10:47 foreman.com pulpcore-worker-2[33539]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '33539@foreman.com' discovered
  391. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: Traceback (most recent call last):
  392. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  393. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  394. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  395. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: return self.main(*args, **kwargs)
  396. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  397. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: rv = self.invoke(ctx)
  398. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  399. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: return ctx.invoke(self.callback, **ctx.params)
  400. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  401. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: return callback(*args, **kwargs)
  402. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  403. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: NewPulpWorker().run_forever()
  404. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  405. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: self.sleep()
  406. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  407. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: self.beat()
  408. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  409. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: self.worker_cleanup()
  410. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  411. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: worker.delete()
  412. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  413. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: value = super().delete(*args, **kwargs)
  414. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  415. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: collector.collect([self], keep_parents=keep_parents)
  416. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  417. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  418. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  419. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: source_attr=field.name, nullable=field.null)
  420. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  421. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  422. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  423. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: sub_objs
  424. Dec 17 09:10:47 foreman.com pulpcore-worker-3[32179]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  425. Dec 17 09:10:48 foreman.com systemd[1]: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
  426. Dec 17 09:10:48 foreman.com systemd[1]: Unit pulpcore-worker@3.service entered failed state.
  427. Dec 17 09:10:48 foreman.com systemd[1]: pulpcore-worker@3.service failed.
  428. Dec 17 09:10:51 foreman.com systemd[1]: pulpcore-worker@3.service holdoff time over, scheduling restart.
  429. Dec 17 09:10:51 foreman.com systemd[1]: Stopped Pulp Worker.
  430. Dec 17 09:10:51 foreman.com systemd[1]: Started Pulp Worker.
  431. Dec 17 09:10:56 foreman.com pulpcore-worker-3[33581]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  432. Dec 17 09:10:56 foreman.com pulpcore-worker-3[33581]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '33581@foreman.com' discovered
  433. Dec 17 09:12:51 foreman.com pulpcore-worker-2[33539]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: Worker '33539@foreman.com' is back online.
  434. Dec 17 09:12:54 foreman.com pulpcore-worker-1[32857]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: Worker '32857@foreman.com' is back online.
  435. Dec 17 09:12:55 foreman.com pulpcore-worker-4[31304]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: Worker '31304@foreman.com' is back online.
  436. Dec 17 09:12:56 foreman.com pulpcore-worker-3[33581]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: Worker '33581@foreman.com' is back online.
  437. Dec 17 09:15:37 foreman.com pulpcore-worker-4[31304]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 31213@foreman.com.
  438. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 32179@foreman.com.
  439. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  440. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: Traceback (most recent call last):
  441. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  442. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  443. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  444. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: return self.main(*args, **kwargs)
  445. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  446. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: rv = self.invoke(ctx)
  447. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  448. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: return ctx.invoke(self.callback, **ctx.params)
  449. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  450. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: return callback(*args, **kwargs)
  451. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  452. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: NewPulpWorker().run_forever()
  453. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  454. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: self.sleep()
  455. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  456. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: self.beat()
  457. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  458. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: self.worker_cleanup()
  459. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  460. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: worker.delete()
  461. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  462. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: value = super().delete(*args, **kwargs)
  463. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  464. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: collector.collect([self], keep_parents=keep_parents)
  465. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  466. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  467. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  468. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: source_attr=field.name, nullable=field.null)
  469. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  470. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  471. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  472. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: sub_objs
  473. Dec 17 09:15:38 foreman.com pulpcore-worker-4[31304]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  474. Dec 17 09:15:47 foreman.com systemd[1]: pulpcore-worker@4.service: main process exited, code=exited, status=1/FAILURE
  475. Dec 17 09:15:47 foreman.com systemd[1]: Unit pulpcore-worker@4.service entered failed state.
  476. Dec 17 09:15:47 foreman.com systemd[1]: pulpcore-worker@4.service failed.
  477. Dec 17 09:15:51 foreman.com systemd[1]: pulpcore-worker@4.service holdoff time over, scheduling restart.
  478. Dec 17 09:15:51 foreman.com systemd[1]: Stopped Pulp Worker.
  479. Dec 17 09:15:51 foreman.com systemd[1]: Started Pulp Worker.
  480. Dec 17 09:16:01 foreman.com pulpcore-worker-4[35748]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  481. Dec 17 09:16:01 foreman.com pulpcore-worker-4[35748]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '35748@foreman.com' discovered
  482. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 31304@foreman.com.
  483. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  484. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: Traceback (most recent call last):
  485. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  486. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  487. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  488. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: return self.main(*args, **kwargs)
  489. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  490. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: rv = self.invoke(ctx)
  491. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  492. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: return ctx.invoke(self.callback, **ctx.params)
  493. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  494. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: return callback(*args, **kwargs)
  495. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  496. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: NewPulpWorker().run_forever()
  497. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  498. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: self.sleep()
  499. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  500. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: self.beat()
  501. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  502. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: self.worker_cleanup()
  503. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  504. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: worker.delete()
  505. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  506. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: value = super().delete(*args, **kwargs)
  507. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  508. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: collector.collect([self], keep_parents=keep_parents)
  509. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  510. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  511. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  512. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: source_attr=field.name, nullable=field.null)
  513. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  514. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  515. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  516. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: sub_objs
  517. Dec 17 09:18:49 foreman.com pulpcore-worker-3[33581]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  518. Dec 17 09:18:50 foreman.com systemd[1]: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
  519. Dec 17 09:18:50 foreman.com systemd[1]: Unit pulpcore-worker@3.service entered failed state.
  520. Dec 17 09:18:50 foreman.com systemd[1]: pulpcore-worker@3.service failed.
  521. Dec 17 09:18:53 foreman.com systemd[1]: pulpcore-worker@3.service holdoff time over, scheduling restart.
  522. Dec 17 09:18:53 foreman.com systemd[1]: Stopped Pulp Worker.
  523. Dec 17 09:18:53 foreman.com systemd[1]: Started Pulp Worker.
  524. Dec 17 09:18:57 foreman.com pulpcore-worker-3[38397]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  525. Dec 17 09:18:57 foreman.com pulpcore-worker-3[38397]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '38397@foreman.com' discovered
  526. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 33581@foreman.com.
  527. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  528. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: Traceback (most recent call last):
  529. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  530. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  531. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  532. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: return self.main(*args, **kwargs)
  533. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  534. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: rv = self.invoke(ctx)
  535. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  536. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: return ctx.invoke(self.callback, **ctx.params)
  537. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  538. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: return callback(*args, **kwargs)
  539. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  540. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: NewPulpWorker().run_forever()
  541. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  542. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: self.sleep()
  543. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  544. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: self.beat()
  545. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  546. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: self.worker_cleanup()
  547. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  548. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: worker.delete()
  549. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  550. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: value = super().delete(*args, **kwargs)
  551. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  552. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: collector.collect([self], keep_parents=keep_parents)
  553. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  554. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  555. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  556. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: source_attr=field.name, nullable=field.null)
  557. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  558. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  559. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  560. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: sub_objs
  561. Dec 17 09:25:52 foreman.com pulpcore-worker-4[35748]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  562. Dec 17 09:25:53 foreman.com systemd[1]: pulpcore-worker@4.service: main process exited, code=exited, status=1/FAILURE
  563. Dec 17 09:25:53 foreman.com systemd[1]: Unit pulpcore-worker@4.service entered failed state.
  564. Dec 17 09:25:53 foreman.com systemd[1]: pulpcore-worker@4.service failed.
  565. Dec 17 09:25:56 foreman.com systemd[1]: pulpcore-worker@4.service holdoff time over, scheduling restart.
  566. Dec 17 09:25:56 foreman.com systemd[1]: Stopped Pulp Worker.
  567. Dec 17 09:25:56 foreman.com systemd[1]: Started Pulp Worker.
  568. Dec 17 09:26:01 foreman.com pulpcore-worker-4[41450]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  569. Dec 17 09:26:01 foreman.com pulpcore-worker-4[41450]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '41450@foreman.com' discovered
  570. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 35748@foreman.com.
  571. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  572. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: Traceback (most recent call last):
  573. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  574. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  575. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  576. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: return self.main(*args, **kwargs)
  577. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  578. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: rv = self.invoke(ctx)
  579. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  580. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: return ctx.invoke(self.callback, **ctx.params)
  581. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  582. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: return callback(*args, **kwargs)
  583. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  584. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: NewPulpWorker().run_forever()
  585. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  586. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: self.sleep()
  587. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  588. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: self.beat()
  589. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  590. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: self.worker_cleanup()
  591. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  592. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: worker.delete()
  593. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  594. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: value = super().delete(*args, **kwargs)
  595. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  596. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: collector.collect([self], keep_parents=keep_parents)
  597. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  598. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  599. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  600. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: source_attr=field.name, nullable=field.null)
  601. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  602. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  603. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  604. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: sub_objs
  605. Dec 17 09:26:47 foreman.com pulpcore-worker-1[32857]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  606. Dec 17 09:26:48 foreman.com systemd[1]: pulpcore-worker@1.service: main process exited, code=exited, status=1/FAILURE
  607. Dec 17 09:26:48 foreman.com systemd[1]: Unit pulpcore-worker@1.service entered failed state.
  608. Dec 17 09:26:48 foreman.com systemd[1]: pulpcore-worker@1.service failed.
  609. Dec 17 09:26:51 foreman.com systemd[1]: pulpcore-worker@1.service holdoff time over, scheduling restart.
  610. Dec 17 09:26:51 foreman.com systemd[1]: Stopped Pulp Worker.
  611. Dec 17 09:26:51 foreman.com systemd[1]: Started Pulp Worker.
  612. Dec 17 09:27:06 foreman.com pulpcore-worker-1[41628]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  613. Dec 17 09:27:16 foreman.com pulpcore-worker-1[41628]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '41628@foreman.com' discovered
  614. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 32857@foreman.com.
  615. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  616. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: Traceback (most recent call last):
  617. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  618. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  619. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  620. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: return self.main(*args, **kwargs)
  621. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  622. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: rv = self.invoke(ctx)
  623. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  624. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: return ctx.invoke(self.callback, **ctx.params)
  625. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  626. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: return callback(*args, **kwargs)
  627. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  628. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: NewPulpWorker().run_forever()
  629. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  630. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: self.sleep()
  631. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  632. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: self.beat()
  633. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  634. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: self.worker_cleanup()
  635. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  636. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: worker.delete()
  637. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  638. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: value = super().delete(*args, **kwargs)
  639. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  640. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: collector.collect([self], keep_parents=keep_parents)
  641. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  642. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  643. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  644. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: source_attr=field.name, nullable=field.null)
  645. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  646. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  647. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  648. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: sub_objs
  649. Dec 17 09:27:34 foreman.com pulpcore-worker-2[33539]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  650. Dec 17 09:27:35 foreman.com systemd[1]: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
  651. Dec 17 09:27:35 foreman.com systemd[1]: Unit pulpcore-worker@2.service entered failed state.
  652. Dec 17 09:27:35 foreman.com systemd[1]: pulpcore-worker@2.service failed.
  653. Dec 17 09:27:38 foreman.com systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
  654. Dec 17 09:27:38 foreman.com systemd[1]: Stopped Pulp Worker.
  655. Dec 17 09:27:38 foreman.com systemd[1]: Started Pulp Worker.
  656. Dec 17 09:27:53 foreman.com pulpcore-worker-2[41816]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  657. Dec 17 09:28:13 foreman.com pulpcore-worker-2[41816]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '41816@foreman.com' discovered
  658. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 33539@foreman.com.
  659. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  660. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: Traceback (most recent call last):
  661. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  662. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  663. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  664. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: return self.main(*args, **kwargs)
  665. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  666. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: rv = self.invoke(ctx)
  667. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  668. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: return ctx.invoke(self.callback, **ctx.params)
  669. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  670. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: return callback(*args, **kwargs)
  671. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  672. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: NewPulpWorker().run_forever()
  673. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  674. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: self.sleep()
  675. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  676. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: self.beat()
  677. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  678. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: self.worker_cleanup()
  679. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  680. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: worker.delete()
  681. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  682. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: value = super().delete(*args, **kwargs)
  683. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  684. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: collector.collect([self], keep_parents=keep_parents)
  685. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  686. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  687. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  688. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: source_attr=field.name, nullable=field.null)
  689. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  690. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  691. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  692. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: sub_objs
  693. Dec 17 09:32:01 foreman.com pulpcore-worker-4[41450]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  694. Dec 17 09:32:02 foreman.com systemd[1]: pulpcore-worker@4.service: main process exited, code=exited, status=1/FAILURE
  695. Dec 17 09:32:02 foreman.com systemd[1]: Unit pulpcore-worker@4.service entered failed state.
  696. Dec 17 09:32:02 foreman.com systemd[1]: pulpcore-worker@4.service failed.
  697. Dec 17 09:32:05 foreman.com systemd[1]: pulpcore-worker@4.service holdoff time over, scheduling restart.
  698. Dec 17 09:32:05 foreman.com systemd[1]: Stopped Pulp Worker.
  699. Dec 17 09:32:05 foreman.com systemd[1]: Started Pulp Worker.
  700. Dec 17 09:32:12 foreman.com pulpcore-worker-4[44976]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  701. Dec 17 09:32:12 foreman.com pulpcore-worker-4[44976]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '44976@foreman.com' discovered
  702. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 41450@foreman.com.
  703. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  704. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: Traceback (most recent call last):
  705. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  706. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  707. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  708. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: return self.main(*args, **kwargs)
  709. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  710. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: rv = self.invoke(ctx)
  711. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  712. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: return ctx.invoke(self.callback, **ctx.params)
  713. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  714. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: return callback(*args, **kwargs)
  715. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  716. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: NewPulpWorker().run_forever()
  717. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  718. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: self.sleep()
  719. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  720. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: self.beat()
  721. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  722. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: self.worker_cleanup()
  723. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  724. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: worker.delete()
  725. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  726. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: value = super().delete(*args, **kwargs)
  727. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  728. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: collector.collect([self], keep_parents=keep_parents)
  729. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  730. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  731. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  732. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: source_attr=field.name, nullable=field.null)
  733. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  734. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  735. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  736. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: sub_objs
  737. Dec 17 09:33:19 foreman.com pulpcore-worker-3[38397]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  738. Dec 17 09:33:19 foreman.com systemd[1]: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
  739. Dec 17 09:33:19 foreman.com systemd[1]: Unit pulpcore-worker@3.service entered failed state.
  740. Dec 17 09:33:19 foreman.com systemd[1]: pulpcore-worker@3.service failed.
  741. Dec 17 09:33:23 foreman.com systemd[1]: pulpcore-worker@3.service holdoff time over, scheduling restart.
  742. Dec 17 09:33:23 foreman.com systemd[1]: Stopped Pulp Worker.
  743. Dec 17 09:33:23 foreman.com systemd[1]: Started Pulp Worker.
  744. Dec 17 09:33:28 foreman.com pulpcore-worker-3[45843]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  745. Dec 17 09:33:28 foreman.com pulpcore-worker-3[45843]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '45843@foreman.com' discovered
  746. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 38397@foreman.com.
  747. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  748. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: Traceback (most recent call last):
  749. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  750. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  751. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  752. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: return self.main(*args, **kwargs)
  753. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  754. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: rv = self.invoke(ctx)
  755. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  756. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: return ctx.invoke(self.callback, **ctx.params)
  757. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  758. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: return callback(*args, **kwargs)
  759. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  760. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: NewPulpWorker().run_forever()
  761. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  762. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: self.sleep()
  763. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  764. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: self.beat()
  765. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  766. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: self.worker_cleanup()
  767. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  768. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: worker.delete()
  769. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  770. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: value = super().delete(*args, **kwargs)
  771. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  772. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: collector.collect([self], keep_parents=keep_parents)
  773. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  774. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  775. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  776. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: source_attr=field.name, nullable=field.null)
  777. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  778. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  779. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  780. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: sub_objs
  781. Dec 17 09:38:14 foreman.com pulpcore-worker-2[41816]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  782. Dec 17 09:38:15 foreman.com systemd[1]: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
  783. Dec 17 09:38:15 foreman.com systemd[1]: Unit pulpcore-worker@2.service entered failed state.
  784. Dec 17 09:38:15 foreman.com systemd[1]: pulpcore-worker@2.service failed.
  785. Dec 17 09:38:17 foreman.com pulpcore-worker-1[41628]: pulp [8b7dda27-bb25-44d8-b3de-4b90191bb982]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 8117b49e-d029-477e-a576-3c46642f9566
  786. Dec 17 09:38:17 foreman.com pulpcore-worker-1[41628]: pulp [8b7dda27-bb25-44d8-b3de-4b90191bb982]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 8117b49e-d029-477e-a576-3c46642f9566
  787. Dec 17 09:38:18 foreman.com systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
  788. Dec 17 09:38:18 foreman.com systemd[1]: Stopped Pulp Worker.
  789. Dec 17 09:38:18 foreman.com systemd[1]: Started Pulp Worker.
  790. Dec 17 09:38:18 foreman.com pulpcore-worker-4[44976]: pulp [8b7dda27-bb25-44d8-b3de-4b90191bb982]: pulpcore.tasking.pulpcore_worker:INFO: Starting task ecae8ea2-deef-4ae4-9cce-b7814e52914b
  791. Dec 17 09:38:18 foreman.com pulpcore-worker-4[44976]: pulp [8b7dda27-bb25-44d8-b3de-4b90191bb982]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=Red_Hat_Enterprise_Linux_7_Workstation_RPMs_x86_64_7_9-23585317 remote=Red_Hat_Enterprise_Linux_7_Workstation_RPMs_x86_64_7_9-23585237
  792. Dec 17 09:38:24 foreman.com pulpcore-worker-2[48408]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  793. Dec 17 09:38:24 foreman.com pulpcore-worker-2[48408]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '48408@foreman.com' discovered
  794. Dec 17 09:42:49 foreman.com pulpcore-worker-4[44976]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 41816@foreman.com.
  795. Dec 17 09:42:49 foreman.com pulpcore-worker-4[44976]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  796. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 44976@foreman.com.
  797. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  798. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: Traceback (most recent call last):
  799. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  800. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  801. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  802. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: return self.main(*args, **kwargs)
  803. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  804. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: rv = self.invoke(ctx)
  805. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  806. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: return ctx.invoke(self.callback, **ctx.params)
  807. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  808. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: return callback(*args, **kwargs)
  809. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  810. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: NewPulpWorker().run_forever()
  811. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  812. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: self.sleep()
  813. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  814. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: self.beat()
  815. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  816. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: self.worker_cleanup()
  817. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  818. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: worker.delete()
  819. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  820. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: value = super().delete(*args, **kwargs)
  821. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  822. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: collector.collect([self], keep_parents=keep_parents)
  823. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  824. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  825. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  826. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: source_attr=field.name, nullable=field.null)
  827. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  828. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  829. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  830. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: sub_objs
  831. Dec 17 09:43:29 foreman.com pulpcore-worker-1[41628]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  832. Dec 17 09:43:29 foreman.com systemd[1]: pulpcore-worker@1.service: main process exited, code=exited, status=1/FAILURE
  833. Dec 17 09:43:29 foreman.com systemd[1]: Unit pulpcore-worker@1.service entered failed state.
  834. Dec 17 09:43:29 foreman.com systemd[1]: pulpcore-worker@1.service failed.
  835. Dec 17 09:43:32 foreman.com systemd[1]: pulpcore-worker@1.service holdoff time over, scheduling restart.
  836. Dec 17 09:43:32 foreman.com systemd[1]: Stopped Pulp Worker.
  837. Dec 17 09:43:32 foreman.com systemd[1]: Started Pulp Worker.
  838. Dec 17 09:43:38 foreman.com pulpcore-worker-1[50958]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  839. Dec 17 09:43:38 foreman.com pulpcore-worker-1[50958]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '50958@foreman.com' discovered
  840. Dec 17 09:43:38 foreman.com pulpcore-worker-1[50958]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Cleaning up task ecae8ea2-deef-4ae4-9cce-b7814e52914b and marking as failed. Reason: Worker has gone missing.
  841. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: Process Process-1:
  842. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: Traceback (most recent call last):
  843. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
  844. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: return self.connection.commit()
  845. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: psycopg2.errors.ForeignKeyViolation: insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
  846. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: DETAIL: Key (version_added_id)=(f4563964-4b15-492a-aaed-99cb81b62bd8) is not present in table "core_repositoryversion".
  847. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: The above exception was the direct cause of the following exception:
  848. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: Traceback (most recent call last):
  849. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
  850. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: loop.run_until_complete(pipeline)
  851. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
  852. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: return future.result()
  853. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
  854. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: await asyncio.gather(*futures)
  855. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
  856. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: await self.run()
  857. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 284, in run
  858. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: self.new_version.add_content(Content.objects.filter(pk__in=to_add))
  859. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 764, in add_content
  860. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: RepositoryContent.objects.bulk_create(repo_content)
  861. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
  862. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: return getattr(self.get_queryset(), name)(*args, **kwargs)
  863. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 480, in bulk_create
  864. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: obj_without_pk._state.db = self.db
  865. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/transaction.py", line 240, in __exit__
  866. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: connection.commit()
  867. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 262, in commit
  868. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: self._commit()
  869. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
  870. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: return self.connection.commit()
  871. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
  872. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: raise dj_exc_value.with_traceback(traceback) from exc_value
  873. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
  874. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: return self.connection.commit()
  875. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: django.db.utils.IntegrityError: insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
  876. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: DETAIL: Key (version_added_id)=(f4563964-4b15-492a-aaed-99cb81b62bd8) is not present in table "core_repositoryversion".
  877. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: During handling of the above exception, another exception occurred:
  878. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: Traceback (most recent call last):
  879. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
  880. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: result = func(*args, **kwargs)
  881. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 490, in synchronize
  882. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: version = dv.create()
  883. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
  884. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: loop.run_until_complete(pipeline)
  885. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib64/python3.6/tempfile.py", line 809, in __exit__
  886. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: self.cleanup()
  887. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib64/python3.6/tempfile.py", line 813, in cleanup
  888. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: _shutil.rmtree(self.name)
  889. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib64/python3.6/shutil.py", line 477, in rmtree
  890. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: onerror(os.lstat, path, sys.exc_info())
  891. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib64/python3.6/shutil.py", line 475, in rmtree
  892. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: orig_st = os.lstat(path)
  893. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: FileNotFoundError: [Errno 2] No such file or directory: './tmpbv7kw3u9'
  894. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: During handling of the above exception, another exception occurred:
  895. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: Traceback (most recent call last):
  896. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
  897. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: self.run()
  898. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
  899. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: self._target(*self._args, **self._kwargs)
  900. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 325, in _perform_task
  901. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: task.set_failed(exc, tb)
  902. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/task.py", line 389, in set_failed
  903. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: raise RuntimeError("Attempt to set a finished task to failed.")
  904. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: RuntimeError: Attempt to set a finished task to failed.
  905. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: Traceback (most recent call last):
  906. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  907. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  908. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  909. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: return self.main(*args, **kwargs)
  910. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  911. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: rv = self.invoke(ctx)
  912. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  913. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: return ctx.invoke(self.callback, **ctx.params)
  914. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  915. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: return callback(*args, **kwargs)
  916. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  917. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: NewPulpWorker().run_forever()
  918. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 273, in run_forever
  919. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: self.supervise_task(task)
  920. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 223, in supervise_task
  921. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: self.beat()
  922. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  923. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: self.worker_cleanup()
  924. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  925. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: worker.delete()
  926. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  927. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: value = super().delete(*args, **kwargs)
  928. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  929. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: collector.collect([self], keep_parents=keep_parents)
  930. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  931. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  932. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  933. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: source_attr=field.name, nullable=field.null)
  934. Dec 17 09:43:41 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  935. Dec 17 09:43:42 foreman.com pulpcore-worker-4[44976]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  936. Dec 17 09:43:42 foreman.com pulpcore-worker-4[44976]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  937. Dec 17 09:43:42 foreman.com pulpcore-worker-4[44976]: sub_objs
  938. Dec 17 09:43:42 foreman.com pulpcore-worker-4[44976]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  939. Dec 17 09:43:42 foreman.com systemd[1]: pulpcore-worker@4.service: main process exited, code=exited, status=1/FAILURE
  940. Dec 17 09:43:42 foreman.com systemd[1]: Unit pulpcore-worker@4.service entered failed state.
  941. Dec 17 09:43:42 foreman.com systemd[1]: pulpcore-worker@4.service failed.
  942. Dec 17 09:43:45 foreman.com pulpcore-worker-2[48408]: pulp [8b7dda27-bb25-44d8-b3de-4b90191bb982]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 5a7e0372-f54d-4584-8f31-b3dee7da6782
  943. Dec 17 09:43:45 foreman.com systemd[1]: pulpcore-worker@4.service holdoff time over, scheduling restart.
  944. Dec 17 09:43:45 foreman.com systemd[1]: Stopped Pulp Worker.
  945. Dec 17 09:43:45 foreman.com systemd[1]: Started Pulp Worker.
  946. Dec 17 09:43:45 foreman.com pulpcore-worker-2[48408]: pulp [8b7dda27-bb25-44d8-b3de-4b90191bb982]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 5a7e0372-f54d-4584-8f31-b3dee7da6782
  947. Dec 17 09:43:51 foreman.com pulpcore-worker-4[51101]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  948. Dec 17 09:43:51 foreman.com pulpcore-worker-4[51101]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '51101@foreman.com' discovered
  949. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 41628@foreman.com.
  950. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  951. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: Traceback (most recent call last):
  952. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  953. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  954. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  955. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: return self.main(*args, **kwargs)
  956. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  957. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: rv = self.invoke(ctx)
  958. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  959. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: return ctx.invoke(self.callback, **ctx.params)
  960. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  961. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: return callback(*args, **kwargs)
  962. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  963. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: NewPulpWorker().run_forever()
  964. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  965. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: self.sleep()
  966. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  967. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: self.beat()
  968. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  969. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: self.worker_cleanup()
  970. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  971. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: worker.delete()
  972. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  973. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: value = super().delete(*args, **kwargs)
  974. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  975. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: collector.collect([self], keep_parents=keep_parents)
  976. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  977. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  978. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  979. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: source_attr=field.name, nullable=field.null)
  980. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  981. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  982. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  983. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: sub_objs
  984. Dec 17 09:44:05 foreman.com pulpcore-worker-3[45843]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  985. Dec 17 09:44:06 foreman.com systemd[1]: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
  986. Dec 17 09:44:06 foreman.com systemd[1]: Unit pulpcore-worker@3.service entered failed state.
  987. Dec 17 09:44:06 foreman.com systemd[1]: pulpcore-worker@3.service failed.
  988. Dec 17 09:44:09 foreman.com systemd[1]: pulpcore-worker@3.service holdoff time over, scheduling restart.
  989. Dec 17 09:44:09 foreman.com systemd[1]: Stopped Pulp Worker.
  990. Dec 17 09:44:09 foreman.com systemd[1]: Started Pulp Worker.
  991. Dec 17 09:44:14 foreman.com pulpcore-worker-3[51271]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  992. Dec 17 09:44:14 foreman.com pulpcore-worker-3[51271]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '51271@foreman.com' discovered
  993. Dec 17 09:44:42 foreman.com pulpcore-worker-2[48408]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 097748a1-ebdc-464e-acac-28417bf0d634
  994. Dec 17 09:44:42 foreman.com pulpcore-worker-2[48408]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 097748a1-ebdc-464e-acac-28417bf0d634
  995. Dec 17 09:44:43 foreman.com pulpcore-worker-4[51101]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 48474026-7d37-4c8f-9f6b-8a111d51fdff
  996. Dec 17 09:44:43 foreman.com pulpcore-worker-4[51101]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=Red_Hat_Enterprise_Linux_7_Workstation_RPMs_x86_64_7_9-23585317 remote=Red_Hat_Enterprise_Linux_7_Workstation_RPMs_x86_64_7_9-23589651
  997. Dec 17 09:48:04 foreman.com pulpcore-worker-4[51101]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 45843@foreman.com.
  998. Dec 17 09:48:04 foreman.com pulpcore-worker-4[51101]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  999. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 51101@foreman.com.
  1000. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  1001. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: Traceback (most recent call last):
  1002. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  1003. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  1004. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  1005. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: return self.main(*args, **kwargs)
  1006. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  1007. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: rv = self.invoke(ctx)
  1008. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  1009. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: return ctx.invoke(self.callback, **ctx.params)
  1010. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  1011. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: return callback(*args, **kwargs)
  1012. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  1013. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: NewPulpWorker().run_forever()
  1014. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  1015. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: self.sleep()
  1016. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  1017. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: self.beat()
  1018. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  1019. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: self.worker_cleanup()
  1020. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  1021. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: worker.delete()
  1022. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  1023. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: value = super().delete(*args, **kwargs)
  1024. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  1025. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: collector.collect([self], keep_parents=keep_parents)
  1026. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  1027. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  1028. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  1029. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: source_attr=field.name, nullable=field.null)
  1030. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  1031. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  1032. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  1033. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: sub_objs
  1034. Dec 17 09:49:04 foreman.com pulpcore-worker-2[48408]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  1035. Dec 17 09:49:04 foreman.com systemd[1]: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
  1036. Dec 17 09:49:04 foreman.com systemd[1]: Unit pulpcore-worker@2.service entered failed state.
  1037. Dec 17 09:49:04 foreman.com systemd[1]: pulpcore-worker@2.service failed.
  1038. Dec 17 09:49:07 foreman.com systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
  1039. Dec 17 09:49:07 foreman.com systemd[1]: Stopped Pulp Worker.
  1040. Dec 17 09:49:07 foreman.com systemd[1]: Started Pulp Worker.
  1041. Dec 17 09:49:12 foreman.com pulpcore-worker-2[53111]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  1042. Dec 17 09:49:12 foreman.com pulpcore-worker-2[53111]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '53111@foreman.com' discovered
  1043. Dec 17 09:49:13 foreman.com pulpcore-worker-2[53111]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Cleaning up task 48474026-7d37-4c8f-9f6b-8a111d51fdff and marking as failed. Reason: Worker has gone missing.
  1044. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: Process Process-1:
  1045. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: Traceback (most recent call last):
  1046. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
  1047. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: return self.connection.commit()
  1048. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: psycopg2.errors.ForeignKeyViolation: insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
  1049. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: DETAIL: Key (version_added_id)=(9ca7c9e7-779f-4d4e-a20d-2966aad54249) is not present in table "core_repositoryversion".
  1050. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: The above exception was the direct cause of the following exception:
  1051. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: Traceback (most recent call last):
  1052. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
  1053. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: loop.run_until_complete(pipeline)
  1054. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
  1055. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: return future.result()
  1056. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
  1057. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: await asyncio.gather(*futures)
  1058. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
  1059. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: await self.run()
  1060. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 284, in run
  1061. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: self.new_version.add_content(Content.objects.filter(pk__in=to_add))
  1062. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 764, in add_content
  1063. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: RepositoryContent.objects.bulk_create(repo_content)
  1064. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
  1065. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: return getattr(self.get_queryset(), name)(*args, **kwargs)
  1066. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 480, in bulk_create
  1067. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: obj_without_pk._state.db = self.db
  1068. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/transaction.py", line 240, in __exit__
  1069. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: connection.commit()
  1070. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 262, in commit
  1071. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: self._commit()
  1072. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
  1073. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: return self.connection.commit()
  1074. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
  1075. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: raise dj_exc_value.with_traceback(traceback) from exc_value
  1076. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
  1077. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: return self.connection.commit()
  1078. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: django.db.utils.IntegrityError: insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
  1079. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: DETAIL: Key (version_added_id)=(9ca7c9e7-779f-4d4e-a20d-2966aad54249) is not present in table "core_repositoryversion".
  1080. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: During handling of the above exception, another exception occurred:
  1081. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: Traceback (most recent call last):
  1082. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
  1083. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: result = func(*args, **kwargs)
  1084. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 490, in synchronize
  1085. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: version = dv.create()
  1086. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
  1087. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: loop.run_until_complete(pipeline)
  1088. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib64/python3.6/tempfile.py", line 809, in __exit__
  1089. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: self.cleanup()
  1090. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib64/python3.6/tempfile.py", line 813, in cleanup
  1091. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: _shutil.rmtree(self.name)
  1092. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib64/python3.6/shutil.py", line 477, in rmtree
  1093. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: onerror(os.lstat, path, sys.exc_info())
  1094. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib64/python3.6/shutil.py", line 475, in rmtree
  1095. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: orig_st = os.lstat(path)
  1096. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: FileNotFoundError: [Errno 2] No such file or directory: './tmpcni8b255'
  1097. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: During handling of the above exception, another exception occurred:
  1098. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: Traceback (most recent call last):
  1099. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
  1100. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: self.run()
  1101. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
  1102. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: self._target(*self._args, **self._kwargs)
  1103. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 325, in _perform_task
  1104. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: task.set_failed(exc, tb)
  1105. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/task.py", line 389, in set_failed
  1106. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: raise RuntimeError("Attempt to set a finished task to failed.")
  1107. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: RuntimeError: Attempt to set a finished task to failed.
  1108. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: Traceback (most recent call last):
  1109. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  1110. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  1111. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  1112. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: return self.main(*args, **kwargs)
  1113. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  1114. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: rv = self.invoke(ctx)
  1115. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  1116. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: return ctx.invoke(self.callback, **ctx.params)
  1117. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  1118. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: return callback(*args, **kwargs)
  1119. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  1120. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: NewPulpWorker().run_forever()
  1121. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 273, in run_forever
  1122. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: self.supervise_task(task)
  1123. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 223, in supervise_task
  1124. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: self.beat()
  1125. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  1126. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: self.worker_cleanup()
  1127. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  1128. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: worker.delete()
  1129. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  1130. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: value = super().delete(*args, **kwargs)
  1131. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  1132. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: collector.collect([self], keep_parents=keep_parents)
  1133. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  1134. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  1135. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  1136. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: source_attr=field.name, nullable=field.null)
  1137. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  1138. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  1139. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  1140. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: sub_objs
  1141. Dec 17 09:49:17 foreman.com pulpcore-worker-4[51101]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  1142. Dec 17 09:49:17 foreman.com systemd[1]: pulpcore-worker@4.service: main process exited, code=exited, status=1/FAILURE
  1143. Dec 17 09:49:17 foreman.com systemd[1]: Unit pulpcore-worker@4.service entered failed state.
  1144. Dec 17 09:49:17 foreman.com systemd[1]: pulpcore-worker@4.service failed.
  1145. Dec 17 09:49:20 foreman.com pulpcore-worker-1[50958]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 7d8176cd-15b9-4c4a-8944-deadd091d53c
  1146. Dec 17 09:49:20 foreman.com pulpcore-worker-1[50958]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 7d8176cd-15b9-4c4a-8944-deadd091d53c
  1147. Dec 17 09:49:21 foreman.com systemd[1]: pulpcore-worker@4.service holdoff time over, scheduling restart.
  1148. Dec 17 09:49:21 foreman.com systemd[1]: Stopped Pulp Worker.
  1149. Dec 17 09:49:21 foreman.com systemd[1]: Started Pulp Worker.
  1150. Dec 17 09:49:23 foreman.com pulpcore-worker-3[51271]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Starting task adf6f53f-74e8-4da4-b833-cf58d3a6a2c3
  1151. Dec 17 09:49:23 foreman.com pulpcore-worker-3[51271]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Red_Hat_Enterprise_Linux_7_Workstation_RPMs_x86_64_7_9-23585317, version=0
  1152. Dec 17 09:49:24 foreman.com pulpcore-worker-3[51271]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulp_rpm.app.tasks.publishing:INFO: Publication: dddf0372-cab0-4a4d-9953-8d7bd00aea32 created
  1153. Dec 17 09:49:24 foreman.com pulpcore-worker-3[51271]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Task completed adf6f53f-74e8-4da4-b833-cf58d3a6a2c3
  1154. Dec 17 09:49:25 foreman.com pulpcore-worker-2[53111]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 3cf3a2bf-6bc9-483a-a098-d3ffb16664f2
  1155. Dec 17 09:49:26 foreman.com pulpcore-worker-2[53111]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 3cf3a2bf-6bc9-483a-a098-d3ffb16664f2
  1156. Dec 17 09:49:26 foreman.com pulpcore-worker-4[53233]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  1157. Dec 17 09:49:26 foreman.com pulpcore-worker-4[53233]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '53233@foreman.com' discovered
  1158. Dec 17 09:49:27 foreman.com pulpcore-worker-3[51271]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Starting task ea3de926-f12b-4e92-b0b6-717eb5cf1742
  1159. Dec 17 09:49:28 foreman.com pulpcore-worker-3[51271]: pulp [edd7e99a-3263-4853-8520-e1932fbfb222]: pulpcore.tasking.pulpcore_worker:INFO: Task completed ea3de926-f12b-4e92-b0b6-717eb5cf1742
  1160. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 48408@foreman.com.
  1161. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Clean offline worker 4895@foreman.com.
  1162. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: Traceback (most recent call last):
  1163. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/bin/pulpcore-worker", line 11, in <module>
  1164. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: load_entry_point('pulpcore==3.14.8', 'console_scripts', 'pulpcore-worker')()
  1165. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  1166. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: return self.main(*args, **kwargs)
  1167. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
  1168. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: rv = self.invoke(ctx)
  1169. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  1170. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: return ctx.invoke(self.callback, **ctx.params)
  1171. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  1172. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: return callback(*args, **kwargs)
  1173. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
  1174. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: NewPulpWorker().run_forever()
  1175. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 277, in run_forever
  1176. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: self.sleep()
  1177. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 187, in sleep
  1178. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: self.beat()
  1179. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 100, in beat
  1180. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: self.worker_cleanup()
  1181. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 90, in worker_cleanup
  1182. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: worker.delete()
  1183. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
  1184. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: value = super().delete(*args, **kwargs)
  1185. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 921, in delete
  1186. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: collector.collect([self], keep_parents=keep_parents)
  1187. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  1188. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  1189. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 16, in CASCADE
  1190. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: source_attr=field.name, nullable=field.null)
  1191. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
  1192. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: field.remote_field.on_delete(self, field, sub_objs, self.using)
  1193. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
  1194. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: sub_objs
  1195. Dec 17 09:49:58 foreman.com pulpcore-worker-1[50958]: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'ReservedResource' because they are referenced through a protected foreign key: 'TaskReservedResource.resource'", <QuerySet [<TaskReservedResource: pk=4e36cff8-2a41-4599-87e3-a2a4a1dc6cb2>, <TaskReservedResource: pk=b48cf587-8e69-4849-8107-54d78b3a2f37>]>)
  1196. Dec 17 09:49:58 foreman.com systemd[1]: pulpcore-worker@1.service: main process exited, code=exited, status=1/FAILURE
  1197. Dec 17 09:49:58 foreman.com systemd[1]: Unit pulpcore-worker@1.service entered failed state.
  1198. Dec 17 09:49:58 foreman.com systemd[1]: pulpcore-worker@1.service failed.
  1199. Dec 17 09:50:02 foreman.com systemd[1]: pulpcore-worker@1.service holdoff time over, scheduling restart.
  1200. Dec 17 09:50:02 foreman.com systemd[1]: Stopped Pulp Worker.
  1201. Dec 17 09:50:02 foreman.com systemd[1]: Started Pulp Worker.
  1202. Dec 17 09:50:07 foreman.com pulpcore-worker-1[53505]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
  1203. Dec 17 09:50:07 foreman.com pulpcore-worker-1[53505]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '53505@foreman.com' discovered
  1204.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!