Guest User

Untitled

a guest
Nov 23rd, 2020
54
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 3.65 KB | None | 0 0
  1. @celery.task(name = 'DynamoJob')
  2. def DynamoJob(source, destination):
  3.  
  4. file_list = os.listdir(source)
  5.  
  6. return {"Source" : source, "Destination" : destination, "File List": file_list}
  7.  
  8. @celery.task(name = 'Migrate')
  9. def Migrate(source, destination, filename):
  10.  
  11. path = os.path.join(source, filename)
  12. #sleep(1)
  13. subprocess.run(['cp', '-a', path, destination])
  14.  
  15. dest_file_list = os.listdir(destination)
  16.  
  17. return "Migrate <--------------"
  18.  
  19. @celery.task(name = 'Transfer')
  20. def Transfer(source, destination, filename):
  21.  
  22. path = os.path.join(source, filename)
  23. #sleep(1)
  24. subprocess.run(['cp', '-a', path, destination])
  25.  
  26. dest_file_list = os.listdir(destination)
  27. return "Transfer <-----------"
  28.  
  29.  
  30.  
  31. @celery.task(name = 'Recall')
  32. def Recall(source, destination, filename):
  33.  
  34. path = os.path.join(source, filename)
  35. #sleep(1)
  36. subprocess.run(['cp', '-a', path, destination])
  37.  
  38. dest_file_list = os.listdir(destination)
  39.  
  40. return "Recall <-----------"
  41.  
  42. --------------------------------------------------------------------------
  43.  
  44. res = chain(
  45. DynamoJob.si(sitex, sitez),
  46. group(
  47. chain(Migrate.si(sitex, s3london, filenames),
  48. Transfer.si(s3london, s3africa, filenames),
  49. Recall.si(s3africa, sitez, filenames)) for filenames in filelist)).delay()
  50.  
  51. --------------------------------------------------------------------------
  52. At the moment I get this output:
  53. 74abeb7f-d432-4173-8323-43623db6ef75(21) ---> Parent Job
  54. 1bcf2ebb-6ce6-4fb4-a270-c3471624381f(20) ----> Group ID
  55. b6b18022-bc36-4bca-9a59-99302807f4e9(0) ----> Chain job
  56. c0222cd5-f1d7-496b-b79d-bf7e9f73819b(0) ----> Chain job
  57. 01726058-7090-4af9-973c-c4ba9d56a0bb(0) ----> Chain job
  58. fef4a1ca-dc36-4e74-aa71-da3bfc3d7c78(0) ----> Chain job
  59. 7ac3e4f6-3c6e-4e6a-bc0a-d04bc11ac937(0) ----> Chain job
  60. 34d1902f-36be-4545-a778-cbfb34622ec4(0) ----> Chain job
  61. 6ff27f56-164e-41af-833a-3b7eecd553bf(0) ----> Chain job
  62. 0c729abe-9c33-4824-9d7e-848d28e30550(0) ----> Chain job
  63. 63e18054-86d6-4e28-b0f4-3532eb3f8a6d(0) ----> Chain job
  64. 874b3e3f-0e75-43d3-8987-a30a128f1b8e(0) ----> Chain job
  65. 4c1001dc-73cb-4190-beff-9c407b4f80a7(0) ----> Chain job
  66. 87b3891e-3b69-4d27-8116-e7eff0be347b(0) ----> Chain job
  67. 53455f94-76e7-427d-96c1-f42d80e05442(0) ----> Chain job
  68. e2496847-d619-49ca-8dd2-a41726c4a7df(0) ----> Chain job
  69. ce76967c-2ca9-4555-a794-c79bd5732d74(0) ----> Chain job
  70. 31cf9a8f-45f3-4678-8a0b-d7fc4e8bdd67(0) ----> Chain job
  71. ec2ca67c-378d-4395-8769-b706aacc79a1(0) ----> Chain job
  72. 3ad82b9b-45e7-4fe1-bcab-5537adeb3a48(0) ----> Chain job
  73. d001a591-22f8-442c-ac5b-4da6daf11c15(0) ----> Chain job
  74. 9ab598bc-8a33-4976-8e9c-9f06d454aba1(0) ----> Chain job
  75.  
  76. How can I get something like this:
  77.  
  78. Parent job ID (DynamoJob)
  79. 1st Group ID
  80. Chain Tasks (Migrate(sitex,s3london, file 1),
  81. Tranfer(s3london, s3africa, file 1),
  82. Recall(s3africa, sitez, file 1))
  83. 2nd Group ID
  84. Chain Tasks (Migrate(sitex,s3london, file 2),
  85. Tranfer(s3london, s3africa, file 2),
  86. Recall(s3africa, sitez, file 2))
  87. 3rd Group ID
  88. Chain Tasks (Migrate(sitex,s3london, file 3),
  89. Tranfer(s3london, s3africa, file 3),
  90. Recall(s3africa, sitez, file 3))
  91.  
  92. and so on......
Advertisement
Add Comment
Please, Sign In to add comment