Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- @celery.task(name = 'DynamoJob')
- def DynamoJob(source, destination):
- file_list = os.listdir(source)
- return {"Source" : source, "Destination" : destination, "File List": file_list}
- @celery.task(name = 'Migrate')
- def Migrate(source, destination, filename):
- path = os.path.join(source, filename)
- #sleep(1)
- subprocess.run(['cp', '-a', path, destination])
- dest_file_list = os.listdir(destination)
- return "Migrate <--------------"
- @celery.task(name = 'Transfer')
- def Transfer(source, destination, filename):
- path = os.path.join(source, filename)
- #sleep(1)
- subprocess.run(['cp', '-a', path, destination])
- dest_file_list = os.listdir(destination)
- return "Transfer <-----------"
- @celery.task(name = 'Recall')
- def Recall(source, destination, filename):
- path = os.path.join(source, filename)
- #sleep(1)
- subprocess.run(['cp', '-a', path, destination])
- dest_file_list = os.listdir(destination)
- return "Recall <-----------"
- --------------------------------------------------------------------------
- res = chain(
- DynamoJob.si(sitex, sitez),
- group(
- chain(Migrate.si(sitex, s3london, filenames),
- Transfer.si(s3london, s3africa, filenames),
- Recall.si(s3africa, sitez, filenames)) for filenames in filelist)).delay()
- --------------------------------------------------------------------------
- At the moment I get this output:
- 74abeb7f-d432-4173-8323-43623db6ef75(21) ---> Parent Job
- 1bcf2ebb-6ce6-4fb4-a270-c3471624381f(20) ----> Group ID
- b6b18022-bc36-4bca-9a59-99302807f4e9(0) ----> Chain job
- c0222cd5-f1d7-496b-b79d-bf7e9f73819b(0) ----> Chain job
- 01726058-7090-4af9-973c-c4ba9d56a0bb(0) ----> Chain job
- fef4a1ca-dc36-4e74-aa71-da3bfc3d7c78(0) ----> Chain job
- 7ac3e4f6-3c6e-4e6a-bc0a-d04bc11ac937(0) ----> Chain job
- 34d1902f-36be-4545-a778-cbfb34622ec4(0) ----> Chain job
- 6ff27f56-164e-41af-833a-3b7eecd553bf(0) ----> Chain job
- 0c729abe-9c33-4824-9d7e-848d28e30550(0) ----> Chain job
- 63e18054-86d6-4e28-b0f4-3532eb3f8a6d(0) ----> Chain job
- 874b3e3f-0e75-43d3-8987-a30a128f1b8e(0) ----> Chain job
- 4c1001dc-73cb-4190-beff-9c407b4f80a7(0) ----> Chain job
- 87b3891e-3b69-4d27-8116-e7eff0be347b(0) ----> Chain job
- 53455f94-76e7-427d-96c1-f42d80e05442(0) ----> Chain job
- e2496847-d619-49ca-8dd2-a41726c4a7df(0) ----> Chain job
- ce76967c-2ca9-4555-a794-c79bd5732d74(0) ----> Chain job
- 31cf9a8f-45f3-4678-8a0b-d7fc4e8bdd67(0) ----> Chain job
- ec2ca67c-378d-4395-8769-b706aacc79a1(0) ----> Chain job
- 3ad82b9b-45e7-4fe1-bcab-5537adeb3a48(0) ----> Chain job
- d001a591-22f8-442c-ac5b-4da6daf11c15(0) ----> Chain job
- 9ab598bc-8a33-4976-8e9c-9f06d454aba1(0) ----> Chain job
- How can I get something like this:
- Parent job ID (DynamoJob)
- 1st Group ID
- Chain Tasks (Migrate(sitex,s3london, file 1),
- Tranfer(s3london, s3africa, file 1),
- Recall(s3africa, sitez, file 1))
- 2nd Group ID
- Chain Tasks (Migrate(sitex,s3london, file 2),
- Tranfer(s3london, s3africa, file 2),
- Recall(s3africa, sitez, file 2))
- 3rd Group ID
- Chain Tasks (Migrate(sitex,s3london, file 3),
- Tranfer(s3london, s3africa, file 3),
- Recall(s3africa, sitez, file 3))
- and so on......
Advertisement
Add Comment
Please, Sign In to add comment