parsl.log

2022-12-13 17:18:01.812 parsl.dataflow.dflow:84 [DEBUG]  Starting DataFlowKernel with config
Config(
    app_cache=True,
    checkpoint_files=None,
    checkpoint_mode=None,
    checkpoint_period=None,
    executors=[HighThroughputExecutor(
        address=None,
        address_probe_timeout=None,
        cores_per_worker=1.0,
        cpu_affinity='none',
        heartbeat_period=30,
        heartbeat_threshold=120,
        interchange_port_range=(55000, 56000),
        label='HighThroughputExecutor',
        launch_cmd='process_worker_pool.py {debug} {max_workers} -a {addresses} -p {prefetch_capacity} -c {cores_per_worker} -m {mem_per_worker} --poll {poll_period} --task_port={task_port} --result_port={result_port} --logdir={logdir} --block_id={{block_id}} --hb_period={heartbeat_period} {address_probe_timeout_string} --hb_threshold={heartbeat_threshold} --cpu-affinity {cpu_affinity} ',
        managed=True,
        max_workers=inf,
        mem_per_worker=None,
        poll_period=10,
        prefetch_capacity=0,
        provider=AdHocProvider(
            channels=[SSHChannel(
                'worker-0',
                envs={},
                gssapi_auth=False,
                host_keys_filename=None,
                key_filename=None,
                password=None,
                port=22,
                script_dir='/home/azureuser',
                skip_auth=False,
                username=None
            )],
            cmd_timeout=30,
            move_files=None,
            parallelism=1,
            worker_init='source $HOME/spack/activate.sh'
        ),
        storage_access=None,
        worker_debug=False,
        worker_logdir_root=None,
        worker_port_range=(54000, 55000),
        worker_ports=None,
        working_dir=None
    )],
    garbage_collect=True,
    initialize_logging=True,
    internal_tasks_max_threads=10,
    max_idletime=120.0,
    monitoring=None,
    retries=0,
    run_dir='runinfo',
    strategy='simple',
    usage_tracking=False
)
2022-12-13 17:18:01.813 parsl.dataflow.dflow:86 [INFO]  Parsl version: 1.1.0
2022-12-13 17:18:01.813 parsl.dataflow.usage_tracking.usage:125 [DEBUG]  Tracking status: False
2022-12-13 17:18:01.813 parsl.dataflow.dflow:114 [INFO]  Run id is: 8b55f012-6259-47ee-9403-b8190a78c106
2022-12-13 17:18:01.996 parsl.dataflow.memoization:164 [INFO]  App caching initialized
2022-12-13 17:18:01.996 parsl.dataflow.strategy:127 [DEBUG]  Scaling strategy: simple
2022-12-13 17:18:01.997 parsl.dataflow.dflow:977 [DEBUG]  Creating script_dir across multiple channels
2022-12-13 17:18:01.997 parsl.channels.ssh.ssh:77 [DEBUG]  connecting to worker-0:22
2022-12-13 17:18:02.386 parsl.executors.high_throughput.executor:473 [DEBUG]  Starting queue management thread
2022-12-13 17:18:02.386 parsl.executors.high_throughput.executor:357 [DEBUG]  [MTHREAD] queue management worker starting
2022-12-13 17:18:02.386 parsl.executors.high_throughput.executor:477 [DEBUG]  Started queue management thread
2022-12-13 17:18:02.399 parsl.executors.high_throughput.executor:318 [DEBUG]  Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 139997067863616)>
2022-12-13 17:18:02.400 parsl.executors.high_throughput.executor:290 [DEBUG]  Launch command: process_worker_pool.py   -a 10.0.2.4,20.25.100.146,127.0.0.1,manager -p 0 -c 1.0 -m None --poll 10 --task_port=54574 --result_port=54028 --logdir=/home/azureuser/runinfo/001/HighThroughputExecutor --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none
2022-12-13 17:18:02.400 parsl.executors.high_throughput.executor:293 [DEBUG]  Starting HighThroughputExecutor with provider:
AdHocProvider(
    channels=[SSHChannel(
        'worker-0',
        envs={},
        gssapi_auth=False,
        host_keys_filename=None,
        key_filename=None,
        password=None,
        port=22,
        script_dir='/home/azureuser',
        skip_auth=False,
        username=None
    )],
    cmd_timeout=30,
    move_files=None,
    parallelism=1,
    worker_init='source $HOME/spack/activate.sh'
)
2022-12-13 17:18:02.402 parsl.providers.ad_hoc.ad_hoc:119 [DEBUG]  Channel_counts : {SSHChannel(
    'worker-0',
    envs={},
    gssapi_auth=False,
    host_keys_filename=None,
    key_filename=None,
    password=None,
    port=22,
    script_dir='/home/azureuser',
    skip_auth=False,
    username=None
): 0}
2022-12-13 17:18:02.402 parsl.providers.ad_hoc.ad_hoc:173 [DEBUG]  Pushing start script
2022-12-13 17:18:02.478 parsl.executors.high_throughput.executor:624 [DEBUG]  Launched block 0->1648
2022-12-13 17:18:02.481 parsl.dataflow.task_status_poller:114 [DEBUG]  Adding executor HighThroughputExecutor
2022-12-13 17:18:02.487 parsl.dataflow.dflow:795 [DEBUG]  Task 0 will be sent to executor HighThroughputExecutor
2022-12-13 17:18:02.487 parsl.dataflow.dflow:624 [DEBUG]  Adding output dependencies
2022-12-13 17:18:02.487 parsl.dataflow.dflow:874 [INFO]  Task 0 submitted for App f, not waiting on any dependency
2022-12-13 17:18:02.489 parsl.dataflow.dflow:882 [DEBUG]  Task 0 set to pending state with AppFuture: <AppFuture super=<AppFuture at 0x7f53aba490f0 state=pending>>
2022-12-13 17:18:02.489 parsl.dataflow.memoization:229 [DEBUG]  Task 0 will not be memoized
2022-12-13 17:18:02.490 parsl.executors.high_throughput.executor:562 [DEBUG]  Pushing function <function f at 0x7f53aba27c70> to queue with args (3,)
2022-12-13 17:18:02.525 parsl.dataflow.dflow:503 [DEBUG]  Got an exception launching task
Traceback (most recent call last):
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 492, in launch_if_ready
    exec_fu = self.launch_task(
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 582, in launch_task
    exec_fu = executor.submit(executable, self.tasks[task_id]['resource_specification'], *args, **kwargs)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/executors/high_throughput/executor.py", line 567, in submit
    fn_buf = pack_apply_message(func, args, kwargs,
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/serialize/facade.py", line 59, in pack_apply_message
    b_func = self.serialize(func, buffer_threshold=buffer_threshold)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/serialize/facade.py", line 105, in serialize
    raise last_exception
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/serialize/facade.py", line 83, in serialize
    serialized = method.serialize(obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/serialize/concretes.py", line 49, in serialize
    x = dill.dumps(data)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 364, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 336, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 620, in dump
    StockPickler.dump(self, obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 487, in dump
    self.save(obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1154, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1765, in save_module
    pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1765, in save_module
    pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1154, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1838, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1140, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 902, in save_tuple
    save(element)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1530, in save_wrapper_descriptor
    pickler.save_reduce(_getattr, (obj.__objclass__, obj.__name__,
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 887, in save_tuple
    save(element)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1851, in save_type
    StockPickler.save_global(pickler, obj, name=obj_name)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 1071, in save_global
    raise PicklingError(
_pickle.PicklingError: Can't pickle <class 'bytes_iterator'>: it's not found as builtins.bytes_iterator
2022-12-13 17:18:02.531 parsl.dataflow.dflow:289 [DEBUG]  Task 0 try 0 failed
2022-12-13 17:18:02.531 parsl.dataflow.dflow:316 [ERROR]  Task 0 failed after 0 retry attempts
Traceback (most recent call last):
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 286, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 439, in _unwrap_remote_exception_wrapper
    result = future.result()
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 492, in launch_if_ready
    exec_fu = self.launch_task(
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 582, in launch_task
    exec_fu = executor.submit(executable, self.tasks[task_id]['resource_specification'], *args, **kwargs)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/executors/high_throughput/executor.py", line 567, in submit
    fn_buf = pack_apply_message(func, args, kwargs,
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/serialize/facade.py", line 59, in pack_apply_message
    b_func = self.serialize(func, buffer_threshold=buffer_threshold)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/serialize/facade.py", line 105, in serialize
    raise last_exception
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/serialize/facade.py", line 83, in serialize
    serialized = method.serialize(obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/parsl/serialize/concretes.py", line 49, in serialize
    x = dill.dumps(data)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 364, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 336, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 620, in dump
    StockPickler.dump(self, obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 487, in dump
    self.save(obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1154, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1765, in save_module
    pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1765, in save_module
    pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1154, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1838, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1140, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 902, in save_tuple
    save(element)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1530, in save_wrapper_descriptor
    pickler.save_reduce(_getattr, (obj.__objclass__, obj.__name__,
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 887, in save_tuple
    save(element)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/site-packages/dill/_dill.py", line 1851, in save_type
    StockPickler.save_global(pickler, obj, name=obj_name)
  File "/home/azureuser/spack/var/spack/environments/wf-reg-test/.spack-env/view/lib/python3.10/pickle.py", line 1071, in save_global
    raise PicklingError(
_pickle.PicklingError: Can't pickle <class 'bytes_iterator'>: it's not found as builtins.bytes_iterator
2022-12-13 17:18:02.549 parsl.dataflow.dflow:1030 [INFO]  DFK cleanup initiated
2022-12-13 17:18:02.549 parsl.dataflow.dflow:920 [INFO]  Summary of tasks in DFK:
2022-12-13 17:18:02.549 parsl.dataflow.dflow:934 [INFO]  Tasks in state States.failed: 1
2022-12-13 17:18:02.549 parsl.dataflow.dflow:940 [INFO]  End of summary
2022-12-13 17:18:02.549 parsl.dataflow.dflow:1054 [INFO]  Terminating flow_control and strategy threads
2022-12-13 17:18:02.550 parsl.executors.high_throughput.executor:692 [DEBUG]  Current blocks : {'0': '1648'}
2022-12-13 17:18:02.550 parsl.providers.ad_hoc.ad_hoc:228 [DEBUG]  Cancelling jobs: []
2022-12-13 17:18:02.550 parsl.executors.high_throughput.executor:731 [INFO]  Attempting HighThroughputExecutor shutdown
2022-12-13 17:18:02.552 parsl.executors.high_throughput.executor:733 [INFO]  Finished HighThroughputExecutor shutdown attempt
2022-12-13 17:18:02.552 parsl.executors.threads:105 [DEBUG]  Done with executor shutdown
2022-12-13 17:18:02.553 parsl.dataflow.dflow:1083 [INFO]  DFK cleanup complete