[2023-05-02 09:20:39,837] [WARNING] [runner.py:186:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-05-02 09:20:39,864] [INFO] [runner.py:550:main] cmd = /local_disk0/.ephemeral_nfs/envs/pythonEnv-e1f29474-0171-420e-8b03-5f4b03ffc5cd/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgM119 --master_addr=127.0.0.1 --master_port=29500 --module --enable_each_rank_log=None training.trainer --input-model Databricks/dolly-v2-7b --deepspeed /Workspace/Repos/opyate@gmail.com/dolly/config/ds_z3_bf16_config.json --epochs 2 --local-output-dir /local_disk0/dolly_training/dolly_mydataset-dolly15kFormat-noJSONSchema__2023-05-02T09:20:32 --dbfs-output-dir /dbfs/dolly_training/dolly_mydataset-dolly15kFormat-noJSONSchema__2023-05-02T09:20:32 --per-device-train-batch-size 3 --per-device-eval-batch-size 3 --logging-steps 10 --save-steps 200 --save-total-limit 20 --eval-steps 50 --warmup-steps 50 --test-size 10 --lr 5e-07
[2023-05-02 09:20:43,216] [INFO] [launch.py:142:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3]}
[2023-05-02 09:20:43,216] [INFO] [launch.py:148:main] nnodes=1, num_local_procs=4, node_rank=0
[2023-05-02 09:20:43,216] [INFO] [launch.py:161:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3]})
[2023-05-02 09:20:43,217] [INFO] [launch.py:162:main] dist_world_size=4
[2023-05-02 09:20:43,217] [INFO] [launch.py:164:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
2023-05-02 09:20:45.707091: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-02 09:20:45.735839: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-02 09:20:45.756864: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-02 09:20:45.778718: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-02 09:20:53 INFO [__main__] Loading tokenizer for Databricks/dolly-v2-7b
2023-05-02 09:20:53 INFO [__main__] Loading tokenizer for Databricks/dolly-v2-7b
2023-05-02 09:20:53 INFO [__main__] Loading tokenizer for Databricks/dolly-v2-7b
2023-05-02 09:20:53 INFO [__main__] Loading tokenizer for Databricks/dolly-v2-7b
2023-05-02 09:20:54 INFO [__main__] Loading model for Databricks/dolly-v2-7b
2023-05-02 09:20:54 INFO [__main__] Loading model for Databricks/dolly-v2-7b
2023-05-02 09:20:54 INFO [__main__] Loading model for Databricks/dolly-v2-7b
2023-05-02 09:20:54 INFO [__main__] Loading model for Databricks/dolly-v2-7b
2023-05-02 09:24:48 INFO [__main__] Found max lenth: 2048
2023-05-02 09:24:48 INFO [__main__] Checking if dataset is specific via env var DATASET_NAME
2023-05-02 09:24:48 INFO [__main__] Yes: opyate/mydataset-dolly15kFormat-noJSONSchema
2023-05-02 09:24:49 WARNING [datasets.builder] Found cached dataset parquet (/root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 200.42it/s]
2023-05-02 09:24:49 INFO [__main__] Found 1032 rows
2023-05-02 09:24:50 INFO [__main__] Preprocessing dataset
2023-05-02 09:24:50 INFO [__main__] Found max lenth: 2048
2023-05-02 09:24:50 INFO [__main__] Checking if dataset is specific via env var DATASET_NAME
2023-05-02 09:24:50 INFO [__main__] Yes: opyate/mydataset-dolly15kFormat-noJSONSchema
Map:   0%|                                      | 0/1032 [00:00<?, ? examples/s]2023-05-02 09:24:50 INFO [__main__] Found max lenth: 2048
2023-05-02 09:24:50 INFO [__main__] Checking if dataset is specific via env var DATASET_NAME
2023-05-02 09:24:50 INFO [__main__] Yes: opyate/mydataset-dolly15kFormat-noJSONSchema
2023-05-02 09:24:50 INFO [__main__] Found max lenth: 2048
2023-05-02 09:24:50 INFO [__main__] Checking if dataset is specific via env var DATASET_NAME
2023-05-02 09:24:50 INFO [__main__] Yes: opyate/mydataset-dolly15kFormat-noJSONSchema
2023-05-02 09:24:51 WARNING [datasets.builder] Found cached dataset parquet (/root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 556.05it/s]
2023-05-02 09:24:51 INFO [__main__] Found 1032 rows
2023-05-02 09:24:51 WARNING [datasets.arrow_dataset] Loading cached processed dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-373a08c6145e0816.arrow
2023-05-02 09:24:51 INFO [__main__] Preprocessing dataset
2023-05-02 09:24:51 WARNING [datasets.builder] Found cached dataset parquet (/root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 505.09it/s]
2023-05-02 09:24:51 INFO [__main__] Found 1032 rows
2023-05-02 09:24:51 WARNING [datasets.arrow_dataset] Loading cached processed dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-373a08c6145e0816.arrow
2023-05-02 09:24:51 INFO [__main__] Preprocessing dataset
Map:   0%|                                      | 0/1032 [00:00<?, ? examples/s]2023-05-02 09:24:51 WARNING [datasets.builder] Found cached dataset parquet (/root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 559.61it/s]
2023-05-02 09:24:51 INFO [__main__] Found 1032 rows
2023-05-02 09:24:51 WARNING [datasets.arrow_dataset] Loading cached processed dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-373a08c6145e0816.arrow
2023-05-02 09:24:51 INFO [__main__] Preprocessing dataset
2023-05-02 09:24:51 INFO [__main__] Processed dataset has 1032 rows
2023-05-02 09:24:52 INFO [__main__] Processed dataset has 1030 rows after filtering for truncated records
2023-05-02 09:24:52 INFO [__main__] Shuffling dataset
2023-05-02 09:24:52 INFO [__main__] Done preprocessing
2023-05-02 09:24:52 INFO [__main__] Train data size: 1020
2023-05-02 09:24:52 INFO [__main__] Test data size: 10
2023-05-02 09:24:52 INFO [__main__] Processed dataset has 1032 rows
2023-05-02 09:24:52 WARNING [datasets.arrow_dataset] Loading cached processed dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-462aabd08dd4ec96.arrow
2023-05-02 09:24:52 INFO [__main__] Processed dataset has 1030 rows after filtering for truncated records
2023-05-02 09:24:52 INFO [__main__] Shuffling dataset
2023-05-02 09:24:52 WARNING [datasets.arrow_dataset] Loading cached shuffled indices for dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-2ef5f0cbbbb9df9d.arrow
2023-05-02 09:24:52 INFO [__main__] Done preprocessing
2023-05-02 09:24:52 WARNING [datasets.arrow_dataset] Loading cached split indices for dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-8ed636b6f0cf0253.arrow and /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-5b275d686d6d0049.arrow
2023-05-02 09:24:52 INFO [__main__] Train data size: 1020
2023-05-02 09:24:52 INFO [__main__] Test data size: 10
2023-05-02 09:24:52 INFO [__main__] Processed dataset has 1032 rows
2023-05-02 09:24:52 WARNING [datasets.arrow_dataset] Loading cached processed dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-462aabd08dd4ec96.arrow
2023-05-02 09:24:52 INFO [__main__] Processed dataset has 1030 rows after filtering for truncated records
2023-05-02 09:24:52 INFO [__main__] Shuffling dataset
2023-05-02 09:24:52 WARNING [datasets.arrow_dataset] Loading cached shuffled indices for dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-2ef5f0cbbbb9df9d.arrow
2023-05-02 09:24:52 INFO [__main__] Done preprocessing
2023-05-02 09:24:52 WARNING [datasets.arrow_dataset] Loading cached split indices for dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-8ed636b6f0cf0253.arrow and /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-5b275d686d6d0049.arrow
2023-05-02 09:24:52 INFO [__main__] Train data size: 1020
2023-05-02 09:24:52 INFO [__main__] Test data size: 10
2023-05-02 09:24:52 INFO [__main__] Processed dataset has 1032 rows
2023-05-02 09:24:52 WARNING [datasets.arrow_dataset] Loading cached processed dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-462aabd08dd4ec96.arrow
2023-05-02 09:24:52 INFO [__main__] Processed dataset has 1030 rows after filtering for truncated records
2023-05-02 09:24:52 INFO [__main__] Shuffling dataset
2023-05-02 09:24:52 WARNING [datasets.arrow_dataset] Loading cached shuffled indices for dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-2ef5f0cbbbb9df9d.arrow
2023-05-02 09:24:52 INFO [__main__] Done preprocessing
2023-05-02 09:24:52 WARNING [datasets.arrow_dataset] Loading cached split indices for dataset at /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-8ed636b6f0cf0253.arrow and /root/.cache/huggingface/datasets/opyate___parquet/opyate--mydataset-dolly15kFormat-noJSONSchema-66b5aa1793cd296a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-5b275d686d6d0049.arrow
2023-05-02 09:24:52 INFO [__main__] Train data size: 1020
2023-05-02 09:24:52 INFO [__main__] Test data size: 10
[2023-05-02 09:24:52,838] [INFO] [comm.py:652:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
2023-05-02 09:24:52 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:1 to store for rank: 3
2023-05-02 09:24:53 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:1 to store for rank: 2
2023-05-02 09:24:53 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:1 to store for rank: 1
2023-05-02 09:24:53 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:1 to store for rank: 0
2023-05-02 09:24:53 INFO [torch.distributed.distributed_c10d] Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-05-02 09:24:53 INFO [torch.distributed.distributed_c10d] Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-05-02 09:24:53 INFO [torch.distributed.distributed_c10d] Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-05-02 09:24:53 INFO [torch.distributed.distributed_c10d] Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-05-02 09:24:54 INFO [__main__] Instantiating Trainer
2023-05-02 09:24:54 INFO [__main__] Instantiating Trainer
2023-05-02 09:24:54 INFO [__main__] Instantiating Trainer
2023-05-02 09:24:54 INFO [__main__] Instantiating Trainer
2023-05-02 09:24:54 INFO [__main__] Training
2023-05-02 09:24:54 INFO [__main__] Training
2023-05-02 09:24:54 INFO [__main__] Training
2023-05-02 09:24:54 INFO [__main__] Training
2023-05-02 09:25:02 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:2 to store for rank: 1
2023-05-02 09:25:02 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:2 to store for rank: 0
2023-05-02 09:25:02 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:2 to store for rank: 2
2023-05-02 09:25:02 INFO [torch.distributed.distributed_c10d] Added key: store_based_barrier_key:2 to store for rank: 3
2023-05-02 09:25:02 INFO [torch.distributed.distributed_c10d] Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-05-02 09:25:02 INFO [torch.distributed.distributed_c10d] Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-05-02 09:25:02 INFO [torch.distributed.distributed_c10d] Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-05-02 09:25:02 INFO [torch.distributed.distributed_c10d] Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
Installed CUDA version 11.3 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.3 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.3 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.3 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py39_cu117/cpu_adam...
Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py39_cu117/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/local_disk0/.ephemeral_nfs/envs/pythonEnv-e1f29474-0171-420e-8b03-5f4b03ffc5cd/lib/python3.9/site-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /databricks/python/lib/python3.9/site-packages/torch/include -isystem /databricks/python/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /databricks/python/lib/python3.9/site-packages/torch/include/TH -isystem /databricks/python/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86 -c /local_disk0/.ephemeral_nfs/envs/pythonEnv-e1f29474-0171-420e-8b03-5f4b03ffc5cd/lib/python3.9/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o
[2/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/local_disk0/.ephemeral_nfs/envs/pythonEnv-e1f29474-0171-420e-8b03-5f4b03ffc5cd/lib/python3.9/site-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /databricks/python/lib/python3.9/site-packages/torch/include -isystem /databricks/python/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /databricks/python/lib/python3.9/site-packages/torch/include/TH -isystem /databricks/python/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -g -Wno-reorder -L/usr/local/cuda/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -c /local_disk0/.ephemeral_nfs/envs/pythonEnv-e1f29474-0171-420e-8b03-5f4b03ffc5cd/lib/python3.9/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
[3/3] c++ cpu_adam.o custom_cuda_kernel.cuda.o -shared -lcurand -L/databricks/python/lib/python3.9/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o cpu_adam.so
Loading extension module cpu_adam...
Time to load cpu_adam op: 33.10828900337219 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 33.150460958480835 seconds
Loading extension module cpu_adam...Loading extension module cpu_adam...

Time to load cpu_adam op: 33.173842906951904 secondsTime to load cpu_adam op: 33.16837549209595 seconds

Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py39_cu117/utils...
Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py39_cu117/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF flatten_unflatten.o.d -DTORCH_EXTENSION_NAME=utils -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /databricks/python/lib/python3.9/site-packages/torch/include -isystem /databricks/python/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /databricks/python/lib/python3.9/site-packages/torch/include/TH -isystem /databricks/python/lib/python3.9/site-packages/torch/include/THC -isystem /usr/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /local_disk0/.ephemeral_nfs/envs/pythonEnv-e1f29474-0171-420e-8b03-5f4b03ffc5cd/lib/python3.9/site-packages/deepspeed/ops/csrc/utils/flatten_unflatten.cpp -o flatten_unflatten.o
[2/2] c++ flatten_unflatten.o -shared -L/databricks/python/lib/python3.9/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o utils.so
Loading extension module utils...
Time to load utils op: 15.778066158294678 seconds
Loading extension module utils...
Time to load utils op: 15.824144840240479 seconds
Loading extension module utils...
Time to load utils op: 15.824395895004272 seconds
Loading extension module utils...
Time to load utils op: 15.723725318908691 seconds
Parameter Offload: Total persistent parameters: 1712128 in 258 params
Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...

Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...No modifications detected for re-loaded extension module utils, skipping build step...

Loading extension module utils...
Loading extension module utils...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.003479480743408203 seconds
Time to load utils op: 0.0035200119018554688 seconds
Time to load utils op: 0.0037059783935546875 seconds
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
Using /root/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0003936290740966797 seconds
/databricks/python/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
  warnings.warn(
/databricks/python/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
  warnings.warn(
/databricks/python/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
  warnings.warn(
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
/databricks/python/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
  warnings.warn(
/databricks/python/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
  warnings.warn(
/databricks/python/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
  warnings.warn(
/databricks/python/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
  warnings.warn(
/databricks/python/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
  warnings.warn(
{'loss': 1.6039, 'learning_rate': 2.942959550338895e-07, 'epoch': 0.12}
{'loss': 1.4676, 'learning_rate': 3.828878651016684e-07, 'epoch': 0.24}
{'loss': 1.0807, 'learning_rate': 4.347108103585802e-07, 'epoch': 0.35}
{'loss': 0.7331, 'learning_rate': 4.7147977516944737e-07, 'epoch': 0.47}
{'loss': 0.4184, 'learning_rate': 5e-07, 'epoch': 0.59}
{'eval_loss': 0.6129394769668579, 'eval_runtime': 3.6325, 'eval_samples_per_second': 2.753, 'eval_steps_per_second': 0.275, 'epoch': 0.59}
[2023-05-02 09:43:47,839] [WARNING] [stage3.py:1942:step] 1 pytorch allocator cache flushes since last step. this happens when there is high memory pressure and is detrimental to performance. if this is happening frequently consider adjusting settings to reduce memory consumption. If you are unable to make the cache flushes go away consider adding get_accelerator().empty_cache() calls in your training loop to ensure that all ranks flush their caches at the same time
{'loss': 0.2071, 'learning_rate': 5e-07, 'epoch': 0.71}
{'loss': 0.1072, 'learning_rate': 5e-07, 'epoch': 0.82}
{'loss': 0.1105, 'learning_rate': 5e-07, 'epoch': 0.94}
{'loss': 0.077, 'learning_rate': 5e-07, 'epoch': 1.06}
{'loss': 0.1277, 'learning_rate': 5e-07, 'epoch': 1.18}
{'eval_loss': 0.398681640625, 'eval_runtime': 3.1007, 'eval_samples_per_second': 3.225, 'eval_steps_per_second': 0.323, 'epoch': 1.18}
{'loss': 0.0476, 'learning_rate': 5e-07, 'epoch': 1.29}
[2023-05-02 10:02:12,287] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 2762
[2023-05-02 10:02:15,511] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 2763
[2023-05-02 10:02:18,768] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 2764
[2023-05-02 10:02:18,768] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 2765
[2023-05-02 10:02:22,024] [ERROR] [launch.py:324:sigkill_handler] ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e1f29474-0171-420e-8b03-5f4b03ffc5cd/bin/python', '-u', '-m', 'training.trainer', '--local_rank=3', '--input-model', 'Databricks/dolly-v2-7b', '--deepspeed', '/Workspace/Repos/opyate@gmail.com/dolly/config/ds_z3_bf16_config.json', '--epochs', '2', '--local-output-dir', '/local_disk0/dolly_training/dolly_mydataset-dolly15kFormat-noJSONSchema__2023-05-02T09:20:32', '--dbfs-output-dir', '/dbfs/dolly_training/dolly_mydataset-dolly15kFormat-noJSONSchema__2023-05-02T09:20:32', '--per-device-train-batch-size', '3', '--per-device-eval-batch-size', '3', '--logging-steps', '10', '--save-steps', '200', '--save-total-limit', '20', '--eval-steps', '50', '--warmup-steps', '50', '--test-size', '10', '--lr', '5e-07'] exits with return code = -9