API tools faq
paste
CookiePPP

PN Tac2 Training

CookiePPP
Sep 30th, 2020 (edited)
82
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 13.17 KB | None | 0 0
raw download clone embed print report
  1. (base) cookie@pop-os:/media/cookie/Samsung PM961/TensorflowTTS$ CUDA_VISIBLE_DEVICES=3 python3 examples/tacotron2/train_tacotron2.py --train-dir ../dump/train/ --dev-dir ../dump/valid/ --outdir ./tac2out --config ./examples/tacotron2/conf/tacotron2.v1_ms_me48.yaml --use-norm 1 --mixed_precision 0
  2. 2020-10-01 04:13:30.506148: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
  3. 2020-10-01 04:13:32.143801: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
  4. 2020-10-01 04:13:32.186689: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  5. 2020-10-01 04:13:32.188430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
  6. pciBusID: 0000:43:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
  7. coreClock: 1.683GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
  8. 2020-10-01 04:13:32.188451: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
  9. 2020-10-01 04:13:32.189814: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
  10. 2020-10-01 04:13:32.191233: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
  11. 2020-10-01 04:13:32.191411: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
  12. 2020-10-01 04:13:32.192743: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
  13. 2020-10-01 04:13:32.193424: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
  14. 2020-10-01 04:13:32.197289: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
  15. 2020-10-01 04:13:32.197484: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  16. 2020-10-01 04:13:32.200285: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  17. 2020-10-01 04:13:32.201843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
  18. 2020-10-01 04:13:33.871710: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
  19. To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  20. 2020-10-01 04:13:33.896915: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3492995000 Hz
  21. 2020-10-01 04:13:33.898465: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55eeb1847680 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
  22. 2020-10-01 04:13:33.898489: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
  23. 2020-10-01 04:13:34.027573: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  24. 2020-10-01 04:13:34.028418: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55eeb1408110 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
  25. 2020-10-01 04:13:34.028445: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
  26. 2020-10-01 04:13:34.028688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  27. 2020-10-01 04:13:34.029902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
  28. pciBusID: 0000:43:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
  29. coreClock: 1.683GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
  30. 2020-10-01 04:13:34.029943: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
  31. 2020-10-01 04:13:34.029980: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
  32. 2020-10-01 04:13:34.030001: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
  33. 2020-10-01 04:13:34.030021: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
  34. 2020-10-01 04:13:34.030043: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
  35. 2020-10-01 04:13:34.030063: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
  36. 2020-10-01 04:13:34.030085: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
  37. 2020-10-01 04:13:34.030165: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  38. 2020-10-01 04:13:34.031605: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  39. 2020-10-01 04:13:34.033076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
  40. 2020-10-01 04:13:34.033117: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
  41. 2020-10-01 04:13:34.404235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
  42. 2020-10-01 04:13:34.404281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
  43. 2020-10-01 04:13:34.404287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
  44. 2020-10-01 04:13:34.404536: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  45. 2020-10-01 04:13:34.405327: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  46. 2020-10-01 04:13:34.406044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9986 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:43:00.0, compute capability: 6.1)
  47. 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: hop_size = 600
  48. 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: format = npy
  49. 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: model_type = tacotron2
  50. 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: tacotron2_params = {'dataset': 'ljspeech', 'embedding_hidden_size': 512, 'initializer_range': 0.02, 'embedding_dropout_prob': 0.1, 'n_speakers': 136, 'n_emotions': 22, 'n_conv_encoder': 5, 'encoder_conv_filters': 512, 'encoder_conv_kernel_sizes': 5, 'encoder_conv_activation': 'relu', 'encoder_conv_dropout_rate': 0.5, 'encoder_lstm_units': 256, 'n_prenet_layers': 2, 'prenet_units': 256, 'prenet_activation': 'relu', 'prenet_dropout_rate': 0.5, 'n_lstm_decoder': 1, 'reduction_factor': 1, 'decoder_lstm_units': 1024, 'attention_dim': 128, 'attention_filters': 32, 'attention_kernel': 31, 'n_mels': 80, 'n_conv_postnet': 5, 'postnet_conv_filters': 512, 'postnet_conv_kernel_sizes': 5, 'postnet_dropout_rate': 0.1, 'attention_type': 'lsa'}
  51. 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: batch_size = 32
  52. 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: remove_short_samples = True
  53. 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: allow_cache = True
  54. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: mel_length_threshold = 32
  55. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: is_shuffle = True
  56. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: use_fixed_shapes = True
  57. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: optimizer_params = {'initial_learning_rate': 0.001, 'end_learning_rate': 1e-05, 'decay_steps': 150000, 'warmup_proportion': 0.02, 'weight_decay': 0.001}
  58. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: train_max_steps = 200000
  59. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: save_interval_steps = 2000
  60. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: eval_interval_steps = 500
  61. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: log_interval_steps = 200
  62. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: start_schedule_teacher_forcing = 200001
  63. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: start_ratio_value = 0.5
  64. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: schedule_decay_steps = 50000
  65. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: end_ratio_value = 0.0
  66. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: num_save_intermediate_results = 1
  67. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: train_dir = ../dump/train/
  68. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: dev_dir = ../dump/valid/
  69. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: use_norm = True
  70. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: outdir = ./tac2out
  71. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: config = ./examples/tacotron2/conf/tacotron2.v1_ms_me48.yaml
  72. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: resume =
  73. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: verbose = 1
  74. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: mixed_precision = False
  75. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: pretrained =
  76. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: version = 0.0
  77. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: max_mel_length = 796
  78. 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: max_char_length = 139
  79. 2020-10-01 04:13:48.728974: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
  80. 2020-10-01 04:13:48.865129: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
  81. Model: "tacotron2"
  82. _________________________________________________________________
  83. Layer (type) Output Shape Param #
  84. =================================================================
  85. encoder (TFTacotronEncoder) multiple 9432064
  86. _________________________________________________________________
  87. decoder_cell (TFTacotronDeco multiple 18246402
  88. _________________________________________________________________
  89. post_net (TFTacotronPostnet) multiple 5460480
  90. _________________________________________________________________
  91. residual_projection (Dense) multiple 41040
  92. =================================================================
  93. Total params: 33,179,986
  94. Trainable params: 33,169,746
  95. Non-trainable params: 10,240
  96. _________________________________________________________________
  97. [train]: 0%| | 0/200000 [00:00<?, ?it/s]2020-10-01 04:13:53.097802: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:906] Skipping loop optimization for Merge node with control input: cond/branch_executed/_8
  98. 2020-10-01 04:14:03.006500: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:172] Filling up shuffle buffer (this may take a while): 3404 of 22881
  99. 2020-10-01 04:14:13.018470: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:172] Filling up shuffle buffer (this may take a while): 6820 of 22881
  100. 2020-10-01 04:14:23.004688: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:172] Filling up shuffle buffer (this may take a while): 10245 of 22881
  101. 2020-10-01 04:14:33.004991: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:172] Filling up shuffle buffer (this may take a while): 13665 of 22881
  102. 2020-10-01 04:14:43.026199: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:172] Filling up shuffle buffer (this may take a while): 17070 of 22881
  103. 2020-10-01 04:14:53.006538: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:172] Filling up shuffle buffer (this may take a while): 20479 of 22881
  104. 2020-10-01 04:15:00.057589: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:221] Shuffle buffer filled.
  105. [train]: 0%| | 35/200000 [03:11<174:55:45, 3.15s/it]
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!