API tools faq
paste
grikdotnet

yas3fs error

grikdotnet
Nov 22nd, 2015
267
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 11.73 KB | None | 0 0
raw download clone embed print report
  1. gri@ubuntu:~$ yas3fs -df s3://sdamtest/ $(pwd)/s3mount/ --cache-path $(pwd)/s3cache/ --cache-on-disk 10240
  2. 2015-11-22 21:55:52,834 DEBUG options = Namespace(aws_managed_encryption=False, buffer_prefetch=0, buffer_size=10240, cache_check=5, cache_disk_size=1024, cache_entries=100000, cache_mem_size=128, cache_on_disk=10240, cache_path='/home/gri/s3cache/', debug=True, download_num=4, download_retries_num=60, download_retries_sleep=1, expiration=2592000, foreground=True, gid=None, hostname=None, id=None, log=None, log_backup_count=10, log_backup_gzip=False, log_mb_size=100, mkdir=False, mountpoint='/home/gri/s3mount/', mp_num=4, mp_retries=3, mp_size=100, new_queue=False, new_queue_with_hostname=False, no_allow_other=False, no_metadata=False, nonempty=False, port=None, prefetch=False, prefetch_num=2, queue=None, queue_polling=0, queue_wait=20, read_only=False, read_retries_num=10, read_retries_sleep=1, recheck_s3=False, region='us-east-1', requester_pays=False, s3_num=32, s3_retries=3, s3_retries_sleep=1, s3path='s3://sdamtest/', st_blksize=None, topic=None, uid=None, umask=None, use_ec2_hostname=False, with_plugin_class=None, with_plugin_file=None)
  3. 2015-11-22 21:55:52,835 INFO Version: 2.3.2
  4. 2015-11-22 21:55:52,835 INFO s3-retries: '3'
  5. 2015-11-22 21:55:52,835 INFO s3-retries-sleep: '1' seconds
  6. 2015-11-22 21:55:52,835 INFO S3 bucket: 'sdamtest'
  7. 2015-11-22 21:55:52,836 INFO S3 prefix (can be empty): ''
  8. 2015-11-22 21:55:52,836 INFO Cache entries: '100000'
  9. 2015-11-22 21:55:52,836 INFO Cache memory size (in bytes): '134217728'
  10. 2015-11-22 21:55:52,836 INFO Cache disk size (in bytes): '1073741824'
  11. 2015-11-22 21:55:52,836 INFO Cache on disk if file size greater than (in bytes): '10240'
  12. 2015-11-22 21:55:52,837 INFO Cache check interval (in seconds): '5'
  13. 2015-11-22 21:55:52,837 INFO Cache ENOENT rechecks S3: False
  14. 2015-11-22 21:55:52,837 INFO AWS Managed Encryption enabled: False
  15. 2015-11-22 21:55:52,837 INFO AWS Managed Encryption enabled: False
  16. 2015-11-22 21:55:52,838 INFO Number of parallel S3 threads (0 to disable writeback): '32'
  17. 2015-11-22 21:55:52,838 INFO Number of parallel downloading threads: '4'
  18. 2015-11-22 21:55:52,838 INFO Number download retry attempts: '60'
  19. 2015-11-22 21:55:52,838 INFO Download retry sleep time seconds: '1'
  20. 2015-11-22 21:55:52,839 INFO Number read retry attempts: '10'
  21. 2015-11-22 21:55:52,839 INFO Read retry sleep time seconds: '1'
  22. 2015-11-22 21:55:52,839 INFO Number of parallel prefetching threads: '2'
  23. 2015-11-22 21:55:52,839 INFO Download buffer size (in KB, 0 to disable buffering): '10485760'
  24. 2015-11-22 21:55:52,839 INFO Number of buffers to prefetch: '0'
  25. 2015-11-22 21:55:52,840 INFO Write metadata (file system attr/xattr) on S3: 'True'
  26. 2015-11-22 21:55:52,840 INFO Download prefetch: 'False'
  27. 2015-11-22 21:55:52,840 INFO Multipart size: '104857600'
  28. 2015-11-22 21:55:52,840 INFO Multipart maximum number of parallel threads: '4'
  29. 2015-11-22 21:55:52,840 INFO Multipart maximum number of retries per part: '3'
  30. 2015-11-22 21:55:52,840 INFO Default expiration for signed URLs via xattrs: '2592000'
  31. 2015-11-22 21:55:52,841 INFO S3 Request Payer: 'False'
  32. 2015-11-22 21:55:52,841 INFO Cache path (on disk): '/home/gri/s3cache/'
  33. 2015-11-22 21:55:53,185 INFO Unique node ID: '837a9029-b05e-40e3-adda-89a33dace88a'
  34. 2015-11-22 21:55:53,194 DEBUG init '/'
  35. 2015-11-22 21:55:53,194 DEBUG check_threads 'True'
  36. 2015-11-22 21:55:53,195 DEBUG Starting S3 thread #0
  37. 2015-11-22 21:55:53,196 DEBUG Starting S3 thread #1
  38. 2015-11-22 21:55:53,197 DEBUG Starting S3 thread #2
  39. 2015-11-22 21:55:53,197 DEBUG Starting S3 thread #3
  40. 2015-11-22 21:55:53,198 DEBUG Starting S3 thread #4
  41. 2015-11-22 21:55:53,199 DEBUG Starting S3 thread #5
  42. 2015-11-22 21:55:53,200 DEBUG Starting S3 thread #6
  43. 2015-11-22 21:55:53,200 DEBUG Starting S3 thread #7
  44. 2015-11-22 21:55:53,201 DEBUG Starting S3 thread #8
  45. 2015-11-22 21:55:53,202 DEBUG Starting S3 thread #9
  46. 2015-11-22 21:55:53,203 DEBUG Starting S3 thread #10
  47. 2015-11-22 21:55:53,204 DEBUG Starting S3 thread #11
  48. 2015-11-22 21:55:53,205 DEBUG Starting S3 thread #12
  49. 2015-11-22 21:55:53,205 DEBUG Starting S3 thread #13
  50. 2015-11-22 21:55:53,206 DEBUG Starting S3 thread #14
  51. 2015-11-22 21:55:53,207 DEBUG Starting S3 thread #15
  52. 2015-11-22 21:55:53,208 DEBUG Starting S3 thread #16
  53. 2015-11-22 21:55:53,209 DEBUG Starting S3 thread #17
  54. 2015-11-22 21:55:53,209 DEBUG Starting S3 thread #18
  55. 2015-11-22 21:55:53,210 DEBUG Starting S3 thread #19
  56. 2015-11-22 21:55:53,210 DEBUG Starting S3 thread #20
  57. 2015-11-22 21:55:53,211 DEBUG Starting S3 thread #21
  58. 2015-11-22 21:55:53,211 DEBUG Starting S3 thread #22
  59. 2015-11-22 21:55:53,211 DEBUG Starting S3 thread #23
  60. 2015-11-22 21:55:53,212 DEBUG Starting S3 thread #24
  61. 2015-11-22 21:55:53,212 DEBUG Starting S3 thread #25
  62. 2015-11-22 21:55:53,212 DEBUG Starting S3 thread #26
  63. 2015-11-22 21:55:53,213 DEBUG Starting S3 thread #27
  64. 2015-11-22 21:55:53,213 DEBUG Starting S3 thread #28
  65. 2015-11-22 21:55:53,213 DEBUG Starting S3 thread #29
  66. 2015-11-22 21:55:53,214 DEBUG Starting S3 thread #30
  67. 2015-11-22 21:55:53,214 DEBUG Starting S3 thread #31
  68. 2015-11-22 21:55:53,214 DEBUG Starting download thread #0
  69. 2015-11-22 21:55:53,215 DEBUG Starting download thread #1
  70. 2015-11-22 21:55:53,215 DEBUG Starting download thread #2
  71. 2015-11-22 21:55:53,215 DEBUG Starting download thread #3
  72. 2015-11-22 21:55:53,216 DEBUG Starting prefetch thread #0
  73. 2015-11-22 21:55:53,216 DEBUG Starting prefetch thread #1
  74. 2015-11-22 21:55:53,216 DEBUG Starting check cache thread
  75. 2015-11-22 21:55:53,217 DEBUG check_cache_size
  76. 2015-11-22 21:55:53,217 DEBUG check_status
  77. 2015-11-22 21:55:57,321 DEBUG getattr -> '/' 'None'
  78. 2015-11-22 21:55:57,322 DEBUG get_metadata -> '/' 'attr' 'None'
  79. 2015-11-22 21:55:57,323 DEBUG get_key /
  80. 2015-11-22 21:55:57,323 DEBUG get_key from S3 #1 '/'
  81. 2015-11-22 21:55:57,395 DEBUG get_key no '/'
  82. 2015-11-22 21:55:57,396 DEBUG mkdir '/' '493'
  83. 2015-11-22 21:55:57,396 DEBUG get_key /
  84. 2015-11-22 21:55:57,397 DEBUG get_key from S3 #1 '/'
  85. 2015-11-22 21:55:57,506 DEBUG get_key no '/'
  86. 2015-11-22 21:55:57,507 DEBUG set_metadata '/' 'attr' 'None'
  87. 2015-11-22 21:55:57,507 DEBUG set_metadata '/' 'xattr' 'None'
  88. 2015-11-22 21:55:57,508 DEBUG mkdir '/' '493' '.' S3
  89. 2015-11-22 21:55:57,509 DEBUG get_metadata -> '/' 'attr' First time mount
  90. 2015-11-22 21:55:57,509 DEBUG getattr <- '/' 'None' '{'st_ctime': 1448214957.0, 'st_mtime': 1448214957.0, 'st_nlink': 1, 'st_atime': 1448214957.0, 'st_size': 4096, 'st_gid': 1000, 'st_uid': 1000, 'st_mode': 16877}'
  91. 2015-11-22 21:55:57,510 DEBUG getattr -> '/1.txt' 'None'
  92. 2015-11-22 21:55:57,510 DEBUG get_metadata -> '/1.txt' 'attr' 'None'
  93. 2015-11-22 21:55:57,511 DEBUG get_key /1.txt
  94. 2015-11-22 21:55:57,511 DEBUG get_key from S3 #1 '/1.txt'
  95. 2015-11-22 21:55:57,531 DEBUG do_cmd_on_s3_now_w_retries try 1 action 'set_contents_from_string' key '.' args '['']' kargs '{'headers': {'Content-Type': 'application/x-directory'}}'
  96. 2015-11-22 21:55:57,531 DEBUG do_cmd_on_s3_now action 'set_contents_from_string' key '.' args '['']' kargs '{'headers': {'Content-Type': 'application/x-directory'}}'
  97. 2015-11-22 21:55:57,596 DEBUG get_key to cache '/1.txt'
  98. 2015-11-22 21:55:57,597 DEBUG get_metadata '/1.txt' 'attr' '1.txt' S3 found
  99. 2015-11-22 21:55:57,598 DEBUG get_metadata <- '/1.txt' 'attr' '1.txt' '{u'st_ctime': 1448214560.0, u'st_mtime': 1448214560.0, u'st_nlink': 1, u'st_gid': 1000, u'st_size': 0, u'st_mode': 33204, u'st_uid': 1000, u'st_atime': 1448214560.0}'
  100. 2015-11-22 21:55:57,598 DEBUG getattr <- '/1.txt' 'None' '{u'st_ctime': 1448214560.0, u'st_mtime': 1448214560.0, u'st_nlink': 1, u'st_gid': 1000, u'st_size': 0, u'st_mode': 33204, u'st_uid': 1000, u'st_atime': 1448214560.0}'
  101. 2015-11-22 21:55:57,599 DEBUG open '/1.txt' '32769'
  102. 2015-11-22 21:55:57,599 DEBUG check_data '/1.txt'
  103. 2015-11-22 21:55:57,600 DEBUG get_key /1.txt
  104. 2015-11-22 21:55:57,600 DEBUG get_key from S3 #1 '/1.txt'
  105. 2015-11-22 21:55:57,676 DEBUG get_key to cache '/1.txt'
  106. 2015-11-22 21:55:57,677 DEBUG check_data '/1.txt' nothing to download
  107. 2015-11-22 21:55:57,677 DEBUG open '/1.txt' '32769' '1'
  108. 2015-11-22 21:55:57,678 DEBUG getxattr '/1.txt' 'security.capability' '0'
  109. 2015-11-22 21:55:57,679 DEBUG get_key from cache '/1.txt'
  110. 2015-11-22 21:55:57,679 DEBUG get_metadata -> '/1.txt' 'xattr' 'None'
  111. 2015-11-22 21:55:57,679 DEBUG get_key from cache '/1.txt'
  112. 2015-11-22 21:55:57,680 DEBUG get_metadata '/1.txt' 'xattr' '1.txt' S3 found
  113. 2015-11-22 21:55:57,680 DEBUG get_metadata <- '/1.txt' 'xattr' '1.txt' '{}'
  114. 2015-11-22 21:55:57,681 DEBUG truncate '/1.txt' '0'
  115. 2015-11-22 21:55:57,681 DEBUG check_data '/1.txt'
  116. 2015-11-22 21:55:57,681 DEBUG check_data '/1.txt' data already in place
  117. 2015-11-22 21:55:57,682 DEBUG get_metadata -> '/1.txt' 'attr' 'None'
  118. 2015-11-22 21:55:57,682 DEBUG get_metadata <- '/1.txt' 'attr' 'None' '{u'st_ctime': 1448214560.0, u'st_mtime': 1448214560.0, u'st_nlink': 1, u'st_gid': 1000, u'st_size': 0, u'st_mode': 33204, u'st_uid': 1000, u'st_atime': 1448214560.0}'
  119. 2015-11-22 21:55:57,682 DEBUG getattr -> '/1.txt' 'None'
  120. 2015-11-22 21:55:57,682 DEBUG get_metadata -> '/1.txt' 'attr' 'None'
  121. 2015-11-22 21:55:57,682 DEBUG get_metadata <- '/1.txt' 'attr' 'None' '{u'st_ctime': 1448214560.0, u'st_mtime': 1448214957.0, u'st_nlink': 1, u'st_gid': 1000, u'st_size': 0, u'st_mode': 33204, u'st_uid': 1000, u'st_atime': 1448214957.0}'
  122. 2015-11-22 21:55:57,683 DEBUG getattr <- '/1.txt' 'None' '{u'st_ctime': 1448214560.0, u'st_mtime': 1448214957.0, u'st_nlink': 1, u'st_gid': 1000, u'st_size': 0, u'st_mode': 33204, u'st_uid': 1000, u'st_atime': 1448214957.0}'
  123. 2015-11-22 21:55:57,684 DEBUG write '/1.txt' '12' '0' '0'
  124. Traceback (most recent call last):
  125. File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 414, in _wrapper
  126. return func(*args, **kwargs) or 0
  127. File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 515, in write
  128. offset, fh)
  129. File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 881, in __call__
  130. ret = getattr(self, op)(path, *args)
  131. File "/usr/local/lib/python2.7/dist-packages/yas3fs/__init__.py", line 2560, in write
  132. logger.debug("write '%s' '%i' '%i' '%s' '%s' content" % (path, len(new_data), offset, fh, data.content.name.decode('utf-8')))
  133. AttributeError: '_io.BytesIO' object has no attribute 'name'
  134. 2015-11-22 21:55:57,689 DEBUG release '/1.txt' '0'
  135. 2015-11-22 21:55:57,690 DEBUG upload_to_s3 '/1.txt'
  136. 2015-11-22 21:55:57,690 DEBUG get_key from cache '/1.txt'
  137. 2015-11-22 21:55:57,690 DEBUG get_metadata -> '/1.txt' 'attr' '1.txt'
  138. 2015-11-22 21:55:57,690 DEBUG get_metadata <- '/1.txt' 'attr' '1.txt' '{u'st_ctime': 1448214560.0, u'st_mtime': 1448214957.0, u'st_nlink': 1, u'st_gid': 1000, u'st_size': 0, u'st_mode': 33204, u'st_uid': 1000, u'st_atime': 1448214957.0}'
  139. 2015-11-22 21:55:57,690 DEBUG set_metadata '/1.txt' 'attr' '1.txt'
  140. 2015-11-22 21:55:57,690 DEBUG set_metadata '/1.txt' 'xattr' '1.txt'
  141. 2015-11-22 21:55:57,693 DEBUG multipart test: key '/1.txt' mp-num '4' st_size '0' mp-size '104857600'
  142. 2015-11-22 21:55:57,693 DEBUG upload_to_s3 '/1.txt' '1.txt' 'text/plain' S3
  143. 2015-11-22 21:55:57,693 DEBUG upload_to_s3 '/1.txt' done
  144. 2015-11-22 21:55:57,694 DEBUG release '/1.txt' '0' 'None'
  145. 2015-11-22 21:55:57,723 DEBUG do_cmd_on_s3_now_w_retries try 1 action 'set_contents_from_file' key '1.txt' args '[<yas3fs.FSData instance at 0x7ff37c1f4fc8>]' kargs '{'headers': {'Content-Type': 'text/plain'}}'
  146. 2015-11-22 21:55:57,724 DEBUG do_cmd_on_s3_now action 'set_contents_from_file' key '1.txt' args '[<yas3fs.FSData instance at 0x7ff37c1f4fc8>]' kargs '{'headers': {'Content-Type': 'text/plain'}}'
  147. 2015-11-22 21:55:57,806 DEBUG do_cmd_on_s3_now action 'set_contents_from_file' key '1.txt' args '[<yas3fs.FSData instance at 0x7ff37c1f4fc8>]' kargs '{'headers': {'Content-Type': 'text/plain'}}' done
  148. 2015-11-22 21:55:57,819 DEBUG do_cmd_on_s3_now action 'set_contents_from_string' key '.' args '['']' kargs '{'headers': {'Content-Type': 'application/x-directory'}}' done
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!