daily pastebin goal
69%
SHARE
TWEET

Untitled

a guest Sep 21st, 2018 65 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. import s3fs
  2. import sys
  3.  
  4. # Expect aws credentials to exist in path
  5. def fetch_filtered_filekeys(bucket_name: str, dir_name: str, prefix_file: str="",suffix: str="") -> list:
  6.     S3 = s3fs.S3FileSystem()
  7.     s3_dataset_path = "/".join([bucket_name, dir_name])
  8.     list_available_file = [p for p in S3.walk(s3_dataset_path)
  9.         if p.endswith(suffix_file) and p.startswith(prefix_file)]
  10.     return list_available_file
  11.    
  12. if __name__ == "__main__":
  13.   # usage python3 count_files_s3_dir_and_subdir.py <bucket_name> <dir_name> [prefix_file] [suffix_file]
  14.   print(len(fetch_filtered_filekeys(**sys.argv[1:])))
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top