Guest User

Untitled

a guest
Feb 21st, 2018
113
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 4.60 KB | None | 0 0
  1. <#
  2. .SYNOPSIS
  3. Compute Amazon S3 ETag for a local file
  4.  
  5. .DESCRIPTION
  6. ETags represent a hash of the content of a file stored in S3.
  7. Comaparing ETags can be used to determine
  8. - If a file in S3 is the same as one you are going to upload
  9. - Following an upload, whether the file was successfully uploaded.
  10.  
  11. .PARAMETER Path
  12. Path to file to compute the ETag for
  13.  
  14. .PARAMETER MinimumPartSize
  15. Minimum size in bytes of a part for mulitpart upload (default 5MB as per Write-S3Object)
  16.  
  17. .PARAMETER MinimumSizeBeforeMultipartUpload
  18. Minimum file size, above which multipart upload will be selected (default 16MB as per Write-S3Object)
  19.  
  20. .OUTPUTS
  21. [string] The computed ETag
  22.  
  23. .NOTES
  24. The defaults for MinimumPartSize and MinimumSizeBeforeMultipartUpload are based on the values used by Write-S3Object.
  25. Other clients (eg. AWS CLI) may use different values, and application code may use any values as determined by the application's developers.
  26.  
  27. For a single part upload, the ETag is simply the MD5 of the whole file converted to hex string.
  28. For multipart, concatentate the binary MD5 hash of each part, then compute MD5 hash of the concatenated binary data, convert to hex string and append the number of parts to this.
  29. #>
  30. param
  31. (
  32. [Parameter(Mandatory=$true)]
  33. [string]$Path,
  34.  
  35. [long]$MinimumPartSize = 5MB,
  36.  
  37. [long]$MinimumSizeBeforeMultipartUpload = 16MB
  38. )
  39.  
  40. try
  41. {
  42. # Get file size
  43. $fileSize = (Get-Item $Path).Length
  44.  
  45. # Calculate part size
  46. $partSize = [long]([Math]::Max([Math]::Ceiling([double]$fileSize / 10000.0), $MinimumPartSize))
  47.  
  48. # Calculate number of parts
  49. $numberOfParts = [int]($fileSize / $partSize)
  50.  
  51. if ($fileSize % $partSize -gt 0)
  52. {
  53. ++$numberOfParts
  54. }
  55.  
  56. # Buffer to read file parts into
  57. $buf = New-Object byte[] $partSize
  58.  
  59. # Will hold the final hash result.
  60. $md5Hash = $null
  61.  
  62. # Create MD5 hash algorithm
  63. $md5 = [Security.Cryptography.HashAlgorithm]::Create("MD5")
  64.  
  65. # Open input file for reading
  66. $inputStream = [System.IO.File]::OpenRead($Path)
  67.  
  68. # Stream to write part hashes to as we compute them
  69. $hashBuffer = $null
  70.  
  71. # Is file large enough to do a multipart upload?
  72. if ($fileSize -ge $minimumSizeBeforeMultipartUpload)
  73. {
  74. # File name for status messages
  75. $filename = [IO.Path]::GetFileName($Path)
  76.  
  77. # Create the stream that will concatenate chuck hashes.
  78. $hashBuffer = New-Object System.IO.MemoryStream
  79.  
  80. # Counter for number of parts read so far
  81. $partsRead = 0
  82.  
  83. if ([Environment]::UserInteractive)
  84. {
  85. # Show progress if running at the command line
  86. Write-Progress -Activity "Computing ETag" -Status $filename -PercentComplete 0
  87. }
  88.  
  89. # Read each part
  90. while (($bytesRead = $inputStream.Read($buf, 0, $buf.length)) -ne 0)
  91. {
  92. if ([Environment]::UserInteractive -and $partsRead % 10 -eq 0)
  93. {
  94. # Show progress every 10 parts read if running at the command line
  95. Write-Progress -Activity "Computing ETag" -Status "$filename - Part $($partsRead)/$($numberOfParts)" -PercentComplete ($partsRead * 100 / $numberOfParts)
  96. }
  97.  
  98. ++$partsRead
  99.  
  100. # Hash the part
  101. $partMd5Hash = $md5.ComputeHash($buf, 0, $bytesRead)
  102.  
  103. # and write to the buffer.
  104. $hashBuffer.Write($partMd5Hash, 0, $partMd5Hash.Length)
  105. }
  106.  
  107. # Seek to start of the buffer
  108. $hashBuffer.Seek(0, 'Begin') | Out-Null
  109.  
  110. # and compute the hash of hashes
  111. $md5Hash = $md5.ComputeHash($hashBuffer)
  112.  
  113. if ([Environment]::UserInteractive)
  114. {
  115. # Remove progress bar if running at the command line
  116. Write-Progress -Activity "Computing ETag" -Completed
  117. }
  118. }
  119. else
  120. {
  121. # Single part upload - ETag is just MD5 of the whole file.
  122. $partsRead = 1
  123. $md5Hash = $md5.ComputeHash($inputStream)
  124. }
  125.  
  126. # Build the ETag
  127. $eTag = [System.BitConverter]::ToString( $md5Hash ).Replace('-', [string]::Empty).ToLower()
  128.  
  129. if ($partsRead -gt 1)
  130. {
  131. # For multipart ETag, append the number of parts.
  132. $eTag = $eTag + "-$partsRead"
  133. }
  134.  
  135. # Emit result
  136. $eTag
  137. }
  138. finally
  139. {
  140. # Dispose any IDisposable CLR objects created.
  141. ($inputStream, $md5, $hashBuffer) |
  142. foreach {
  143.  
  144. if ($_)
  145. {
  146. $_.Dispose()
  147. }
  148. }
  149. }
Add Comment
Please, Sign In to add comment