NLinker

Read files with akka stream

Dec 14th, 2017
153
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Scala 0.96 KB | None | 0 0
  1.     import akka.stream.scaladsl.{Flow, Framing}
  2.    
  3.     // Given a stream of bytestrings delimited by the system line separator we can get lines represented as Strings
  4.     val lines = Flow[ByteString]
  5.       .via(Framing.delimiter(ByteString(System.lineSeparator), 10000, allowTruncation = true))
  6.       .map(bs => bs.utf8String)
  7.  
  8.     // given as stream of Paths we read those files and count the number of lines
  9.     val lineCounter = Flow[Path]
  10.       .flatMapConcat(path => FileIO.fromPath(path).via(lines))
  11.       .fold(0l)((count, line) => count + 1)
  12.       .toMat(Sink.head)(Keep.right)
  13.  
  14.     // Here's our test data source (replace paths with real paths)
  15.     val testFiles = Source(List("somePathToFile1", "somePathToFile2").map(new File(_).toPath))
  16.  
  17.     // Runs the line counter over the test files, returns a Future, which contains the number of lines, which we then print out to the console when it completes
  18.     testFiles.runWith(lineCounter).foreach(println)
Add Comment
Please, Sign In to add comment