#High-concurrency web crawling implementations (generally Scala/Akka based)

https://github.com/fhopf/akka-crawler-example

http://stackoverflow.com/questions/29950299/distributed-web-crawling-using-apache-spark

https://github.com/reggoodwin/ferrit

https://www.google.com/search?q=(spark+OR+akka+OR+hadoop+OR+mapreduce)+scala+%22web+crawler%22