Advertisement
Guest User

Untitled

a guest
Dec 16th, 2014
156
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Scala 1.18 KB | None | 0 0
  1. val modelFile   = DistributedCacheFile(sumacArgs.model.getAbsolutePath)
  2.   val featureFile = DistributedCacheFile(sumacArgs.features.getAbsolutePath)
  3.  
  4.   val trainFile   = DistributedCacheFile(sumacArgs.trainFeatures.getAbsolutePath)
  5.  
  6.   @transient
  7.   private lazy val ethnicityModel =
  8.     EthnicityFS.loadModel(llModelFile = new File(modelFile.path), featureFile = new File(featureFile.path))
  9.  
  10.   @transient
  11.   private lazy val validFeatures: Set[Int] = {
  12.     Source.fromFile(trainFile.path).getLines.map { line =>
  13.       line.toInt
  14.     }.toSet
  15.   }
  16.  
  17.  
  18. /**
  19. hadoop jar transform/target/scala-2.10/transform-assembly-0.1-SNAPSHOT.jar com.twitter.scalding.Tool -Dmapred.map.child.java.opts="-Xmx4500m -Dfile.encoding=UTF-8” -Dmapred.reduce.child.java.opts="-Xmx5500m -Dfile.encoding=UTF-8” -Dmapred.map.tasks=24 -Dmapred.reduce.tasks=8 com.qf.util.fbcrawlexplorer.EthFeatureExplorer --env dev --hdfs --inputFile /mapr/mapr-prod/data/rawJoins/tw/20141213_1418494555086/ --model /mapr/mapr-dev/data/exports3/models/eth2/ethnicityModel.ser.gz --features /mapr/mapr-dev/data/exports3/models/eth2/ethnicityFeatures.ser.gz --sample 0.2  --trainFeatures /mapr/mapr-dev/data/pierre/validFeatures.txt
  20. */
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement