Advertisement
Guest User

Untitled

a guest
Jan 2nd, 2020
149
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.03 KB | None | 0 0
  1. job.name=TestParquet
  2. job.group=Test
  3. job.description=this is a test job that pulls from a test source and writes to Parquet
  4. job.lock.enabled=false
  5. task.execution.synchronousExecutionModel=false
  6.  
  7. source.class=org.apache.gobblin.test.SequentialTestSource
  8. source.numParallelism=2
  9. source.inMemFormat=JSON
  10.  
  11. fs.uri=file:///
  12. work.dir=/Users/sdas/gobblin-work/parquetWorkDir
  13. converter.classes=org.apache.gobblin.converter.parquet.JsonIntermediateToParquetGroupConverter
  14.  
  15. extract.table.name=TestData
  16. extract.namespace=org.apache.gobblin.example
  17. extract.table.type=APPEND_ONLY
  18.  
  19.  
  20. state.store.enabled=true
  21. state.store.fs.uri=${fs.uri}
  22. state.store.dir=${work.dir}/store
  23.  
  24.  
  25. writer.destination.type=HDFS
  26. writer.output.format=PARQUET
  27. writer.fs.uri=${fs.uri}
  28. writer.builder.class=org.apache.gobblin.writer.ParquetDataWriterBuilder
  29.  
  30. data.publisher.fs.uri=${fs.uri}
  31. data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher
  32. data.publisher.final.dir=${work.dir}/test_data
  33. data.publisher.metadata.output.dir=${work.dir}/metadata
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement