SHARE
TWEET

Untitled

a guest Jan 2nd, 2020 78 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. job.name=TestParquet
  2. job.group=Test
  3. job.description=this is a test job that pulls from a test source and writes to Parquet
  4. job.lock.enabled=false
  5. task.execution.synchronousExecutionModel=false
  6.  
  7. source.class=org.apache.gobblin.test.SequentialTestSource
  8. source.numParallelism=2
  9. source.inMemFormat=JSON
  10.  
  11. fs.uri=file:///
  12. work.dir=/Users/sdas/gobblin-work/parquetWorkDir
  13. converter.classes=org.apache.gobblin.converter.parquet.JsonIntermediateToParquetGroupConverter
  14.  
  15. extract.table.name=TestData
  16. extract.namespace=org.apache.gobblin.example
  17. extract.table.type=APPEND_ONLY
  18.  
  19.  
  20. state.store.enabled=true
  21. state.store.fs.uri=${fs.uri}
  22. state.store.dir=${work.dir}/store
  23.  
  24.  
  25. writer.destination.type=HDFS
  26. writer.output.format=PARQUET
  27. writer.fs.uri=${fs.uri}
  28. writer.builder.class=org.apache.gobblin.writer.ParquetDataWriterBuilder
  29.  
  30. data.publisher.fs.uri=${fs.uri}
  31. data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher
  32. data.publisher.final.dir=${work.dir}/test_data
  33. data.publisher.metadata.output.dir=${work.dir}/metadata
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
Top