SHARE
TWEET

streamingpro_test

hivefans Apr 6th, 2017 21 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. {
  2.   "convert_data_parquet": {
  3.     "desc": "测试",
  4.     "strategy": "streaming.core.strategy.SparkStreamingStrategy",
  5.     "algorithm": [],
  6.     "ref": [],
  7.     "compositor": [
  8.       {
  9.         "name": "streaming.core.compositor.spark.source.HDFSSourceCompositor",
  10.         "params": [
  11.           {
  12.             "path":"hdfs://....."
  13.           }
  14.         ]
  15.       },
  16.       {
  17.         "name": "streaming.core.compositor.spark.transformation.SingleColumnJSONCompositor",
  18.         "params": [
  19.           {
  20.             "name": "raw"
  21.           }
  22.         ]
  23.       },
  24.       {
  25.         "name": "streaming.core.compositor.spark.transformation.JSONTableCompositor",
  26.         "params": [
  27.           {
  28.             "tableName": "test"
  29.           }
  30.         ]
  31.       },
  32.       {
  33.         "name": "streaming.core.compositor.spark.transformation.SQLCompositor",
  34.         "params": [
  35.           {
  36.             "sql": "select split(raw,'\\007|\\001')[2] from test limit 100"
  37.           }
  38.         ]
  39.       },
  40.       {
  41.         "name": "streaming.core.compositor.spark.output.SQLOutputCompositor",
  42.         "params": [
  43.           {
  44.             "format": "parquet",
  45.             "path":"",
  46.             "mode": ""
  47.           }
  48.         ]
  49.       }
  50.     ],
  51.     "configParams": {
  52.     }
  53.   }
  54. }
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top