Adding the Jar into the job.addFileToClassPath("/home/anji/web/simple-json.jar").
You can use on of the three options to put the org.json.* classes into your cluster:
Pack the org.json.* classes into your jar file (can easily be done using GUI IDE). This is the option I used in my answer
Put the jar file containing org.json.* classes on each of the cluster nodes into one of the CLASSPATH directories
(see yarn.application.classpath)
Put the jar file containing org.json.* into HDFS (hdfs dfs -put <org.json jar> <hdfs path>)
and use job.addFileToClassPath call for this jar file to be available for all of the tasks executing your job on the cluster.
In my answer you should add job.addFileToClassPath(new Path("<jar_file_on_hdfs_location>")); to the main
======================================================================================================
======================================================================================================
The mapper code is completely abstracted from the csv format now. It received an custom writable object as an input. The key is the txnid and the value is the TxnRecordWritable. And it retrieves the required fields by calling getter methods.
Reducer code remains same as in the earlier post.
Let's export the driver, mapper and reducers code into a jar file called groupings.jar
Now we will run the program as follows.
We will need to add the dataformts.jar file to the HADOOP_CLASSPATH first for the driver code to find it. And pass it in -libjars option to be added to classpath of the map and reduce jvms.
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/root/localrepo/dataformats.jar
Let's run the program now.
yarn jar grouping.jar com.bigdataleap.samples.customs.GroupingDriver
-libjars /root/lab/programs/dataformats.jar
/lab/mr/txns /lab/output/gr3