elcome to the Team 5 WikiStats tool.
README
The WikiStats tool is used for analyzing statistics about page views for WikiPedia.
CONFIGURATION
To configure the tool, open "wikistats.pbs". Here, modify the number of nodes to use for this run. In addition, specify the max amount of time (wall time) to run for. Modify the below lines in "wikistats.pbs"
PBS -l nodes=NO_OF_NODES:ppn=8,pmem=2gb
PBS -l walltime=HH:MM:SS
Next, edit the command line arguments section (lines 33-37). The arguments are: ARGSLANGS : The M popular languages to analyze the spikes for. ARGSSPIKES : The N top page spikes to view for each of the popular languages. ARGSODAYS : The O day period to use for this run. This will be every O day period over the input data. ARGSNODES : The number of nodes to use for this run. This is required as an argument in order to calculate the ideal number of reducers to use. ARGSMAXDAYS : The number of days to consider data over.
Next, go to LOCALINPUT in line 50. Set this to the directory where the data is stored for input. Ideally, this directory will contain the same number of days as set in the max days args. Any additional files in the directory will be read over, but the data within will not be considered.
Finally, go to LOCALOUTPUT in line 54. Set this to the directory where the output should be stored. The output will come in one file.
For a collection of Wikistats data, see: /lustre/cs562178/wikistats The data in this directory is not guarenteed to be accurate or persistent.
EXECUTION
To run the program, execute: chmod u+x runit.sh ./runit.sh
INPUT:
Files within the given input directory should be named according to the following protocol: pagecounts-yyyymmdd-hh0000.gz Where yyyy is the year, mm is the month, dd is the day, and hh is the 24-hr associated with the corresponding wikistats data.
Each input file should contain data with each line according to the following protocol: ln pgName pageViews bytes Where ln is the language code, pgName is the name of the Wikipedia page, pageViews are the number of page views for that time section, and bytes is the size of the page. Language codes other than two characters long will be ignored.
OUTPUT:
The output for this tool will come in one file, with the following format: ln1 pageName1 pageSpike1 ... ln1 pageNameN pageSpikeN Language: ln1 uniquePages ... lnM pageName1 pageSpike1 ... lnM pageNameN pageSpikeN Language: lnM uniquePages
The pages with the highest spikes for a language are output first, followed by that language and the unique pages for the dataset. This is repeated for each of desired top languages.
For sample output, see 5-10-1-5.txt. This file uses parameters of: M = 5, N = 10, and O = 1. This is for 5 days of input data.