API tools faq
paste
ottodude125

apache-nutch-2.3.log

ottodude125
Feb 26th, 2015
396
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 459.80 KB | None | 0 0
raw download clone embed print report
  1.  
  2. 2015-02-26 10:22:27,928 INFO crawl.InjectorJob - InjectorJob: starting at 2015-02-26 10:22:27
  3. 2015-02-26 10:22:27,929 INFO crawl.InjectorJob - InjectorJob: Injecting urlDir: urls/seeds.txt
  4. 2015-02-26 10:22:28,142 DEBUG security.Groups - Creating new Groups object
  5. 2015-02-26 10:22:28,151 DEBUG security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
  6. 2015-02-26 10:22:28,226 DEBUG security.UserGroupInformation - hadoop login
  7. 2015-02-26 10:22:28,226 DEBUG security.UserGroupInformation - hadoop login commit
  8. 2015-02-26 10:22:28,231 DEBUG security.UserGroupInformation - using local user:UnixPrincipal: katonj
  9. 2015-02-26 10:22:28,233 DEBUG security.UserGroupInformation - UGI loginUser:katonj
  10. 2015-02-26 10:22:28,307 DEBUG fs.FileSystem - Creating filesystem for file:///
  11. 2015-02-26 10:22:28,941 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'webpage' in cluster 'Test Cluster' on host 'localhost:9160'.
  12. 2015-02-26 10:22:28,941 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'host' in cluster 'Test Cluster' on host 'localhost:9160'.
  13. 2015-02-26 10:22:28,941 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.WebPage' for Keyspace 'webpage'.
  14. 2015-02-26 10:22:28,942 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.Host' for Keyspace 'host'.
  15. 2015-02-26 10:22:28,942 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  16. 2015-02-26 10:22:28,943 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  17. 2015-02-26 10:22:28,943 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  18. 2015-02-26 10:22:28,943 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  19. 2015-02-26 10:22:28,943 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  20. 2015-02-26 10:22:28,943 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  21. 2015-02-26 10:22:28,943 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  22. 2015-02-26 10:22:28,979 DEBUG store.CassandraMapping - Located column family: 'p'
  23. 2015-02-26 10:22:28,979 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  24. 2015-02-26 10:22:28,979 DEBUG store.CassandraMapping - Located column family: 'f'
  25. 2015-02-26 10:22:28,979 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  26. 2015-02-26 10:22:28,979 DEBUG store.CassandraMapping - Located column family: 'sc'
  27. 2015-02-26 10:22:28,979 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  28. 2015-02-26 10:22:28,980 DEBUG store.CassandraMapping - Located super column family
  29. 2015-02-26 10:22:28,980 DEBUG store.CassandraMapping - Added super column family: 'sc'
  30. 2015-02-26 10:22:29,041 INFO connection.CassandraHostRetryService - Downed Host Retry service started with queue size -1 and retry delay 10s
  31. 2015-02-26 10:22:29,128 INFO service.JmxMonitor - Registering JMX me.prettyprint.cassandra.service_Test Cluster:ServiceType=hector,MonitorType=hector
  32. 2015-02-26 10:22:29,166 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  33. 2015-02-26 10:22:29,175 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  34. 2015-02-26 10:22:29,175 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  35. 2015-02-26 10:22:29,176 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  36. 2015-02-26 10:22:29,176 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  37. 2015-02-26 10:22:29,176 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  38. 2015-02-26 10:22:29,177 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  39. 2015-02-26 10:22:29,177 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  40. 2015-02-26 10:22:29,177 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  41. 2015-02-26 10:22:29,178 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  42. 2015-02-26 10:22:29,178 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  43. 2015-02-26 10:22:29,178 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  44. 2015-02-26 10:22:29,179 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  45. 2015-02-26 10:22:29,179 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  46. 2015-02-26 10:22:29,179 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  47. 2015-02-26 10:22:29,180 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  48. 2015-02-26 10:22:29,180 DEBUG connection.ConcurrentHClientPool - Concurrent Host pool started with 16 active clients; max: 50 exhausted wait: 0
  49. 2015-02-26 10:22:29,493 DEBUG store.CassandraClient - ColumnFamily consistency level configured to 'QUORUM'.
  50. 2015-02-26 10:22:29,493 ERROR store.CassandraStore -
  51. 2015-02-26 10:22:29,494 ERROR store.CassandraStore - [Ljava.lang.StackTraceElement;@7481ca2
  52. 2015-02-26 10:22:29,510 INFO crawl.InjectorJob - InjectorJob: Using class org.apache.gora.cassandra.store.CassandraStore as the Gora storage class.
  53. 2015-02-26 10:22:29,511 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapreduce.Job.connect(Job.java:561)
  54. 2015-02-26 10:22:29,546 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  55. 2015-02-26 10:22:29,557 DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
  56. 2015-02-26 10:22:29,558 DEBUG util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
  57. 2015-02-26 10:22:29,558 DEBUG util.NativeCodeLoader - java.library.path=/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/lib/native/Linux-amd64-64
  58. 2015-02-26 10:22:29,558 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  59. 2015-02-26 10:22:29,566 DEBUG mapred.JobClient - adding the following namenodes' delegation tokens:null
  60. 2015-02-26 10:22:29,566 DEBUG mapred.JobClient - default FileSystem: file:///
  61. 2015-02-26 10:22:29,631 DEBUG mapred.JobClient - Creating splits at file:/tmp/hadoop-katonj/mapred/staging/katonj217590972/.staging/job_local217590972_0001
  62. 2015-02-26 10:22:29,644 INFO input.FileInputFormat - Total input paths to process : 1
  63. 2015-02-26 10:22:29,660 WARN snappy.LoadSnappy - Snappy native library not loaded
  64. 2015-02-26 10:22:29,662 DEBUG input.FileInputFormat - Total # of splits: 1
  65. 2015-02-26 10:22:29,892 DEBUG mapred.JobClient - Printing tokens for job: job_local217590972_0001
  66. 2015-02-26 10:22:29,917 DEBUG mapred.TaskRunner - Fully deleting contents of /tmp/hadoop-katonj/mapred/local/localRunner
  67. 2015-02-26 10:22:30,033 INFO mapred.JobClient - Running job: job_local217590972_0001
  68. 2015-02-26 10:22:30,068 DEBUG mapred.LocalJobRunner - Starting thread pool executor.
  69. 2015-02-26 10:22:30,069 DEBUG mapred.LocalJobRunner - Max local threads: 1
  70. 2015-02-26 10:22:30,069 DEBUG mapred.LocalJobRunner - Map tasks to process: 1
  71. 2015-02-26 10:22:30,069 INFO mapred.LocalJobRunner - Waiting for map tasks
  72. 2015-02-26 10:22:30,073 INFO mapred.LocalJobRunner - Starting task: attempt_local217590972_0001_m_000000_0
  73. 2015-02-26 10:22:30,091 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  74. 2015-02-26 10:22:30,103 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  75. 2015-02-26 10:22:30,103 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  76. 2015-02-26 10:22:30,122 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local217590972_0001/attempt_local217590972_0001_m_000000_0
  77. 2015-02-26 10:22:30,127 DEBUG mapred.Task - using new api for output committer
  78. 2015-02-26 10:22:30,145 INFO util.ProcessTree - setsid exited with exit code 0
  79. 2015-02-26 10:22:30,151 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@381369f0
  80. 2015-02-26 10:22:30,157 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  81. 2015-02-26 10:22:30,157 INFO mapred.MapTask - Processing split: file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/urls/seeds.txt:0+24
  82. 2015-02-26 10:22:30,158 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  83. 2015-02-26 10:22:30,162 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  84. 2015-02-26 10:22:30,162 DEBUG mapred.Counters - Adding BYTES_READ
  85. 2015-02-26 10:22:30,164 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  86. 2015-02-26 10:22:30,167 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  87. 2015-02-26 10:22:30,167 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  88. 2015-02-26 10:22:30,170 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  89. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  90. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  91. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  92. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  93. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  94. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  95. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located column family: 'p'
  96. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  97. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located column family: 'f'
  98. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  99. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located column family: 'sc'
  100. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  101. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Located super column family
  102. 2015-02-26 10:22:30,170 DEBUG store.CassandraMapping - Added super column family: 'sc'
  103. 2015-02-26 10:22:30,178 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  104. 2015-02-26 10:22:30,178 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  105. 2015-02-26 10:22:30,178 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  106. 2015-02-26 10:22:30,189 INFO mapreduce.GoraRecordWriter - gora.buffer.write.limit = 10000
  107. 2015-02-26 10:22:30,200 INFO plugin.PluginRepository - Plugins: looking in: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins
  108. 2015-02-26 10:22:30,201 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-xml/plugin.xml
  109. 2015-02-26 10:22:30,209 DEBUG plugin.PluginRepository - plugin: id=lib-xml name=XML Libraries version=1.0 provider=org.apache.nutch.xmlclass=null
  110. 2015-02-26 10:22:30,210 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-sftp/plugin.xml
  111. 2015-02-26 10:22:30,213 DEBUG plugin.PluginRepository - plugin: id=protocol-sftp name=Sftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  112. 2015-02-26 10:22:30,213 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.sftp.Sftp
  113. 2015-02-26 10:22:30,215 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-automaton/plugin.xml
  114. 2015-02-26 10:22:30,218 DEBUG plugin.PluginRepository - plugin: id=urlfilter-automaton name=Automaton URL Filter version=1.0.0 provider=nutch.orgclass=null
  115. 2015-02-26 10:22:30,218 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.automaton.AutomatonURLFilter
  116. 2015-02-26 10:22:30,219 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-tika/plugin.xml
  117. 2015-02-26 10:22:30,223 DEBUG plugin.PluginRepository - plugin: id=parse-tika name=Tika Parser Plug-in version=1.0.0 provider=nutch.orgclass=null
  118. 2015-02-26 10:22:30,224 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.tika.TikaParser
  119. 2015-02-26 10:22:30,235 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-elastic/plugin.xml
  120. 2015-02-26 10:22:30,239 DEBUG plugin.PluginRepository - plugin: id=indexer-elastic name=ElasticIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  121. 2015-02-26 10:22:30,239 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
  122. 2015-02-26 10:22:30,244 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/microformats-reltag/plugin.xml
  123. 2015-02-26 10:22:30,247 DEBUG plugin.PluginRepository - plugin: id=microformats-reltag name=Rel-Tag microformat Parser/Indexer/Querier version=1.0.0 provider=nutch.orgclass=null
  124. 2015-02-26 10:22:30,247 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.microformats.reltag.RelTagParser
  125. 2015-02-26 10:22:30,247 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.microformats.reltag.RelTagIndexingFilter
  126. 2015-02-26 10:22:30,248 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-html/plugin.xml
  127. 2015-02-26 10:22:30,251 DEBUG plugin.PluginRepository - plugin: id=parse-html name=Html Parse Plug-in version=1.0.0 provider=nutch.orgclass=null
  128. 2015-02-26 10:22:30,251 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.html.HtmlParser
  129. 2015-02-26 10:22:30,252 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/subcollection/plugin.xml
  130. 2015-02-26 10:22:30,255 DEBUG plugin.PluginRepository - plugin: id=subcollection name=Subcollection indexing and query filter version=1.0.0 provider=apache.orgclass=null
  131. 2015-02-26 10:22:30,255 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.subcollection.SubcollectionIndexingFilter
  132. 2015-02-26 10:22:30,256 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-metadata/plugin.xml
  133. 2015-02-26 10:22:30,258 DEBUG plugin.PluginRepository - plugin: id=index-metadata name=Metadata Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  134. 2015-02-26 10:22:30,259 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.metadata.MetadataIndexer
  135. 2015-02-26 10:22:30,259 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-httpclient/plugin.xml
  136. 2015-02-26 10:22:30,262 DEBUG plugin.PluginRepository - plugin: id=protocol-httpclient name=Http / Https Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  137. 2015-02-26 10:22:30,262 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  138. 2015-02-26 10:22:30,263 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  139. 2015-02-26 10:22:30,263 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/nutch-extensionpoints/plugin.xml
  140. 2015-02-26 10:22:30,266 DEBUG plugin.PluginRepository - plugin: id=nutch-extensionpoints name=the nutch core extension points version=2.0.0 provider=nutch.orgclass=null
  141. 2015-02-26 10:22:30,268 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-pass/plugin.xml
  142. 2015-02-26 10:22:30,271 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-pass name=Pass-through URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  143. 2015-02-26 10:22:30,271 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer
  144. 2015-02-26 10:22:30,271 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-regex/plugin.xml
  145. 2015-02-26 10:22:30,274 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-regex name=Regex URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  146. 2015-02-26 10:22:30,274 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
  147. 2015-02-26 10:22:30,275 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-ftp/plugin.xml
  148. 2015-02-26 10:22:30,278 DEBUG plugin.PluginRepository - plugin: id=protocol-ftp name=Ftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  149. 2015-02-26 10:22:30,278 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.ftp.Ftp
  150. 2015-02-26 10:22:30,279 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-nekohtml/plugin.xml
  151. 2015-02-26 10:22:30,281 DEBUG plugin.PluginRepository - plugin: id=lib-nekohtml name=CyberNeko HTML Parser version=1.9.19 provider=net.sourceforge.nekohtmlclass=null
  152. 2015-02-26 10:22:30,282 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-http/plugin.xml
  153. 2015-02-26 10:22:30,285 DEBUG plugin.PluginRepository - plugin: id=protocol-http name=Http Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  154. 2015-02-26 10:22:30,285 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  155. 2015-02-26 10:22:30,285 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  156. 2015-02-26 10:22:30,286 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-solr/plugin.xml
  157. 2015-02-26 10:22:30,289 DEBUG plugin.PluginRepository - plugin: id=indexer-solr name=SOLRIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  158. 2015-02-26 10:22:30,290 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.solr.SolrIndexWriter
  159. 2015-02-26 10:22:30,292 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/language-identifier/plugin.xml
  160. 2015-02-26 10:22:30,294 DEBUG plugin.PluginRepository - plugin: id=language-identifier name=Language Identification Parser/Filter version=1.0.0 provider=nutch.orgclass=null
  161. 2015-02-26 10:22:30,295 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.analysis.lang.HTMLLanguageParser
  162. 2015-02-26 10:22:30,295 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.analysis.lang.LanguageIndexingFilter
  163. 2015-02-26 10:22:30,299 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-basic/plugin.xml
  164. 2015-02-26 10:22:30,301 DEBUG plugin.PluginRepository - plugin: id=index-basic name=Basic Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  165. 2015-02-26 10:22:30,302 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.basic.BasicIndexingFilter
  166. 2015-02-26 10:22:30,302 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-link/plugin.xml
  167. 2015-02-26 10:22:30,305 DEBUG plugin.PluginRepository - plugin: id=scoring-link name=Link Analysis Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  168. 2015-02-26 10:22:30,305 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.link.LinkAnalysisScoringFilter
  169. 2015-02-26 10:22:30,306 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-http/plugin.xml
  170. 2015-02-26 10:22:30,308 DEBUG plugin.PluginRepository - plugin: id=lib-http name=HTTP Framework version=1.0 provider=org.apache.nutchclass=null
  171. 2015-02-26 10:22:30,309 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-regex-filter/plugin.xml
  172. 2015-02-26 10:22:30,312 DEBUG plugin.PluginRepository - plugin: id=lib-regex-filter name=Regex URL Filter Framework version=1.0 provider=org.apache.nutchclass=null
  173. 2015-02-26 10:22:30,312 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-prefix/plugin.xml
  174. 2015-02-26 10:22:30,315 DEBUG plugin.PluginRepository - plugin: id=urlfilter-prefix name=Prefix URL Filter version=1.0.0 provider=nutch.orgclass=null
  175. 2015-02-26 10:22:30,315 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.prefix.PrefixURLFilter
  176. 2015-02-26 10:22:30,316 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-basic/plugin.xml
  177. 2015-02-26 10:22:30,319 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-basic name=Basic URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  178. 2015-02-26 10:22:30,319 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
  179. 2015-02-26 10:22:30,320 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/tld/plugin.xml
  180. 2015-02-26 10:22:30,322 DEBUG plugin.PluginRepository - plugin: id=tld name=Top Level Domain Plugin version=1.0.0 provider=nutch.orgclass=null
  181. 2015-02-26 10:22:30,323 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.tld.TLDIndexingFilter
  182. 2015-02-26 10:22:30,323 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.tld.TLDScoringFilter
  183. 2015-02-26 10:22:30,323 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-anchor/plugin.xml
  184. 2015-02-26 10:22:30,326 DEBUG plugin.PluginRepository - plugin: id=index-anchor name=Anchor Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  185. 2015-02-26 10:22:30,326 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.anchor.AnchorIndexingFilter
  186. 2015-02-26 10:22:30,328 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-suffix/plugin.xml
  187. 2015-02-26 10:22:30,331 DEBUG plugin.PluginRepository - plugin: id=urlfilter-suffix name=Suffix URL Filter version=1.0.0 provider=nutch.orgclass=null
  188. 2015-02-26 10:22:30,331 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.suffix.SuffixURLFilter
  189. 2015-02-26 10:22:30,332 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-regex/plugin.xml
  190. 2015-02-26 10:22:30,334 DEBUG plugin.PluginRepository - plugin: id=urlfilter-regex name=Regex URL Filter version=1.0.0 provider=nutch.orgclass=null
  191. 2015-02-26 10:22:30,335 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.regex.RegexURLFilter
  192. 2015-02-26 10:22:30,335 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-metatags/plugin.xml
  193. 2015-02-26 10:22:30,338 DEBUG plugin.PluginRepository - plugin: id=parse-metatags name=MetaTags version=1.0 provider=digitalpebble.comclass=null
  194. 2015-02-26 10:22:30,338 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.metatags.MetaTagsParser
  195. 2015-02-26 10:22:30,339 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-domain/plugin.xml
  196. 2015-02-26 10:22:30,341 DEBUG plugin.PluginRepository - plugin: id=urlfilter-domain name=Domain URL Filter version=1.0.0 provider=nutch.orgclass=null
  197. 2015-02-26 10:22:30,342 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.domain.DomainURLFilter
  198. 2015-02-26 10:22:30,342 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-opic/plugin.xml
  199. 2015-02-26 10:22:30,345 DEBUG plugin.PluginRepository - plugin: id=scoring-opic name=OPIC Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  200. 2015-02-26 10:22:30,345 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.opic.OPICScoringFilter
  201. 2015-02-26 10:22:30,346 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-more/plugin.xml
  202. 2015-02-26 10:22:30,349 DEBUG plugin.PluginRepository - plugin: id=index-more name=More Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  203. 2015-02-26 10:22:30,349 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.more.MoreIndexingFilter
  204. 2015-02-26 10:22:30,349 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/creativecommons/plugin.xml
  205. 2015-02-26 10:22:30,352 DEBUG plugin.PluginRepository - plugin: id=creativecommons name=Creative Commons Plugins version=1.0.0 provider=nutch.orgclass=null
  206. 2015-02-26 10:22:30,353 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.creativecommons.nutch.CCParseFilter
  207. 2015-02-26 10:22:30,353 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.creativecommons.nutch.CCIndexingFilter
  208. 2015-02-26 10:22:30,353 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-file/plugin.xml
  209. 2015-02-26 10:22:30,356 DEBUG plugin.PluginRepository - plugin: id=protocol-file name=File Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  210. 2015-02-26 10:22:30,357 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.file.File
  211. 2015-02-26 10:22:30,357 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-js/plugin.xml
  212. 2015-02-26 10:22:30,360 DEBUG plugin.PluginRepository - plugin: id=parse-js name=JavaScript Parser version=1.0.0 provider=nutch.orgclass=null
  213. 2015-02-26 10:22:30,361 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.js.JSParseFilter
  214. 2015-02-26 10:22:30,361 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.js.JSParseFilter
  215. 2015-02-26 10:22:30,362 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-validator/plugin.xml
  216. 2015-02-26 10:22:30,364 DEBUG plugin.PluginRepository - plugin: id=urlfilter-validator name=URL Validator version=1.0.0 provider=nutch.orgclass=null
  217. 2015-02-26 10:22:30,364 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.validator.UrlValidator
  218. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: urlfilter-suffix
  219. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: language-identifier
  220. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: protocol-httpclient
  221. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: scoring-link
  222. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: indexer-elastic
  223. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: creativecommons
  224. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: index-more
  225. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: lib-http
  226. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: lib-xml
  227. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: subcollection
  228. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: urlfilter-prefix
  229. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: lib-regex-filter
  230. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: nutch-extensionpoints
  231. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: urlfilter-automaton
  232. 2015-02-26 10:22:30,366 DEBUG plugin.PluginRepository - not including: urlfilter-validator
  233. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - not including: protocol-sftp
  234. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - not including: protocol-ftp
  235. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - not including: lib-nekohtml
  236. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - not including: parse-js
  237. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - not including: protocol-file
  238. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - not including: urlfilter-domain
  239. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - not including: microformats-reltag
  240. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - not including: tld
  241. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexingFilter
  242. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexCleaningFilter
  243. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.Parser
  244. 2015-02-26 10:22:30,367 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.ParseFilter
  245. 2015-02-26 10:22:30,368 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.protocol.Protocol
  246. 2015-02-26 10:22:30,368 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLFilter
  247. 2015-02-26 10:22:30,368 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLNormalizer
  248. 2015-02-26 10:22:30,368 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.scoring.ScoringFilter
  249. 2015-02-26 10:22:30,368 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexWriter
  250. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
  251. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Registered Plugins:
  252. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
  253. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Basic URL Normalizer (urlnormalizer-basic)
  254. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
  255. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
  256. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - SOLRIndexWriter (indexer-solr)
  257. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - HTTP Framework (lib-http)
  258. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
  259. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Pass-through URL Normalizer (urlnormalizer-pass)
  260. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Http Protocol Plug-in (protocol-http)
  261. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Regex URL Normalizer (urlnormalizer-regex)
  262. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
  263. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Tika Parser Plug-in (parse-tika)
  264. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
  265. 2015-02-26 10:22:30,368 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
  266. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
  267. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - MetaTags (parse-metatags)
  268. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Metadata Indexing Filter (index-metadata)
  269. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Registered Extension-Points:
  270. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
  271. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Nutch Index Cleaning Filter (org.apache.nutch.indexer.IndexCleaningFilter)
  272. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
  273. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Parse Filter (org.apache.nutch.parse.ParseFilter)
  274. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
  275. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Nutch Index Writer (org.apache.nutch.indexer.IndexWriter)
  276. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
  277. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
  278. 2015-02-26 10:22:30,369 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
  279. 2015-02-26 10:22:30,370 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local217590972_0001.xml, instantiating a new object cache
  280. 2015-02-26 10:22:30,395 INFO conf.Configuration - found resource regex-normalize.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/regex-normalize.xml
  281. 2015-02-26 10:22:30,411 INFO conf.Configuration - found resource regex-urlfilter.txt at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/regex-urlfilter.txt
  282. 2015-02-26 10:22:30,428 INFO regex.RegexURLNormalizer - can't find rules for scope 'inject', using default
  283. 2015-02-26 10:22:30,474 DEBUG mapred.Counters - Creating group injector with nothing
  284. 2015-02-26 10:22:30,474 DEBUG mapred.Counters - Adding urls_injected
  285. 2015-02-26 10:22:30,482 DEBUG store.CassandraStore - close
  286. 2015-02-26 10:22:30,483 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.LongSerializer selected as appropriate Serializer.
  287. 2015-02-26 10:22:30,483 DEBUG store.CassandraClient - fieldName:fetchTime columnName:ts
  288. 2015-02-26 10:22:30,483 DEBUG store.CassandraClient - ttl for field:fetchTime is 0
  289. 2015-02-26 10:22:30,504 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  290. 2015-02-26 10:22:30,529 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  291. 2015-02-26 10:22:30,530 DEBUG store.CassandraClient - fieldName:fetchInterval columnName:fi
  292. 2015-02-26 10:22:30,530 DEBUG store.CassandraClient - ttl for field:fetchInterval is 0
  293. 2015-02-26 10:22:30,530 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  294. 2015-02-26 10:22:30,533 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.FloatSerializer selected as appropriate Serializer.
  295. 2015-02-26 10:22:30,533 DEBUG store.CassandraClient - fieldName:score columnName:s
  296. 2015-02-26 10:22:30,533 DEBUG store.CassandraClient - ttl for field:score is 0
  297. 2015-02-26 10:22:30,533 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  298. 2015-02-26 10:22:30,559 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  299. 2015-02-26 10:22:30,570 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  300. 2015-02-26 10:22:30,570 DEBUG store.CassandraClient - ttl for field:markers is 0
  301. 2015-02-26 10:22:30,575 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  302. 2015-02-26 10:22:30,578 DEBUG store.CassandraClient - class org.apache.gora.cassandra.serializers.CharSequenceSerializer selected as appropriate Serializer.
  303. 2015-02-26 10:22:30,578 DEBUG store.CassandraClient - ttl for field:markers is 0
  304. 2015-02-26 10:22:30,579 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  305. 2015-02-26 10:22:30,580 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  306. 2015-02-26 10:22:30,580 DEBUG store.CassandraClient - ttl for field:markers is 0
  307. 2015-02-26 10:22:30,580 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  308. 2015-02-26 10:22:30,582 DEBUG store.CassandraClient - class org.apache.gora.cassandra.serializers.CharSequenceSerializer selected as appropriate Serializer.
  309. 2015-02-26 10:22:30,582 DEBUG store.CassandraClient - ttl for field:markers is 0
  310. 2015-02-26 10:22:30,582 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  311. 2015-02-26 10:22:30,584 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  312. 2015-02-26 10:22:30,585 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  313. 2015-02-26 10:22:30,585 DEBUG store.CassandraClient - ttl for field:metadata is 0
  314. 2015-02-26 10:22:30,585 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  315. 2015-02-26 10:22:30,587 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.ByteBufferSerializer selected as appropriate Serializer.
  316. 2015-02-26 10:22:30,587 DEBUG store.CassandraClient - ttl for field:metadata is 0
  317. 2015-02-26 10:22:30,587 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  318. 2015-02-26 10:22:30,589 INFO mapred.Task - Task:attempt_local217590972_0001_m_000000_0 is done. And is in the process of commiting
  319. 2015-02-26 10:22:30,592 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  320. 2015-02-26 10:22:30,593 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  321. 2015-02-26 10:22:30,593 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  322. 2015-02-26 10:22:30,593 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  323. 2015-02-26 10:22:30,593 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  324. 2015-02-26 10:22:30,593 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  325. 2015-02-26 10:22:30,593 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  326. 2015-02-26 10:22:30,595 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  327. 2015-02-26 10:22:30,595 DEBUG mapred.Counters - Creating group injector with nothing
  328. 2015-02-26 10:22:30,595 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  329. 2015-02-26 10:22:30,595 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  330. 2015-02-26 10:22:30,595 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  331. 2015-02-26 10:22:30,596 INFO mapred.LocalJobRunner -
  332. 2015-02-26 10:22:30,596 INFO mapred.Task - Task 'attempt_local217590972_0001_m_000000_0' done.
  333. 2015-02-26 10:22:30,596 INFO mapred.LocalJobRunner - Finishing task: attempt_local217590972_0001_m_000000_0
  334. 2015-02-26 10:22:30,596 INFO mapred.LocalJobRunner - Map task executor complete.
  335. 2015-02-26 10:22:30,597 WARN mapred.FileOutputCommitter - Output path is null in cleanup
  336. 2015-02-26 10:22:31,036 INFO mapred.JobClient - map 100% reduce 0%
  337. 2015-02-26 10:22:31,038 INFO mapred.JobClient - Job complete: job_local217590972_0001
  338. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  339. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  340. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Creating group injector with nothing
  341. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Adding urls_injected
  342. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  343. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Adding BYTES_READ
  344. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  345. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  346. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  347. 2015-02-26 10:22:31,038 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  348. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  349. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  350. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  351. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  352. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  353. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  354. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  355. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  356. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  357. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  358. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Creating group injector with nothing
  359. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding urls_injected
  360. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  361. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  362. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  363. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  364. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding BYTES_READ
  365. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  366. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  367. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  368. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  369. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  370. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  371. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  372. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  373. 2015-02-26 10:22:31,039 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  374. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Counters: 13
  375. 2015-02-26 10:22:31,040 INFO mapred.JobClient - File Output Format Counters
  376. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Bytes Written=0
  377. 2015-02-26 10:22:31,040 INFO mapred.JobClient - injector
  378. 2015-02-26 10:22:31,040 INFO mapred.JobClient - urls_injected=1
  379. 2015-02-26 10:22:31,040 INFO mapred.JobClient - File Input Format Counters
  380. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Bytes Read=24
  381. 2015-02-26 10:22:31,040 INFO mapred.JobClient - FileSystemCounters
  382. 2015-02-26 10:22:31,040 INFO mapred.JobClient - FILE_BYTES_READ=608085
  383. 2015-02-26 10:22:31,040 INFO mapred.JobClient - FILE_BYTES_WRITTEN=693328
  384. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Map-Reduce Framework
  385. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Map input records=1
  386. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Physical memory (bytes) snapshot=0
  387. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Spilled Records=0
  388. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Total committed heap usage (bytes)=758120448
  389. 2015-02-26 10:22:31,040 INFO mapred.JobClient - CPU time spent (ms)=0
  390. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Virtual memory (bytes) snapshot=0
  391. 2015-02-26 10:22:31,040 INFO mapred.JobClient - SPLIT_RAW_BYTES=160
  392. 2015-02-26 10:22:31,040 INFO mapred.JobClient - Map output records=1
  393. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  394. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  395. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Creating group injector with nothing
  396. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Adding urls_injected
  397. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  398. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Adding BYTES_READ
  399. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  400. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  401. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  402. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  403. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  404. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  405. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  406. 2015-02-26 10:22:31,041 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  407. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  408. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  409. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  410. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  411. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  412. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  413. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Creating group injector with nothing
  414. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding urls_injected
  415. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  416. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  417. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  418. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  419. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding BYTES_READ
  420. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  421. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  422. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  423. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  424. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  425. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  426. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  427. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  428. 2015-02-26 10:22:31,042 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  429. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  430. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  431. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Creating group injector with nothing
  432. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding urls_injected
  433. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  434. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding BYTES_READ
  435. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  436. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  437. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  438. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  439. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  440. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  441. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  442. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  443. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  444. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  445. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  446. 2015-02-26 10:22:31,043 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  447. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  448. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  449. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Creating group injector with nothing
  450. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding urls_injected
  451. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  452. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  453. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  454. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  455. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding BYTES_READ
  456. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  457. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  458. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  459. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  460. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  461. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  462. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  463. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  464. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  465. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  466. 2015-02-26 10:22:31,044 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  467. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Creating group injector with nothing
  468. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding urls_injected
  469. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  470. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding BYTES_READ
  471. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  472. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  473. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  474. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  475. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  476. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  477. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  478. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  479. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  480. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  481. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  482. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  483. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  484. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  485. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Creating group injector with nothing
  486. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding urls_injected
  487. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  488. 2015-02-26 10:22:31,045 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  489. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  490. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  491. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding BYTES_READ
  492. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  493. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  494. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  495. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  496. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  497. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  498. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  499. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  500. 2015-02-26 10:22:31,046 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  501. 2015-02-26 10:22:31,046 INFO crawl.InjectorJob - InjectorJob: total number of urls rejected by filters: 0
  502. 2015-02-26 10:22:31,046 INFO crawl.InjectorJob - InjectorJob: total number of urls injected after normalization and filtering: 1
  503. 2015-02-26 10:22:31,047 INFO crawl.InjectorJob - Injector: finished at 2015-02-26 10:22:31, elapsed: 00:00:03
  504. 2015-02-26 10:22:31,048 DEBUG fs.FileSystem - Starting clear of FileSystem cache with 1 elements.
  505. 2015-02-26 10:22:31,048 DEBUG fs.FileSystem - Removing filesystem for file:///
  506. 2015-02-26 10:22:31,048 DEBUG fs.FileSystem - Removing filesystem for file:///
  507. 2015-02-26 10:22:31,048 DEBUG fs.FileSystem - Done clearing cache
  508. 2015-02-26 10:22:32,396 INFO crawl.GeneratorJob - GeneratorJob: starting at 2015-02-26 10:22:32
  509. 2015-02-26 10:22:32,397 INFO crawl.GeneratorJob - GeneratorJob: Selecting best-scoring urls due for fetch.
  510. 2015-02-26 10:22:32,397 INFO crawl.GeneratorJob - GeneratorJob: starting
  511. 2015-02-26 10:22:32,397 INFO crawl.GeneratorJob - GeneratorJob: filtering: false
  512. 2015-02-26 10:22:32,397 INFO crawl.GeneratorJob - GeneratorJob: normalizing: false
  513. 2015-02-26 10:22:32,397 INFO crawl.GeneratorJob - GeneratorJob: topN: 50000
  514. 2015-02-26 10:22:32,603 DEBUG security.Groups - Creating new Groups object
  515. 2015-02-26 10:22:32,606 DEBUG security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
  516. 2015-02-26 10:22:32,672 DEBUG security.UserGroupInformation - hadoop login
  517. 2015-02-26 10:22:32,673 DEBUG security.UserGroupInformation - hadoop login commit
  518. 2015-02-26 10:22:32,679 DEBUG security.UserGroupInformation - using local user:UnixPrincipal: katonj
  519. 2015-02-26 10:22:32,682 DEBUG security.UserGroupInformation - UGI loginUser:katonj
  520. 2015-02-26 10:22:32,758 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, nutch-default.xml, nutch-site.xml, instantiating a new object cache
  521. 2015-02-26 10:22:32,758 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
  522. 2015-02-26 10:22:32,759 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
  523. 2015-02-26 10:22:32,759 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
  524. 2015-02-26 10:22:33,402 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'webpage' in cluster 'Test Cluster' on host 'localhost:9160'.
  525. 2015-02-26 10:22:33,402 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'host' in cluster 'Test Cluster' on host 'localhost:9160'.
  526. 2015-02-26 10:22:33,402 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.WebPage' for Keyspace 'webpage'.
  527. 2015-02-26 10:22:33,402 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.Host' for Keyspace 'host'.
  528. 2015-02-26 10:22:33,402 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  529. 2015-02-26 10:22:33,404 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  530. 2015-02-26 10:22:33,404 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  531. 2015-02-26 10:22:33,404 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  532. 2015-02-26 10:22:33,404 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  533. 2015-02-26 10:22:33,404 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  534. 2015-02-26 10:22:33,404 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  535. 2015-02-26 10:22:33,443 DEBUG store.CassandraMapping - Located column family: 'p'
  536. 2015-02-26 10:22:33,443 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  537. 2015-02-26 10:22:33,443 DEBUG store.CassandraMapping - Located column family: 'f'
  538. 2015-02-26 10:22:33,443 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  539. 2015-02-26 10:22:33,443 DEBUG store.CassandraMapping - Located column family: 'sc'
  540. 2015-02-26 10:22:33,443 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  541. 2015-02-26 10:22:33,443 DEBUG store.CassandraMapping - Located super column family
  542. 2015-02-26 10:22:33,443 DEBUG store.CassandraMapping - Added super column family: 'sc'
  543. 2015-02-26 10:22:33,505 INFO connection.CassandraHostRetryService - Downed Host Retry service started with queue size -1 and retry delay 10s
  544. 2015-02-26 10:22:33,589 INFO service.JmxMonitor - Registering JMX me.prettyprint.cassandra.service_Test Cluster:ServiceType=hector,MonitorType=hector
  545. 2015-02-26 10:22:33,618 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  546. 2015-02-26 10:22:33,627 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  547. 2015-02-26 10:22:33,628 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  548. 2015-02-26 10:22:33,628 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  549. 2015-02-26 10:22:33,628 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  550. 2015-02-26 10:22:33,629 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  551. 2015-02-26 10:22:33,629 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  552. 2015-02-26 10:22:33,630 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  553. 2015-02-26 10:22:33,630 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  554. 2015-02-26 10:22:33,630 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  555. 2015-02-26 10:22:33,631 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  556. 2015-02-26 10:22:33,631 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  557. 2015-02-26 10:22:33,631 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  558. 2015-02-26 10:22:33,632 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  559. 2015-02-26 10:22:33,632 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  560. 2015-02-26 10:22:33,633 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  561. 2015-02-26 10:22:33,633 DEBUG connection.ConcurrentHClientPool - Concurrent Host pool started with 16 active clients; max: 50 exhausted wait: 0
  562. 2015-02-26 10:22:33,712 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  563. 2015-02-26 10:22:33,712 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  564. 2015-02-26 10:22:33,712 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  565. 2015-02-26 10:22:33,785 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  566. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  567. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  568. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  569. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  570. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  571. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  572. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located column family: 'p'
  573. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  574. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located column family: 'f'
  575. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  576. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located column family: 'sc'
  577. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  578. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Located super column family
  579. 2015-02-26 10:22:33,786 DEBUG store.CassandraMapping - Added super column family: 'sc'
  580. 2015-02-26 10:22:33,789 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  581. 2015-02-26 10:22:33,789 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  582. 2015-02-26 10:22:33,789 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  583. 2015-02-26 10:22:33,790 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapreduce.Job.connect(Job.java:561)
  584. 2015-02-26 10:22:33,819 DEBUG fs.FileSystem - Creating filesystem for file:///
  585. 2015-02-26 10:22:33,836 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  586. 2015-02-26 10:22:33,846 DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
  587. 2015-02-26 10:22:33,847 DEBUG util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
  588. 2015-02-26 10:22:33,847 DEBUG util.NativeCodeLoader - java.library.path=/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/lib/native/Linux-amd64-64
  589. 2015-02-26 10:22:33,847 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  590. 2015-02-26 10:22:33,848 DEBUG mapred.JobClient - adding the following namenodes' delegation tokens:null
  591. 2015-02-26 10:22:33,849 DEBUG mapred.JobClient - default FileSystem: file:///
  592. 2015-02-26 10:22:33,908 DEBUG mapred.JobClient - Creating splits at file:/tmp/hadoop-katonj/mapred/staging/katonj1860589617/.staging/job_local1860589617_0001
  593. 2015-02-26 10:22:33,913 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  594. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  595. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  596. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  597. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  598. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  599. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  600. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located column family: 'p'
  601. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  602. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located column family: 'f'
  603. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  604. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located column family: 'sc'
  605. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  606. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Located super column family
  607. 2015-02-26 10:22:33,914 DEBUG store.CassandraMapping - Added super column family: 'sc'
  608. 2015-02-26 10:22:33,916 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  609. 2015-02-26 10:22:33,916 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  610. 2015-02-26 10:22:33,916 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  611. 2015-02-26 10:22:34,119 DEBUG mapred.JobClient - Printing tokens for job: job_local1860589617_0001
  612. 2015-02-26 10:22:34,143 DEBUG mapred.TaskRunner - Fully deleting contents of /tmp/hadoop-katonj/mapred/local/localRunner
  613. 2015-02-26 10:22:34,232 INFO mapred.JobClient - Running job: job_local1860589617_0001
  614. 2015-02-26 10:22:34,268 DEBUG mapred.LocalJobRunner - Starting thread pool executor.
  615. 2015-02-26 10:22:34,268 DEBUG mapred.LocalJobRunner - Max local threads: 1
  616. 2015-02-26 10:22:34,268 DEBUG mapred.LocalJobRunner - Map tasks to process: 1
  617. 2015-02-26 10:22:34,269 INFO mapred.LocalJobRunner - Waiting for map tasks
  618. 2015-02-26 10:22:34,271 INFO mapred.LocalJobRunner - Starting task: attempt_local1860589617_0001_m_000000_0
  619. 2015-02-26 10:22:34,283 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  620. 2015-02-26 10:22:34,291 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  621. 2015-02-26 10:22:34,291 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  622. 2015-02-26 10:22:34,303 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local1860589617_0001/attempt_local1860589617_0001_m_000000_0
  623. 2015-02-26 10:22:34,306 DEBUG mapred.Task - using new api for output committer
  624. 2015-02-26 10:22:34,317 INFO util.ProcessTree - setsid exited with exit code 0
  625. 2015-02-26 10:22:34,320 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ec2a43e
  626. 2015-02-26 10:22:34,324 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  627. 2015-02-26 10:22:34,324 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  628. 2015-02-26 10:22:34,324 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  629. 2015-02-26 10:22:34,324 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  630. 2015-02-26 10:22:34,324 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  631. 2015-02-26 10:22:34,324 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  632. 2015-02-26 10:22:34,324 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  633. 2015-02-26 10:22:34,324 DEBUG store.CassandraMapping - Located column family: 'p'
  634. 2015-02-26 10:22:34,324 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  635. 2015-02-26 10:22:34,324 DEBUG store.CassandraMapping - Located column family: 'f'
  636. 2015-02-26 10:22:34,325 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  637. 2015-02-26 10:22:34,325 DEBUG store.CassandraMapping - Located column family: 'sc'
  638. 2015-02-26 10:22:34,325 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  639. 2015-02-26 10:22:34,325 DEBUG store.CassandraMapping - Located super column family
  640. 2015-02-26 10:22:34,325 DEBUG store.CassandraMapping - Added super column family: 'sc'
  641. 2015-02-26 10:22:34,326 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  642. 2015-02-26 10:22:34,326 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  643. 2015-02-26 10:22:34,326 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  644. 2015-02-26 10:22:34,331 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  645. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  646. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  647. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  648. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  649. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  650. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  651. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located column family: 'p'
  652. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  653. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located column family: 'f'
  654. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  655. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located column family: 'sc'
  656. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  657. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Located super column family
  658. 2015-02-26 10:22:34,331 DEBUG store.CassandraMapping - Added super column family: 'sc'
  659. 2015-02-26 10:22:34,333 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  660. 2015-02-26 10:22:34,333 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  661. 2015-02-26 10:22:34,333 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  662. 2015-02-26 10:22:34,334 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  663. 2015-02-26 10:22:34,334 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  664. 2015-02-26 10:22:34,334 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  665. 2015-02-26 10:22:34,334 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  666. 2015-02-26 10:22:34,334 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  667. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  668. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  669. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - Located column family: 'p'
  670. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  671. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - Located column family: 'f'
  672. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  673. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - Located column family: 'sc'
  674. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  675. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - Located super column family
  676. 2015-02-26 10:22:34,335 DEBUG store.CassandraMapping - Added super column family: 'sc'
  677. 2015-02-26 10:22:34,336 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  678. 2015-02-26 10:22:34,336 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  679. 2015-02-26 10:22:34,336 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  680. 2015-02-26 10:22:34,336 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  681. 2015-02-26 10:22:34,336 INFO mapred.MapTask - Processing split: org.apache.gora.mapreduce.GoraInputSplit@78e1b737
  682. 2015-02-26 10:22:34,337 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  683. 2015-02-26 10:22:34,339 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  684. 2015-02-26 10:22:34,339 DEBUG mapred.Counters - Adding BYTES_READ
  685. 2015-02-26 10:22:34,341 INFO mapreduce.GoraRecordReader - gora.buffer.read.limit = 10000
  686. 2015-02-26 10:22:34,348 INFO mapred.MapTask - io.sort.mb = 100
  687. 2015-02-26 10:22:34,401 INFO mapred.MapTask - data buffer = 79691776/99614720
  688. 2015-02-26 10:22:34,401 INFO mapred.MapTask - record buffer = 262144/327680
  689. 2015-02-26 10:22:34,404 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  690. 2015-02-26 10:22:34,405 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  691. 2015-02-26 10:22:34,405 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  692. 2015-02-26 10:22:34,405 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  693. 2015-02-26 10:22:34,405 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  694. 2015-02-26 10:22:34,417 INFO plugin.PluginRepository - Plugins: looking in: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins
  695. 2015-02-26 10:22:34,418 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-xml/plugin.xml
  696. 2015-02-26 10:22:34,425 DEBUG plugin.PluginRepository - plugin: id=lib-xml name=XML Libraries version=1.0 provider=org.apache.nutch.xmlclass=null
  697. 2015-02-26 10:22:34,426 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-sftp/plugin.xml
  698. 2015-02-26 10:22:34,429 DEBUG plugin.PluginRepository - plugin: id=protocol-sftp name=Sftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  699. 2015-02-26 10:22:34,429 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.sftp.Sftp
  700. 2015-02-26 10:22:34,430 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-automaton/plugin.xml
  701. 2015-02-26 10:22:34,433 DEBUG plugin.PluginRepository - plugin: id=urlfilter-automaton name=Automaton URL Filter version=1.0.0 provider=nutch.orgclass=null
  702. 2015-02-26 10:22:34,433 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.automaton.AutomatonURLFilter
  703. 2015-02-26 10:22:34,434 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-tika/plugin.xml
  704. 2015-02-26 10:22:34,437 DEBUG plugin.PluginRepository - plugin: id=parse-tika name=Tika Parser Plug-in version=1.0.0 provider=nutch.orgclass=null
  705. 2015-02-26 10:22:34,438 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.tika.TikaParser
  706. 2015-02-26 10:22:34,443 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-elastic/plugin.xml
  707. 2015-02-26 10:22:34,446 DEBUG plugin.PluginRepository - plugin: id=indexer-elastic name=ElasticIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  708. 2015-02-26 10:22:34,446 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
  709. 2015-02-26 10:22:34,449 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/microformats-reltag/plugin.xml
  710. 2015-02-26 10:22:34,451 DEBUG plugin.PluginRepository - plugin: id=microformats-reltag name=Rel-Tag microformat Parser/Indexer/Querier version=1.0.0 provider=nutch.orgclass=null
  711. 2015-02-26 10:22:34,451 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.microformats.reltag.RelTagParser
  712. 2015-02-26 10:22:34,451 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.microformats.reltag.RelTagIndexingFilter
  713. 2015-02-26 10:22:34,452 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-html/plugin.xml
  714. 2015-02-26 10:22:34,455 DEBUG plugin.PluginRepository - plugin: id=parse-html name=Html Parse Plug-in version=1.0.0 provider=nutch.orgclass=null
  715. 2015-02-26 10:22:34,455 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.html.HtmlParser
  716. 2015-02-26 10:22:34,456 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/subcollection/plugin.xml
  717. 2015-02-26 10:22:34,458 DEBUG plugin.PluginRepository - plugin: id=subcollection name=Subcollection indexing and query filter version=1.0.0 provider=apache.orgclass=null
  718. 2015-02-26 10:22:34,458 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.subcollection.SubcollectionIndexingFilter
  719. 2015-02-26 10:22:34,459 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-metadata/plugin.xml
  720. 2015-02-26 10:22:34,461 DEBUG plugin.PluginRepository - plugin: id=index-metadata name=Metadata Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  721. 2015-02-26 10:22:34,461 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.metadata.MetadataIndexer
  722. 2015-02-26 10:22:34,462 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-httpclient/plugin.xml
  723. 2015-02-26 10:22:34,464 DEBUG plugin.PluginRepository - plugin: id=protocol-httpclient name=Http / Https Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  724. 2015-02-26 10:22:34,465 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  725. 2015-02-26 10:22:34,465 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  726. 2015-02-26 10:22:34,465 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/nutch-extensionpoints/plugin.xml
  727. 2015-02-26 10:22:34,468 DEBUG plugin.PluginRepository - plugin: id=nutch-extensionpoints name=the nutch core extension points version=2.0.0 provider=nutch.orgclass=null
  728. 2015-02-26 10:22:34,469 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-pass/plugin.xml
  729. 2015-02-26 10:22:34,471 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-pass name=Pass-through URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  730. 2015-02-26 10:22:34,472 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer
  731. 2015-02-26 10:22:34,472 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-regex/plugin.xml
  732. 2015-02-26 10:22:34,474 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-regex name=Regex URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  733. 2015-02-26 10:22:34,474 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
  734. 2015-02-26 10:22:34,475 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-ftp/plugin.xml
  735. 2015-02-26 10:22:34,477 DEBUG plugin.PluginRepository - plugin: id=protocol-ftp name=Ftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  736. 2015-02-26 10:22:34,477 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.ftp.Ftp
  737. 2015-02-26 10:22:34,478 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-nekohtml/plugin.xml
  738. 2015-02-26 10:22:34,480 DEBUG plugin.PluginRepository - plugin: id=lib-nekohtml name=CyberNeko HTML Parser version=1.9.19 provider=net.sourceforge.nekohtmlclass=null
  739. 2015-02-26 10:22:34,480 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-http/plugin.xml
  740. 2015-02-26 10:22:34,483 DEBUG plugin.PluginRepository - plugin: id=protocol-http name=Http Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  741. 2015-02-26 10:22:34,483 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  742. 2015-02-26 10:22:34,483 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  743. 2015-02-26 10:22:34,484 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-solr/plugin.xml
  744. 2015-02-26 10:22:34,486 DEBUG plugin.PluginRepository - plugin: id=indexer-solr name=SOLRIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  745. 2015-02-26 10:22:34,487 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.solr.SolrIndexWriter
  746. 2015-02-26 10:22:34,488 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/language-identifier/plugin.xml
  747. 2015-02-26 10:22:34,490 DEBUG plugin.PluginRepository - plugin: id=language-identifier name=Language Identification Parser/Filter version=1.0.0 provider=nutch.orgclass=null
  748. 2015-02-26 10:22:34,491 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.analysis.lang.HTMLLanguageParser
  749. 2015-02-26 10:22:34,491 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.analysis.lang.LanguageIndexingFilter
  750. 2015-02-26 10:22:34,491 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-basic/plugin.xml
  751. 2015-02-26 10:22:34,493 DEBUG plugin.PluginRepository - plugin: id=index-basic name=Basic Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  752. 2015-02-26 10:22:34,494 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.basic.BasicIndexingFilter
  753. 2015-02-26 10:22:34,494 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-link/plugin.xml
  754. 2015-02-26 10:22:34,496 DEBUG plugin.PluginRepository - plugin: id=scoring-link name=Link Analysis Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  755. 2015-02-26 10:22:34,496 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.link.LinkAnalysisScoringFilter
  756. 2015-02-26 10:22:34,497 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-http/plugin.xml
  757. 2015-02-26 10:22:34,499 DEBUG plugin.PluginRepository - plugin: id=lib-http name=HTTP Framework version=1.0 provider=org.apache.nutchclass=null
  758. 2015-02-26 10:22:34,499 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-regex-filter/plugin.xml
  759. 2015-02-26 10:22:34,501 DEBUG plugin.PluginRepository - plugin: id=lib-regex-filter name=Regex URL Filter Framework version=1.0 provider=org.apache.nutchclass=null
  760. 2015-02-26 10:22:34,502 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-prefix/plugin.xml
  761. 2015-02-26 10:22:34,504 DEBUG plugin.PluginRepository - plugin: id=urlfilter-prefix name=Prefix URL Filter version=1.0.0 provider=nutch.orgclass=null
  762. 2015-02-26 10:22:34,504 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.prefix.PrefixURLFilter
  763. 2015-02-26 10:22:34,504 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-basic/plugin.xml
  764. 2015-02-26 10:22:34,507 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-basic name=Basic URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  765. 2015-02-26 10:22:34,507 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
  766. 2015-02-26 10:22:34,507 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/tld/plugin.xml
  767. 2015-02-26 10:22:34,510 DEBUG plugin.PluginRepository - plugin: id=tld name=Top Level Domain Plugin version=1.0.0 provider=nutch.orgclass=null
  768. 2015-02-26 10:22:34,510 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.tld.TLDIndexingFilter
  769. 2015-02-26 10:22:34,510 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.tld.TLDScoringFilter
  770. 2015-02-26 10:22:34,510 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-anchor/plugin.xml
  771. 2015-02-26 10:22:34,512 DEBUG plugin.PluginRepository - plugin: id=index-anchor name=Anchor Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  772. 2015-02-26 10:22:34,513 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.anchor.AnchorIndexingFilter
  773. 2015-02-26 10:22:34,513 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-suffix/plugin.xml
  774. 2015-02-26 10:22:34,515 DEBUG plugin.PluginRepository - plugin: id=urlfilter-suffix name=Suffix URL Filter version=1.0.0 provider=nutch.orgclass=null
  775. 2015-02-26 10:22:34,516 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.suffix.SuffixURLFilter
  776. 2015-02-26 10:22:34,516 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-regex/plugin.xml
  777. 2015-02-26 10:22:34,518 DEBUG plugin.PluginRepository - plugin: id=urlfilter-regex name=Regex URL Filter version=1.0.0 provider=nutch.orgclass=null
  778. 2015-02-26 10:22:34,518 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.regex.RegexURLFilter
  779. 2015-02-26 10:22:34,519 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-metatags/plugin.xml
  780. 2015-02-26 10:22:34,521 DEBUG plugin.PluginRepository - plugin: id=parse-metatags name=MetaTags version=1.0 provider=digitalpebble.comclass=null
  781. 2015-02-26 10:22:34,521 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.metatags.MetaTagsParser
  782. 2015-02-26 10:22:34,521 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-domain/plugin.xml
  783. 2015-02-26 10:22:34,523 DEBUG plugin.PluginRepository - plugin: id=urlfilter-domain name=Domain URL Filter version=1.0.0 provider=nutch.orgclass=null
  784. 2015-02-26 10:22:34,524 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.domain.DomainURLFilter
  785. 2015-02-26 10:22:34,524 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-opic/plugin.xml
  786. 2015-02-26 10:22:34,526 DEBUG plugin.PluginRepository - plugin: id=scoring-opic name=OPIC Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  787. 2015-02-26 10:22:34,527 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.opic.OPICScoringFilter
  788. 2015-02-26 10:22:34,527 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-more/plugin.xml
  789. 2015-02-26 10:22:34,529 DEBUG plugin.PluginRepository - plugin: id=index-more name=More Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  790. 2015-02-26 10:22:34,529 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.more.MoreIndexingFilter
  791. 2015-02-26 10:22:34,530 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/creativecommons/plugin.xml
  792. 2015-02-26 10:22:34,532 DEBUG plugin.PluginRepository - plugin: id=creativecommons name=Creative Commons Plugins version=1.0.0 provider=nutch.orgclass=null
  793. 2015-02-26 10:22:34,532 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.creativecommons.nutch.CCParseFilter
  794. 2015-02-26 10:22:34,533 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.creativecommons.nutch.CCIndexingFilter
  795. 2015-02-26 10:22:34,533 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-file/plugin.xml
  796. 2015-02-26 10:22:34,535 DEBUG plugin.PluginRepository - plugin: id=protocol-file name=File Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  797. 2015-02-26 10:22:34,535 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.file.File
  798. 2015-02-26 10:22:34,536 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-js/plugin.xml
  799. 2015-02-26 10:22:34,538 DEBUG plugin.PluginRepository - plugin: id=parse-js name=JavaScript Parser version=1.0.0 provider=nutch.orgclass=null
  800. 2015-02-26 10:22:34,539 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.js.JSParseFilter
  801. 2015-02-26 10:22:34,539 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.js.JSParseFilter
  802. 2015-02-26 10:22:34,539 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-validator/plugin.xml
  803. 2015-02-26 10:22:34,542 DEBUG plugin.PluginRepository - plugin: id=urlfilter-validator name=URL Validator version=1.0.0 provider=nutch.orgclass=null
  804. 2015-02-26 10:22:34,542 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.validator.UrlValidator
  805. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: urlfilter-suffix
  806. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: language-identifier
  807. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: protocol-httpclient
  808. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: scoring-link
  809. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: indexer-elastic
  810. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: creativecommons
  811. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: index-more
  812. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: lib-http
  813. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: lib-xml
  814. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: subcollection
  815. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: urlfilter-prefix
  816. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: lib-regex-filter
  817. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: nutch-extensionpoints
  818. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: urlfilter-automaton
  819. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: urlfilter-validator
  820. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: protocol-sftp
  821. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: protocol-ftp
  822. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: lib-nekohtml
  823. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: parse-js
  824. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: protocol-file
  825. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: urlfilter-domain
  826. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: microformats-reltag
  827. 2015-02-26 10:22:34,543 DEBUG plugin.PluginRepository - not including: tld
  828. 2015-02-26 10:22:34,544 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexingFilter
  829. 2015-02-26 10:22:34,544 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexCleaningFilter
  830. 2015-02-26 10:22:34,544 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.Parser
  831. 2015-02-26 10:22:34,544 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.ParseFilter
  832. 2015-02-26 10:22:34,544 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.protocol.Protocol
  833. 2015-02-26 10:22:34,544 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLFilter
  834. 2015-02-26 10:22:34,544 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLNormalizer
  835. 2015-02-26 10:22:34,544 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.scoring.ScoringFilter
  836. 2015-02-26 10:22:34,544 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexWriter
  837. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
  838. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Registered Plugins:
  839. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
  840. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Basic URL Normalizer (urlnormalizer-basic)
  841. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
  842. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
  843. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - SOLRIndexWriter (indexer-solr)
  844. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - HTTP Framework (lib-http)
  845. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
  846. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Pass-through URL Normalizer (urlnormalizer-pass)
  847. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Http Protocol Plug-in (protocol-http)
  848. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Regex URL Normalizer (urlnormalizer-regex)
  849. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
  850. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Tika Parser Plug-in (parse-tika)
  851. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
  852. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
  853. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
  854. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - MetaTags (parse-metatags)
  855. 2015-02-26 10:22:34,544 INFO plugin.PluginRepository - Metadata Indexing Filter (index-metadata)
  856. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Registered Extension-Points:
  857. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
  858. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Nutch Index Cleaning Filter (org.apache.nutch.indexer.IndexCleaningFilter)
  859. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
  860. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Parse Filter (org.apache.nutch.parse.ParseFilter)
  861. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
  862. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Nutch Index Writer (org.apache.nutch.indexer.IndexWriter)
  863. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
  864. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
  865. 2015-02-26 10:22:34,545 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
  866. 2015-02-26 10:22:34,545 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local1860589617_0001.xml, instantiating a new object cache
  867. 2015-02-26 10:22:34,570 INFO conf.Configuration - found resource regex-normalize.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/regex-normalize.xml
  868. 2015-02-26 10:22:34,579 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local1860589617_0001.xml, instantiating a new object cache
  869. 2015-02-26 10:22:34,579 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
  870. 2015-02-26 10:22:34,579 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
  871. 2015-02-26 10:22:34,579 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
  872. 2015-02-26 10:22:34,653 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  873. 2015-02-26 10:22:34,703 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  874. 2015-02-26 10:22:34,732 INFO mapred.MapTask - Starting flush of map output
  875. 2015-02-26 10:22:34,748 INFO compress.CodecPool - Got brand-new compressor
  876. 2015-02-26 10:22:34,749 INFO mapred.MapTask - Finished spill 0
  877. 2015-02-26 10:22:34,752 INFO mapred.Task - Task:attempt_local1860589617_0001_m_000000_0 is done. And is in the process of commiting
  878. 2015-02-26 10:22:34,756 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  879. 2015-02-26 10:22:34,756 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  880. 2015-02-26 10:22:34,756 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  881. 2015-02-26 10:22:34,756 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  882. 2015-02-26 10:22:34,756 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  883. 2015-02-26 10:22:34,756 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  884. 2015-02-26 10:22:34,756 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  885. 2015-02-26 10:22:34,758 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  886. 2015-02-26 10:22:34,758 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  887. 2015-02-26 10:22:34,758 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  888. 2015-02-26 10:22:34,759 INFO mapred.LocalJobRunner -
  889. 2015-02-26 10:22:34,759 INFO mapred.Task - Task 'attempt_local1860589617_0001_m_000000_0' done.
  890. 2015-02-26 10:22:34,759 INFO mapred.LocalJobRunner - Finishing task: attempt_local1860589617_0001_m_000000_0
  891. 2015-02-26 10:22:34,759 INFO mapred.LocalJobRunner - Map task executor complete.
  892. 2015-02-26 10:22:34,766 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  893. 2015-02-26 10:22:34,766 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  894. 2015-02-26 10:22:34,766 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  895. 2015-02-26 10:22:34,766 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  896. 2015-02-26 10:22:34,766 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  897. 2015-02-26 10:22:34,766 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  898. 2015-02-26 10:22:34,767 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  899. 2015-02-26 10:22:34,767 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  900. 2015-02-26 10:22:34,768 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local1860589617_0001/attempt_local1860589617_0001_r_000000_0
  901. 2015-02-26 10:22:34,773 DEBUG mapred.Task - using new api for output committer
  902. 2015-02-26 10:22:34,774 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@476b9c59
  903. 2015-02-26 10:22:34,774 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  904. 2015-02-26 10:22:34,775 INFO mapred.LocalJobRunner -
  905. 2015-02-26 10:22:34,781 INFO mapred.Merger - Merging 1 sorted segments
  906. 2015-02-26 10:22:34,787 INFO compress.CodecPool - Got brand-new decompressor
  907. 2015-02-26 10:22:34,788 INFO mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 82 bytes
  908. 2015-02-26 10:22:34,789 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  909. 2015-02-26 10:22:34,789 INFO mapred.LocalJobRunner -
  910. 2015-02-26 10:22:34,795 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  911. 2015-02-26 10:22:34,795 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  912. 2015-02-26 10:22:34,798 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  913. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  914. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  915. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  916. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  917. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  918. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  919. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located column family: 'p'
  920. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  921. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located column family: 'f'
  922. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  923. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located column family: 'sc'
  924. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  925. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Located super column family
  926. 2015-02-26 10:22:34,798 DEBUG store.CassandraMapping - Added super column family: 'sc'
  927. 2015-02-26 10:22:34,801 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  928. 2015-02-26 10:22:34,801 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  929. 2015-02-26 10:22:34,801 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  930. 2015-02-26 10:22:34,801 INFO mapreduce.GoraRecordWriter - gora.buffer.write.limit = 10000
  931. 2015-02-26 10:22:34,820 DEBUG mapred.Counters - Creating group Generator with nothing
  932. 2015-02-26 10:22:34,821 DEBUG mapred.Counters - Adding GENERATE_MARK
  933. 2015-02-26 10:22:34,821 DEBUG store.CassandraStore - close
  934. 2015-02-26 10:22:34,828 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  935. 2015-02-26 10:22:34,839 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  936. 2015-02-26 10:22:34,839 DEBUG store.CassandraClient - ttl for field:markers is 0
  937. 2015-02-26 10:22:34,845 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  938. 2015-02-26 10:22:34,856 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  939. 2015-02-26 10:22:34,856 DEBUG store.CassandraClient - ttl for field:markers is 0
  940. 2015-02-26 10:22:34,856 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  941. 2015-02-26 10:22:34,858 DEBUG store.CassandraClient - class org.apache.gora.cassandra.serializers.CharSequenceSerializer selected as appropriate Serializer.
  942. 2015-02-26 10:22:34,858 DEBUG store.CassandraClient - ttl for field:markers is 0
  943. 2015-02-26 10:22:34,858 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  944. 2015-02-26 10:22:34,859 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  945. 2015-02-26 10:22:34,859 DEBUG store.CassandraClient - ttl for field:markers is 0
  946. 2015-02-26 10:22:34,859 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  947. 2015-02-26 10:22:34,861 DEBUG store.CassandraClient - class org.apache.gora.cassandra.serializers.CharSequenceSerializer selected as appropriate Serializer.
  948. 2015-02-26 10:22:34,861 DEBUG store.CassandraClient - ttl for field:markers is 0
  949. 2015-02-26 10:22:34,861 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  950. 2015-02-26 10:22:34,862 DEBUG store.CassandraClient - class org.apache.gora.cassandra.serializers.CharSequenceSerializer selected as appropriate Serializer.
  951. 2015-02-26 10:22:34,862 DEBUG store.CassandraClient - ttl for field:markers is 0
  952. 2015-02-26 10:22:34,863 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  953. 2015-02-26 10:22:34,864 DEBUG store.CassandraStore - Union with value: 1424964151-3322 at index: 1 supported for field: batchId
  954. 2015-02-26 10:22:34,864 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  955. 2015-02-26 10:22:34,864 DEBUG store.CassandraClient - fieldName:batchId_UnionIndex columnName:batchId_UnionIndex
  956. 2015-02-26 10:22:34,864 DEBUG store.CassandraClient - ttl was not set for field:batchId_UnionIndex .Using 0
  957. 2015-02-26 10:22:34,864 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  958. 2015-02-26 10:22:34,866 DEBUG store.CassandraClient - class org.apache.gora.cassandra.serializers.CharSequenceSerializer selected as appropriate Serializer.
  959. 2015-02-26 10:22:34,866 DEBUG store.CassandraClient - fieldName:batchId columnName:bid
  960. 2015-02-26 10:22:34,866 DEBUG store.CassandraClient - ttl for field:batchId is 0
  961. 2015-02-26 10:22:34,866 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  962. 2015-02-26 10:22:34,867 INFO mapred.Task - Task:attempt_local1860589617_0001_r_000000_0 is done. And is in the process of commiting
  963. 2015-02-26 10:22:34,867 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  964. 2015-02-26 10:22:34,867 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  965. 2015-02-26 10:22:34,867 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  966. 2015-02-26 10:22:34,868 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  967. 2015-02-26 10:22:34,868 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  968. 2015-02-26 10:22:34,868 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  969. 2015-02-26 10:22:34,868 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  970. 2015-02-26 10:22:34,869 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  971. 2015-02-26 10:22:34,869 DEBUG mapred.Counters - Creating group Generator with nothing
  972. 2015-02-26 10:22:34,869 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  973. 2015-02-26 10:22:34,869 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  974. 2015-02-26 10:22:34,869 INFO mapred.LocalJobRunner - reduce > reduce
  975. 2015-02-26 10:22:34,869 INFO mapred.Task - Task 'attempt_local1860589617_0001_r_000000_0' done.
  976. 2015-02-26 10:22:34,871 WARN mapred.FileOutputCommitter - Output path is null in cleanup
  977. 2015-02-26 10:22:35,234 INFO mapred.JobClient - map 100% reduce 100%
  978. 2015-02-26 10:22:35,235 INFO mapred.JobClient - Job complete: job_local1860589617_0001
  979. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  980. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding BYTES_READ
  981. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  982. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  983. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  984. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  985. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  986. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  987. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  988. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  989. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  990. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  991. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  992. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  993. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  994. 2015-02-26 10:22:35,236 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  995. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  996. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  997. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  998. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  999. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1000. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1001. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding BYTES_READ
  1002. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1003. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1004. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1005. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1006. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1007. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1008. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1009. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1010. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1011. 2015-02-26 10:22:35,237 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1012. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1013. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1014. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1015. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  1016. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  1017. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Creating group Generator with nothing
  1018. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Adding GENERATE_MARK
  1019. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  1020. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  1021. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  1022. 2015-02-26 10:22:35,238 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  1023. 2015-02-26 10:22:35,238 INFO mapred.JobClient - Counters: 21
  1024. 2015-02-26 10:22:35,238 INFO mapred.JobClient - File Output Format Counters
  1025. 2015-02-26 10:22:35,238 INFO mapred.JobClient - Bytes Written=0
  1026. 2015-02-26 10:22:35,238 INFO mapred.JobClient - Generator
  1027. 2015-02-26 10:22:35,239 INFO mapred.JobClient - GENERATE_MARK=1
  1028. 2015-02-26 10:22:35,239 INFO mapred.JobClient - File Input Format Counters
  1029. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Bytes Read=0
  1030. 2015-02-26 10:22:35,239 INFO mapred.JobClient - FileSystemCounters
  1031. 2015-02-26 10:22:35,239 INFO mapred.JobClient - FILE_BYTES_READ=1217334
  1032. 2015-02-26 10:22:35,239 INFO mapred.JobClient - FILE_BYTES_WRITTEN=1393382
  1033. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Map-Reduce Framework
  1034. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Reduce input groups=1
  1035. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Map output materialized bytes=86
  1036. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Combine output records=0
  1037. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Map input records=1
  1038. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Reduce shuffle bytes=0
  1039. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Physical memory (bytes) snapshot=0
  1040. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Reduce output records=1
  1041. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Spilled Records=2
  1042. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Map output bytes=85
  1043. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Total committed heap usage (bytes)=1516240896
  1044. 2015-02-26 10:22:35,239 INFO mapred.JobClient - CPU time spent (ms)=0
  1045. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Virtual memory (bytes) snapshot=0
  1046. 2015-02-26 10:22:35,239 INFO mapred.JobClient - SPLIT_RAW_BYTES=729
  1047. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Map output records=1
  1048. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Combine input records=0
  1049. 2015-02-26 10:22:35,239 INFO mapred.JobClient - Reduce input records=1
  1050. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1051. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding BYTES_READ
  1052. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1053. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1054. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1055. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1056. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1057. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1058. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1059. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1060. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1061. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1062. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1063. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1064. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1065. 2015-02-26 10:22:35,241 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1066. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1067. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1068. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1069. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1070. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1071. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1072. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding BYTES_READ
  1073. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1074. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1075. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1076. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1077. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1078. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1079. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1080. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1081. 2015-02-26 10:22:35,242 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1082. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1083. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1084. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1085. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1086. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  1087. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  1088. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Creating group Generator with nothing
  1089. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding GENERATE_MARK
  1090. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  1091. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  1092. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  1093. 2015-02-26 10:22:35,243 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  1094. 2015-02-26 10:22:35,244 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1095. 2015-02-26 10:22:35,244 DEBUG mapred.Counters - Adding BYTES_READ
  1096. 2015-02-26 10:22:35,244 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1097. 2015-02-26 10:22:35,244 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1098. 2015-02-26 10:22:35,244 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1099. 2015-02-26 10:22:35,244 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1100. 2015-02-26 10:22:35,244 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1101. 2015-02-26 10:22:35,244 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1102. 2015-02-26 10:22:35,244 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1103. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1104. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1105. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1106. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1107. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1108. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1109. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1110. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1111. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1112. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1113. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1114. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1115. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1116. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding BYTES_READ
  1117. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1118. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1119. 2015-02-26 10:22:35,245 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1120. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1121. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1122. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1123. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1124. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1125. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1126. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1127. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1128. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1129. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1130. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  1131. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  1132. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Creating group Generator with nothing
  1133. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding GENERATE_MARK
  1134. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  1135. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  1136. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  1137. 2015-02-26 10:22:35,246 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  1138. 2015-02-26 10:22:35,248 INFO crawl.GeneratorJob - GeneratorJob: finished at 2015-02-26 10:22:35, time elapsed: 00:00:02
  1139. 2015-02-26 10:22:35,248 INFO crawl.GeneratorJob - GeneratorJob: generated batch id: 1424964151-3322 containing 1 URLs
  1140. 2015-02-26 10:22:35,248 DEBUG fs.FileSystem - Starting clear of FileSystem cache with 1 elements.
  1141. 2015-02-26 10:22:35,249 DEBUG fs.FileSystem - Removing filesystem for file:///
  1142. 2015-02-26 10:22:35,249 DEBUG fs.FileSystem - Removing filesystem for file:///
  1143. 2015-02-26 10:22:35,249 DEBUG fs.FileSystem - Done clearing cache
  1144. 2015-02-26 10:22:36,642 INFO fetcher.FetcherJob - FetcherJob: starting at 2015-02-26 10:22:36
  1145. 2015-02-26 10:22:36,642 INFO fetcher.FetcherJob - FetcherJob: batchId: 1424964151-3322
  1146. 2015-02-26 10:22:36,645 INFO fetcher.FetcherJob - FetcherJob: threads: 50
  1147. 2015-02-26 10:22:36,645 INFO fetcher.FetcherJob - FetcherJob: parsing: false
  1148. 2015-02-26 10:22:36,645 INFO fetcher.FetcherJob - FetcherJob: resuming: false
  1149. 2015-02-26 10:22:36,646 INFO fetcher.FetcherJob - FetcherJob : timelimit set for : 1424974956645
  1150. 2015-02-26 10:22:36,780 DEBUG security.Groups - Creating new Groups object
  1151. 2015-02-26 10:22:36,785 DEBUG security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
  1152. 2015-02-26 10:22:36,827 DEBUG security.UserGroupInformation - hadoop login
  1153. 2015-02-26 10:22:36,828 DEBUG security.UserGroupInformation - hadoop login commit
  1154. 2015-02-26 10:22:36,834 DEBUG security.UserGroupInformation - using local user:UnixPrincipal: katonj
  1155. 2015-02-26 10:22:36,837 DEBUG security.UserGroupInformation - UGI loginUser:katonj
  1156. 2015-02-26 10:22:36,908 INFO plugin.PluginRepository - Plugins: looking in: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins
  1157. 2015-02-26 10:22:36,908 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-xml/plugin.xml
  1158. 2015-02-26 10:22:36,913 DEBUG plugin.PluginRepository - plugin: id=lib-xml name=XML Libraries version=1.0 provider=org.apache.nutch.xmlclass=null
  1159. 2015-02-26 10:22:36,915 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-sftp/plugin.xml
  1160. 2015-02-26 10:22:36,917 DEBUG plugin.PluginRepository - plugin: id=protocol-sftp name=Sftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1161. 2015-02-26 10:22:36,917 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.sftp.Sftp
  1162. 2015-02-26 10:22:36,918 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-automaton/plugin.xml
  1163. 2015-02-26 10:22:36,920 DEBUG plugin.PluginRepository - plugin: id=urlfilter-automaton name=Automaton URL Filter version=1.0.0 provider=nutch.orgclass=null
  1164. 2015-02-26 10:22:36,920 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.automaton.AutomatonURLFilter
  1165. 2015-02-26 10:22:36,921 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-tika/plugin.xml
  1166. 2015-02-26 10:22:36,923 DEBUG plugin.PluginRepository - plugin: id=parse-tika name=Tika Parser Plug-in version=1.0.0 provider=nutch.orgclass=null
  1167. 2015-02-26 10:22:36,924 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.tika.TikaParser
  1168. 2015-02-26 10:22:36,932 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-elastic/plugin.xml
  1169. 2015-02-26 10:22:36,934 DEBUG plugin.PluginRepository - plugin: id=indexer-elastic name=ElasticIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  1170. 2015-02-26 10:22:36,934 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
  1171. 2015-02-26 10:22:36,938 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/microformats-reltag/plugin.xml
  1172. 2015-02-26 10:22:36,940 DEBUG plugin.PluginRepository - plugin: id=microformats-reltag name=Rel-Tag microformat Parser/Indexer/Querier version=1.0.0 provider=nutch.orgclass=null
  1173. 2015-02-26 10:22:36,940 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.microformats.reltag.RelTagParser
  1174. 2015-02-26 10:22:36,940 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.microformats.reltag.RelTagIndexingFilter
  1175. 2015-02-26 10:22:36,940 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-html/plugin.xml
  1176. 2015-02-26 10:22:36,942 DEBUG plugin.PluginRepository - plugin: id=parse-html name=Html Parse Plug-in version=1.0.0 provider=nutch.orgclass=null
  1177. 2015-02-26 10:22:36,942 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.html.HtmlParser
  1178. 2015-02-26 10:22:36,943 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/subcollection/plugin.xml
  1179. 2015-02-26 10:22:36,945 DEBUG plugin.PluginRepository - plugin: id=subcollection name=Subcollection indexing and query filter version=1.0.0 provider=apache.orgclass=null
  1180. 2015-02-26 10:22:36,945 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.subcollection.SubcollectionIndexingFilter
  1181. 2015-02-26 10:22:36,945 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-metadata/plugin.xml
  1182. 2015-02-26 10:22:36,947 DEBUG plugin.PluginRepository - plugin: id=index-metadata name=Metadata Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  1183. 2015-02-26 10:22:36,947 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.metadata.MetadataIndexer
  1184. 2015-02-26 10:22:36,947 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-httpclient/plugin.xml
  1185. 2015-02-26 10:22:36,949 DEBUG plugin.PluginRepository - plugin: id=protocol-httpclient name=Http / Https Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1186. 2015-02-26 10:22:36,950 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  1187. 2015-02-26 10:22:36,950 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  1188. 2015-02-26 10:22:36,950 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/nutch-extensionpoints/plugin.xml
  1189. 2015-02-26 10:22:36,952 DEBUG plugin.PluginRepository - plugin: id=nutch-extensionpoints name=the nutch core extension points version=2.0.0 provider=nutch.orgclass=null
  1190. 2015-02-26 10:22:36,953 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-pass/plugin.xml
  1191. 2015-02-26 10:22:36,955 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-pass name=Pass-through URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  1192. 2015-02-26 10:22:36,955 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer
  1193. 2015-02-26 10:22:36,955 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-regex/plugin.xml
  1194. 2015-02-26 10:22:36,957 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-regex name=Regex URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  1195. 2015-02-26 10:22:36,957 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
  1196. 2015-02-26 10:22:36,957 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-ftp/plugin.xml
  1197. 2015-02-26 10:22:36,959 DEBUG plugin.PluginRepository - plugin: id=protocol-ftp name=Ftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1198. 2015-02-26 10:22:36,959 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.ftp.Ftp
  1199. 2015-02-26 10:22:36,959 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-nekohtml/plugin.xml
  1200. 2015-02-26 10:22:36,961 DEBUG plugin.PluginRepository - plugin: id=lib-nekohtml name=CyberNeko HTML Parser version=1.9.19 provider=net.sourceforge.nekohtmlclass=null
  1201. 2015-02-26 10:22:36,961 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-http/plugin.xml
  1202. 2015-02-26 10:22:36,963 DEBUG plugin.PluginRepository - plugin: id=protocol-http name=Http Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1203. 2015-02-26 10:22:36,963 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  1204. 2015-02-26 10:22:36,963 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  1205. 2015-02-26 10:22:36,964 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-solr/plugin.xml
  1206. 2015-02-26 10:22:36,965 DEBUG plugin.PluginRepository - plugin: id=indexer-solr name=SOLRIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  1207. 2015-02-26 10:22:36,966 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.solr.SolrIndexWriter
  1208. 2015-02-26 10:22:36,967 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/language-identifier/plugin.xml
  1209. 2015-02-26 10:22:36,969 DEBUG plugin.PluginRepository - plugin: id=language-identifier name=Language Identification Parser/Filter version=1.0.0 provider=nutch.orgclass=null
  1210. 2015-02-26 10:22:36,969 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.analysis.lang.HTMLLanguageParser
  1211. 2015-02-26 10:22:36,969 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.analysis.lang.LanguageIndexingFilter
  1212. 2015-02-26 10:22:36,969 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-basic/plugin.xml
  1213. 2015-02-26 10:22:36,971 DEBUG plugin.PluginRepository - plugin: id=index-basic name=Basic Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  1214. 2015-02-26 10:22:36,971 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.basic.BasicIndexingFilter
  1215. 2015-02-26 10:22:36,971 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-link/plugin.xml
  1216. 2015-02-26 10:22:36,973 DEBUG plugin.PluginRepository - plugin: id=scoring-link name=Link Analysis Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  1217. 2015-02-26 10:22:36,973 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.link.LinkAnalysisScoringFilter
  1218. 2015-02-26 10:22:36,973 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-http/plugin.xml
  1219. 2015-02-26 10:22:36,975 DEBUG plugin.PluginRepository - plugin: id=lib-http name=HTTP Framework version=1.0 provider=org.apache.nutchclass=null
  1220. 2015-02-26 10:22:36,975 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-regex-filter/plugin.xml
  1221. 2015-02-26 10:22:36,976 DEBUG plugin.PluginRepository - plugin: id=lib-regex-filter name=Regex URL Filter Framework version=1.0 provider=org.apache.nutchclass=null
  1222. 2015-02-26 10:22:36,977 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-prefix/plugin.xml
  1223. 2015-02-26 10:22:36,978 DEBUG plugin.PluginRepository - plugin: id=urlfilter-prefix name=Prefix URL Filter version=1.0.0 provider=nutch.orgclass=null
  1224. 2015-02-26 10:22:36,978 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.prefix.PrefixURLFilter
  1225. 2015-02-26 10:22:36,979 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-basic/plugin.xml
  1226. 2015-02-26 10:22:36,980 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-basic name=Basic URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  1227. 2015-02-26 10:22:36,980 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
  1228. 2015-02-26 10:22:36,981 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/tld/plugin.xml
  1229. 2015-02-26 10:22:36,982 DEBUG plugin.PluginRepository - plugin: id=tld name=Top Level Domain Plugin version=1.0.0 provider=nutch.orgclass=null
  1230. 2015-02-26 10:22:36,982 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.tld.TLDIndexingFilter
  1231. 2015-02-26 10:22:36,982 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.tld.TLDScoringFilter
  1232. 2015-02-26 10:22:36,983 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-anchor/plugin.xml
  1233. 2015-02-26 10:22:36,985 DEBUG plugin.PluginRepository - plugin: id=index-anchor name=Anchor Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  1234. 2015-02-26 10:22:36,985 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.anchor.AnchorIndexingFilter
  1235. 2015-02-26 10:22:36,985 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-suffix/plugin.xml
  1236. 2015-02-26 10:22:36,987 DEBUG plugin.PluginRepository - plugin: id=urlfilter-suffix name=Suffix URL Filter version=1.0.0 provider=nutch.orgclass=null
  1237. 2015-02-26 10:22:36,987 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.suffix.SuffixURLFilter
  1238. 2015-02-26 10:22:36,987 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-regex/plugin.xml
  1239. 2015-02-26 10:22:36,989 DEBUG plugin.PluginRepository - plugin: id=urlfilter-regex name=Regex URL Filter version=1.0.0 provider=nutch.orgclass=null
  1240. 2015-02-26 10:22:36,989 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.regex.RegexURLFilter
  1241. 2015-02-26 10:22:36,989 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-metatags/plugin.xml
  1242. 2015-02-26 10:22:36,991 DEBUG plugin.PluginRepository - plugin: id=parse-metatags name=MetaTags version=1.0 provider=digitalpebble.comclass=null
  1243. 2015-02-26 10:22:36,991 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.metatags.MetaTagsParser
  1244. 2015-02-26 10:22:36,991 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-domain/plugin.xml
  1245. 2015-02-26 10:22:36,993 DEBUG plugin.PluginRepository - plugin: id=urlfilter-domain name=Domain URL Filter version=1.0.0 provider=nutch.orgclass=null
  1246. 2015-02-26 10:22:36,993 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.domain.DomainURLFilter
  1247. 2015-02-26 10:22:36,993 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-opic/plugin.xml
  1248. 2015-02-26 10:22:36,995 DEBUG plugin.PluginRepository - plugin: id=scoring-opic name=OPIC Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  1249. 2015-02-26 10:22:36,995 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.opic.OPICScoringFilter
  1250. 2015-02-26 10:22:36,995 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-more/plugin.xml
  1251. 2015-02-26 10:22:36,997 DEBUG plugin.PluginRepository - plugin: id=index-more name=More Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  1252. 2015-02-26 10:22:36,997 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.more.MoreIndexingFilter
  1253. 2015-02-26 10:22:36,997 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/creativecommons/plugin.xml
  1254. 2015-02-26 10:22:36,999 DEBUG plugin.PluginRepository - plugin: id=creativecommons name=Creative Commons Plugins version=1.0.0 provider=nutch.orgclass=null
  1255. 2015-02-26 10:22:36,999 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.creativecommons.nutch.CCParseFilter
  1256. 2015-02-26 10:22:36,999 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.creativecommons.nutch.CCIndexingFilter
  1257. 2015-02-26 10:22:36,999 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-file/plugin.xml
  1258. 2015-02-26 10:22:37,001 DEBUG plugin.PluginRepository - plugin: id=protocol-file name=File Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1259. 2015-02-26 10:22:37,001 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.file.File
  1260. 2015-02-26 10:22:37,001 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-js/plugin.xml
  1261. 2015-02-26 10:22:37,003 DEBUG plugin.PluginRepository - plugin: id=parse-js name=JavaScript Parser version=1.0.0 provider=nutch.orgclass=null
  1262. 2015-02-26 10:22:37,003 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.js.JSParseFilter
  1263. 2015-02-26 10:22:37,004 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.js.JSParseFilter
  1264. 2015-02-26 10:22:37,004 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-validator/plugin.xml
  1265. 2015-02-26 10:22:37,005 DEBUG plugin.PluginRepository - plugin: id=urlfilter-validator name=URL Validator version=1.0.0 provider=nutch.orgclass=null
  1266. 2015-02-26 10:22:37,006 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.validator.UrlValidator
  1267. 2015-02-26 10:22:37,006 DEBUG plugin.PluginRepository - not including: urlfilter-suffix
  1268. 2015-02-26 10:22:37,006 DEBUG plugin.PluginRepository - not including: language-identifier
  1269. 2015-02-26 10:22:37,006 DEBUG plugin.PluginRepository - not including: protocol-httpclient
  1270. 2015-02-26 10:22:37,006 DEBUG plugin.PluginRepository - not including: scoring-link
  1271. 2015-02-26 10:22:37,006 DEBUG plugin.PluginRepository - not including: indexer-elastic
  1272. 2015-02-26 10:22:37,006 DEBUG plugin.PluginRepository - not including: creativecommons
  1273. 2015-02-26 10:22:37,006 DEBUG plugin.PluginRepository - not including: index-more
  1274. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: lib-http
  1275. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: lib-xml
  1276. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: subcollection
  1277. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: urlfilter-prefix
  1278. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: lib-regex-filter
  1279. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: nutch-extensionpoints
  1280. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: urlfilter-automaton
  1281. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: urlfilter-validator
  1282. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: protocol-sftp
  1283. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: protocol-ftp
  1284. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: lib-nekohtml
  1285. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: parse-js
  1286. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: protocol-file
  1287. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: urlfilter-domain
  1288. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: microformats-reltag
  1289. 2015-02-26 10:22:37,007 DEBUG plugin.PluginRepository - not including: tld
  1290. 2015-02-26 10:22:37,008 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexingFilter
  1291. 2015-02-26 10:22:37,008 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexCleaningFilter
  1292. 2015-02-26 10:22:37,008 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.Parser
  1293. 2015-02-26 10:22:37,008 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.ParseFilter
  1294. 2015-02-26 10:22:37,008 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.protocol.Protocol
  1295. 2015-02-26 10:22:37,008 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLFilter
  1296. 2015-02-26 10:22:37,008 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLNormalizer
  1297. 2015-02-26 10:22:37,008 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.scoring.ScoringFilter
  1298. 2015-02-26 10:22:37,008 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexWriter
  1299. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
  1300. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Registered Plugins:
  1301. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
  1302. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Basic URL Normalizer (urlnormalizer-basic)
  1303. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
  1304. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
  1305. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - SOLRIndexWriter (indexer-solr)
  1306. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - HTTP Framework (lib-http)
  1307. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
  1308. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Pass-through URL Normalizer (urlnormalizer-pass)
  1309. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Http Protocol Plug-in (protocol-http)
  1310. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Regex URL Normalizer (urlnormalizer-regex)
  1311. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
  1312. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Tika Parser Plug-in (parse-tika)
  1313. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
  1314. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
  1315. 2015-02-26 10:22:37,008 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
  1316. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - MetaTags (parse-metatags)
  1317. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Metadata Indexing Filter (index-metadata)
  1318. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Registered Extension-Points:
  1319. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
  1320. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Nutch Index Cleaning Filter (org.apache.nutch.indexer.IndexCleaningFilter)
  1321. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
  1322. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Parse Filter (org.apache.nutch.parse.ParseFilter)
  1323. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
  1324. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Nutch Index Writer (org.apache.nutch.indexer.IndexWriter)
  1325. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
  1326. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
  1327. 2015-02-26 10:22:37,009 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
  1328. 2015-02-26 10:22:37,373 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, nutch-default.xml, nutch-site.xml, instantiating a new object cache
  1329. 2015-02-26 10:22:37,475 INFO http.Http - http.proxy.host = null
  1330. 2015-02-26 10:22:37,475 INFO http.Http - http.proxy.port = 8080
  1331. 2015-02-26 10:22:37,475 INFO http.Http - http.timeout = 10000
  1332. 2015-02-26 10:22:37,475 INFO http.Http - http.content.limit = 65536
  1333. 2015-02-26 10:22:37,475 INFO http.Http - http.agent = My Test/Nutch-2.3
  1334. 2015-02-26 10:22:37,475 INFO http.Http - http.accept.language = en-us,en-gb,en;q=0.7,*;q=0.3
  1335. 2015-02-26 10:22:37,475 INFO http.Http - http.accept = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  1336. 2015-02-26 10:22:37,476 INFO http.Http - http.proxy.host = null
  1337. 2015-02-26 10:22:37,476 INFO http.Http - http.proxy.port = 8080
  1338. 2015-02-26 10:22:37,476 INFO http.Http - http.timeout = 10000
  1339. 2015-02-26 10:22:37,476 INFO http.Http - http.content.limit = 65536
  1340. 2015-02-26 10:22:37,476 INFO http.Http - http.agent = My Test/Nutch-2.3
  1341. 2015-02-26 10:22:37,476 INFO http.Http - http.accept.language = en-us,en-gb,en;q=0.7,*;q=0.3
  1342. 2015-02-26 10:22:37,476 INFO http.Http - http.accept = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  1343. 2015-02-26 10:22:37,779 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'webpage' in cluster 'Test Cluster' on host 'localhost:9160'.
  1344. 2015-02-26 10:22:37,779 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'host' in cluster 'Test Cluster' on host 'localhost:9160'.
  1345. 2015-02-26 10:22:37,779 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.WebPage' for Keyspace 'webpage'.
  1346. 2015-02-26 10:22:37,779 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.Host' for Keyspace 'host'.
  1347. 2015-02-26 10:22:37,779 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  1348. 2015-02-26 10:22:37,780 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  1349. 2015-02-26 10:22:37,780 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  1350. 2015-02-26 10:22:37,780 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  1351. 2015-02-26 10:22:37,780 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  1352. 2015-02-26 10:22:37,780 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  1353. 2015-02-26 10:22:37,780 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  1354. 2015-02-26 10:22:37,805 DEBUG store.CassandraMapping - Located column family: 'p'
  1355. 2015-02-26 10:22:37,805 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1356. 2015-02-26 10:22:37,805 DEBUG store.CassandraMapping - Located column family: 'f'
  1357. 2015-02-26 10:22:37,805 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1358. 2015-02-26 10:22:37,805 DEBUG store.CassandraMapping - Located column family: 'sc'
  1359. 2015-02-26 10:22:37,805 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1360. 2015-02-26 10:22:37,805 DEBUG store.CassandraMapping - Located super column family
  1361. 2015-02-26 10:22:37,805 DEBUG store.CassandraMapping - Added super column family: 'sc'
  1362. 2015-02-26 10:22:37,845 INFO connection.CassandraHostRetryService - Downed Host Retry service started with queue size -1 and retry delay 10s
  1363. 2015-02-26 10:22:37,897 INFO service.JmxMonitor - Registering JMX me.prettyprint.cassandra.service_Test Cluster:ServiceType=hector,MonitorType=hector
  1364. 2015-02-26 10:22:37,917 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1365. 2015-02-26 10:22:37,923 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1366. 2015-02-26 10:22:37,923 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1367. 2015-02-26 10:22:37,923 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1368. 2015-02-26 10:22:37,923 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1369. 2015-02-26 10:22:37,924 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1370. 2015-02-26 10:22:37,924 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1371. 2015-02-26 10:22:37,924 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1372. 2015-02-26 10:22:37,924 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1373. 2015-02-26 10:22:37,924 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1374. 2015-02-26 10:22:37,925 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1375. 2015-02-26 10:22:37,925 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1376. 2015-02-26 10:22:37,925 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1377. 2015-02-26 10:22:37,925 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1378. 2015-02-26 10:22:37,925 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1379. 2015-02-26 10:22:37,926 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1380. 2015-02-26 10:22:37,926 DEBUG connection.ConcurrentHClientPool - Concurrent Host pool started with 16 active clients; max: 50 exhausted wait: 0
  1381. 2015-02-26 10:22:37,979 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  1382. 2015-02-26 10:22:37,979 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  1383. 2015-02-26 10:22:37,979 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  1384. 2015-02-26 10:22:38,182 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  1385. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  1386. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  1387. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  1388. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  1389. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  1390. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  1391. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located column family: 'p'
  1392. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1393. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located column family: 'f'
  1394. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1395. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located column family: 'sc'
  1396. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1397. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Located super column family
  1398. 2015-02-26 10:22:38,183 DEBUG store.CassandraMapping - Added super column family: 'sc'
  1399. 2015-02-26 10:22:38,185 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  1400. 2015-02-26 10:22:38,185 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  1401. 2015-02-26 10:22:38,185 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  1402. 2015-02-26 10:22:38,186 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapreduce.Job.connect(Job.java:561)
  1403. 2015-02-26 10:22:38,218 DEBUG fs.FileSystem - Creating filesystem for file:///
  1404. 2015-02-26 10:22:38,230 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  1405. 2015-02-26 10:22:38,238 DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
  1406. 2015-02-26 10:22:38,238 DEBUG util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
  1407. 2015-02-26 10:22:38,238 DEBUG util.NativeCodeLoader - java.library.path=/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/lib/native/Linux-amd64-64
  1408. 2015-02-26 10:22:38,238 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  1409. 2015-02-26 10:22:38,239 DEBUG mapred.JobClient - adding the following namenodes' delegation tokens:null
  1410. 2015-02-26 10:22:38,240 DEBUG mapred.JobClient - default FileSystem: file:///
  1411. 2015-02-26 10:22:38,286 DEBUG mapred.JobClient - Creating splits at file:/tmp/hadoop-katonj/mapred/staging/katonj133920565/.staging/job_local133920565_0001
  1412. 2015-02-26 10:22:38,290 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  1413. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  1414. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  1415. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  1416. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  1417. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  1418. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  1419. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located column family: 'p'
  1420. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1421. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located column family: 'f'
  1422. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1423. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located column family: 'sc'
  1424. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1425. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Located super column family
  1426. 2015-02-26 10:22:38,290 DEBUG store.CassandraMapping - Added super column family: 'sc'
  1427. 2015-02-26 10:22:38,292 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  1428. 2015-02-26 10:22:38,292 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  1429. 2015-02-26 10:22:38,292 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  1430. 2015-02-26 10:22:38,448 DEBUG mapred.JobClient - Printing tokens for job: job_local133920565_0001
  1431. 2015-02-26 10:22:38,463 DEBUG mapred.TaskRunner - Fully deleting contents of /tmp/hadoop-katonj/mapred/local/localRunner
  1432. 2015-02-26 10:22:38,560 INFO mapred.JobClient - Running job: job_local133920565_0001
  1433. 2015-02-26 10:22:38,603 DEBUG mapred.LocalJobRunner - Starting thread pool executor.
  1434. 2015-02-26 10:22:38,603 DEBUG mapred.LocalJobRunner - Max local threads: 1
  1435. 2015-02-26 10:22:38,603 DEBUG mapred.LocalJobRunner - Map tasks to process: 1
  1436. 2015-02-26 10:22:38,604 INFO mapred.LocalJobRunner - Waiting for map tasks
  1437. 2015-02-26 10:22:38,617 INFO mapred.LocalJobRunner - Starting task: attempt_local133920565_0001_m_000000_0
  1438. 2015-02-26 10:22:38,629 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  1439. 2015-02-26 10:22:38,636 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1440. 2015-02-26 10:22:38,636 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1441. 2015-02-26 10:22:38,660 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local133920565_0001/attempt_local133920565_0001_m_000000_0
  1442. 2015-02-26 10:22:38,663 DEBUG mapred.Task - using new api for output committer
  1443. 2015-02-26 10:22:38,677 INFO util.ProcessTree - setsid exited with exit code 0
  1444. 2015-02-26 10:22:38,689 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@951cfdd
  1445. 2015-02-26 10:22:38,696 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  1446. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  1447. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  1448. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  1449. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  1450. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  1451. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  1452. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located column family: 'p'
  1453. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1454. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located column family: 'f'
  1455. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1456. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located column family: 'sc'
  1457. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1458. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Located super column family
  1459. 2015-02-26 10:22:38,696 DEBUG store.CassandraMapping - Added super column family: 'sc'
  1460. 2015-02-26 10:22:38,699 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  1461. 2015-02-26 10:22:38,699 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  1462. 2015-02-26 10:22:38,699 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  1463. 2015-02-26 10:22:38,706 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  1464. 2015-02-26 10:22:38,706 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  1465. 2015-02-26 10:22:38,706 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  1466. 2015-02-26 10:22:38,706 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  1467. 2015-02-26 10:22:38,706 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  1468. 2015-02-26 10:22:38,706 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  1469. 2015-02-26 10:22:38,706 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  1470. 2015-02-26 10:22:38,707 DEBUG store.CassandraMapping - Located column family: 'p'
  1471. 2015-02-26 10:22:38,707 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1472. 2015-02-26 10:22:38,707 DEBUG store.CassandraMapping - Located column family: 'f'
  1473. 2015-02-26 10:22:38,707 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1474. 2015-02-26 10:22:38,707 DEBUG store.CassandraMapping - Located column family: 'sc'
  1475. 2015-02-26 10:22:38,707 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1476. 2015-02-26 10:22:38,707 DEBUG store.CassandraMapping - Located super column family
  1477. 2015-02-26 10:22:38,707 DEBUG store.CassandraMapping - Added super column family: 'sc'
  1478. 2015-02-26 10:22:38,709 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  1479. 2015-02-26 10:22:38,709 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  1480. 2015-02-26 10:22:38,709 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  1481. 2015-02-26 10:22:38,712 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  1482. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  1483. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  1484. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  1485. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  1486. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  1487. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  1488. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located column family: 'p'
  1489. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1490. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located column family: 'f'
  1491. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1492. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located column family: 'sc'
  1493. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1494. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Located super column family
  1495. 2015-02-26 10:22:38,712 DEBUG store.CassandraMapping - Added super column family: 'sc'
  1496. 2015-02-26 10:22:38,714 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  1497. 2015-02-26 10:22:38,715 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  1498. 2015-02-26 10:22:38,715 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  1499. 2015-02-26 10:22:38,715 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1500. 2015-02-26 10:22:38,715 INFO mapred.MapTask - Processing split: org.apache.gora.mapreduce.GoraInputSplit@5aa22f2c
  1501. 2015-02-26 10:22:38,716 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1502. 2015-02-26 10:22:38,719 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1503. 2015-02-26 10:22:38,719 DEBUG mapred.Counters - Adding BYTES_READ
  1504. 2015-02-26 10:22:38,721 INFO mapreduce.GoraRecordReader - gora.buffer.read.limit = 10000
  1505. 2015-02-26 10:22:38,732 INFO mapred.MapTask - io.sort.mb = 100
  1506. 2015-02-26 10:22:39,029 INFO mapred.MapTask - data buffer = 79691776/99614720
  1507. 2015-02-26 10:22:39,029 INFO mapred.MapTask - record buffer = 262144/327680
  1508. 2015-02-26 10:22:39,030 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1509. 2015-02-26 10:22:39,030 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1510. 2015-02-26 10:22:39,030 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1511. 2015-02-26 10:22:39,031 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1512. 2015-02-26 10:22:39,031 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1513. 2015-02-26 10:22:39,043 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local133920565_0001.xml, instantiating a new object cache
  1514. 2015-02-26 10:22:39,078 INFO conf.Configuration - found resource regex-normalize.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/regex-normalize.xml
  1515. 2015-02-26 10:22:39,174 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  1516. 2015-02-26 10:22:39,214 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  1517. 2015-02-26 10:22:39,234 INFO mapred.MapTask - Starting flush of map output
  1518. 2015-02-26 10:22:39,244 INFO compress.CodecPool - Got brand-new compressor
  1519. 2015-02-26 10:22:39,245 INFO mapred.MapTask - Finished spill 0
  1520. 2015-02-26 10:22:39,247 INFO mapred.Task - Task:attempt_local133920565_0001_m_000000_0 is done. And is in the process of commiting
  1521. 2015-02-26 10:22:39,250 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1522. 2015-02-26 10:22:39,250 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1523. 2015-02-26 10:22:39,250 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1524. 2015-02-26 10:22:39,250 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1525. 2015-02-26 10:22:39,250 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1526. 2015-02-26 10:22:39,250 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1527. 2015-02-26 10:22:39,250 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1528. 2015-02-26 10:22:39,251 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1529. 2015-02-26 10:22:39,251 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1530. 2015-02-26 10:22:39,252 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1531. 2015-02-26 10:22:39,252 INFO mapred.LocalJobRunner -
  1532. 2015-02-26 10:22:39,252 INFO mapred.Task - Task 'attempt_local133920565_0001_m_000000_0' done.
  1533. 2015-02-26 10:22:39,252 INFO mapred.LocalJobRunner - Finishing task: attempt_local133920565_0001_m_000000_0
  1534. 2015-02-26 10:22:39,252 INFO mapred.LocalJobRunner - Map task executor complete.
  1535. 2015-02-26 10:22:39,258 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  1536. 2015-02-26 10:22:39,259 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1537. 2015-02-26 10:22:39,259 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1538. 2015-02-26 10:22:39,259 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  1539. 2015-02-26 10:22:39,259 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  1540. 2015-02-26 10:22:39,259 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  1541. 2015-02-26 10:22:39,259 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  1542. 2015-02-26 10:22:39,259 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1543. 2015-02-26 10:22:39,261 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local133920565_0001/attempt_local133920565_0001_r_000000_0
  1544. 2015-02-26 10:22:39,266 DEBUG mapred.Task - using new api for output committer
  1545. 2015-02-26 10:22:39,266 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@66ae1312
  1546. 2015-02-26 10:22:39,266 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1547. 2015-02-26 10:22:39,267 INFO mapred.LocalJobRunner -
  1548. 2015-02-26 10:22:39,272 INFO mapred.Merger - Merging 1 sorted segments
  1549. 2015-02-26 10:22:39,278 INFO compress.CodecPool - Got brand-new decompressor
  1550. 2015-02-26 10:22:39,279 INFO mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 96 bytes
  1551. 2015-02-26 10:22:39,279 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1552. 2015-02-26 10:22:39,280 INFO mapred.LocalJobRunner -
  1553. 2015-02-26 10:22:39,285 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  1554. 2015-02-26 10:22:39,285 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  1555. 2015-02-26 10:22:39,287 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  1556. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  1557. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  1558. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  1559. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  1560. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  1561. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  1562. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located column family: 'p'
  1563. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1564. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located column family: 'f'
  1565. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1566. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located column family: 'sc'
  1567. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  1568. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Located super column family
  1569. 2015-02-26 10:22:39,287 DEBUG store.CassandraMapping - Added super column family: 'sc'
  1570. 2015-02-26 10:22:39,289 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  1571. 2015-02-26 10:22:39,289 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  1572. 2015-02-26 10:22:39,289 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  1573. 2015-02-26 10:22:39,289 INFO mapreduce.GoraRecordWriter - gora.buffer.write.limit = 10000
  1574. 2015-02-26 10:22:39,292 INFO fetcher.FetcherJob - Using queue mode : byHost
  1575. 2015-02-26 10:22:39,292 INFO fetcher.FetcherJob - Fetcher: threads: 50
  1576. 2015-02-26 10:22:39,300 DEBUG fetcher.FetcherJob - -feeding 2500 input urls ...
  1577. 2015-02-26 10:22:39,301 INFO fetcher.FetcherJob - QueueFeeder finished: total 1 records. Hit by time limit :0
  1578. 2015-02-26 10:22:39,302 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local133920565_0001.xml, instantiating a new object cache
  1579. 2015-02-26 10:22:39,303 DEBUG mapred.Counters - Creating group FetcherStatus with nothing
  1580. 2015-02-26 10:22:39,303 DEBUG mapred.Counters - Adding HitByTimeLimit-QueueFeeder
  1581. 2015-02-26 10:22:39,309 INFO conf.Configuration - found resource regex-urlfilter.txt at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/regex-urlfilter.txt
  1582. 2015-02-26 10:22:39,311 INFO conf.Configuration - found resource regex-normalize.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/regex-normalize.xml
  1583. 2015-02-26 10:22:39,313 INFO fetcher.FetcherJob - fetching http://nutch.apache.org/ (queue crawl delay=5000ms)
  1584. 2015-02-26 10:22:39,314 INFO fetcher.FetcherJob - -finishing thread FetcherThread1, activeThreads=1
  1585. 2015-02-26 10:22:39,314 INFO fetcher.FetcherJob - -finishing thread FetcherThread2, activeThreads=1
  1586. 2015-02-26 10:22:39,314 INFO fetcher.FetcherJob - -finishing thread FetcherThread3, activeThreads=1
  1587. 2015-02-26 10:22:39,314 INFO fetcher.FetcherJob - -finishing thread FetcherThread4, activeThreads=1
  1588. 2015-02-26 10:22:39,344 INFO fetcher.FetcherJob - -finishing thread FetcherThread5, activeThreads=1
  1589. 2015-02-26 10:22:39,344 INFO fetcher.FetcherJob - -finishing thread FetcherThread6, activeThreads=1
  1590. 2015-02-26 10:22:39,345 INFO http.Http - http.proxy.host = null
  1591. 2015-02-26 10:22:39,345 INFO http.Http - http.proxy.port = 8080
  1592. 2015-02-26 10:22:39,345 INFO http.Http - http.timeout = 10000
  1593. 2015-02-26 10:22:39,345 INFO http.Http - http.content.limit = 65536
  1594. 2015-02-26 10:22:39,345 INFO http.Http - http.agent = My Test/Nutch-2.3
  1595. 2015-02-26 10:22:39,345 INFO http.Http - http.accept.language = en-us,en-gb,en;q=0.7,*;q=0.3
  1596. 2015-02-26 10:22:39,345 INFO http.Http - http.accept = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  1597. 2015-02-26 10:22:39,345 INFO fetcher.FetcherJob - -finishing thread FetcherThread7, activeThreads=1
  1598. 2015-02-26 10:22:39,345 INFO fetcher.FetcherJob - -finishing thread FetcherThread8, activeThreads=1
  1599. 2015-02-26 10:22:39,345 INFO fetcher.FetcherJob - -finishing thread FetcherThread9, activeThreads=1
  1600. 2015-02-26 10:22:39,346 INFO fetcher.FetcherJob - -finishing thread FetcherThread10, activeThreads=1
  1601. 2015-02-26 10:22:39,346 INFO fetcher.FetcherJob - -finishing thread FetcherThread11, activeThreads=1
  1602. 2015-02-26 10:22:39,346 INFO fetcher.FetcherJob - -finishing thread FetcherThread12, activeThreads=1
  1603. 2015-02-26 10:22:39,346 INFO fetcher.FetcherJob - -finishing thread FetcherThread13, activeThreads=1
  1604. 2015-02-26 10:22:39,347 INFO fetcher.FetcherJob - -finishing thread FetcherThread14, activeThreads=1
  1605. 2015-02-26 10:22:39,348 INFO fetcher.FetcherJob - -finishing thread FetcherThread15, activeThreads=1
  1606. 2015-02-26 10:22:39,348 INFO fetcher.FetcherJob - -finishing thread FetcherThread16, activeThreads=1
  1607. 2015-02-26 10:22:39,348 INFO fetcher.FetcherJob - -finishing thread FetcherThread17, activeThreads=1
  1608. 2015-02-26 10:22:39,349 INFO fetcher.FetcherJob - -finishing thread FetcherThread18, activeThreads=1
  1609. 2015-02-26 10:22:39,349 INFO fetcher.FetcherJob - -finishing thread FetcherThread19, activeThreads=1
  1610. 2015-02-26 10:22:39,349 INFO fetcher.FetcherJob - -finishing thread FetcherThread20, activeThreads=1
  1611. 2015-02-26 10:22:39,349 INFO fetcher.FetcherJob - -finishing thread FetcherThread21, activeThreads=1
  1612. 2015-02-26 10:22:39,350 INFO fetcher.FetcherJob - -finishing thread FetcherThread22, activeThreads=1
  1613. 2015-02-26 10:22:39,350 INFO fetcher.FetcherJob - -finishing thread FetcherThread23, activeThreads=1
  1614. 2015-02-26 10:22:39,350 INFO fetcher.FetcherJob - -finishing thread FetcherThread24, activeThreads=1
  1615. 2015-02-26 10:22:39,351 INFO fetcher.FetcherJob - -finishing thread FetcherThread25, activeThreads=1
  1616. 2015-02-26 10:22:39,351 INFO fetcher.FetcherJob - -finishing thread FetcherThread26, activeThreads=1
  1617. 2015-02-26 10:22:39,351 INFO fetcher.FetcherJob - -finishing thread FetcherThread27, activeThreads=1
  1618. 2015-02-26 10:22:39,351 INFO fetcher.FetcherJob - -finishing thread FetcherThread28, activeThreads=1
  1619. 2015-02-26 10:22:39,352 INFO fetcher.FetcherJob - -finishing thread FetcherThread29, activeThreads=1
  1620. 2015-02-26 10:22:39,352 INFO fetcher.FetcherJob - -finishing thread FetcherThread30, activeThreads=1
  1621. 2015-02-26 10:22:39,352 INFO fetcher.FetcherJob - -finishing thread FetcherThread31, activeThreads=1
  1622. 2015-02-26 10:22:39,352 INFO fetcher.FetcherJob - -finishing thread FetcherThread32, activeThreads=1
  1623. 2015-02-26 10:22:39,353 INFO fetcher.FetcherJob - -finishing thread FetcherThread33, activeThreads=1
  1624. 2015-02-26 10:22:39,353 INFO fetcher.FetcherJob - -finishing thread FetcherThread34, activeThreads=1
  1625. 2015-02-26 10:22:39,353 INFO fetcher.FetcherJob - -finishing thread FetcherThread35, activeThreads=1
  1626. 2015-02-26 10:22:39,353 INFO fetcher.FetcherJob - -finishing thread FetcherThread36, activeThreads=1
  1627. 2015-02-26 10:22:39,354 INFO fetcher.FetcherJob - -finishing thread FetcherThread37, activeThreads=1
  1628. 2015-02-26 10:22:39,354 INFO fetcher.FetcherJob - -finishing thread FetcherThread38, activeThreads=1
  1629. 2015-02-26 10:22:39,354 INFO fetcher.FetcherJob - -finishing thread FetcherThread39, activeThreads=1
  1630. 2015-02-26 10:22:39,354 INFO fetcher.FetcherJob - -finishing thread FetcherThread40, activeThreads=1
  1631. 2015-02-26 10:22:39,355 INFO fetcher.FetcherJob - -finishing thread FetcherThread41, activeThreads=1
  1632. 2015-02-26 10:22:39,355 INFO fetcher.FetcherJob - -finishing thread FetcherThread42, activeThreads=1
  1633. 2015-02-26 10:22:39,355 INFO fetcher.FetcherJob - -finishing thread FetcherThread43, activeThreads=1
  1634. 2015-02-26 10:22:39,355 INFO fetcher.FetcherJob - -finishing thread FetcherThread44, activeThreads=1
  1635. 2015-02-26 10:22:39,355 INFO fetcher.FetcherJob - -finishing thread FetcherThread45, activeThreads=1
  1636. 2015-02-26 10:22:39,355 INFO fetcher.FetcherJob - -finishing thread FetcherThread46, activeThreads=1
  1637. 2015-02-26 10:22:39,356 INFO fetcher.FetcherJob - -finishing thread FetcherThread47, activeThreads=1
  1638. 2015-02-26 10:22:39,356 INFO fetcher.FetcherJob - -finishing thread FetcherThread48, activeThreads=1
  1639. 2015-02-26 10:22:39,356 INFO fetcher.FetcherJob - Fetcher: throughput threshold: -1
  1640. 2015-02-26 10:22:39,356 INFO fetcher.FetcherJob - -finishing thread FetcherThread49, activeThreads=1
  1641. 2015-02-26 10:22:39,356 INFO fetcher.FetcherJob - Fetcher: throughput threshold sequence: 5
  1642. 2015-02-26 10:22:39,562 INFO mapred.JobClient - map 100% reduce 0%
  1643. 2015-02-26 10:22:39,598 DEBUG mapred.Counters - Adding SUCCESS
  1644. 2015-02-26 10:22:39,600 INFO fetcher.FetcherJob - -finishing thread FetcherThread0, activeThreads=0
  1645. 2015-02-26 10:22:44,357 INFO fetcher.FetcherJob - 0/0 spinwaiting/active, 1 pages, 0 errors, 0.2 0 pages/s, 82 82 kb/s, 0 URLs in 0 queues
  1646. 2015-02-26 10:22:44,357 INFO fetcher.FetcherJob - -activeThreads=0
  1647. 2015-02-26 10:22:44,357 DEBUG store.CassandraStore - close
  1648. 2015-02-26 10:22:44,357 DEBUG store.CassandraStore - Union with value: http://nutch.apache.org/ at index: 1 supported for field: baseUrl
  1649. 2015-02-26 10:22:44,357 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  1650. 2015-02-26 10:22:44,358 DEBUG store.CassandraClient - fieldName:baseUrl_UnionIndex columnName:baseUrl_UnionIndex
  1651. 2015-02-26 10:22:44,358 DEBUG store.CassandraClient - ttl was not set for field:baseUrl_UnionIndex .Using 0
  1652. 2015-02-26 10:22:44,366 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  1653. 2015-02-26 10:22:44,376 DEBUG store.CassandraClient - class org.apache.gora.cassandra.serializers.CharSequenceSerializer selected as appropriate Serializer.
  1654. 2015-02-26 10:22:44,376 DEBUG store.CassandraClient - fieldName:baseUrl columnName:bas
  1655. 2015-02-26 10:22:44,376 DEBUG store.CassandraClient - ttl for field:baseUrl is 0
  1656. 2015-02-26 10:22:44,377 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  1657. 2015-02-26 10:22:44,378 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  1658. 2015-02-26 10:22:44,378 DEBUG store.CassandraClient - fieldName:status columnName:st
  1659. 2015-02-26 10:22:44,378 DEBUG store.CassandraClient - ttl for field:status is 0
  1660. 2015-02-26 10:22:44,378 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  1661. 2015-02-26 10:22:44,379 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.LongSerializer selected as appropriate Serializer.
  1662. 2015-02-26 10:22:44,380 DEBUG store.CassandraClient - fieldName:fetchTime columnName:ts
  1663. 2015-02-26 10:22:44,380 DEBUG store.CassandraClient - ttl for field:fetchTime is 0
  1664. 2015-02-26 10:22:44,380 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  1665. 2015-02-26 10:22:44,381 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.LongSerializer selected as appropriate Serializer.
  1666. 2015-02-26 10:22:44,381 DEBUG store.CassandraClient - fieldName:prevFetchTime columnName:pts
  1667. 2015-02-26 10:22:44,381 DEBUG store.CassandraClient - ttl for field:prevFetchTime is 0
  1668. 2015-02-26 10:22:44,382 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  1669. 2015-02-26 10:22:44,383 DEBUG store.CassandraStore - Union with value: {"code": 1, "args": [], "lastModified": 0} at index: 1 supported for field: protocolStatus
  1670. 2015-02-26 10:22:44,383 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  1671. 2015-02-26 10:22:44,383 DEBUG store.CassandraClient - ttl was not set for field:protocolStatus_UnionIndex .Using 0
  1672. 2015-02-26 10:22:44,384 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  1673. 2015-02-26 10:22:44,387 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.BytesArraySerializer selected as appropriate Serializer.
  1674. 2015-02-26 10:22:44,387 DEBUG store.CassandraClient - fieldName:protocolStatus columnName:prs
  1675. 2015-02-26 10:22:44,387 DEBUG store.CassandraClient - ttl for field:protocolStatus is 0
  1676. 2015-02-26 10:22:44,387 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  1677. 2015-02-26 10:22:44,391 DEBUG client.HThriftClient - Closing client CassandraClient<localhost:9160-16>
  1678. 2015-02-26 10:22:44,392 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  1679. 2015-02-26 10:22:44,392 WARN mapreduce.GoraRecordWriter - Exception at GoraRecordWriter.class while closing datastore.InvalidRequestException(why:supercolumn parameter is not optional for super CF sc)
  1680. 2015-02-26 10:22:44,392 INFO mapred.Task - Task:attempt_local133920565_0001_r_000000_0 is done. And is in the process of commiting
  1681. 2015-02-26 10:22:44,392 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1682. 2015-02-26 10:22:44,392 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1683. 2015-02-26 10:22:44,392 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1684. 2015-02-26 10:22:44,392 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1685. 2015-02-26 10:22:44,392 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1686. 2015-02-26 10:22:44,392 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1687. 2015-02-26 10:22:44,392 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1688. 2015-02-26 10:22:44,393 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  1689. 2015-02-26 10:22:44,394 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1690. 2015-02-26 10:22:44,394 DEBUG mapred.Counters - Creating group FetcherStatus with nothing
  1691. 2015-02-26 10:22:44,394 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1692. 2015-02-26 10:22:44,394 INFO mapred.LocalJobRunner - 0/0 spinwaiting/active, 1 pages, 0 errors, 0.2 0 pages/s, 82 82 kb/s, 0 URLs in 0 queues > reduce
  1693. 2015-02-26 10:22:44,394 INFO mapred.Task - Task 'attempt_local133920565_0001_r_000000_0' done.
  1694. 2015-02-26 10:22:44,395 WARN mapred.FileOutputCommitter - Output path is null in cleanup
  1695. 2015-02-26 10:22:44,564 INFO mapred.JobClient - map 100% reduce 100%
  1696. 2015-02-26 10:22:44,564 INFO mapred.JobClient - Job complete: job_local133920565_0001
  1697. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1698. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding BYTES_READ
  1699. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1700. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1701. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1702. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1703. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1704. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1705. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1706. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1707. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1708. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1709. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1710. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1711. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1712. 2015-02-26 10:22:44,565 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1713. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1714. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1715. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1716. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1717. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1718. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1719. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding BYTES_READ
  1720. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1721. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1722. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1723. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1724. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1725. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1726. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1727. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1728. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1729. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1730. 2015-02-26 10:22:44,566 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1731. 2015-02-26 10:22:44,567 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1732. 2015-02-26 10:22:44,567 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1733. 2015-02-26 10:22:44,567 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  1734. 2015-02-26 10:22:44,568 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  1735. 2015-02-26 10:22:44,568 DEBUG mapred.Counters - Creating group FetcherStatus with nothing
  1736. 2015-02-26 10:22:44,569 DEBUG mapred.Counters - Adding HitByTimeLimit-QueueFeeder
  1737. 2015-02-26 10:22:44,569 DEBUG mapred.Counters - Adding SUCCESS
  1738. 2015-02-26 10:22:44,569 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  1739. 2015-02-26 10:22:44,569 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  1740. 2015-02-26 10:22:44,569 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  1741. 2015-02-26 10:22:44,570 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  1742. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Counters: 22
  1743. 2015-02-26 10:22:44,570 INFO mapred.JobClient - File Output Format Counters
  1744. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Bytes Written=0
  1745. 2015-02-26 10:22:44,570 INFO mapred.JobClient - File Input Format Counters
  1746. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Bytes Read=0
  1747. 2015-02-26 10:22:44,570 INFO mapred.JobClient - FileSystemCounters
  1748. 2015-02-26 10:22:44,570 INFO mapred.JobClient - FILE_BYTES_READ=1217522
  1749. 2015-02-26 10:22:44,570 INFO mapred.JobClient - FILE_BYTES_WRITTEN=1392760
  1750. 2015-02-26 10:22:44,570 INFO mapred.JobClient - FetcherStatus
  1751. 2015-02-26 10:22:44,570 INFO mapred.JobClient - HitByTimeLimit-QueueFeeder=0
  1752. 2015-02-26 10:22:44,570 INFO mapred.JobClient - SUCCESS=1
  1753. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Map-Reduce Framework
  1754. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Reduce input groups=1
  1755. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Map output materialized bytes=100
  1756. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Combine output records=0
  1757. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Map input records=1
  1758. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Reduce shuffle bytes=0
  1759. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Physical memory (bytes) snapshot=0
  1760. 2015-02-26 10:22:44,570 INFO mapred.JobClient - Reduce output records=1
  1761. 2015-02-26 10:22:44,571 INFO mapred.JobClient - Spilled Records=2
  1762. 2015-02-26 10:22:44,571 INFO mapred.JobClient - Map output bytes=106
  1763. 2015-02-26 10:22:44,571 INFO mapred.JobClient - Total committed heap usage (bytes)=1516240896
  1764. 2015-02-26 10:22:44,571 INFO mapred.JobClient - CPU time spent (ms)=0
  1765. 2015-02-26 10:22:44,571 INFO mapred.JobClient - Virtual memory (bytes) snapshot=0
  1766. 2015-02-26 10:22:44,571 INFO mapred.JobClient - SPLIT_RAW_BYTES=816
  1767. 2015-02-26 10:22:44,571 INFO mapred.JobClient - Map output records=1
  1768. 2015-02-26 10:22:44,571 INFO mapred.JobClient - Combine input records=0
  1769. 2015-02-26 10:22:44,571 INFO mapred.JobClient - Reduce input records=1
  1770. 2015-02-26 10:22:44,572 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1771. 2015-02-26 10:22:44,572 DEBUG mapred.Counters - Adding BYTES_READ
  1772. 2015-02-26 10:22:44,572 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1773. 2015-02-26 10:22:44,572 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1774. 2015-02-26 10:22:44,572 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1775. 2015-02-26 10:22:44,572 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1776. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1777. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1778. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1779. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1780. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1781. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1782. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1783. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1784. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1785. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1786. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1787. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1788. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  1789. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  1790. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  1791. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  1792. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Adding BYTES_READ
  1793. 2015-02-26 10:22:44,573 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  1794. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  1795. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  1796. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  1797. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  1798. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  1799. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  1800. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  1801. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  1802. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  1803. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  1804. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  1805. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  1806. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  1807. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  1808. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Creating group FetcherStatus with nothing
  1809. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding HitByTimeLimit-QueueFeeder
  1810. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding SUCCESS
  1811. 2015-02-26 10:22:44,574 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  1812. 2015-02-26 10:22:44,575 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  1813. 2015-02-26 10:22:44,575 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  1814. 2015-02-26 10:22:44,575 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  1815. 2015-02-26 10:22:44,576 INFO fetcher.FetcherJob - FetcherJob: finished at 2015-02-26 10:22:44, time elapsed: 00:00:07
  1816. 2015-02-26 10:22:44,577 DEBUG fs.FileSystem - Starting clear of FileSystem cache with 1 elements.
  1817. 2015-02-26 10:22:44,577 DEBUG fs.FileSystem - Removing filesystem for file:///
  1818. 2015-02-26 10:22:44,577 DEBUG fs.FileSystem - Removing filesystem for file:///
  1819. 2015-02-26 10:22:44,577 DEBUG fs.FileSystem - Done clearing cache
  1820. 2015-02-26 10:22:46,084 INFO parse.ParserJob - ParserJob: starting at 2015-02-26 10:22:46
  1821. 2015-02-26 10:22:46,087 INFO parse.ParserJob - ParserJob: resuming: false
  1822. 2015-02-26 10:22:46,087 INFO parse.ParserJob - ParserJob: forced reparse: false
  1823. 2015-02-26 10:22:46,087 INFO parse.ParserJob - ParserJob: batchId: 1424964151-3322
  1824. 2015-02-26 10:22:46,291 DEBUG security.Groups - Creating new Groups object
  1825. 2015-02-26 10:22:46,300 DEBUG security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
  1826. 2015-02-26 10:22:46,366 DEBUG security.UserGroupInformation - hadoop login
  1827. 2015-02-26 10:22:46,367 DEBUG security.UserGroupInformation - hadoop login commit
  1828. 2015-02-26 10:22:46,374 DEBUG security.UserGroupInformation - using local user:UnixPrincipal: katonj
  1829. 2015-02-26 10:22:46,377 DEBUG security.UserGroupInformation - UGI loginUser:katonj
  1830. 2015-02-26 10:22:46,453 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, nutch-default.xml, nutch-site.xml, instantiating a new object cache
  1831. 2015-02-26 10:22:46,462 INFO plugin.PluginRepository - Plugins: looking in: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins
  1832. 2015-02-26 10:22:46,463 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-xml/plugin.xml
  1833. 2015-02-26 10:22:46,470 DEBUG plugin.PluginRepository - plugin: id=lib-xml name=XML Libraries version=1.0 provider=org.apache.nutch.xmlclass=null
  1834. 2015-02-26 10:22:46,473 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-sftp/plugin.xml
  1835. 2015-02-26 10:22:46,476 DEBUG plugin.PluginRepository - plugin: id=protocol-sftp name=Sftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1836. 2015-02-26 10:22:46,476 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.sftp.Sftp
  1837. 2015-02-26 10:22:46,478 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-automaton/plugin.xml
  1838. 2015-02-26 10:22:46,481 DEBUG plugin.PluginRepository - plugin: id=urlfilter-automaton name=Automaton URL Filter version=1.0.0 provider=nutch.orgclass=null
  1839. 2015-02-26 10:22:46,481 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.automaton.AutomatonURLFilter
  1840. 2015-02-26 10:22:46,482 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-tika/plugin.xml
  1841. 2015-02-26 10:22:46,486 DEBUG plugin.PluginRepository - plugin: id=parse-tika name=Tika Parser Plug-in version=1.0.0 provider=nutch.orgclass=null
  1842. 2015-02-26 10:22:46,487 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.tika.TikaParser
  1843. 2015-02-26 10:22:46,499 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-elastic/plugin.xml
  1844. 2015-02-26 10:22:46,502 DEBUG plugin.PluginRepository - plugin: id=indexer-elastic name=ElasticIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  1845. 2015-02-26 10:22:46,503 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
  1846. 2015-02-26 10:22:46,508 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/microformats-reltag/plugin.xml
  1847. 2015-02-26 10:22:46,511 DEBUG plugin.PluginRepository - plugin: id=microformats-reltag name=Rel-Tag microformat Parser/Indexer/Querier version=1.0.0 provider=nutch.orgclass=null
  1848. 2015-02-26 10:22:46,511 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.microformats.reltag.RelTagParser
  1849. 2015-02-26 10:22:46,511 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.microformats.reltag.RelTagIndexingFilter
  1850. 2015-02-26 10:22:46,512 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-html/plugin.xml
  1851. 2015-02-26 10:22:46,515 DEBUG plugin.PluginRepository - plugin: id=parse-html name=Html Parse Plug-in version=1.0.0 provider=nutch.orgclass=null
  1852. 2015-02-26 10:22:46,515 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.html.HtmlParser
  1853. 2015-02-26 10:22:46,516 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/subcollection/plugin.xml
  1854. 2015-02-26 10:22:46,519 DEBUG plugin.PluginRepository - plugin: id=subcollection name=Subcollection indexing and query filter version=1.0.0 provider=apache.orgclass=null
  1855. 2015-02-26 10:22:46,519 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.subcollection.SubcollectionIndexingFilter
  1856. 2015-02-26 10:22:46,519 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-metadata/plugin.xml
  1857. 2015-02-26 10:22:46,522 DEBUG plugin.PluginRepository - plugin: id=index-metadata name=Metadata Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  1858. 2015-02-26 10:22:46,522 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.metadata.MetadataIndexer
  1859. 2015-02-26 10:22:46,523 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-httpclient/plugin.xml
  1860. 2015-02-26 10:22:46,526 DEBUG plugin.PluginRepository - plugin: id=protocol-httpclient name=Http / Https Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1861. 2015-02-26 10:22:46,526 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  1862. 2015-02-26 10:22:46,526 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  1863. 2015-02-26 10:22:46,527 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/nutch-extensionpoints/plugin.xml
  1864. 2015-02-26 10:22:46,530 DEBUG plugin.PluginRepository - plugin: id=nutch-extensionpoints name=the nutch core extension points version=2.0.0 provider=nutch.orgclass=null
  1865. 2015-02-26 10:22:46,531 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-pass/plugin.xml
  1866. 2015-02-26 10:22:46,534 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-pass name=Pass-through URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  1867. 2015-02-26 10:22:46,535 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer
  1868. 2015-02-26 10:22:46,535 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-regex/plugin.xml
  1869. 2015-02-26 10:22:46,538 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-regex name=Regex URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  1870. 2015-02-26 10:22:46,538 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
  1871. 2015-02-26 10:22:46,539 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-ftp/plugin.xml
  1872. 2015-02-26 10:22:46,541 DEBUG plugin.PluginRepository - plugin: id=protocol-ftp name=Ftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1873. 2015-02-26 10:22:46,542 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.ftp.Ftp
  1874. 2015-02-26 10:22:46,543 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-nekohtml/plugin.xml
  1875. 2015-02-26 10:22:46,545 DEBUG plugin.PluginRepository - plugin: id=lib-nekohtml name=CyberNeko HTML Parser version=1.9.19 provider=net.sourceforge.nekohtmlclass=null
  1876. 2015-02-26 10:22:46,546 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-http/plugin.xml
  1877. 2015-02-26 10:22:46,548 DEBUG plugin.PluginRepository - plugin: id=protocol-http name=Http Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1878. 2015-02-26 10:22:46,549 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  1879. 2015-02-26 10:22:46,549 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  1880. 2015-02-26 10:22:46,550 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-solr/plugin.xml
  1881. 2015-02-26 10:22:46,553 DEBUG plugin.PluginRepository - plugin: id=indexer-solr name=SOLRIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  1882. 2015-02-26 10:22:46,553 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.solr.SolrIndexWriter
  1883. 2015-02-26 10:22:46,556 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/language-identifier/plugin.xml
  1884. 2015-02-26 10:22:46,559 DEBUG plugin.PluginRepository - plugin: id=language-identifier name=Language Identification Parser/Filter version=1.0.0 provider=nutch.orgclass=null
  1885. 2015-02-26 10:22:46,560 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.analysis.lang.HTMLLanguageParser
  1886. 2015-02-26 10:22:46,560 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.analysis.lang.LanguageIndexingFilter
  1887. 2015-02-26 10:22:46,560 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-basic/plugin.xml
  1888. 2015-02-26 10:22:46,563 DEBUG plugin.PluginRepository - plugin: id=index-basic name=Basic Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  1889. 2015-02-26 10:22:46,563 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.basic.BasicIndexingFilter
  1890. 2015-02-26 10:22:46,564 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-link/plugin.xml
  1891. 2015-02-26 10:22:46,566 DEBUG plugin.PluginRepository - plugin: id=scoring-link name=Link Analysis Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  1892. 2015-02-26 10:22:46,567 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.link.LinkAnalysisScoringFilter
  1893. 2015-02-26 10:22:46,567 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-http/plugin.xml
  1894. 2015-02-26 10:22:46,570 DEBUG plugin.PluginRepository - plugin: id=lib-http name=HTTP Framework version=1.0 provider=org.apache.nutchclass=null
  1895. 2015-02-26 10:22:46,570 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-regex-filter/plugin.xml
  1896. 2015-02-26 10:22:46,573 DEBUG plugin.PluginRepository - plugin: id=lib-regex-filter name=Regex URL Filter Framework version=1.0 provider=org.apache.nutchclass=null
  1897. 2015-02-26 10:22:46,574 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-prefix/plugin.xml
  1898. 2015-02-26 10:22:46,576 DEBUG plugin.PluginRepository - plugin: id=urlfilter-prefix name=Prefix URL Filter version=1.0.0 provider=nutch.orgclass=null
  1899. 2015-02-26 10:22:46,577 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.prefix.PrefixURLFilter
  1900. 2015-02-26 10:22:46,577 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-basic/plugin.xml
  1901. 2015-02-26 10:22:46,580 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-basic name=Basic URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  1902. 2015-02-26 10:22:46,580 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
  1903. 2015-02-26 10:22:46,581 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/tld/plugin.xml
  1904. 2015-02-26 10:22:46,584 DEBUG plugin.PluginRepository - plugin: id=tld name=Top Level Domain Plugin version=1.0.0 provider=nutch.orgclass=null
  1905. 2015-02-26 10:22:46,584 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.tld.TLDIndexingFilter
  1906. 2015-02-26 10:22:46,584 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.tld.TLDScoringFilter
  1907. 2015-02-26 10:22:46,585 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-anchor/plugin.xml
  1908. 2015-02-26 10:22:46,587 DEBUG plugin.PluginRepository - plugin: id=index-anchor name=Anchor Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  1909. 2015-02-26 10:22:46,588 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.anchor.AnchorIndexingFilter
  1910. 2015-02-26 10:22:46,588 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-suffix/plugin.xml
  1911. 2015-02-26 10:22:46,591 DEBUG plugin.PluginRepository - plugin: id=urlfilter-suffix name=Suffix URL Filter version=1.0.0 provider=nutch.orgclass=null
  1912. 2015-02-26 10:22:46,591 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.suffix.SuffixURLFilter
  1913. 2015-02-26 10:22:46,592 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-regex/plugin.xml
  1914. 2015-02-26 10:22:46,594 DEBUG plugin.PluginRepository - plugin: id=urlfilter-regex name=Regex URL Filter version=1.0.0 provider=nutch.orgclass=null
  1915. 2015-02-26 10:22:46,595 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.regex.RegexURLFilter
  1916. 2015-02-26 10:22:46,595 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-metatags/plugin.xml
  1917. 2015-02-26 10:22:46,598 DEBUG plugin.PluginRepository - plugin: id=parse-metatags name=MetaTags version=1.0 provider=digitalpebble.comclass=null
  1918. 2015-02-26 10:22:46,598 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.metatags.MetaTagsParser
  1919. 2015-02-26 10:22:46,598 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-domain/plugin.xml
  1920. 2015-02-26 10:22:46,601 DEBUG plugin.PluginRepository - plugin: id=urlfilter-domain name=Domain URL Filter version=1.0.0 provider=nutch.orgclass=null
  1921. 2015-02-26 10:22:46,601 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.domain.DomainURLFilter
  1922. 2015-02-26 10:22:46,602 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-opic/plugin.xml
  1923. 2015-02-26 10:22:46,604 DEBUG plugin.PluginRepository - plugin: id=scoring-opic name=OPIC Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  1924. 2015-02-26 10:22:46,605 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.opic.OPICScoringFilter
  1925. 2015-02-26 10:22:46,605 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-more/plugin.xml
  1926. 2015-02-26 10:22:46,607 DEBUG plugin.PluginRepository - plugin: id=index-more name=More Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  1927. 2015-02-26 10:22:46,608 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.more.MoreIndexingFilter
  1928. 2015-02-26 10:22:46,608 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/creativecommons/plugin.xml
  1929. 2015-02-26 10:22:46,611 DEBUG plugin.PluginRepository - plugin: id=creativecommons name=Creative Commons Plugins version=1.0.0 provider=nutch.orgclass=null
  1930. 2015-02-26 10:22:46,611 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.creativecommons.nutch.CCParseFilter
  1931. 2015-02-26 10:22:46,611 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.creativecommons.nutch.CCIndexingFilter
  1932. 2015-02-26 10:22:46,612 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-file/plugin.xml
  1933. 2015-02-26 10:22:46,614 DEBUG plugin.PluginRepository - plugin: id=protocol-file name=File Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  1934. 2015-02-26 10:22:46,614 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.file.File
  1935. 2015-02-26 10:22:46,615 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-js/plugin.xml
  1936. 2015-02-26 10:22:46,618 DEBUG plugin.PluginRepository - plugin: id=parse-js name=JavaScript Parser version=1.0.0 provider=nutch.orgclass=null
  1937. 2015-02-26 10:22:46,618 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.js.JSParseFilter
  1938. 2015-02-26 10:22:46,618 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.js.JSParseFilter
  1939. 2015-02-26 10:22:46,619 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-validator/plugin.xml
  1940. 2015-02-26 10:22:46,621 DEBUG plugin.PluginRepository - plugin: id=urlfilter-validator name=URL Validator version=1.0.0 provider=nutch.orgclass=null
  1941. 2015-02-26 10:22:46,621 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.validator.UrlValidator
  1942. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: urlfilter-suffix
  1943. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: language-identifier
  1944. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: protocol-httpclient
  1945. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: scoring-link
  1946. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: indexer-elastic
  1947. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: creativecommons
  1948. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: index-more
  1949. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: lib-http
  1950. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: lib-xml
  1951. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: subcollection
  1952. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: urlfilter-prefix
  1953. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: lib-regex-filter
  1954. 2015-02-26 10:22:46,623 DEBUG plugin.PluginRepository - not including: nutch-extensionpoints
  1955. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: urlfilter-automaton
  1956. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: urlfilter-validator
  1957. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: protocol-sftp
  1958. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: protocol-ftp
  1959. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: lib-nekohtml
  1960. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: parse-js
  1961. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: protocol-file
  1962. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: urlfilter-domain
  1963. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: microformats-reltag
  1964. 2015-02-26 10:22:46,624 DEBUG plugin.PluginRepository - not including: tld
  1965. 2015-02-26 10:22:46,625 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexingFilter
  1966. 2015-02-26 10:22:46,625 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexCleaningFilter
  1967. 2015-02-26 10:22:46,625 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.Parser
  1968. 2015-02-26 10:22:46,625 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.ParseFilter
  1969. 2015-02-26 10:22:46,625 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.protocol.Protocol
  1970. 2015-02-26 10:22:46,625 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLFilter
  1971. 2015-02-26 10:22:46,625 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLNormalizer
  1972. 2015-02-26 10:22:46,625 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.scoring.ScoringFilter
  1973. 2015-02-26 10:22:46,625 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexWriter
  1974. 2015-02-26 10:22:46,625 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
  1975. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Registered Plugins:
  1976. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
  1977. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Basic URL Normalizer (urlnormalizer-basic)
  1978. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
  1979. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
  1980. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - SOLRIndexWriter (indexer-solr)
  1981. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - HTTP Framework (lib-http)
  1982. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
  1983. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Pass-through URL Normalizer (urlnormalizer-pass)
  1984. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Http Protocol Plug-in (protocol-http)
  1985. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Regex URL Normalizer (urlnormalizer-regex)
  1986. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
  1987. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Tika Parser Plug-in (parse-tika)
  1988. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
  1989. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
  1990. 2015-02-26 10:22:46,626 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
  1991. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - MetaTags (parse-metatags)
  1992. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Metadata Indexing Filter (index-metadata)
  1993. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Registered Extension-Points:
  1994. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
  1995. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Nutch Index Cleaning Filter (org.apache.nutch.indexer.IndexCleaningFilter)
  1996. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
  1997. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Parse Filter (org.apache.nutch.parse.ParseFilter)
  1998. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
  1999. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Nutch Index Writer (org.apache.nutch.indexer.IndexWriter)
  2000. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
  2001. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
  2002. 2015-02-26 10:22:46,627 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
  2003. 2015-02-26 10:22:46,631 INFO conf.Configuration - found resource parse-plugins.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/parse-plugins.xml
  2004. 2015-02-26 10:22:46,649 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, nutch-default.xml, nutch-site.xml, instantiating a new object cache
  2005. 2015-02-26 10:22:47,560 INFO crawl.SignatureFactory - Using Signature impl: org.apache.nutch.crawl.MD5Signature
  2006. 2015-02-26 10:22:47,861 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'webpage' in cluster 'Test Cluster' on host 'localhost:9160'.
  2007. 2015-02-26 10:22:47,861 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'host' in cluster 'Test Cluster' on host 'localhost:9160'.
  2008. 2015-02-26 10:22:47,861 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.WebPage' for Keyspace 'webpage'.
  2009. 2015-02-26 10:22:47,861 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.Host' for Keyspace 'host'.
  2010. 2015-02-26 10:22:47,861 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2011. 2015-02-26 10:22:47,862 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2012. 2015-02-26 10:22:47,862 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2013. 2015-02-26 10:22:47,862 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2014. 2015-02-26 10:22:47,862 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2015. 2015-02-26 10:22:47,862 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2016. 2015-02-26 10:22:47,862 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2017. 2015-02-26 10:22:47,884 DEBUG store.CassandraMapping - Located column family: 'p'
  2018. 2015-02-26 10:22:47,884 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2019. 2015-02-26 10:22:47,884 DEBUG store.CassandraMapping - Located column family: 'f'
  2020. 2015-02-26 10:22:47,885 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2021. 2015-02-26 10:22:47,885 DEBUG store.CassandraMapping - Located column family: 'sc'
  2022. 2015-02-26 10:22:47,885 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2023. 2015-02-26 10:22:47,885 DEBUG store.CassandraMapping - Located super column family
  2024. 2015-02-26 10:22:47,885 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2025. 2015-02-26 10:22:47,920 INFO connection.CassandraHostRetryService - Downed Host Retry service started with queue size -1 and retry delay 10s
  2026. 2015-02-26 10:22:47,970 INFO service.JmxMonitor - Registering JMX me.prettyprint.cassandra.service_Test Cluster:ServiceType=hector,MonitorType=hector
  2027. 2015-02-26 10:22:47,989 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2028. 2015-02-26 10:22:47,994 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2029. 2015-02-26 10:22:47,994 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2030. 2015-02-26 10:22:47,994 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2031. 2015-02-26 10:22:47,995 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2032. 2015-02-26 10:22:47,995 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2033. 2015-02-26 10:22:47,995 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2034. 2015-02-26 10:22:47,995 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2035. 2015-02-26 10:22:47,995 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2036. 2015-02-26 10:22:47,996 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2037. 2015-02-26 10:22:47,996 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2038. 2015-02-26 10:22:47,996 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2039. 2015-02-26 10:22:47,996 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2040. 2015-02-26 10:22:47,996 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2041. 2015-02-26 10:22:47,997 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2042. 2015-02-26 10:22:47,997 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2043. 2015-02-26 10:22:47,997 DEBUG connection.ConcurrentHClientPool - Concurrent Host pool started with 16 active clients; max: 50 exhausted wait: 0
  2044. 2015-02-26 10:22:48,049 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2045. 2015-02-26 10:22:48,049 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2046. 2015-02-26 10:22:48,049 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2047. 2015-02-26 10:22:48,094 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2048. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2049. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2050. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2051. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2052. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2053. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2054. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located column family: 'p'
  2055. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2056. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located column family: 'f'
  2057. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2058. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located column family: 'sc'
  2059. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2060. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Located super column family
  2061. 2015-02-26 10:22:48,094 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2062. 2015-02-26 10:22:48,096 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2063. 2015-02-26 10:22:48,096 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2064. 2015-02-26 10:22:48,096 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2065. 2015-02-26 10:22:48,097 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapreduce.Job.connect(Job.java:561)
  2066. 2015-02-26 10:22:48,116 DEBUG fs.FileSystem - Creating filesystem for file:///
  2067. 2015-02-26 10:22:48,128 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  2068. 2015-02-26 10:22:48,134 DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
  2069. 2015-02-26 10:22:48,134 DEBUG util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
  2070. 2015-02-26 10:22:48,134 DEBUG util.NativeCodeLoader - java.library.path=/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/lib/native/Linux-amd64-64
  2071. 2015-02-26 10:22:48,134 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2072. 2015-02-26 10:22:48,135 DEBUG mapred.JobClient - adding the following namenodes' delegation tokens:null
  2073. 2015-02-26 10:22:48,136 DEBUG mapred.JobClient - default FileSystem: file:///
  2074. 2015-02-26 10:22:48,180 DEBUG mapred.JobClient - Creating splits at file:/tmp/hadoop-katonj/mapred/staging/katonj1824480531/.staging/job_local1824480531_0001
  2075. 2015-02-26 10:22:48,183 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2076. 2015-02-26 10:22:48,183 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2077. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2078. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2079. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2080. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2081. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2082. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located column family: 'p'
  2083. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2084. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located column family: 'f'
  2085. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2086. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located column family: 'sc'
  2087. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2088. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Located super column family
  2089. 2015-02-26 10:22:48,184 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2090. 2015-02-26 10:22:48,185 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2091. 2015-02-26 10:22:48,185 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2092. 2015-02-26 10:22:48,186 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2093. 2015-02-26 10:22:48,315 DEBUG mapred.JobClient - Printing tokens for job: job_local1824480531_0001
  2094. 2015-02-26 10:22:48,330 DEBUG mapred.TaskRunner - Fully deleting contents of /tmp/hadoop-katonj/mapred/local/localRunner
  2095. 2015-02-26 10:22:48,434 INFO mapred.JobClient - Running job: job_local1824480531_0001
  2096. 2015-02-26 10:22:48,484 DEBUG mapred.LocalJobRunner - Starting thread pool executor.
  2097. 2015-02-26 10:22:48,484 DEBUG mapred.LocalJobRunner - Max local threads: 1
  2098. 2015-02-26 10:22:48,484 DEBUG mapred.LocalJobRunner - Map tasks to process: 1
  2099. 2015-02-26 10:22:48,485 INFO mapred.LocalJobRunner - Waiting for map tasks
  2100. 2015-02-26 10:22:48,489 INFO mapred.LocalJobRunner - Starting task: attempt_local1824480531_0001_m_000000_0
  2101. 2015-02-26 10:22:48,505 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  2102. 2015-02-26 10:22:48,517 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2103. 2015-02-26 10:22:48,517 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2104. 2015-02-26 10:22:48,534 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local1824480531_0001/attempt_local1824480531_0001_m_000000_0
  2105. 2015-02-26 10:22:48,539 DEBUG mapred.Task - using new api for output committer
  2106. 2015-02-26 10:22:48,555 INFO util.ProcessTree - setsid exited with exit code 0
  2107. 2015-02-26 10:22:48,560 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@cce6203
  2108. 2015-02-26 10:22:48,563 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2109. 2015-02-26 10:22:48,563 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2110. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2111. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2112. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2113. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2114. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2115. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located column family: 'p'
  2116. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2117. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located column family: 'f'
  2118. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2119. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located column family: 'sc'
  2120. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2121. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Located super column family
  2122. 2015-02-26 10:22:48,564 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2123. 2015-02-26 10:22:48,565 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2124. 2015-02-26 10:22:48,565 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2125. 2015-02-26 10:22:48,566 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2126. 2015-02-26 10:22:48,571 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2127. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2128. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2129. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2130. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2131. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2132. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2133. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located column family: 'p'
  2134. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2135. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located column family: 'f'
  2136. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2137. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located column family: 'sc'
  2138. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2139. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Located super column family
  2140. 2015-02-26 10:22:48,571 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2141. 2015-02-26 10:22:48,573 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2142. 2015-02-26 10:22:48,573 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2143. 2015-02-26 10:22:48,573 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2144. 2015-02-26 10:22:48,575 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2145. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2146. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2147. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2148. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2149. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2150. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2151. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located column family: 'p'
  2152. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2153. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located column family: 'f'
  2154. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2155. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located column family: 'sc'
  2156. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2157. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Located super column family
  2158. 2015-02-26 10:22:48,575 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2159. 2015-02-26 10:22:48,576 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2160. 2015-02-26 10:22:48,576 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2161. 2015-02-26 10:22:48,576 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2162. 2015-02-26 10:22:48,577 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2163. 2015-02-26 10:22:48,577 INFO mapred.MapTask - Processing split: org.apache.gora.mapreduce.GoraInputSplit@7c4392c2
  2164. 2015-02-26 10:22:48,578 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2165. 2015-02-26 10:22:48,580 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2166. 2015-02-26 10:22:48,580 DEBUG mapred.Counters - Adding BYTES_READ
  2167. 2015-02-26 10:22:48,581 INFO mapreduce.GoraRecordReader - gora.buffer.read.limit = 10000
  2168. 2015-02-26 10:22:48,582 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2169. 2015-02-26 10:22:48,584 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2170. 2015-02-26 10:22:48,584 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  2171. 2015-02-26 10:22:48,585 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2172. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2173. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2174. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2175. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2176. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2177. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2178. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located column family: 'p'
  2179. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2180. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located column family: 'f'
  2181. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2182. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located column family: 'sc'
  2183. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2184. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Located super column family
  2185. 2015-02-26 10:22:48,586 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2186. 2015-02-26 10:22:48,587 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2187. 2015-02-26 10:22:48,587 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2188. 2015-02-26 10:22:48,587 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2189. 2015-02-26 10:22:48,588 INFO mapreduce.GoraRecordWriter - gora.buffer.write.limit = 10000
  2190. 2015-02-26 10:22:48,590 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local1824480531_0001.xml, instantiating a new object cache
  2191. 2015-02-26 10:22:48,590 INFO conf.Configuration - found resource parse-plugins.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/parse-plugins.xml
  2192. 2015-02-26 10:22:48,593 INFO crawl.SignatureFactory - Using Signature impl: org.apache.nutch.crawl.MD5Signature
  2193. 2015-02-26 10:22:48,599 INFO conf.Configuration - found resource regex-urlfilter.txt at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/regex-urlfilter.txt
  2194. 2015-02-26 10:22:48,616 INFO conf.Configuration - found resource regex-normalize.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/regex-normalize.xml
  2195. 2015-02-26 10:22:48,679 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2196. 2015-02-26 10:22:48,731 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2197. 2015-02-26 10:22:48,736 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2198. 2015-02-26 10:22:48,755 DEBUG store.CassandraStore - close
  2199. 2015-02-26 10:22:48,756 INFO mapred.Task - Task:attempt_local1824480531_0001_m_000000_0 is done. And is in the process of commiting
  2200. 2015-02-26 10:22:48,759 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2201. 2015-02-26 10:22:48,759 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2202. 2015-02-26 10:22:48,759 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2203. 2015-02-26 10:22:48,759 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2204. 2015-02-26 10:22:48,760 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2205. 2015-02-26 10:22:48,760 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2206. 2015-02-26 10:22:48,760 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2207. 2015-02-26 10:22:48,761 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2208. 2015-02-26 10:22:48,761 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2209. 2015-02-26 10:22:48,761 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2210. 2015-02-26 10:22:48,762 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2211. 2015-02-26 10:22:48,762 INFO mapred.LocalJobRunner -
  2212. 2015-02-26 10:22:48,762 INFO mapred.Task - Task 'attempt_local1824480531_0001_m_000000_0' done.
  2213. 2015-02-26 10:22:48,762 INFO mapred.LocalJobRunner - Finishing task: attempt_local1824480531_0001_m_000000_0
  2214. 2015-02-26 10:22:48,762 INFO mapred.LocalJobRunner - Map task executor complete.
  2215. 2015-02-26 10:22:48,763 WARN mapred.FileOutputCommitter - Output path is null in cleanup
  2216. 2015-02-26 10:22:49,436 INFO mapred.JobClient - map 100% reduce 0%
  2217. 2015-02-26 10:22:49,438 INFO mapred.JobClient - Job complete: job_local1824480531_0001
  2218. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2219. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  2220. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2221. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding BYTES_READ
  2222. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2223. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2224. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2225. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2226. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2227. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2228. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2229. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2230. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2231. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2232. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2233. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2234. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2235. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  2236. 2015-02-26 10:22:49,438 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2237. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2238. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2239. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2240. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding BYTES_READ
  2241. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2242. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2243. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2244. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2245. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2246. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2247. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2248. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2249. 2015-02-26 10:22:49,439 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2250. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Counters: 12
  2251. 2015-02-26 10:22:49,439 INFO mapred.JobClient - File Output Format Counters
  2252. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Bytes Written=0
  2253. 2015-02-26 10:22:49,439 INFO mapred.JobClient - File Input Format Counters
  2254. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Bytes Read=0
  2255. 2015-02-26 10:22:49,439 INFO mapred.JobClient - FileSystemCounters
  2256. 2015-02-26 10:22:49,439 INFO mapred.JobClient - FILE_BYTES_READ=608756
  2257. 2015-02-26 10:22:49,439 INFO mapred.JobClient - FILE_BYTES_WRITTEN=695813
  2258. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Map-Reduce Framework
  2259. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Map input records=0
  2260. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Physical memory (bytes) snapshot=0
  2261. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Spilled Records=0
  2262. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Total committed heap usage (bytes)=758120448
  2263. 2015-02-26 10:22:49,439 INFO mapred.JobClient - CPU time spent (ms)=0
  2264. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Virtual memory (bytes) snapshot=0
  2265. 2015-02-26 10:22:49,439 INFO mapred.JobClient - SPLIT_RAW_BYTES=861
  2266. 2015-02-26 10:22:49,439 INFO mapred.JobClient - Map output records=0
  2267. 2015-02-26 10:22:49,440 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2268. 2015-02-26 10:22:49,440 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  2269. 2015-02-26 10:22:49,440 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2270. 2015-02-26 10:22:49,440 DEBUG mapred.Counters - Adding BYTES_READ
  2271. 2015-02-26 10:22:49,440 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2272. 2015-02-26 10:22:49,440 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2273. 2015-02-26 10:22:49,440 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2274. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2275. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2276. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2277. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2278. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2279. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2280. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2281. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2282. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2283. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2284. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  2285. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2286. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2287. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2288. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2289. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding BYTES_READ
  2290. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2291. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2292. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2293. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2294. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2295. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2296. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2297. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2298. 2015-02-26 10:22:49,441 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2299. 2015-02-26 10:22:49,442 INFO parse.ParserJob - ParserJob: success
  2300. 2015-02-26 10:22:49,442 INFO parse.ParserJob - ParserJob: finished at 2015-02-26 10:22:49, time elapsed: 00:00:03
  2301. 2015-02-26 10:22:49,443 DEBUG fs.FileSystem - Starting clear of FileSystem cache with 1 elements.
  2302. 2015-02-26 10:22:49,443 DEBUG fs.FileSystem - Removing filesystem for file:///
  2303. 2015-02-26 10:22:49,443 DEBUG fs.FileSystem - Removing filesystem for file:///
  2304. 2015-02-26 10:22:49,443 DEBUG fs.FileSystem - Done clearing cache
  2305. 2015-02-26 10:22:50,670 INFO crawl.DbUpdaterJob - DbUpdaterJob: starting at 2015-02-26 10:22:50
  2306. 2015-02-26 10:22:50,671 INFO crawl.DbUpdaterJob - DbUpdaterJob: batchId: 1424964151-3322
  2307. 2015-02-26 10:22:50,676 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, nutch-default.xml, nutch-site.xml, instantiating a new object cache
  2308. 2015-02-26 10:22:50,686 INFO plugin.PluginRepository - Plugins: looking in: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins
  2309. 2015-02-26 10:22:50,687 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-xml/plugin.xml
  2310. 2015-02-26 10:22:50,695 DEBUG plugin.PluginRepository - plugin: id=lib-xml name=XML Libraries version=1.0 provider=org.apache.nutch.xmlclass=null
  2311. 2015-02-26 10:22:50,697 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-sftp/plugin.xml
  2312. 2015-02-26 10:22:50,700 DEBUG plugin.PluginRepository - plugin: id=protocol-sftp name=Sftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  2313. 2015-02-26 10:22:50,701 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.sftp.Sftp
  2314. 2015-02-26 10:22:50,702 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-automaton/plugin.xml
  2315. 2015-02-26 10:22:50,706 DEBUG plugin.PluginRepository - plugin: id=urlfilter-automaton name=Automaton URL Filter version=1.0.0 provider=nutch.orgclass=null
  2316. 2015-02-26 10:22:50,706 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.automaton.AutomatonURLFilter
  2317. 2015-02-26 10:22:50,707 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-tika/plugin.xml
  2318. 2015-02-26 10:22:50,712 DEBUG plugin.PluginRepository - plugin: id=parse-tika name=Tika Parser Plug-in version=1.0.0 provider=nutch.orgclass=null
  2319. 2015-02-26 10:22:50,714 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.tika.TikaParser
  2320. 2015-02-26 10:22:50,724 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-elastic/plugin.xml
  2321. 2015-02-26 10:22:50,728 DEBUG plugin.PluginRepository - plugin: id=indexer-elastic name=ElasticIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  2322. 2015-02-26 10:22:50,729 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
  2323. 2015-02-26 10:22:50,732 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/microformats-reltag/plugin.xml
  2324. 2015-02-26 10:22:50,736 DEBUG plugin.PluginRepository - plugin: id=microformats-reltag name=Rel-Tag microformat Parser/Indexer/Querier version=1.0.0 provider=nutch.orgclass=null
  2325. 2015-02-26 10:22:50,736 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.microformats.reltag.RelTagParser
  2326. 2015-02-26 10:22:50,736 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.microformats.reltag.RelTagIndexingFilter
  2327. 2015-02-26 10:22:50,737 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-html/plugin.xml
  2328. 2015-02-26 10:22:50,740 DEBUG plugin.PluginRepository - plugin: id=parse-html name=Html Parse Plug-in version=1.0.0 provider=nutch.orgclass=null
  2329. 2015-02-26 10:22:50,741 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.html.HtmlParser
  2330. 2015-02-26 10:22:50,741 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/subcollection/plugin.xml
  2331. 2015-02-26 10:22:50,744 DEBUG plugin.PluginRepository - plugin: id=subcollection name=Subcollection indexing and query filter version=1.0.0 provider=apache.orgclass=null
  2332. 2015-02-26 10:22:50,745 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.subcollection.SubcollectionIndexingFilter
  2333. 2015-02-26 10:22:50,745 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-metadata/plugin.xml
  2334. 2015-02-26 10:22:50,748 DEBUG plugin.PluginRepository - plugin: id=index-metadata name=Metadata Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  2335. 2015-02-26 10:22:50,748 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.metadata.MetadataIndexer
  2336. 2015-02-26 10:22:50,749 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-httpclient/plugin.xml
  2337. 2015-02-26 10:22:50,757 DEBUG plugin.PluginRepository - plugin: id=protocol-httpclient name=Http / Https Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  2338. 2015-02-26 10:22:50,757 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  2339. 2015-02-26 10:22:50,757 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  2340. 2015-02-26 10:22:50,758 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/nutch-extensionpoints/plugin.xml
  2341. 2015-02-26 10:22:50,761 DEBUG plugin.PluginRepository - plugin: id=nutch-extensionpoints name=the nutch core extension points version=2.0.0 provider=nutch.orgclass=null
  2342. 2015-02-26 10:22:50,763 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-pass/plugin.xml
  2343. 2015-02-26 10:22:50,766 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-pass name=Pass-through URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  2344. 2015-02-26 10:22:50,766 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer
  2345. 2015-02-26 10:22:50,766 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-regex/plugin.xml
  2346. 2015-02-26 10:22:50,770 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-regex name=Regex URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  2347. 2015-02-26 10:22:50,770 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
  2348. 2015-02-26 10:22:50,770 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-ftp/plugin.xml
  2349. 2015-02-26 10:22:50,773 DEBUG plugin.PluginRepository - plugin: id=protocol-ftp name=Ftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  2350. 2015-02-26 10:22:50,774 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.ftp.Ftp
  2351. 2015-02-26 10:22:50,774 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-nekohtml/plugin.xml
  2352. 2015-02-26 10:22:50,777 DEBUG plugin.PluginRepository - plugin: id=lib-nekohtml name=CyberNeko HTML Parser version=1.9.19 provider=net.sourceforge.nekohtmlclass=null
  2353. 2015-02-26 10:22:50,777 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-http/plugin.xml
  2354. 2015-02-26 10:22:50,780 DEBUG plugin.PluginRepository - plugin: id=protocol-http name=Http Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  2355. 2015-02-26 10:22:50,781 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  2356. 2015-02-26 10:22:50,781 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  2357. 2015-02-26 10:22:50,782 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-solr/plugin.xml
  2358. 2015-02-26 10:22:50,785 DEBUG plugin.PluginRepository - plugin: id=indexer-solr name=SOLRIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  2359. 2015-02-26 10:22:50,786 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.solr.SolrIndexWriter
  2360. 2015-02-26 10:22:50,787 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/language-identifier/plugin.xml
  2361. 2015-02-26 10:22:50,790 DEBUG plugin.PluginRepository - plugin: id=language-identifier name=Language Identification Parser/Filter version=1.0.0 provider=nutch.orgclass=null
  2362. 2015-02-26 10:22:50,791 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.analysis.lang.HTMLLanguageParser
  2363. 2015-02-26 10:22:50,791 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.analysis.lang.LanguageIndexingFilter
  2364. 2015-02-26 10:22:50,791 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-basic/plugin.xml
  2365. 2015-02-26 10:22:50,794 DEBUG plugin.PluginRepository - plugin: id=index-basic name=Basic Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  2366. 2015-02-26 10:22:50,794 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.basic.BasicIndexingFilter
  2367. 2015-02-26 10:22:50,795 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-link/plugin.xml
  2368. 2015-02-26 10:22:50,797 DEBUG plugin.PluginRepository - plugin: id=scoring-link name=Link Analysis Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  2369. 2015-02-26 10:22:50,797 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.link.LinkAnalysisScoringFilter
  2370. 2015-02-26 10:22:50,798 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-http/plugin.xml
  2371. 2015-02-26 10:22:50,800 DEBUG plugin.PluginRepository - plugin: id=lib-http name=HTTP Framework version=1.0 provider=org.apache.nutchclass=null
  2372. 2015-02-26 10:22:50,801 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-regex-filter/plugin.xml
  2373. 2015-02-26 10:22:50,803 DEBUG plugin.PluginRepository - plugin: id=lib-regex-filter name=Regex URL Filter Framework version=1.0 provider=org.apache.nutchclass=null
  2374. 2015-02-26 10:22:50,803 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-prefix/plugin.xml
  2375. 2015-02-26 10:22:50,806 DEBUG plugin.PluginRepository - plugin: id=urlfilter-prefix name=Prefix URL Filter version=1.0.0 provider=nutch.orgclass=null
  2376. 2015-02-26 10:22:50,806 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.prefix.PrefixURLFilter
  2377. 2015-02-26 10:22:50,807 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-basic/plugin.xml
  2378. 2015-02-26 10:22:50,809 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-basic name=Basic URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  2379. 2015-02-26 10:22:50,810 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
  2380. 2015-02-26 10:22:50,810 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/tld/plugin.xml
  2381. 2015-02-26 10:22:50,813 DEBUG plugin.PluginRepository - plugin: id=tld name=Top Level Domain Plugin version=1.0.0 provider=nutch.orgclass=null
  2382. 2015-02-26 10:22:50,813 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.tld.TLDIndexingFilter
  2383. 2015-02-26 10:22:50,813 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.tld.TLDScoringFilter
  2384. 2015-02-26 10:22:50,814 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-anchor/plugin.xml
  2385. 2015-02-26 10:22:50,816 DEBUG plugin.PluginRepository - plugin: id=index-anchor name=Anchor Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  2386. 2015-02-26 10:22:50,817 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.anchor.AnchorIndexingFilter
  2387. 2015-02-26 10:22:50,817 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-suffix/plugin.xml
  2388. 2015-02-26 10:22:50,820 DEBUG plugin.PluginRepository - plugin: id=urlfilter-suffix name=Suffix URL Filter version=1.0.0 provider=nutch.orgclass=null
  2389. 2015-02-26 10:22:50,820 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.suffix.SuffixURLFilter
  2390. 2015-02-26 10:22:50,820 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-regex/plugin.xml
  2391. 2015-02-26 10:22:50,823 DEBUG plugin.PluginRepository - plugin: id=urlfilter-regex name=Regex URL Filter version=1.0.0 provider=nutch.orgclass=null
  2392. 2015-02-26 10:22:50,823 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.regex.RegexURLFilter
  2393. 2015-02-26 10:22:50,824 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-metatags/plugin.xml
  2394. 2015-02-26 10:22:50,826 DEBUG plugin.PluginRepository - plugin: id=parse-metatags name=MetaTags version=1.0 provider=digitalpebble.comclass=null
  2395. 2015-02-26 10:22:50,826 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.metatags.MetaTagsParser
  2396. 2015-02-26 10:22:50,827 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-domain/plugin.xml
  2397. 2015-02-26 10:22:50,829 DEBUG plugin.PluginRepository - plugin: id=urlfilter-domain name=Domain URL Filter version=1.0.0 provider=nutch.orgclass=null
  2398. 2015-02-26 10:22:50,830 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.domain.DomainURLFilter
  2399. 2015-02-26 10:22:50,830 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-opic/plugin.xml
  2400. 2015-02-26 10:22:50,832 DEBUG plugin.PluginRepository - plugin: id=scoring-opic name=OPIC Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  2401. 2015-02-26 10:22:50,833 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.opic.OPICScoringFilter
  2402. 2015-02-26 10:22:50,833 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-more/plugin.xml
  2403. 2015-02-26 10:22:50,836 DEBUG plugin.PluginRepository - plugin: id=index-more name=More Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  2404. 2015-02-26 10:22:50,836 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.more.MoreIndexingFilter
  2405. 2015-02-26 10:22:50,836 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/creativecommons/plugin.xml
  2406. 2015-02-26 10:22:50,839 DEBUG plugin.PluginRepository - plugin: id=creativecommons name=Creative Commons Plugins version=1.0.0 provider=nutch.orgclass=null
  2407. 2015-02-26 10:22:50,839 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.creativecommons.nutch.CCParseFilter
  2408. 2015-02-26 10:22:50,840 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.creativecommons.nutch.CCIndexingFilter
  2409. 2015-02-26 10:22:50,840 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-file/plugin.xml
  2410. 2015-02-26 10:22:50,842 DEBUG plugin.PluginRepository - plugin: id=protocol-file name=File Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  2411. 2015-02-26 10:22:50,843 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.file.File
  2412. 2015-02-26 10:22:50,843 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-js/plugin.xml
  2413. 2015-02-26 10:22:50,846 DEBUG plugin.PluginRepository - plugin: id=parse-js name=JavaScript Parser version=1.0.0 provider=nutch.orgclass=null
  2414. 2015-02-26 10:22:50,846 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.js.JSParseFilter
  2415. 2015-02-26 10:22:50,847 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.js.JSParseFilter
  2416. 2015-02-26 10:22:50,847 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-validator/plugin.xml
  2417. 2015-02-26 10:22:50,850 DEBUG plugin.PluginRepository - plugin: id=urlfilter-validator name=URL Validator version=1.0.0 provider=nutch.orgclass=null
  2418. 2015-02-26 10:22:50,850 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.validator.UrlValidator
  2419. 2015-02-26 10:22:50,851 DEBUG plugin.PluginRepository - not including: urlfilter-suffix
  2420. 2015-02-26 10:22:50,851 DEBUG plugin.PluginRepository - not including: language-identifier
  2421. 2015-02-26 10:22:50,851 DEBUG plugin.PluginRepository - not including: protocol-httpclient
  2422. 2015-02-26 10:22:50,851 DEBUG plugin.PluginRepository - not including: scoring-link
  2423. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: indexer-elastic
  2424. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: creativecommons
  2425. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: index-more
  2426. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: lib-http
  2427. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: lib-xml
  2428. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: subcollection
  2429. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: urlfilter-prefix
  2430. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: lib-regex-filter
  2431. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: nutch-extensionpoints
  2432. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: urlfilter-automaton
  2433. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: urlfilter-validator
  2434. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: protocol-sftp
  2435. 2015-02-26 10:22:50,852 DEBUG plugin.PluginRepository - not including: protocol-ftp
  2436. 2015-02-26 10:22:50,853 DEBUG plugin.PluginRepository - not including: lib-nekohtml
  2437. 2015-02-26 10:22:50,853 DEBUG plugin.PluginRepository - not including: parse-js
  2438. 2015-02-26 10:22:50,853 DEBUG plugin.PluginRepository - not including: protocol-file
  2439. 2015-02-26 10:22:50,853 DEBUG plugin.PluginRepository - not including: urlfilter-domain
  2440. 2015-02-26 10:22:50,853 DEBUG plugin.PluginRepository - not including: microformats-reltag
  2441. 2015-02-26 10:22:50,853 DEBUG plugin.PluginRepository - not including: tld
  2442. 2015-02-26 10:22:50,854 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexingFilter
  2443. 2015-02-26 10:22:50,854 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexCleaningFilter
  2444. 2015-02-26 10:22:50,854 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.Parser
  2445. 2015-02-26 10:22:50,854 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.ParseFilter
  2446. 2015-02-26 10:22:50,854 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.protocol.Protocol
  2447. 2015-02-26 10:22:50,854 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLFilter
  2448. 2015-02-26 10:22:50,854 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLNormalizer
  2449. 2015-02-26 10:22:50,854 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.scoring.ScoringFilter
  2450. 2015-02-26 10:22:50,854 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexWriter
  2451. 2015-02-26 10:22:50,854 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
  2452. 2015-02-26 10:22:50,854 INFO plugin.PluginRepository - Registered Plugins:
  2453. 2015-02-26 10:22:50,854 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
  2454. 2015-02-26 10:22:50,854 INFO plugin.PluginRepository - Basic URL Normalizer (urlnormalizer-basic)
  2455. 2015-02-26 10:22:50,854 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
  2456. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
  2457. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - SOLRIndexWriter (indexer-solr)
  2458. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - HTTP Framework (lib-http)
  2459. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
  2460. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Pass-through URL Normalizer (urlnormalizer-pass)
  2461. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Http Protocol Plug-in (protocol-http)
  2462. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Regex URL Normalizer (urlnormalizer-regex)
  2463. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
  2464. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Tika Parser Plug-in (parse-tika)
  2465. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
  2466. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
  2467. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
  2468. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - MetaTags (parse-metatags)
  2469. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Metadata Indexing Filter (index-metadata)
  2470. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Registered Extension-Points:
  2471. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
  2472. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Nutch Index Cleaning Filter (org.apache.nutch.indexer.IndexCleaningFilter)
  2473. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
  2474. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Parse Filter (org.apache.nutch.parse.ParseFilter)
  2475. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
  2476. 2015-02-26 10:22:50,855 INFO plugin.PluginRepository - Nutch Index Writer (org.apache.nutch.indexer.IndexWriter)
  2477. 2015-02-26 10:22:50,856 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
  2478. 2015-02-26 10:22:50,856 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
  2479. 2015-02-26 10:22:50,856 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
  2480. 2015-02-26 10:22:51,030 DEBUG security.Groups - Creating new Groups object
  2481. 2015-02-26 10:22:51,039 DEBUG security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
  2482. 2015-02-26 10:22:51,098 DEBUG security.UserGroupInformation - hadoop login
  2483. 2015-02-26 10:22:51,098 DEBUG security.UserGroupInformation - hadoop login commit
  2484. 2015-02-26 10:22:51,104 DEBUG security.UserGroupInformation - using local user:UnixPrincipal: katonj
  2485. 2015-02-26 10:22:51,107 DEBUG security.UserGroupInformation - UGI loginUser:katonj
  2486. 2015-02-26 10:22:51,802 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'webpage' in cluster 'Test Cluster' on host 'localhost:9160'.
  2487. 2015-02-26 10:22:51,802 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'host' in cluster 'Test Cluster' on host 'localhost:9160'.
  2488. 2015-02-26 10:22:51,802 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.WebPage' for Keyspace 'webpage'.
  2489. 2015-02-26 10:22:51,802 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.Host' for Keyspace 'host'.
  2490. 2015-02-26 10:22:51,802 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2491. 2015-02-26 10:22:51,804 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2492. 2015-02-26 10:22:51,804 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2493. 2015-02-26 10:22:51,804 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2494. 2015-02-26 10:22:51,804 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2495. 2015-02-26 10:22:51,804 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2496. 2015-02-26 10:22:51,804 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2497. 2015-02-26 10:22:51,840 DEBUG store.CassandraMapping - Located column family: 'p'
  2498. 2015-02-26 10:22:51,840 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2499. 2015-02-26 10:22:51,840 DEBUG store.CassandraMapping - Located column family: 'f'
  2500. 2015-02-26 10:22:51,840 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2501. 2015-02-26 10:22:51,840 DEBUG store.CassandraMapping - Located column family: 'sc'
  2502. 2015-02-26 10:22:51,840 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2503. 2015-02-26 10:22:51,840 DEBUG store.CassandraMapping - Located super column family
  2504. 2015-02-26 10:22:51,840 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2505. 2015-02-26 10:22:51,896 INFO connection.CassandraHostRetryService - Downed Host Retry service started with queue size -1 and retry delay 10s
  2506. 2015-02-26 10:22:51,976 INFO service.JmxMonitor - Registering JMX me.prettyprint.cassandra.service_Test Cluster:ServiceType=hector,MonitorType=hector
  2507. 2015-02-26 10:22:52,005 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2508. 2015-02-26 10:22:52,014 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2509. 2015-02-26 10:22:52,014 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2510. 2015-02-26 10:22:52,014 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2511. 2015-02-26 10:22:52,015 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2512. 2015-02-26 10:22:52,015 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2513. 2015-02-26 10:22:52,015 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2514. 2015-02-26 10:22:52,015 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2515. 2015-02-26 10:22:52,016 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2516. 2015-02-26 10:22:52,016 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2517. 2015-02-26 10:22:52,016 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2518. 2015-02-26 10:22:52,017 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2519. 2015-02-26 10:22:52,017 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2520. 2015-02-26 10:22:52,017 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2521. 2015-02-26 10:22:52,017 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2522. 2015-02-26 10:22:52,018 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  2523. 2015-02-26 10:22:52,018 DEBUG connection.ConcurrentHClientPool - Concurrent Host pool started with 16 active clients; max: 50 exhausted wait: 0
  2524. 2015-02-26 10:22:52,098 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2525. 2015-02-26 10:22:52,098 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2526. 2015-02-26 10:22:52,098 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2527. 2015-02-26 10:22:52,168 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2528. 2015-02-26 10:22:52,168 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2529. 2015-02-26 10:22:52,168 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2530. 2015-02-26 10:22:52,168 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2531. 2015-02-26 10:22:52,168 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2532. 2015-02-26 10:22:52,168 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2533. 2015-02-26 10:22:52,168 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2534. 2015-02-26 10:22:52,169 DEBUG store.CassandraMapping - Located column family: 'p'
  2535. 2015-02-26 10:22:52,169 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2536. 2015-02-26 10:22:52,169 DEBUG store.CassandraMapping - Located column family: 'f'
  2537. 2015-02-26 10:22:52,169 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2538. 2015-02-26 10:22:52,169 DEBUG store.CassandraMapping - Located column family: 'sc'
  2539. 2015-02-26 10:22:52,169 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2540. 2015-02-26 10:22:52,169 DEBUG store.CassandraMapping - Located super column family
  2541. 2015-02-26 10:22:52,169 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2542. 2015-02-26 10:22:52,170 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2543. 2015-02-26 10:22:52,170 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2544. 2015-02-26 10:22:52,170 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2545. 2015-02-26 10:22:52,171 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapreduce.Job.connect(Job.java:561)
  2546. 2015-02-26 10:22:52,191 DEBUG fs.FileSystem - Creating filesystem for file:///
  2547. 2015-02-26 10:22:52,203 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  2548. 2015-02-26 10:22:52,209 DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
  2549. 2015-02-26 10:22:52,210 DEBUG util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
  2550. 2015-02-26 10:22:52,210 DEBUG util.NativeCodeLoader - java.library.path=/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/lib/native/Linux-amd64-64
  2551. 2015-02-26 10:22:52,210 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2552. 2015-02-26 10:22:52,211 DEBUG mapred.JobClient - adding the following namenodes' delegation tokens:null
  2553. 2015-02-26 10:22:52,211 DEBUG mapred.JobClient - default FileSystem: file:///
  2554. 2015-02-26 10:22:52,250 DEBUG mapred.JobClient - Creating splits at file:/tmp/hadoop-katonj/mapred/staging/katonj112922449/.staging/job_local112922449_0001
  2555. 2015-02-26 10:22:52,254 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2556. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2557. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2558. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2559. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2560. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2561. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2562. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located column family: 'p'
  2563. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2564. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located column family: 'f'
  2565. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2566. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located column family: 'sc'
  2567. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2568. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Located super column family
  2569. 2015-02-26 10:22:52,254 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2570. 2015-02-26 10:22:52,256 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2571. 2015-02-26 10:22:52,256 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2572. 2015-02-26 10:22:52,256 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2573. 2015-02-26 10:22:52,390 DEBUG mapred.JobClient - Printing tokens for job: job_local112922449_0001
  2574. 2015-02-26 10:22:52,405 DEBUG mapred.TaskRunner - Fully deleting contents of /tmp/hadoop-katonj/mapred/local/localRunner
  2575. 2015-02-26 10:22:52,479 INFO mapred.JobClient - Running job: job_local112922449_0001
  2576. 2015-02-26 10:22:52,514 DEBUG mapred.LocalJobRunner - Starting thread pool executor.
  2577. 2015-02-26 10:22:52,514 DEBUG mapred.LocalJobRunner - Max local threads: 1
  2578. 2015-02-26 10:22:52,514 DEBUG mapred.LocalJobRunner - Map tasks to process: 1
  2579. 2015-02-26 10:22:52,515 INFO mapred.LocalJobRunner - Waiting for map tasks
  2580. 2015-02-26 10:22:52,518 INFO mapred.LocalJobRunner - Starting task: attempt_local112922449_0001_m_000000_0
  2581. 2015-02-26 10:22:52,531 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  2582. 2015-02-26 10:22:52,540 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2583. 2015-02-26 10:22:52,540 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2584. 2015-02-26 10:22:52,557 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local112922449_0001/attempt_local112922449_0001_m_000000_0
  2585. 2015-02-26 10:22:52,560 DEBUG mapred.Task - using new api for output committer
  2586. 2015-02-26 10:22:52,572 INFO util.ProcessTree - setsid exited with exit code 0
  2587. 2015-02-26 10:22:52,577 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b4addd
  2588. 2015-02-26 10:22:52,585 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2589. 2015-02-26 10:22:52,585 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2590. 2015-02-26 10:22:52,585 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2591. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2592. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2593. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2594. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2595. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Located column family: 'p'
  2596. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2597. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Located column family: 'f'
  2598. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2599. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Located column family: 'sc'
  2600. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2601. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Located super column family
  2602. 2015-02-26 10:22:52,586 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2603. 2015-02-26 10:22:52,589 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2604. 2015-02-26 10:22:52,589 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2605. 2015-02-26 10:22:52,589 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2606. 2015-02-26 10:22:52,596 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2607. 2015-02-26 10:22:52,596 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2608. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2609. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2610. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2611. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2612. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2613. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located column family: 'p'
  2614. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2615. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located column family: 'f'
  2616. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2617. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located column family: 'sc'
  2618. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2619. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Located super column family
  2620. 2015-02-26 10:22:52,597 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2621. 2015-02-26 10:22:52,599 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2622. 2015-02-26 10:22:52,599 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2623. 2015-02-26 10:22:52,599 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2624. 2015-02-26 10:22:52,602 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2625. 2015-02-26 10:22:52,602 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2626. 2015-02-26 10:22:52,602 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2627. 2015-02-26 10:22:52,602 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2628. 2015-02-26 10:22:52,602 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2629. 2015-02-26 10:22:52,602 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2630. 2015-02-26 10:22:52,602 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2631. 2015-02-26 10:22:52,602 DEBUG store.CassandraMapping - Located column family: 'p'
  2632. 2015-02-26 10:22:52,602 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2633. 2015-02-26 10:22:52,603 DEBUG store.CassandraMapping - Located column family: 'f'
  2634. 2015-02-26 10:22:52,603 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2635. 2015-02-26 10:22:52,603 DEBUG store.CassandraMapping - Located column family: 'sc'
  2636. 2015-02-26 10:22:52,603 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2637. 2015-02-26 10:22:52,603 DEBUG store.CassandraMapping - Located super column family
  2638. 2015-02-26 10:22:52,603 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2639. 2015-02-26 10:22:52,605 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2640. 2015-02-26 10:22:52,605 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2641. 2015-02-26 10:22:52,605 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2642. 2015-02-26 10:22:52,605 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2643. 2015-02-26 10:22:52,606 INFO mapred.MapTask - Processing split: org.apache.gora.mapreduce.GoraInputSplit@4c6b39a3
  2644. 2015-02-26 10:22:52,606 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2645. 2015-02-26 10:22:52,610 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2646. 2015-02-26 10:22:52,610 DEBUG mapred.Counters - Adding BYTES_READ
  2647. 2015-02-26 10:22:52,612 INFO mapreduce.GoraRecordReader - gora.buffer.read.limit = 10000
  2648. 2015-02-26 10:22:52,623 INFO mapred.MapTask - io.sort.mb = 100
  2649. 2015-02-26 10:22:52,681 INFO mapred.MapTask - data buffer = 79691776/99614720
  2650. 2015-02-26 10:22:52,681 INFO mapred.MapTask - record buffer = 262144/327680
  2651. 2015-02-26 10:22:52,682 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  2652. 2015-02-26 10:22:52,682 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2653. 2015-02-26 10:22:52,682 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  2654. 2015-02-26 10:22:52,682 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  2655. 2015-02-26 10:22:52,682 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  2656. 2015-02-26 10:22:52,693 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local112922449_0001.xml, instantiating a new object cache
  2657. 2015-02-26 10:22:52,822 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2658. 2015-02-26 10:22:52,877 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2659. 2015-02-26 10:22:52,882 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2660. 2015-02-26 10:22:52,917 INFO mapred.MapTask - Starting flush of map output
  2661. 2015-02-26 10:22:52,934 INFO compress.CodecPool - Got brand-new compressor
  2662. 2015-02-26 10:22:52,935 INFO mapred.MapTask - Finished spill 0
  2663. 2015-02-26 10:22:52,938 INFO mapred.Task - Task:attempt_local112922449_0001_m_000000_0 is done. And is in the process of commiting
  2664. 2015-02-26 10:22:52,942 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2665. 2015-02-26 10:22:52,942 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2666. 2015-02-26 10:22:52,942 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2667. 2015-02-26 10:22:52,942 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2668. 2015-02-26 10:22:52,942 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2669. 2015-02-26 10:22:52,942 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2670. 2015-02-26 10:22:52,942 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2671. 2015-02-26 10:22:52,944 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2672. 2015-02-26 10:22:52,945 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2673. 2015-02-26 10:22:52,945 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2674. 2015-02-26 10:22:52,945 INFO mapred.LocalJobRunner -
  2675. 2015-02-26 10:22:52,945 INFO mapred.Task - Task 'attempt_local112922449_0001_m_000000_0' done.
  2676. 2015-02-26 10:22:52,945 INFO mapred.LocalJobRunner - Finishing task: attempt_local112922449_0001_m_000000_0
  2677. 2015-02-26 10:22:52,946 INFO mapred.LocalJobRunner - Map task executor complete.
  2678. 2015-02-26 10:22:52,952 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  2679. 2015-02-26 10:22:52,952 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2680. 2015-02-26 10:22:52,952 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2681. 2015-02-26 10:22:52,952 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  2682. 2015-02-26 10:22:52,952 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  2683. 2015-02-26 10:22:52,953 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  2684. 2015-02-26 10:22:52,953 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  2685. 2015-02-26 10:22:52,953 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  2686. 2015-02-26 10:22:52,955 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local112922449_0001/attempt_local112922449_0001_r_000000_0
  2687. 2015-02-26 10:22:52,959 DEBUG mapred.Task - using new api for output committer
  2688. 2015-02-26 10:22:52,960 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4b55cd6b
  2689. 2015-02-26 10:22:52,960 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2690. 2015-02-26 10:22:52,961 INFO mapred.LocalJobRunner -
  2691. 2015-02-26 10:22:52,967 INFO mapred.Merger - Merging 1 sorted segments
  2692. 2015-02-26 10:22:52,973 INFO compress.CodecPool - Got brand-new decompressor
  2693. 2015-02-26 10:22:52,974 INFO mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 120 bytes
  2694. 2015-02-26 10:22:52,974 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2695. 2015-02-26 10:22:52,974 INFO mapred.LocalJobRunner -
  2696. 2015-02-26 10:22:52,981 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2697. 2015-02-26 10:22:52,981 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  2698. 2015-02-26 10:22:52,984 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  2699. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  2700. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  2701. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  2702. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  2703. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  2704. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  2705. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located column family: 'p'
  2706. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2707. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located column family: 'f'
  2708. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2709. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located column family: 'sc'
  2710. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  2711. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Located super column family
  2712. 2015-02-26 10:22:52,984 DEBUG store.CassandraMapping - Added super column family: 'sc'
  2713. 2015-02-26 10:22:52,987 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  2714. 2015-02-26 10:22:52,987 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  2715. 2015-02-26 10:22:52,987 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  2716. 2015-02-26 10:22:52,987 INFO mapreduce.GoraRecordWriter - gora.buffer.write.limit = 10000
  2717. 2015-02-26 10:22:52,993 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local112922449_0001.xml, instantiating a new object cache
  2718. 2015-02-26 10:22:52,993 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
  2719. 2015-02-26 10:22:52,994 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
  2720. 2015-02-26 10:22:52,994 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
  2721. 2015-02-26 10:22:53,008 DEBUG store.CassandraStore - close
  2722. 2015-02-26 10:22:53,008 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.LongSerializer selected as appropriate Serializer.
  2723. 2015-02-26 10:22:53,008 DEBUG store.CassandraClient - fieldName:fetchTime columnName:ts
  2724. 2015-02-26 10:22:53,008 DEBUG store.CassandraClient - ttl for field:fetchTime is 0
  2725. 2015-02-26 10:22:53,016 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2726. 2015-02-26 10:22:53,027 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  2727. 2015-02-26 10:22:53,027 DEBUG store.CassandraClient - fieldName:retriesSinceFetch columnName:rsf
  2728. 2015-02-26 10:22:53,027 DEBUG store.CassandraClient - ttl for field:retriesSinceFetch is 0
  2729. 2015-02-26 10:22:53,027 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2730. 2015-02-26 10:22:53,029 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.LongSerializer selected as appropriate Serializer.
  2731. 2015-02-26 10:22:53,029 DEBUG store.CassandraClient - fieldName:modifiedTime columnName:mod
  2732. 2015-02-26 10:22:53,029 DEBUG store.CassandraClient - ttl for field:modifiedTime is 0
  2733. 2015-02-26 10:22:53,029 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2734. 2015-02-26 10:22:53,030 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.LongSerializer selected as appropriate Serializer.
  2735. 2015-02-26 10:22:53,031 DEBUG store.CassandraClient - fieldName:prevModifiedTime columnName:pmod
  2736. 2015-02-26 10:22:53,031 DEBUG store.CassandraClient - ttl for field:prevModifiedTime is 0
  2737. 2015-02-26 10:22:53,031 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2738. 2015-02-26 10:22:53,032 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.FloatSerializer selected as appropriate Serializer.
  2739. 2015-02-26 10:22:53,032 DEBUG store.CassandraClient - fieldName:score columnName:s
  2740. 2015-02-26 10:22:53,032 DEBUG store.CassandraClient - ttl for field:score is 0
  2741. 2015-02-26 10:22:53,032 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2742. 2015-02-26 10:22:53,039 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2743. 2015-02-26 10:22:53,050 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  2744. 2015-02-26 10:22:53,050 DEBUG store.CassandraClient - ttl for field:markers is 0
  2745. 2015-02-26 10:22:53,050 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2746. 2015-02-26 10:22:53,052 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  2747. 2015-02-26 10:22:53,052 DEBUG store.CassandraClient - ttl for field:markers is 0
  2748. 2015-02-26 10:22:53,052 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2749. 2015-02-26 10:22:53,053 DEBUG store.CassandraClient - class org.apache.gora.cassandra.serializers.CharSequenceSerializer selected as appropriate Serializer.
  2750. 2015-02-26 10:22:53,053 DEBUG store.CassandraClient - ttl for field:markers is 0
  2751. 2015-02-26 10:22:53,053 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2752. 2015-02-26 10:22:53,055 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  2753. 2015-02-26 10:22:53,055 DEBUG store.CassandraClient - ttl for field:markers is 0
  2754. 2015-02-26 10:22:53,056 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2755. 2015-02-26 10:22:53,057 DEBUG store.CassandraClient - class org.apache.gora.cassandra.serializers.CharSequenceSerializer selected as appropriate Serializer.
  2756. 2015-02-26 10:22:53,057 DEBUG store.CassandraClient - ttl for field:markers is 0
  2757. 2015-02-26 10:22:53,057 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2758. 2015-02-26 10:22:53,059 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  2759. 2015-02-26 10:22:53,060 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.IntegerSerializer selected as appropriate Serializer.
  2760. 2015-02-26 10:22:53,060 DEBUG store.CassandraClient - ttl for field:metadata is 0
  2761. 2015-02-26 10:22:53,061 DEBUG store.CassandraClient - class me.prettyprint.cassandra.serializers.ByteBufferSerializer selected as appropriate Serializer.
  2762. 2015-02-26 10:22:53,061 DEBUG store.CassandraClient - ttl for field:metadata is 0
  2763. 2015-02-26 10:22:53,063 INFO mapred.Task - Task:attempt_local112922449_0001_r_000000_0 is done. And is in the process of commiting
  2764. 2015-02-26 10:22:53,063 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2765. 2015-02-26 10:22:53,063 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2766. 2015-02-26 10:22:53,063 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2767. 2015-02-26 10:22:53,063 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2768. 2015-02-26 10:22:53,063 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2769. 2015-02-26 10:22:53,063 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2770. 2015-02-26 10:22:53,063 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2771. 2015-02-26 10:22:53,064 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2772. 2015-02-26 10:22:53,064 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2773. 2015-02-26 10:22:53,065 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2774. 2015-02-26 10:22:53,065 INFO mapred.LocalJobRunner - reduce > reduce
  2775. 2015-02-26 10:22:53,065 INFO mapred.Task - Task 'attempt_local112922449_0001_r_000000_0' done.
  2776. 2015-02-26 10:22:53,066 WARN mapred.FileOutputCommitter - Output path is null in cleanup
  2777. 2015-02-26 10:22:53,481 INFO mapred.JobClient - map 100% reduce 100%
  2778. 2015-02-26 10:22:53,482 INFO mapred.JobClient - Job complete: job_local112922449_0001
  2779. 2015-02-26 10:22:53,482 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2780. 2015-02-26 10:22:53,482 DEBUG mapred.Counters - Adding BYTES_READ
  2781. 2015-02-26 10:22:53,482 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2782. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2783. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2784. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2785. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  2786. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  2787. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2788. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2789. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2790. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  2791. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2792. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2793. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2794. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2795. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2796. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  2797. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2798. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2799. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2800. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2801. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding BYTES_READ
  2802. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2803. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  2804. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  2805. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2806. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2807. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2808. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  2809. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2810. 2015-02-26 10:22:53,483 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2811. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2812. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  2813. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2814. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2815. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2816. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  2817. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  2818. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  2819. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  2820. 2015-02-26 10:22:53,484 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  2821. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Counters: 20
  2822. 2015-02-26 10:22:53,484 INFO mapred.JobClient - File Output Format Counters
  2823. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Bytes Written=0
  2824. 2015-02-26 10:22:53,484 INFO mapred.JobClient - File Input Format Counters
  2825. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Bytes Read=0
  2826. 2015-02-26 10:22:53,484 INFO mapred.JobClient - FileSystemCounters
  2827. 2015-02-26 10:22:53,484 INFO mapred.JobClient - FILE_BYTES_READ=1217782
  2828. 2015-02-26 10:22:53,484 INFO mapred.JobClient - FILE_BYTES_WRITTEN=1394272
  2829. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Map-Reduce Framework
  2830. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Reduce input groups=1
  2831. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Map output materialized bytes=124
  2832. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Combine output records=0
  2833. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Map input records=1
  2834. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Reduce shuffle bytes=0
  2835. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Physical memory (bytes) snapshot=0
  2836. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Reduce output records=1
  2837. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Spilled Records=2
  2838. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Map output bytes=128
  2839. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Total committed heap usage (bytes)=1516240896
  2840. 2015-02-26 10:22:53,484 INFO mapred.JobClient - CPU time spent (ms)=0
  2841. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Virtual memory (bytes) snapshot=0
  2842. 2015-02-26 10:22:53,484 INFO mapred.JobClient - SPLIT_RAW_BYTES=934
  2843. 2015-02-26 10:22:53,484 INFO mapred.JobClient - Map output records=1
  2844. 2015-02-26 10:22:53,485 INFO mapred.JobClient - Combine input records=0
  2845. 2015-02-26 10:22:53,485 INFO mapred.JobClient - Reduce input records=1
  2846. 2015-02-26 10:22:53,485 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2847. 2015-02-26 10:22:53,485 DEBUG mapred.Counters - Adding BYTES_READ
  2848. 2015-02-26 10:22:53,485 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2849. 2015-02-26 10:22:53,485 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2850. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2851. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2852. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  2853. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  2854. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2855. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2856. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2857. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  2858. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2859. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2860. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2861. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2862. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2863. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  2864. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  2865. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  2866. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  2867. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  2868. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding BYTES_READ
  2869. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  2870. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  2871. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  2872. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  2873. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  2874. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  2875. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  2876. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  2877. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  2878. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  2879. 2015-02-26 10:22:53,486 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  2880. 2015-02-26 10:22:53,487 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  2881. 2015-02-26 10:22:53,487 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  2882. 2015-02-26 10:22:53,487 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  2883. 2015-02-26 10:22:53,487 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  2884. 2015-02-26 10:22:53,487 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  2885. 2015-02-26 10:22:53,487 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  2886. 2015-02-26 10:22:53,487 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  2887. 2015-02-26 10:22:53,487 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  2888. 2015-02-26 10:22:53,488 INFO crawl.DbUpdaterJob - DbUpdaterJob: finished at 2015-02-26 10:22:53, time elapsed: 00:00:02
  2889. 2015-02-26 10:22:53,488 DEBUG fs.FileSystem - Starting clear of FileSystem cache with 1 elements.
  2890. 2015-02-26 10:22:53,489 DEBUG fs.FileSystem - Removing filesystem for file:///
  2891. 2015-02-26 10:22:53,489 DEBUG fs.FileSystem - Removing filesystem for file:///
  2892. 2015-02-26 10:22:53,489 DEBUG fs.FileSystem - Done clearing cache
  2893. 2015-02-26 10:22:54,749 INFO indexer.IndexingJob - IndexingJob: starting
  2894. 2015-02-26 10:22:54,939 DEBUG security.Groups - Creating new Groups object
  2895. 2015-02-26 10:22:54,948 DEBUG security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
  2896. 2015-02-26 10:22:55,009 DEBUG security.UserGroupInformation - hadoop login
  2897. 2015-02-26 10:22:55,009 DEBUG security.UserGroupInformation - hadoop login commit
  2898. 2015-02-26 10:22:55,016 DEBUG security.UserGroupInformation - using local user:UnixPrincipal: katonj
  2899. 2015-02-26 10:22:55,019 DEBUG security.UserGroupInformation - UGI loginUser:katonj
  2900. 2015-02-26 10:22:55,095 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, nutch-default.xml, nutch-site.xml, instantiating a new object cache
  2901. 2015-02-26 10:22:55,104 INFO plugin.PluginRepository - Plugins: looking in: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins
  2902. 2015-02-26 10:22:55,105 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-xml/plugin.xml
  2903. 2015-02-26 10:22:55,112 DEBUG plugin.PluginRepository - plugin: id=lib-xml name=XML Libraries version=1.0 provider=org.apache.nutch.xmlclass=null
  2904. 2015-02-26 10:22:55,114 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-sftp/plugin.xml
  2905. 2015-02-26 10:22:55,117 DEBUG plugin.PluginRepository - plugin: id=protocol-sftp name=Sftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  2906. 2015-02-26 10:22:55,117 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.sftp.Sftp
  2907. 2015-02-26 10:22:55,119 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-automaton/plugin.xml
  2908. 2015-02-26 10:22:55,122 DEBUG plugin.PluginRepository - plugin: id=urlfilter-automaton name=Automaton URL Filter version=1.0.0 provider=nutch.orgclass=null
  2909. 2015-02-26 10:22:55,122 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.automaton.AutomatonURLFilter
  2910. 2015-02-26 10:22:55,123 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-tika/plugin.xml
  2911. 2015-02-26 10:22:55,126 DEBUG plugin.PluginRepository - plugin: id=parse-tika name=Tika Parser Plug-in version=1.0.0 provider=nutch.orgclass=null
  2912. 2015-02-26 10:22:55,128 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.tika.TikaParser
  2913. 2015-02-26 10:22:55,139 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-elastic/plugin.xml
  2914. 2015-02-26 10:22:55,142 DEBUG plugin.PluginRepository - plugin: id=indexer-elastic name=ElasticIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  2915. 2015-02-26 10:22:55,142 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
  2916. 2015-02-26 10:22:55,147 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/microformats-reltag/plugin.xml
  2917. 2015-02-26 10:22:55,150 DEBUG plugin.PluginRepository - plugin: id=microformats-reltag name=Rel-Tag microformat Parser/Indexer/Querier version=1.0.0 provider=nutch.orgclass=null
  2918. 2015-02-26 10:22:55,151 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.microformats.reltag.RelTagParser
  2919. 2015-02-26 10:22:55,151 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.microformats.reltag.RelTagIndexingFilter
  2920. 2015-02-26 10:22:55,151 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-html/plugin.xml
  2921. 2015-02-26 10:22:55,154 DEBUG plugin.PluginRepository - plugin: id=parse-html name=Html Parse Plug-in version=1.0.0 provider=nutch.orgclass=null
  2922. 2015-02-26 10:22:55,154 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.html.HtmlParser
  2923. 2015-02-26 10:22:55,155 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/subcollection/plugin.xml
  2924. 2015-02-26 10:22:55,158 DEBUG plugin.PluginRepository - plugin: id=subcollection name=Subcollection indexing and query filter version=1.0.0 provider=apache.orgclass=null
  2925. 2015-02-26 10:22:55,158 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.subcollection.SubcollectionIndexingFilter
  2926. 2015-02-26 10:22:55,158 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-metadata/plugin.xml
  2927. 2015-02-26 10:22:55,161 DEBUG plugin.PluginRepository - plugin: id=index-metadata name=Metadata Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  2928. 2015-02-26 10:22:55,161 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.metadata.MetadataIndexer
  2929. 2015-02-26 10:22:55,162 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-httpclient/plugin.xml
  2930. 2015-02-26 10:22:55,164 DEBUG plugin.PluginRepository - plugin: id=protocol-httpclient name=Http / Https Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  2931. 2015-02-26 10:22:55,165 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  2932. 2015-02-26 10:22:55,165 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  2933. 2015-02-26 10:22:55,165 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/nutch-extensionpoints/plugin.xml
  2934. 2015-02-26 10:22:55,168 DEBUG plugin.PluginRepository - plugin: id=nutch-extensionpoints name=the nutch core extension points version=2.0.0 provider=nutch.orgclass=null
  2935. 2015-02-26 10:22:55,170 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-pass/plugin.xml
  2936. 2015-02-26 10:22:55,173 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-pass name=Pass-through URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  2937. 2015-02-26 10:22:55,173 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer
  2938. 2015-02-26 10:22:55,173 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-regex/plugin.xml
  2939. 2015-02-26 10:22:55,176 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-regex name=Regex URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  2940. 2015-02-26 10:22:55,176 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
  2941. 2015-02-26 10:22:55,177 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-ftp/plugin.xml
  2942. 2015-02-26 10:22:55,179 DEBUG plugin.PluginRepository - plugin: id=protocol-ftp name=Ftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  2943. 2015-02-26 10:22:55,180 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.ftp.Ftp
  2944. 2015-02-26 10:22:55,180 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-nekohtml/plugin.xml
  2945. 2015-02-26 10:22:55,183 DEBUG plugin.PluginRepository - plugin: id=lib-nekohtml name=CyberNeko HTML Parser version=1.9.19 provider=net.sourceforge.nekohtmlclass=null
  2946. 2015-02-26 10:22:55,183 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-http/plugin.xml
  2947. 2015-02-26 10:22:55,186 DEBUG plugin.PluginRepository - plugin: id=protocol-http name=Http Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  2948. 2015-02-26 10:22:55,186 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  2949. 2015-02-26 10:22:55,186 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  2950. 2015-02-26 10:22:55,187 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-solr/plugin.xml
  2951. 2015-02-26 10:22:55,190 DEBUG plugin.PluginRepository - plugin: id=indexer-solr name=SOLRIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  2952. 2015-02-26 10:22:55,190 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.solr.SolrIndexWriter
  2953. 2015-02-26 10:22:55,193 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/language-identifier/plugin.xml
  2954. 2015-02-26 10:22:55,196 DEBUG plugin.PluginRepository - plugin: id=language-identifier name=Language Identification Parser/Filter version=1.0.0 provider=nutch.orgclass=null
  2955. 2015-02-26 10:22:55,196 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.analysis.lang.HTMLLanguageParser
  2956. 2015-02-26 10:22:55,196 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.analysis.lang.LanguageIndexingFilter
  2957. 2015-02-26 10:22:55,197 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-basic/plugin.xml
  2958. 2015-02-26 10:22:55,199 DEBUG plugin.PluginRepository - plugin: id=index-basic name=Basic Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  2959. 2015-02-26 10:22:55,200 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.basic.BasicIndexingFilter
  2960. 2015-02-26 10:22:55,200 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-link/plugin.xml
  2961. 2015-02-26 10:22:55,202 DEBUG plugin.PluginRepository - plugin: id=scoring-link name=Link Analysis Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  2962. 2015-02-26 10:22:55,203 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.link.LinkAnalysisScoringFilter
  2963. 2015-02-26 10:22:55,203 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-http/plugin.xml
  2964. 2015-02-26 10:22:55,205 DEBUG plugin.PluginRepository - plugin: id=lib-http name=HTTP Framework version=1.0 provider=org.apache.nutchclass=null
  2965. 2015-02-26 10:22:55,206 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-regex-filter/plugin.xml
  2966. 2015-02-26 10:22:55,208 DEBUG plugin.PluginRepository - plugin: id=lib-regex-filter name=Regex URL Filter Framework version=1.0 provider=org.apache.nutchclass=null
  2967. 2015-02-26 10:22:55,209 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-prefix/plugin.xml
  2968. 2015-02-26 10:22:55,211 DEBUG plugin.PluginRepository - plugin: id=urlfilter-prefix name=Prefix URL Filter version=1.0.0 provider=nutch.orgclass=null
  2969. 2015-02-26 10:22:55,211 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.prefix.PrefixURLFilter
  2970. 2015-02-26 10:22:55,212 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-basic/plugin.xml
  2971. 2015-02-26 10:22:55,215 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-basic name=Basic URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  2972. 2015-02-26 10:22:55,215 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
  2973. 2015-02-26 10:22:55,215 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/tld/plugin.xml
  2974. 2015-02-26 10:22:55,218 DEBUG plugin.PluginRepository - plugin: id=tld name=Top Level Domain Plugin version=1.0.0 provider=nutch.orgclass=null
  2975. 2015-02-26 10:22:55,218 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.tld.TLDIndexingFilter
  2976. 2015-02-26 10:22:55,218 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.tld.TLDScoringFilter
  2977. 2015-02-26 10:22:55,219 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-anchor/plugin.xml
  2978. 2015-02-26 10:22:55,222 DEBUG plugin.PluginRepository - plugin: id=index-anchor name=Anchor Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  2979. 2015-02-26 10:22:55,222 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.anchor.AnchorIndexingFilter
  2980. 2015-02-26 10:22:55,223 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-suffix/plugin.xml
  2981. 2015-02-26 10:22:55,230 DEBUG plugin.PluginRepository - plugin: id=urlfilter-suffix name=Suffix URL Filter version=1.0.0 provider=nutch.orgclass=null
  2982. 2015-02-26 10:22:55,230 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.suffix.SuffixURLFilter
  2983. 2015-02-26 10:22:55,231 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-regex/plugin.xml
  2984. 2015-02-26 10:22:55,233 DEBUG plugin.PluginRepository - plugin: id=urlfilter-regex name=Regex URL Filter version=1.0.0 provider=nutch.orgclass=null
  2985. 2015-02-26 10:22:55,234 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.regex.RegexURLFilter
  2986. 2015-02-26 10:22:55,234 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-metatags/plugin.xml
  2987. 2015-02-26 10:22:55,237 DEBUG plugin.PluginRepository - plugin: id=parse-metatags name=MetaTags version=1.0 provider=digitalpebble.comclass=null
  2988. 2015-02-26 10:22:55,237 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.metatags.MetaTagsParser
  2989. 2015-02-26 10:22:55,238 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-domain/plugin.xml
  2990. 2015-02-26 10:22:55,241 DEBUG plugin.PluginRepository - plugin: id=urlfilter-domain name=Domain URL Filter version=1.0.0 provider=nutch.orgclass=null
  2991. 2015-02-26 10:22:55,241 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.domain.DomainURLFilter
  2992. 2015-02-26 10:22:55,242 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-opic/plugin.xml
  2993. 2015-02-26 10:22:55,244 DEBUG plugin.PluginRepository - plugin: id=scoring-opic name=OPIC Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  2994. 2015-02-26 10:22:55,245 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.opic.OPICScoringFilter
  2995. 2015-02-26 10:22:55,245 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-more/plugin.xml
  2996. 2015-02-26 10:22:55,248 DEBUG plugin.PluginRepository - plugin: id=index-more name=More Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  2997. 2015-02-26 10:22:55,248 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.more.MoreIndexingFilter
  2998. 2015-02-26 10:22:55,248 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/creativecommons/plugin.xml
  2999. 2015-02-26 10:22:55,251 DEBUG plugin.PluginRepository - plugin: id=creativecommons name=Creative Commons Plugins version=1.0.0 provider=nutch.orgclass=null
  3000. 2015-02-26 10:22:55,252 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.creativecommons.nutch.CCParseFilter
  3001. 2015-02-26 10:22:55,252 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.creativecommons.nutch.CCIndexingFilter
  3002. 2015-02-26 10:22:55,252 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-file/plugin.xml
  3003. 2015-02-26 10:22:55,255 DEBUG plugin.PluginRepository - plugin: id=protocol-file name=File Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  3004. 2015-02-26 10:22:55,255 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.file.File
  3005. 2015-02-26 10:22:55,256 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-js/plugin.xml
  3006. 2015-02-26 10:22:55,258 DEBUG plugin.PluginRepository - plugin: id=parse-js name=JavaScript Parser version=1.0.0 provider=nutch.orgclass=null
  3007. 2015-02-26 10:22:55,259 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.js.JSParseFilter
  3008. 2015-02-26 10:22:55,259 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.js.JSParseFilter
  3009. 2015-02-26 10:22:55,260 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-validator/plugin.xml
  3010. 2015-02-26 10:22:55,262 DEBUG plugin.PluginRepository - plugin: id=urlfilter-validator name=URL Validator version=1.0.0 provider=nutch.orgclass=null
  3011. 2015-02-26 10:22:55,262 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.validator.UrlValidator
  3012. 2015-02-26 10:22:55,263 DEBUG plugin.PluginRepository - not including: urlfilter-suffix
  3013. 2015-02-26 10:22:55,263 DEBUG plugin.PluginRepository - not including: language-identifier
  3014. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: protocol-httpclient
  3015. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: scoring-link
  3016. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: indexer-elastic
  3017. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: creativecommons
  3018. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: index-more
  3019. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: lib-http
  3020. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: lib-xml
  3021. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: subcollection
  3022. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: urlfilter-prefix
  3023. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: lib-regex-filter
  3024. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: nutch-extensionpoints
  3025. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: urlfilter-automaton
  3026. 2015-02-26 10:22:55,264 DEBUG plugin.PluginRepository - not including: urlfilter-validator
  3027. 2015-02-26 10:22:55,265 DEBUG plugin.PluginRepository - not including: protocol-sftp
  3028. 2015-02-26 10:22:55,265 DEBUG plugin.PluginRepository - not including: protocol-ftp
  3029. 2015-02-26 10:22:55,265 DEBUG plugin.PluginRepository - not including: lib-nekohtml
  3030. 2015-02-26 10:22:55,265 DEBUG plugin.PluginRepository - not including: parse-js
  3031. 2015-02-26 10:22:55,265 DEBUG plugin.PluginRepository - not including: protocol-file
  3032. 2015-02-26 10:22:55,265 DEBUG plugin.PluginRepository - not including: urlfilter-domain
  3033. 2015-02-26 10:22:55,265 DEBUG plugin.PluginRepository - not including: microformats-reltag
  3034. 2015-02-26 10:22:55,265 DEBUG plugin.PluginRepository - not including: tld
  3035. 2015-02-26 10:22:55,265 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexingFilter
  3036. 2015-02-26 10:22:55,266 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexCleaningFilter
  3037. 2015-02-26 10:22:55,266 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.Parser
  3038. 2015-02-26 10:22:55,266 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.ParseFilter
  3039. 2015-02-26 10:22:55,266 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.protocol.Protocol
  3040. 2015-02-26 10:22:55,266 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLFilter
  3041. 2015-02-26 10:22:55,266 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLNormalizer
  3042. 2015-02-26 10:22:55,266 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.scoring.ScoringFilter
  3043. 2015-02-26 10:22:55,266 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexWriter
  3044. 2015-02-26 10:22:55,266 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
  3045. 2015-02-26 10:22:55,266 INFO plugin.PluginRepository - Registered Plugins:
  3046. 2015-02-26 10:22:55,266 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
  3047. 2015-02-26 10:22:55,266 INFO plugin.PluginRepository - Basic URL Normalizer (urlnormalizer-basic)
  3048. 2015-02-26 10:22:55,266 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
  3049. 2015-02-26 10:22:55,266 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
  3050. 2015-02-26 10:22:55,266 INFO plugin.PluginRepository - SOLRIndexWriter (indexer-solr)
  3051. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - HTTP Framework (lib-http)
  3052. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
  3053. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Pass-through URL Normalizer (urlnormalizer-pass)
  3054. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Http Protocol Plug-in (protocol-http)
  3055. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Regex URL Normalizer (urlnormalizer-regex)
  3056. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
  3057. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Tika Parser Plug-in (parse-tika)
  3058. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
  3059. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
  3060. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
  3061. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - MetaTags (parse-metatags)
  3062. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Metadata Indexing Filter (index-metadata)
  3063. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Registered Extension-Points:
  3064. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
  3065. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Nutch Index Cleaning Filter (org.apache.nutch.indexer.IndexCleaningFilter)
  3066. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
  3067. 2015-02-26 10:22:55,267 INFO plugin.PluginRepository - Parse Filter (org.apache.nutch.parse.ParseFilter)
  3068. 2015-02-26 10:22:55,268 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
  3069. 2015-02-26 10:22:55,268 INFO plugin.PluginRepository - Nutch Index Writer (org.apache.nutch.indexer.IndexWriter)
  3070. 2015-02-26 10:22:55,268 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
  3071. 2015-02-26 10:22:55,268 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
  3072. 2015-02-26 10:22:55,268 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
  3073. 2015-02-26 10:22:55,272 INFO basic.BasicIndexingFilter - Maximum title length for indexing set to: 100
  3074. 2015-02-26 10:22:55,272 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter
  3075. 2015-02-26 10:22:55,276 INFO anchor.AnchorIndexingFilter - Anchor deduplication is: off
  3076. 2015-02-26 10:22:55,276 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter
  3077. 2015-02-26 10:22:55,283 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.metadata.MetadataIndexer
  3078. 2015-02-26 10:22:55,863 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'webpage' in cluster 'Test Cluster' on host 'localhost:9160'.
  3079. 2015-02-26 10:22:55,863 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'host' in cluster 'Test Cluster' on host 'localhost:9160'.
  3080. 2015-02-26 10:22:55,863 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.WebPage' for Keyspace 'webpage'.
  3081. 2015-02-26 10:22:55,863 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.Host' for Keyspace 'host'.
  3082. 2015-02-26 10:22:55,863 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3083. 2015-02-26 10:22:55,864 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3084. 2015-02-26 10:22:55,864 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3085. 2015-02-26 10:22:55,864 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3086. 2015-02-26 10:22:55,864 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3087. 2015-02-26 10:22:55,864 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3088. 2015-02-26 10:22:55,864 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3089. 2015-02-26 10:22:55,895 DEBUG store.CassandraMapping - Located column family: 'p'
  3090. 2015-02-26 10:22:55,895 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3091. 2015-02-26 10:22:55,895 DEBUG store.CassandraMapping - Located column family: 'f'
  3092. 2015-02-26 10:22:55,895 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3093. 2015-02-26 10:22:55,895 DEBUG store.CassandraMapping - Located column family: 'sc'
  3094. 2015-02-26 10:22:55,895 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3095. 2015-02-26 10:22:55,895 DEBUG store.CassandraMapping - Located super column family
  3096. 2015-02-26 10:22:55,895 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3097. 2015-02-26 10:22:55,943 INFO connection.CassandraHostRetryService - Downed Host Retry service started with queue size -1 and retry delay 10s
  3098. 2015-02-26 10:22:56,012 INFO service.JmxMonitor - Registering JMX me.prettyprint.cassandra.service_Test Cluster:ServiceType=hector,MonitorType=hector
  3099. 2015-02-26 10:22:56,036 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3100. 2015-02-26 10:22:56,044 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3101. 2015-02-26 10:22:56,044 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3102. 2015-02-26 10:22:56,044 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3103. 2015-02-26 10:22:56,044 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3104. 2015-02-26 10:22:56,045 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3105. 2015-02-26 10:22:56,045 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3106. 2015-02-26 10:22:56,045 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3107. 2015-02-26 10:22:56,045 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3108. 2015-02-26 10:22:56,046 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3109. 2015-02-26 10:22:56,046 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3110. 2015-02-26 10:22:56,046 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3111. 2015-02-26 10:22:56,046 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3112. 2015-02-26 10:22:56,047 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3113. 2015-02-26 10:22:56,047 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3114. 2015-02-26 10:22:56,047 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3115. 2015-02-26 10:22:56,047 DEBUG connection.ConcurrentHClientPool - Concurrent Host pool started with 16 active clients; max: 50 exhausted wait: 0
  3116. 2015-02-26 10:22:56,115 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3117. 2015-02-26 10:22:56,116 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3118. 2015-02-26 10:22:56,116 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3119. 2015-02-26 10:22:56,170 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapreduce.Job.connect(Job.java:561)
  3120. 2015-02-26 10:22:56,189 DEBUG fs.FileSystem - Creating filesystem for file:///
  3121. 2015-02-26 10:22:56,200 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  3122. 2015-02-26 10:22:56,207 DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
  3123. 2015-02-26 10:22:56,207 DEBUG util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
  3124. 2015-02-26 10:22:56,207 DEBUG util.NativeCodeLoader - java.library.path=/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/lib/native/Linux-amd64-64
  3125. 2015-02-26 10:22:56,207 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  3126. 2015-02-26 10:22:56,208 DEBUG mapred.JobClient - adding the following namenodes' delegation tokens:null
  3127. 2015-02-26 10:22:56,208 DEBUG mapred.JobClient - default FileSystem: file:///
  3128. 2015-02-26 10:22:56,246 DEBUG mapred.JobClient - Creating splits at file:/tmp/hadoop-katonj/mapred/staging/katonj152972145/.staging/job_local152972145_0001
  3129. 2015-02-26 10:22:56,250 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3130. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3131. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3132. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3133. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3134. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3135. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3136. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located column family: 'p'
  3137. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3138. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located column family: 'f'
  3139. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3140. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located column family: 'sc'
  3141. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3142. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Located super column family
  3143. 2015-02-26 10:22:56,250 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3144. 2015-02-26 10:22:56,252 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3145. 2015-02-26 10:22:56,252 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3146. 2015-02-26 10:22:56,252 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3147. 2015-02-26 10:22:56,380 DEBUG mapred.JobClient - Printing tokens for job: job_local152972145_0001
  3148. 2015-02-26 10:22:56,395 DEBUG mapred.TaskRunner - Fully deleting contents of /tmp/hadoop-katonj/mapred/local/localRunner
  3149. 2015-02-26 10:22:56,486 INFO mapred.JobClient - Running job: job_local152972145_0001
  3150. 2015-02-26 10:22:56,511 DEBUG mapred.LocalJobRunner - Starting thread pool executor.
  3151. 2015-02-26 10:22:56,512 DEBUG mapred.LocalJobRunner - Max local threads: 1
  3152. 2015-02-26 10:22:56,512 DEBUG mapred.LocalJobRunner - Map tasks to process: 1
  3153. 2015-02-26 10:22:56,512 INFO mapred.LocalJobRunner - Waiting for map tasks
  3154. 2015-02-26 10:22:56,515 INFO mapred.LocalJobRunner - Starting task: attempt_local152972145_0001_m_000000_0
  3155. 2015-02-26 10:22:56,526 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  3156. 2015-02-26 10:22:56,534 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3157. 2015-02-26 10:22:56,534 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3158. 2015-02-26 10:22:56,545 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local152972145_0001/attempt_local152972145_0001_m_000000_0
  3159. 2015-02-26 10:22:56,547 DEBUG mapred.Task - using new api for output committer
  3160. 2015-02-26 10:22:56,559 INFO util.ProcessTree - setsid exited with exit code 0
  3161. 2015-02-26 10:22:56,563 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5469716
  3162. 2015-02-26 10:22:56,566 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3163. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3164. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3165. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3166. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3167. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3168. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3169. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - Located column family: 'p'
  3170. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3171. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - Located column family: 'f'
  3172. 2015-02-26 10:22:56,566 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3173. 2015-02-26 10:22:56,567 DEBUG store.CassandraMapping - Located column family: 'sc'
  3174. 2015-02-26 10:22:56,567 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3175. 2015-02-26 10:22:56,567 DEBUG store.CassandraMapping - Located super column family
  3176. 2015-02-26 10:22:56,567 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3177. 2015-02-26 10:22:56,568 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3178. 2015-02-26 10:22:56,568 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3179. 2015-02-26 10:22:56,568 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3180. 2015-02-26 10:22:56,573 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3181. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3182. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3183. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3184. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3185. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3186. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3187. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - Located column family: 'p'
  3188. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3189. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - Located column family: 'f'
  3190. 2015-02-26 10:22:56,573 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3191. 2015-02-26 10:22:56,574 DEBUG store.CassandraMapping - Located column family: 'sc'
  3192. 2015-02-26 10:22:56,574 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3193. 2015-02-26 10:22:56,574 DEBUG store.CassandraMapping - Located super column family
  3194. 2015-02-26 10:22:56,574 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3195. 2015-02-26 10:22:56,575 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3196. 2015-02-26 10:22:56,575 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3197. 2015-02-26 10:22:56,575 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3198. 2015-02-26 10:22:56,577 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3199. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3200. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3201. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3202. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3203. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3204. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3205. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located column family: 'p'
  3206. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3207. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located column family: 'f'
  3208. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3209. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located column family: 'sc'
  3210. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3211. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Located super column family
  3212. 2015-02-26 10:22:56,577 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3213. 2015-02-26 10:22:56,578 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3214. 2015-02-26 10:22:56,578 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3215. 2015-02-26 10:22:56,578 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3216. 2015-02-26 10:22:56,579 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3217. 2015-02-26 10:22:56,579 INFO mapred.MapTask - Processing split: org.apache.gora.mapreduce.GoraInputSplit@42710720
  3218. 2015-02-26 10:22:56,579 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3219. 2015-02-26 10:22:56,581 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3220. 2015-02-26 10:22:56,582 DEBUG mapred.Counters - Adding BYTES_READ
  3221. 2015-02-26 10:22:56,583 INFO mapreduce.GoraRecordReader - gora.buffer.read.limit = 10000
  3222. 2015-02-26 10:22:56,583 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3223. 2015-02-26 10:22:56,585 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  3224. 2015-02-26 10:22:56,585 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  3225. 2015-02-26 10:22:56,586 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local152972145_0001.xml, instantiating a new object cache
  3226. 2015-02-26 10:22:56,593 INFO indexer.IndexWriters - Adding org.apache.nutch.indexwriter.solr.SolrIndexWriter
  3227. 2015-02-26 10:22:56,637 INFO conf.Configuration - found resource solrindex-mapping.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/solrindex-mapping.xml
  3228. 2015-02-26 10:22:56,641 INFO solr.SolrMappingReader - source: content dest: content
  3229. 2015-02-26 10:22:56,641 INFO solr.SolrMappingReader - source: title dest: title
  3230. 2015-02-26 10:22:56,641 INFO solr.SolrMappingReader - source: host dest: host
  3231. 2015-02-26 10:22:56,641 INFO solr.SolrMappingReader - source: batchId dest: batchId
  3232. 2015-02-26 10:22:56,641 INFO solr.SolrMappingReader - source: boost dest: boost
  3233. 2015-02-26 10:22:56,641 INFO solr.SolrMappingReader - source: digest dest: digest
  3234. 2015-02-26 10:22:56,641 INFO solr.SolrMappingReader - source: tstamp dest: tstamp
  3235. 2015-02-26 10:22:56,645 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local152972145_0001.xml, instantiating a new object cache
  3236. 2015-02-26 10:22:56,645 INFO basic.BasicIndexingFilter - Maximum title length for indexing set to: 100
  3237. 2015-02-26 10:22:56,645 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter
  3238. 2015-02-26 10:22:56,645 INFO anchor.AnchorIndexingFilter - Anchor deduplication is: off
  3239. 2015-02-26 10:22:56,645 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter
  3240. 2015-02-26 10:22:56,645 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.metadata.MetadataIndexer
  3241. 2015-02-26 10:22:56,648 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3242. 2015-02-26 10:22:56,648 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3243. 2015-02-26 10:22:56,648 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3244. 2015-02-26 10:22:56,648 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3245. 2015-02-26 10:22:56,648 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3246. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3247. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3248. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - Located column family: 'p'
  3249. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3250. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - Located column family: 'f'
  3251. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3252. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - Located column family: 'sc'
  3253. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3254. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - Located super column family
  3255. 2015-02-26 10:22:56,649 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3256. 2015-02-26 10:22:56,651 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3257. 2015-02-26 10:22:56,651 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3258. 2015-02-26 10:22:56,651 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3259. 2015-02-26 10:22:56,731 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  3260. 2015-02-26 10:22:56,781 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  3261. 2015-02-26 10:22:56,786 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  3262. 2015-02-26 10:22:56,806 DEBUG store.CassandraStore - close
  3263. 2015-02-26 10:22:56,807 INFO mapred.Task - Task:attempt_local152972145_0001_m_000000_0 is done. And is in the process of commiting
  3264. 2015-02-26 10:22:56,811 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3265. 2015-02-26 10:22:56,811 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3266. 2015-02-26 10:22:56,811 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3267. 2015-02-26 10:22:56,811 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3268. 2015-02-26 10:22:56,811 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3269. 2015-02-26 10:22:56,811 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3270. 2015-02-26 10:22:56,811 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3271. 2015-02-26 10:22:56,812 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  3272. 2015-02-26 10:22:56,813 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3273. 2015-02-26 10:22:56,813 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3274. 2015-02-26 10:22:56,813 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3275. 2015-02-26 10:22:56,813 INFO mapred.LocalJobRunner -
  3276. 2015-02-26 10:22:56,814 INFO mapred.Task - Task 'attempt_local152972145_0001_m_000000_0' done.
  3277. 2015-02-26 10:22:56,814 INFO mapred.LocalJobRunner - Finishing task: attempt_local152972145_0001_m_000000_0
  3278. 2015-02-26 10:22:56,814 INFO mapred.LocalJobRunner - Map task executor complete.
  3279. 2015-02-26 10:22:56,815 WARN mapred.FileOutputCommitter - Output path is null in cleanup
  3280. 2015-02-26 10:22:57,488 INFO mapred.JobClient - map 100% reduce 0%
  3281. 2015-02-26 10:22:57,489 INFO mapred.JobClient - Job complete: job_local152972145_0001
  3282. 2015-02-26 10:22:57,489 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  3283. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  3284. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3285. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding BYTES_READ
  3286. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3287. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3288. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3289. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3290. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3291. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3292. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3293. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3294. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3295. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3296. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3297. 2015-02-26 10:22:57,490 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3298. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  3299. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  3300. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3301. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3302. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3303. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3304. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding BYTES_READ
  3305. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3306. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3307. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3308. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3309. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3310. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3311. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3312. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3313. 2015-02-26 10:22:57,491 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3314. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Counters: 12
  3315. 2015-02-26 10:22:57,492 INFO mapred.JobClient - File Output Format Counters
  3316. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Bytes Written=0
  3317. 2015-02-26 10:22:57,492 INFO mapred.JobClient - File Input Format Counters
  3318. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Bytes Read=0
  3319. 2015-02-26 10:22:57,492 INFO mapred.JobClient - FileSystemCounters
  3320. 2015-02-26 10:22:57,492 INFO mapred.JobClient - FILE_BYTES_READ=608635
  3321. 2015-02-26 10:22:57,492 INFO mapred.JobClient - FILE_BYTES_WRITTEN=695348
  3322. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Map-Reduce Framework
  3323. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Map input records=1
  3324. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Physical memory (bytes) snapshot=0
  3325. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Spilled Records=0
  3326. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Total committed heap usage (bytes)=758120448
  3327. 2015-02-26 10:22:57,492 INFO mapred.JobClient - CPU time spent (ms)=0
  3328. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Virtual memory (bytes) snapshot=0
  3329. 2015-02-26 10:22:57,492 INFO mapred.JobClient - SPLIT_RAW_BYTES=740
  3330. 2015-02-26 10:22:57,492 INFO mapred.JobClient - Map output records=0
  3331. 2015-02-26 10:22:57,493 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  3332. 2015-02-26 10:22:57,493 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  3333. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3334. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding BYTES_READ
  3335. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3336. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3337. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3338. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3339. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3340. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3341. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3342. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3343. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3344. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3345. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3346. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3347. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  3348. 2015-02-26 10:22:57,494 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  3349. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3350. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3351. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3352. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3353. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding BYTES_READ
  3354. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3355. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3356. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3357. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3358. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3359. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3360. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3361. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3362. 2015-02-26 10:22:57,495 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3363. 2015-02-26 10:22:57,496 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, nutch-default.xml, nutch-site.xml, instantiating a new object cache
  3364. 2015-02-26 10:22:57,496 INFO indexer.IndexWriters - Adding org.apache.nutch.indexwriter.solr.SolrIndexWriter
  3365. 2015-02-26 10:22:57,496 INFO indexer.IndexingJob - Active IndexWriters :
  3366. SOLRIndexWriter
  3367. solr.server.url : URL of the SOLR instance (mandatory)
  3368. solr.commit.size : buffer size when sending to SOLR (default 1000)
  3369. solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
  3370. solr.auth : use authentication (default false)
  3371. solr.auth.username : username for authentication
  3372. solr.auth.password : password for authentication
  3373.  
  3374.  
  3375. 2015-02-26 10:22:57,496 INFO conf.Configuration - found resource solrindex-mapping.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/solrindex-mapping.xml
  3376. 2015-02-26 10:22:57,499 INFO solr.SolrMappingReader - source: content dest: content
  3377. 2015-02-26 10:22:57,499 INFO solr.SolrMappingReader - source: title dest: title
  3378. 2015-02-26 10:22:57,499 INFO solr.SolrMappingReader - source: host dest: host
  3379. 2015-02-26 10:22:57,499 INFO solr.SolrMappingReader - source: batchId dest: batchId
  3380. 2015-02-26 10:22:57,499 INFO solr.SolrMappingReader - source: boost dest: boost
  3381. 2015-02-26 10:22:57,499 INFO solr.SolrMappingReader - source: digest dest: digest
  3382. 2015-02-26 10:22:57,499 INFO solr.SolrMappingReader - source: tstamp dest: tstamp
  3383. 2015-02-26 10:22:57,708 DEBUG conn.BasicClientConnectionManager - Get connection for route {}->http://mylocalhost:8983
  3384. 2015-02-26 10:22:57,728 DEBUG conn.DefaultClientConnectionOperator - Connecting to mylocalhost:8983
  3385. 2015-02-26 10:22:57,754 DEBUG protocol.RequestAddCookies - CookieSpec selected: best-match
  3386. 2015-02-26 10:22:57,771 DEBUG protocol.RequestAuthCache - Auth cache not set in the context
  3387. 2015-02-26 10:22:57,771 DEBUG protocol.RequestTargetAuthentication - Target auth state: UNCHALLENGED
  3388. 2015-02-26 10:22:57,772 DEBUG protocol.RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
  3389. 2015-02-26 10:22:57,772 DEBUG client.DefaultHttpClient - Attempt 1 to execute request
  3390. 2015-02-26 10:22:57,772 DEBUG conn.DefaultClientConnection - Sending request: POST /solr/nutch_crawler/update HTTP/1.1
  3391. 2015-02-26 10:22:57,774 DEBUG http.wire - >> "POST /solr/nutch_crawler/update HTTP/1.1[\r][\n]"
  3392. 2015-02-26 10:22:57,775 DEBUG http.wire - >> "Content-Charset: UTF-8[\r][\n]"
  3393. 2015-02-26 10:22:57,775 DEBUG http.wire - >> "Content-Type: application/x-www-form-urlencoded; charset=UTF-8[\r][\n]"
  3394. 2015-02-26 10:22:57,775 DEBUG http.wire - >> "User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0[\r][\n]"
  3395. 2015-02-26 10:22:57,775 DEBUG http.wire - >> "Content-Length: 67[\r][\n]"
  3396. 2015-02-26 10:22:57,776 DEBUG http.wire - >> "Host: mylocalhost:8983[\r][\n]"
  3397. 2015-02-26 10:22:57,776 DEBUG http.wire - >> "Connection: Keep-Alive[\r][\n]"
  3398. 2015-02-26 10:22:57,776 DEBUG http.wire - >> "[\r][\n]"
  3399. 2015-02-26 10:22:57,776 DEBUG http.headers - >> POST /solr/nutch_crawler/update HTTP/1.1
  3400. 2015-02-26 10:22:57,776 DEBUG http.headers - >> Content-Charset: UTF-8
  3401. 2015-02-26 10:22:57,776 DEBUG http.headers - >> Content-Type: application/x-www-form-urlencoded; charset=UTF-8
  3402. 2015-02-26 10:22:57,776 DEBUG http.headers - >> User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0
  3403. 2015-02-26 10:22:57,776 DEBUG http.headers - >> Content-Length: 67
  3404. 2015-02-26 10:22:57,776 DEBUG http.headers - >> Host: mylocalhost:8983
  3405. 2015-02-26 10:22:57,776 DEBUG http.headers - >> Connection: Keep-Alive
  3406. 2015-02-26 10:22:57,776 DEBUG http.wire - >> "commit=true&softCommit=false&waitSearcher=true&wt=javabin&version=2"
  3407. 2015-02-26 10:22:57,784 DEBUG http.wire - << "HTTP/1.1 200 OK[\r][\n]"
  3408. 2015-02-26 10:22:57,787 DEBUG http.wire - << "Content-Type: application/octet-stream[\r][\n]"
  3409. 2015-02-26 10:22:57,788 DEBUG http.wire - << "Content-Length: 40[\r][\n]"
  3410. 2015-02-26 10:22:57,788 DEBUG http.wire - << "[\r][\n]"
  3411. 2015-02-26 10:22:57,789 DEBUG conn.DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
  3412. 2015-02-26 10:22:57,789 DEBUG http.headers - << HTTP/1.1 200 OK
  3413. 2015-02-26 10:22:57,789 DEBUG http.headers - << Content-Type: application/octet-stream
  3414. 2015-02-26 10:22:57,789 DEBUG http.headers - << Content-Length: 40
  3415. 2015-02-26 10:22:57,794 DEBUG client.DefaultHttpClient - Connection can be kept alive indefinitely
  3416. 2015-02-26 10:22:57,806 DEBUG http.wire - << "[0x2][0xa1][0xe0].responseHeader[0xa2][0xe0]&status[0x6][0x0][0x0][0x0][0x0][0xe0]%QTimeD"
  3417. 2015-02-26 10:22:57,808 DEBUG conn.BasicClientConnectionManager - Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl@210c2ed8
  3418. 2015-02-26 10:22:57,808 DEBUG conn.BasicClientConnectionManager - Connection can be kept alive indefinitely
  3419. 2015-02-26 10:22:57,808 INFO solr.SolrIndexWriter - Total 0 document is added.
  3420. 2015-02-26 10:22:57,808 INFO indexer.IndexingJob - IndexingJob: done.
  3421. 2015-02-26 10:22:57,810 DEBUG fs.FileSystem - Starting clear of FileSystem cache with 1 elements.
  3422. 2015-02-26 10:22:57,810 DEBUG fs.FileSystem - Removing filesystem for file:///
  3423. 2015-02-26 10:22:57,810 DEBUG fs.FileSystem - Removing filesystem for file:///
  3424. 2015-02-26 10:22:57,810 DEBUG fs.FileSystem - Done clearing cache
  3425. 2015-02-26 10:22:59,201 INFO solr.SolrDeleteDuplicates - SolrDeleteDuplicates: starting...
  3426. 2015-02-26 10:22:59,202 INFO solr.SolrDeleteDuplicates - SolrDeleteDuplicates: Solr url: http://mylocalhost:8983/solr/nutch_crawler/
  3427. 2015-02-26 10:22:59,392 DEBUG security.Groups - Creating new Groups object
  3428. 2015-02-26 10:22:59,400 DEBUG security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
  3429. 2015-02-26 10:22:59,463 DEBUG security.UserGroupInformation - hadoop login
  3430. 2015-02-26 10:22:59,464 DEBUG security.UserGroupInformation - hadoop login commit
  3431. 2015-02-26 10:22:59,470 DEBUG security.UserGroupInformation - using local user:UnixPrincipal: katonj
  3432. 2015-02-26 10:22:59,472 DEBUG security.UserGroupInformation - UGI loginUser:katonj
  3433. 2015-02-26 10:22:59,478 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapreduce.Job.connect(Job.java:561)
  3434. 2015-02-26 10:22:59,517 DEBUG fs.FileSystem - Creating filesystem for file:///
  3435. 2015-02-26 10:22:59,538 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  3436. 2015-02-26 10:22:59,548 DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
  3437. 2015-02-26 10:22:59,549 DEBUG util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
  3438. 2015-02-26 10:22:59,549 DEBUG util.NativeCodeLoader - java.library.path=/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/lib/native/Linux-amd64-64
  3439. 2015-02-26 10:22:59,549 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  3440. 2015-02-26 10:22:59,557 DEBUG mapred.JobClient - adding the following namenodes' delegation tokens:null
  3441. 2015-02-26 10:22:59,557 DEBUG mapred.JobClient - default FileSystem: file:///
  3442. 2015-02-26 10:22:59,564 WARN mapred.JobClient - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
  3443. 2015-02-26 10:22:59,574 DEBUG mapred.JobClient - Creating splits at file:/tmp/hadoop-katonj/mapred/staging/katonj1608299630/.staging/job_local1608299630_0001
  3444. 2015-02-26 10:22:59,908 DEBUG conn.BasicClientConnectionManager - Get connection for route {}->http://mylocalhost:8983
  3445. 2015-02-26 10:22:59,930 DEBUG conn.DefaultClientConnectionOperator - Connecting to mylocalhost:8983
  3446. 2015-02-26 10:22:59,968 DEBUG protocol.RequestAddCookies - CookieSpec selected: best-match
  3447. 2015-02-26 10:22:59,989 DEBUG protocol.RequestAuthCache - Auth cache not set in the context
  3448. 2015-02-26 10:22:59,989 DEBUG protocol.RequestTargetAuthentication - Target auth state: UNCHALLENGED
  3449. 2015-02-26 10:22:59,990 DEBUG protocol.RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
  3450. 2015-02-26 10:22:59,990 DEBUG client.DefaultHttpClient - Attempt 1 to execute request
  3451. 2015-02-26 10:22:59,990 DEBUG conn.DefaultClientConnection - Sending request: GET /solr/nutch_crawler/select?q=id%3A%5B*+TO+*%5D&fl=id&rows=1&wt=javabin&version=2 HTTP/1.1
  3452. 2015-02-26 10:22:59,991 DEBUG http.wire - >> "GET /solr/nutch_crawler/select?q=id%3A%5B*+TO+*%5D&fl=id&rows=1&wt=javabin&version=2 HTTP/1.1[\r][\n]"
  3453. 2015-02-26 10:22:59,993 DEBUG http.wire - >> "User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0[\r][\n]"
  3454. 2015-02-26 10:22:59,993 DEBUG http.wire - >> "Host: mylocalhost:8983[\r][\n]"
  3455. 2015-02-26 10:22:59,993 DEBUG http.wire - >> "Connection: Keep-Alive[\r][\n]"
  3456. 2015-02-26 10:22:59,993 DEBUG http.wire - >> "[\r][\n]"
  3457. 2015-02-26 10:22:59,993 DEBUG http.headers - >> GET /solr/nutch_crawler/select?q=id%3A%5B*+TO+*%5D&fl=id&rows=1&wt=javabin&version=2 HTTP/1.1
  3458. 2015-02-26 10:22:59,993 DEBUG http.headers - >> User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0
  3459. 2015-02-26 10:22:59,993 DEBUG http.headers - >> Host: mylocalhost:8983
  3460. 2015-02-26 10:22:59,993 DEBUG http.headers - >> Connection: Keep-Alive
  3461. 2015-02-26 10:22:59,998 DEBUG http.wire - << "HTTP/1.1 200 OK[\r][\n]"
  3462. 2015-02-26 10:23:00,001 DEBUG http.wire - << "Content-Type: application/octet-stream[\r][\n]"
  3463. 2015-02-26 10:23:00,001 DEBUG http.wire - << "Content-Length: 118[\r][\n]"
  3464. 2015-02-26 10:23:00,001 DEBUG http.wire - << "[\r][\n]"
  3465. 2015-02-26 10:23:00,002 DEBUG conn.DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
  3466. 2015-02-26 10:23:00,002 DEBUG http.headers - << HTTP/1.1 200 OK
  3467. 2015-02-26 10:23:00,002 DEBUG http.headers - << Content-Type: application/octet-stream
  3468. 2015-02-26 10:23:00,002 DEBUG http.headers - << Content-Length: 118
  3469. 2015-02-26 10:23:00,006 DEBUG client.DefaultHttpClient - Connection can be kept alive indefinitely
  3470. 2015-02-26 10:23:00,019 DEBUG http.wire - << "[0x2][0xa2][0xe0].responseHeader[0xa3][0xe0]&status[0x6][0x0][0x0][0x0][0x0][0xe0]%QTimeC[0xe0]&params[0xa5][0xe0]"fl"id[0xe0]!q+id:[* TO *][0xe0]"wt'javabin[0xe0]'version!2[0xe0]$rows!1[0xe0](response[0xc][0x83]``[0x0][0x80]"
  3471. 2015-02-26 10:23:00,023 DEBUG conn.BasicClientConnectionManager - Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl@4089314f
  3472. 2015-02-26 10:23:00,023 DEBUG conn.BasicClientConnectionManager - Connection can be kept alive indefinitely
  3473. 2015-02-26 10:23:00,217 DEBUG mapred.JobClient - Printing tokens for job: job_local1608299630_0001
  3474. 2015-02-26 10:23:00,232 DEBUG mapred.TaskRunner - Fully deleting contents of /tmp/hadoop-katonj/mapred/local/localRunner
  3475. 2015-02-26 10:23:00,324 INFO mapred.JobClient - Running job: job_local1608299630_0001
  3476. 2015-02-26 10:23:00,374 DEBUG mapred.LocalJobRunner - Starting thread pool executor.
  3477. 2015-02-26 10:23:00,374 DEBUG mapred.LocalJobRunner - Max local threads: 1
  3478. 2015-02-26 10:23:00,374 DEBUG mapred.LocalJobRunner - Map tasks to process: 2
  3479. 2015-02-26 10:23:00,379 INFO mapred.LocalJobRunner - Waiting for map tasks
  3480. 2015-02-26 10:23:00,382 INFO mapred.LocalJobRunner - Starting task: attempt_local1608299630_0001_m_000000_0
  3481. 2015-02-26 10:23:00,392 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  3482. 2015-02-26 10:23:00,400 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3483. 2015-02-26 10:23:00,400 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3484. 2015-02-26 10:23:00,412 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local1608299630_0001/attempt_local1608299630_0001_m_000000_0
  3485. 2015-02-26 10:23:00,415 DEBUG mapred.Task - using new api for output committer
  3486. 2015-02-26 10:23:00,426 INFO util.ProcessTree - setsid exited with exit code 0
  3487. 2015-02-26 10:23:00,431 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4c50bc82
  3488. 2015-02-26 10:23:00,437 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3489. 2015-02-26 10:23:00,437 INFO mapred.MapTask - Processing split: org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputSplit@7ac3bad1
  3490. 2015-02-26 10:23:00,438 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3491. 2015-02-26 10:23:00,441 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3492. 2015-02-26 10:23:00,442 DEBUG mapred.Counters - Adding BYTES_READ
  3493. 2015-02-26 10:23:00,455 DEBUG conn.BasicClientConnectionManager - Get connection for route {}->http://mylocalhost:8983
  3494. 2015-02-26 10:23:00,455 DEBUG conn.DefaultClientConnectionOperator - Connecting to mylocalhost:8983
  3495. 2015-02-26 10:23:00,456 DEBUG protocol.RequestAddCookies - CookieSpec selected: best-match
  3496. 2015-02-26 10:23:00,456 DEBUG protocol.RequestAuthCache - Auth cache not set in the context
  3497. 2015-02-26 10:23:00,456 DEBUG protocol.RequestTargetAuthentication - Target auth state: UNCHALLENGED
  3498. 2015-02-26 10:23:00,456 DEBUG protocol.RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
  3499. 2015-02-26 10:23:00,457 DEBUG client.DefaultHttpClient - Attempt 1 to execute request
  3500. 2015-02-26 10:23:00,457 DEBUG conn.DefaultClientConnection - Sending request: GET /solr/nutch_crawler/select?q=id%3A%5B*+TO+*%5D&fl=id%2Cboost%2Ctstamp%2Cdigest&start=0&rows=0&wt=javabin&version=2 HTTP/1.1
  3501. 2015-02-26 10:23:00,457 DEBUG http.wire - >> "GET /solr/nutch_crawler/select?q=id%3A%5B*+TO+*%5D&fl=id%2Cboost%2Ctstamp%2Cdigest&start=0&rows=0&wt=javabin&version=2 HTTP/1.1[\r][\n]"
  3502. 2015-02-26 10:23:00,457 DEBUG http.wire - >> "User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0[\r][\n]"
  3503. 2015-02-26 10:23:00,457 DEBUG http.wire - >> "Host: mylocalhost:8983[\r][\n]"
  3504. 2015-02-26 10:23:00,457 DEBUG http.wire - >> "Connection: Keep-Alive[\r][\n]"
  3505. 2015-02-26 10:23:00,457 DEBUG http.wire - >> "[\r][\n]"
  3506. 2015-02-26 10:23:00,457 DEBUG http.headers - >> GET /solr/nutch_crawler/select?q=id%3A%5B*+TO+*%5D&fl=id%2Cboost%2Ctstamp%2Cdigest&start=0&rows=0&wt=javabin&version=2 HTTP/1.1
  3507. 2015-02-26 10:23:00,457 DEBUG http.headers - >> User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0
  3508. 2015-02-26 10:23:00,457 DEBUG http.headers - >> Host: mylocalhost:8983
  3509. 2015-02-26 10:23:00,457 DEBUG http.headers - >> Connection: Keep-Alive
  3510. 2015-02-26 10:23:00,461 DEBUG http.wire - << "HTTP/1.1 200 OK[\r][\n]"
  3511. 2015-02-26 10:23:00,461 DEBUG http.wire - << "Content-Type: application/octet-stream[\r][\n]"
  3512. 2015-02-26 10:23:00,461 DEBUG http.wire - << "Content-Length: 147[\r][\n]"
  3513. 2015-02-26 10:23:00,461 DEBUG http.wire - << "[\r][\n]"
  3514. 2015-02-26 10:23:00,461 DEBUG conn.DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
  3515. 2015-02-26 10:23:00,461 DEBUG http.headers - << HTTP/1.1 200 OK
  3516. 2015-02-26 10:23:00,461 DEBUG http.headers - << Content-Type: application/octet-stream
  3517. 2015-02-26 10:23:00,461 DEBUG http.headers - << Content-Length: 147
  3518. 2015-02-26 10:23:00,461 DEBUG client.DefaultHttpClient - Connection can be kept alive indefinitely
  3519. 2015-02-26 10:23:00,461 DEBUG http.wire - << "[0x2][0xa2][0xe0].responseHeader[0xa3][0xe0]&status[0x6][0x0][0x0][0x0][0x0][0xe0]%QTimeA[0xe0]&params[0xa6][0xe0]"fl6id,boost,tstamp,digest[0xe0]%start!0[0xe0]!q+id:[* TO *][0xe0]"wt'javabin[0xe0]'version!2[0xe0]$rows!0[0xe0](response[0xc][0x83]``[0x0][0x80]"
  3520. 2015-02-26 10:23:00,462 DEBUG conn.BasicClientConnectionManager - Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl@69317d48
  3521. 2015-02-26 10:23:00,462 DEBUG conn.BasicClientConnectionManager - Connection can be kept alive indefinitely
  3522. 2015-02-26 10:23:00,473 INFO mapred.MapTask - io.sort.mb = 100
  3523. 2015-02-26 10:23:00,618 INFO mapred.MapTask - data buffer = 79691776/99614720
  3524. 2015-02-26 10:23:00,618 INFO mapred.MapTask - record buffer = 262144/327680
  3525. 2015-02-26 10:23:00,618 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  3526. 2015-02-26 10:23:00,618 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3527. 2015-02-26 10:23:00,618 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  3528. 2015-02-26 10:23:00,618 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  3529. 2015-02-26 10:23:00,618 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  3530. 2015-02-26 10:23:00,625 DEBUG conn.DefaultClientConnection - Connection 0.0.0.0:52999<->131.101.103.242:8983 closed
  3531. 2015-02-26 10:23:00,629 DEBUG conn.DefaultClientConnection - Connection 0.0.0.0:52998<->131.101.103.242:8983 closed
  3532. 2015-02-26 10:23:00,629 INFO mapred.MapTask - Starting flush of map output
  3533. 2015-02-26 10:23:00,645 INFO compress.CodecPool - Got brand-new compressor
  3534. 2015-02-26 10:23:00,649 INFO mapred.Task - Task:attempt_local1608299630_0001_m_000000_0 is done. And is in the process of commiting
  3535. 2015-02-26 10:23:00,652 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3536. 2015-02-26 10:23:00,652 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3537. 2015-02-26 10:23:00,653 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3538. 2015-02-26 10:23:00,653 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3539. 2015-02-26 10:23:00,653 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3540. 2015-02-26 10:23:00,653 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3541. 2015-02-26 10:23:00,653 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3542. 2015-02-26 10:23:00,655 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3543. 2015-02-26 10:23:00,655 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3544. 2015-02-26 10:23:00,655 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3545. 2015-02-26 10:23:00,656 INFO mapred.LocalJobRunner -
  3546. 2015-02-26 10:23:00,656 INFO mapred.Task - Task 'attempt_local1608299630_0001_m_000000_0' done.
  3547. 2015-02-26 10:23:00,656 INFO mapred.LocalJobRunner - Finishing task: attempt_local1608299630_0001_m_000000_0
  3548. 2015-02-26 10:23:00,656 INFO mapred.LocalJobRunner - Starting task: attempt_local1608299630_0001_m_000001_0
  3549. 2015-02-26 10:23:00,656 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  3550. 2015-02-26 10:23:00,657 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3551. 2015-02-26 10:23:00,657 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3552. 2015-02-26 10:23:00,657 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local1608299630_0001/attempt_local1608299630_0001_m_000001_0
  3553. 2015-02-26 10:23:00,660 DEBUG mapred.Task - using new api for output committer
  3554. 2015-02-26 10:23:00,660 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@294d5a82
  3555. 2015-02-26 10:23:00,663 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3556. 2015-02-26 10:23:00,663 INFO mapred.MapTask - Processing split: org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputSplit@366e740a
  3557. 2015-02-26 10:23:00,663 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3558. 2015-02-26 10:23:00,663 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3559. 2015-02-26 10:23:00,663 DEBUG mapred.Counters - Adding BYTES_READ
  3560. 2015-02-26 10:23:00,674 DEBUG conn.BasicClientConnectionManager - Get connection for route {}->http://mylocalhost:8983
  3561. 2015-02-26 10:23:00,674 DEBUG conn.DefaultClientConnectionOperator - Connecting to mylocalhost:8983
  3562. 2015-02-26 10:23:00,675 DEBUG protocol.RequestAddCookies - CookieSpec selected: best-match
  3563. 2015-02-26 10:23:00,675 DEBUG protocol.RequestAuthCache - Auth cache not set in the context
  3564. 2015-02-26 10:23:00,675 DEBUG protocol.RequestTargetAuthentication - Target auth state: UNCHALLENGED
  3565. 2015-02-26 10:23:00,675 DEBUG protocol.RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
  3566. 2015-02-26 10:23:00,675 DEBUG client.DefaultHttpClient - Attempt 1 to execute request
  3567. 2015-02-26 10:23:00,675 DEBUG conn.DefaultClientConnection - Sending request: GET /solr/nutch_crawler/select?q=id%3A%5B*+TO+*%5D&fl=id%2Cboost%2Ctstamp%2Cdigest&start=0&rows=0&wt=javabin&version=2 HTTP/1.1
  3568. 2015-02-26 10:23:00,675 DEBUG http.wire - >> "GET /solr/nutch_crawler/select?q=id%3A%5B*+TO+*%5D&fl=id%2Cboost%2Ctstamp%2Cdigest&start=0&rows=0&wt=javabin&version=2 HTTP/1.1[\r][\n]"
  3569. 2015-02-26 10:23:00,675 DEBUG http.wire - >> "User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0[\r][\n]"
  3570. 2015-02-26 10:23:00,676 DEBUG http.wire - >> "Host: mylocalhost:8983[\r][\n]"
  3571. 2015-02-26 10:23:00,676 DEBUG http.wire - >> "Connection: Keep-Alive[\r][\n]"
  3572. 2015-02-26 10:23:00,676 DEBUG http.wire - >> "[\r][\n]"
  3573. 2015-02-26 10:23:00,676 DEBUG http.headers - >> GET /solr/nutch_crawler/select?q=id%3A%5B*+TO+*%5D&fl=id%2Cboost%2Ctstamp%2Cdigest&start=0&rows=0&wt=javabin&version=2 HTTP/1.1
  3574. 2015-02-26 10:23:00,676 DEBUG http.headers - >> User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0
  3575. 2015-02-26 10:23:00,676 DEBUG http.headers - >> Host: mylocalhost:8983
  3576. 2015-02-26 10:23:00,676 DEBUG http.headers - >> Connection: Keep-Alive
  3577. 2015-02-26 10:23:00,678 DEBUG http.wire - << "HTTP/1.1 200 OK[\r][\n]"
  3578. 2015-02-26 10:23:00,679 DEBUG http.wire - << "Content-Type: application/octet-stream[\r][\n]"
  3579. 2015-02-26 10:23:00,679 DEBUG http.wire - << "Content-Length: 147[\r][\n]"
  3580. 2015-02-26 10:23:00,679 DEBUG http.wire - << "[\r][\n]"
  3581. 2015-02-26 10:23:00,679 DEBUG conn.DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
  3582. 2015-02-26 10:23:00,679 DEBUG http.headers - << HTTP/1.1 200 OK
  3583. 2015-02-26 10:23:00,679 DEBUG http.headers - << Content-Type: application/octet-stream
  3584. 2015-02-26 10:23:00,679 DEBUG http.headers - << Content-Length: 147
  3585. 2015-02-26 10:23:00,679 DEBUG client.DefaultHttpClient - Connection can be kept alive indefinitely
  3586. 2015-02-26 10:23:00,679 DEBUG http.wire - << "[0x2][0xa2][0xe0].responseHeader[0xa3][0xe0]&status[0x6][0x0][0x0][0x0][0x0][0xe0]%QTimeA[0xe0]&params[0xa6][0xe0]"fl6id,boost,tstamp,digest[0xe0]%start!0[0xe0]!q+id:[* TO *][0xe0]"wt'javabin[0xe0]'version!2[0xe0]$rows!0[0xe0](response[0xc][0x83]``[0x0][0x80]"
  3587. 2015-02-26 10:23:00,679 DEBUG conn.BasicClientConnectionManager - Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl@35de06ec
  3588. 2015-02-26 10:23:00,680 DEBUG conn.BasicClientConnectionManager - Connection can be kept alive indefinitely
  3589. 2015-02-26 10:23:00,680 INFO mapred.MapTask - io.sort.mb = 100
  3590. 2015-02-26 10:23:00,706 INFO mapred.MapTask - data buffer = 79691776/99614720
  3591. 2015-02-26 10:23:00,706 INFO mapred.MapTask - record buffer = 262144/327680
  3592. 2015-02-26 10:23:00,706 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  3593. 2015-02-26 10:23:00,706 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3594. 2015-02-26 10:23:00,706 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  3595. 2015-02-26 10:23:00,706 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  3596. 2015-02-26 10:23:00,706 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  3597. 2015-02-26 10:23:00,708 INFO mapred.MapTask - Starting flush of map output
  3598. 2015-02-26 10:23:00,711 DEBUG compress.CodecPool - Got recycled compressor
  3599. 2015-02-26 10:23:00,712 INFO mapred.Task - Task:attempt_local1608299630_0001_m_000001_0 is done. And is in the process of commiting
  3600. 2015-02-26 10:23:00,713 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3601. 2015-02-26 10:23:00,713 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3602. 2015-02-26 10:23:00,713 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3603. 2015-02-26 10:23:00,713 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3604. 2015-02-26 10:23:00,713 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3605. 2015-02-26 10:23:00,713 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3606. 2015-02-26 10:23:00,713 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3607. 2015-02-26 10:23:00,714 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3608. 2015-02-26 10:23:00,715 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3609. 2015-02-26 10:23:00,715 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3610. 2015-02-26 10:23:00,715 INFO mapred.LocalJobRunner -
  3611. 2015-02-26 10:23:00,715 INFO mapred.Task - Task 'attempt_local1608299630_0001_m_000001_0' done.
  3612. 2015-02-26 10:23:00,715 INFO mapred.LocalJobRunner - Finishing task: attempt_local1608299630_0001_m_000001_0
  3613. 2015-02-26 10:23:00,716 INFO mapred.LocalJobRunner - Map task executor complete.
  3614. 2015-02-26 10:23:00,722 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  3615. 2015-02-26 10:23:00,722 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3616. 2015-02-26 10:23:00,722 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3617. 2015-02-26 10:23:00,722 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  3618. 2015-02-26 10:23:00,722 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  3619. 2015-02-26 10:23:00,722 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  3620. 2015-02-26 10:23:00,722 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  3621. 2015-02-26 10:23:00,722 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  3622. 2015-02-26 10:23:00,724 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local1608299630_0001/attempt_local1608299630_0001_r_000000_0
  3623. 2015-02-26 10:23:00,733 DEBUG mapred.Task - using new api for output committer
  3624. 2015-02-26 10:23:00,733 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@54d2891f
  3625. 2015-02-26 10:23:00,733 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3626. 2015-02-26 10:23:00,734 INFO mapred.LocalJobRunner -
  3627. 2015-02-26 10:23:00,741 INFO mapred.Merger - Merging 2 sorted segments
  3628. 2015-02-26 10:23:00,749 INFO compress.CodecPool - Got brand-new decompressor
  3629. 2015-02-26 10:23:00,751 DEBUG compress.CodecPool - Got recycled decompressor
  3630. 2015-02-26 10:23:00,751 INFO mapred.Merger - Down to the last merge-pass, with 0 segments left of total size: 0 bytes
  3631. 2015-02-26 10:23:00,752 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3632. 2015-02-26 10:23:00,752 INFO mapred.LocalJobRunner -
  3633. 2015-02-26 10:23:00,757 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  3634. 2015-02-26 10:23:00,757 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  3635. 2015-02-26 10:23:00,761 INFO mapred.Task - Task:attempt_local1608299630_0001_r_000000_0 is done. And is in the process of commiting
  3636. 2015-02-26 10:23:00,761 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3637. 2015-02-26 10:23:00,761 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3638. 2015-02-26 10:23:00,761 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3639. 2015-02-26 10:23:00,761 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3640. 2015-02-26 10:23:00,761 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3641. 2015-02-26 10:23:00,761 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3642. 2015-02-26 10:23:00,761 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3643. 2015-02-26 10:23:00,762 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  3644. 2015-02-26 10:23:00,763 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3645. 2015-02-26 10:23:00,763 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3646. 2015-02-26 10:23:00,763 INFO mapred.LocalJobRunner - reduce > reduce
  3647. 2015-02-26 10:23:00,763 INFO mapred.Task - Task 'attempt_local1608299630_0001_r_000000_0' done.
  3648. 2015-02-26 10:23:00,765 WARN mapred.FileOutputCommitter - Output path is null in cleanup
  3649. 2015-02-26 10:23:01,326 INFO mapred.JobClient - map � reduce 100%
  3650. 2015-02-26 10:23:01,328 INFO mapred.JobClient - Job complete: job_local1608299630_0001
  3651. 2015-02-26 10:23:01,328 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3652. 2015-02-26 10:23:01,328 DEBUG mapred.Counters - Adding BYTES_READ
  3653. 2015-02-26 10:23:01,328 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3654. 2015-02-26 10:23:01,328 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3655. 2015-02-26 10:23:01,328 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3656. 2015-02-26 10:23:01,328 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3657. 2015-02-26 10:23:01,328 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  3658. 2015-02-26 10:23:01,328 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  3659. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3660. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3661. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3662. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  3663. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3664. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3665. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3666. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3667. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3668. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  3669. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3670. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3671. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3672. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3673. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Adding BYTES_READ
  3674. 2015-02-26 10:23:01,329 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3675. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  3676. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  3677. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3678. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3679. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3680. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  3681. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3682. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3683. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3684. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  3685. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3686. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3687. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3688. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding BYTES_READ
  3689. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  3690. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  3691. 2015-02-26 10:23:01,330 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  3692. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3693. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  3694. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  3695. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3696. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  3697. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3698. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  3699. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  3700. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  3701. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  3702. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3703. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3704. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  3705. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  3706. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  3707. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  3708. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  3709. 2015-02-26 10:23:01,331 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  3710. 2015-02-26 10:23:01,332 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  3711. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Counters: 20
  3712. 2015-02-26 10:23:01,332 INFO mapred.JobClient - File Output Format Counters
  3713. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Bytes Written=0
  3714. 2015-02-26 10:23:01,332 INFO mapred.JobClient - FileSystemCounters
  3715. 2015-02-26 10:23:01,332 INFO mapred.JobClient - FILE_BYTES_READ=947
  3716. 2015-02-26 10:23:01,332 INFO mapred.JobClient - FILE_BYTES_WRITTEN=235862
  3717. 2015-02-26 10:23:01,332 INFO mapred.JobClient - File Input Format Counters
  3718. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Bytes Read=0
  3719. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Map-Reduce Framework
  3720. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Reduce input groups=0
  3721. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Map output materialized bytes=28
  3722. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Combine output records=0
  3723. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Map input records=0
  3724. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Reduce shuffle bytes=0
  3725. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Physical memory (bytes) snapshot=0
  3726. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Reduce output records=0
  3727. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Spilled Records=0
  3728. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Map output bytes=0
  3729. 2015-02-26 10:23:01,332 INFO mapred.JobClient - CPU time spent (ms)=0
  3730. 2015-02-26 10:23:01,332 INFO mapred.JobClient - Total committed heap usage (bytes)=2274361344
  3731. 2015-02-26 10:23:01,333 INFO mapred.JobClient - Virtual memory (bytes) snapshot=0
  3732. 2015-02-26 10:23:01,333 INFO mapred.JobClient - Combine input records=0
  3733. 2015-02-26 10:23:01,333 INFO mapred.JobClient - Map output records=0
  3734. 2015-02-26 10:23:01,333 INFO mapred.JobClient - SPLIT_RAW_BYTES=148
  3735. 2015-02-26 10:23:01,333 INFO mapred.JobClient - Reduce input records=0
  3736. 2015-02-26 10:23:01,333 INFO solr.SolrDeleteDuplicates - SolrDeleteDuplicates: done.
  3737. 2015-02-26 10:23:01,334 DEBUG fs.FileSystem - Starting clear of FileSystem cache with 1 elements.
  3738. 2015-02-26 10:23:01,334 DEBUG fs.FileSystem - Removing filesystem for file:///
  3739. 2015-02-26 10:23:01,334 DEBUG fs.FileSystem - Removing filesystem for file:///
  3740. 2015-02-26 10:23:01,334 DEBUG fs.FileSystem - Done clearing cache
  3741. 2015-02-26 10:23:02,726 INFO crawl.GeneratorJob - GeneratorJob: starting at 2015-02-26 10:23:02
  3742. 2015-02-26 10:23:02,727 INFO crawl.GeneratorJob - GeneratorJob: Selecting best-scoring urls due for fetch.
  3743. 2015-02-26 10:23:02,727 INFO crawl.GeneratorJob - GeneratorJob: starting
  3744. 2015-02-26 10:23:02,727 INFO crawl.GeneratorJob - GeneratorJob: filtering: false
  3745. 2015-02-26 10:23:02,727 INFO crawl.GeneratorJob - GeneratorJob: normalizing: false
  3746. 2015-02-26 10:23:02,727 INFO crawl.GeneratorJob - GeneratorJob: topN: 50000
  3747. 2015-02-26 10:23:02,934 DEBUG security.Groups - Creating new Groups object
  3748. 2015-02-26 10:23:02,937 DEBUG security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
  3749. 2015-02-26 10:23:03,004 DEBUG security.UserGroupInformation - hadoop login
  3750. 2015-02-26 10:23:03,005 DEBUG security.UserGroupInformation - hadoop login commit
  3751. 2015-02-26 10:23:03,014 DEBUG security.UserGroupInformation - using local user:UnixPrincipal: katonj
  3752. 2015-02-26 10:23:03,018 DEBUG security.UserGroupInformation - UGI loginUser:katonj
  3753. 2015-02-26 10:23:03,094 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, nutch-default.xml, nutch-site.xml, instantiating a new object cache
  3754. 2015-02-26 10:23:03,094 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
  3755. 2015-02-26 10:23:03,095 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
  3756. 2015-02-26 10:23:03,095 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
  3757. 2015-02-26 10:23:03,706 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'webpage' in cluster 'Test Cluster' on host 'localhost:9160'.
  3758. 2015-02-26 10:23:03,706 DEBUG store.CassandraMappingManager - Located Cassandra Keyspace: 'host' in cluster 'Test Cluster' on host 'localhost:9160'.
  3759. 2015-02-26 10:23:03,706 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.WebPage' for Keyspace 'webpage'.
  3760. 2015-02-26 10:23:03,706 DEBUG store.CassandraMappingManager - Located Cassandra Mapping: keyClass: 'java.lang.String' in storage class 'org.apache.nutch.storage.Host' for Keyspace 'host'.
  3761. 2015-02-26 10:23:03,706 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3762. 2015-02-26 10:23:03,707 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3763. 2015-02-26 10:23:03,707 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3764. 2015-02-26 10:23:03,707 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3765. 2015-02-26 10:23:03,708 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3766. 2015-02-26 10:23:03,708 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3767. 2015-02-26 10:23:03,708 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3768. 2015-02-26 10:23:03,742 DEBUG store.CassandraMapping - Located column family: 'p'
  3769. 2015-02-26 10:23:03,742 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3770. 2015-02-26 10:23:03,742 DEBUG store.CassandraMapping - Located column family: 'f'
  3771. 2015-02-26 10:23:03,742 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3772. 2015-02-26 10:23:03,743 DEBUG store.CassandraMapping - Located column family: 'sc'
  3773. 2015-02-26 10:23:03,743 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3774. 2015-02-26 10:23:03,743 DEBUG store.CassandraMapping - Located super column family
  3775. 2015-02-26 10:23:03,743 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3776. 2015-02-26 10:23:03,797 INFO connection.CassandraHostRetryService - Downed Host Retry service started with queue size -1 and retry delay 10s
  3777. 2015-02-26 10:23:03,878 INFO service.JmxMonitor - Registering JMX me.prettyprint.cassandra.service_Test Cluster:ServiceType=hector,MonitorType=hector
  3778. 2015-02-26 10:23:03,906 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3779. 2015-02-26 10:23:03,915 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3780. 2015-02-26 10:23:03,915 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3781. 2015-02-26 10:23:03,916 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3782. 2015-02-26 10:23:03,916 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3783. 2015-02-26 10:23:03,916 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3784. 2015-02-26 10:23:03,917 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3785. 2015-02-26 10:23:03,917 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3786. 2015-02-26 10:23:03,917 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3787. 2015-02-26 10:23:03,918 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3788. 2015-02-26 10:23:03,918 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3789. 2015-02-26 10:23:03,918 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3790. 2015-02-26 10:23:03,919 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3791. 2015-02-26 10:23:03,919 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3792. 2015-02-26 10:23:03,919 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3793. 2015-02-26 10:23:03,920 DEBUG client.HThriftClient - Creating a new thrift connection to localhost(127.0.0.1):9160
  3794. 2015-02-26 10:23:03,920 DEBUG connection.ConcurrentHClientPool - Concurrent Host pool started with 16 active clients; max: 50 exhausted wait: 0
  3795. 2015-02-26 10:23:03,998 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3796. 2015-02-26 10:23:03,998 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3797. 2015-02-26 10:23:03,998 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3798. 2015-02-26 10:23:04,072 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3799. 2015-02-26 10:23:04,072 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3800. 2015-02-26 10:23:04,072 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3801. 2015-02-26 10:23:04,072 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3802. 2015-02-26 10:23:04,072 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3803. 2015-02-26 10:23:04,072 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3804. 2015-02-26 10:23:04,072 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3805. 2015-02-26 10:23:04,072 DEBUG store.CassandraMapping - Located column family: 'p'
  3806. 2015-02-26 10:23:04,072 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3807. 2015-02-26 10:23:04,072 DEBUG store.CassandraMapping - Located column family: 'f'
  3808. 2015-02-26 10:23:04,073 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3809. 2015-02-26 10:23:04,073 DEBUG store.CassandraMapping - Located column family: 'sc'
  3810. 2015-02-26 10:23:04,073 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3811. 2015-02-26 10:23:04,073 DEBUG store.CassandraMapping - Located super column family
  3812. 2015-02-26 10:23:04,073 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3813. 2015-02-26 10:23:04,075 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3814. 2015-02-26 10:23:04,075 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3815. 2015-02-26 10:23:04,075 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3816. 2015-02-26 10:23:04,076 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapreduce.Job.connect(Job.java:561)
  3817. 2015-02-26 10:23:04,106 DEBUG fs.FileSystem - Creating filesystem for file:///
  3818. 2015-02-26 10:23:04,123 DEBUG security.UserGroupInformation - PriviledgedAction as:katonj from:org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  3819. 2015-02-26 10:23:04,134 DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
  3820. 2015-02-26 10:23:04,134 DEBUG util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
  3821. 2015-02-26 10:23:04,134 DEBUG util.NativeCodeLoader - java.library.path=/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/lib/native/Linux-amd64-64
  3822. 2015-02-26 10:23:04,134 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  3823. 2015-02-26 10:23:04,136 DEBUG mapred.JobClient - adding the following namenodes' delegation tokens:null
  3824. 2015-02-26 10:23:04,136 DEBUG mapred.JobClient - default FileSystem: file:///
  3825. 2015-02-26 10:23:04,194 DEBUG mapred.JobClient - Creating splits at file:/tmp/hadoop-katonj/mapred/staging/katonj46914250/.staging/job_local46914250_0001
  3826. 2015-02-26 10:23:04,197 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3827. 2015-02-26 10:23:04,197 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3828. 2015-02-26 10:23:04,197 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3829. 2015-02-26 10:23:04,197 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3830. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3831. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3832. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3833. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - Located column family: 'p'
  3834. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3835. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - Located column family: 'f'
  3836. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3837. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - Located column family: 'sc'
  3838. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3839. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - Located super column family
  3840. 2015-02-26 10:23:04,198 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3841. 2015-02-26 10:23:04,199 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3842. 2015-02-26 10:23:04,200 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3843. 2015-02-26 10:23:04,200 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3844. 2015-02-26 10:23:04,331 DEBUG mapred.JobClient - Printing tokens for job: job_local46914250_0001
  3845. 2015-02-26 10:23:04,346 DEBUG mapred.TaskRunner - Fully deleting contents of /tmp/hadoop-katonj/mapred/local/localRunner
  3846. 2015-02-26 10:23:04,446 INFO mapred.JobClient - Running job: job_local46914250_0001
  3847. 2015-02-26 10:23:04,468 DEBUG mapred.LocalJobRunner - Starting thread pool executor.
  3848. 2015-02-26 10:23:04,468 DEBUG mapred.LocalJobRunner - Max local threads: 1
  3849. 2015-02-26 10:23:04,468 DEBUG mapred.LocalJobRunner - Map tasks to process: 1
  3850. 2015-02-26 10:23:04,468 INFO mapred.LocalJobRunner - Waiting for map tasks
  3851. 2015-02-26 10:23:04,472 INFO mapred.LocalJobRunner - Starting task: attempt_local46914250_0001_m_000000_0
  3852. 2015-02-26 10:23:04,489 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  3853. 2015-02-26 10:23:04,501 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  3854. 2015-02-26 10:23:04,501 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  3855. 2015-02-26 10:23:04,519 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local46914250_0001/attempt_local46914250_0001_m_000000_0
  3856. 2015-02-26 10:23:04,523 DEBUG mapred.Task - using new api for output committer
  3857. 2015-02-26 10:23:04,537 INFO util.ProcessTree - setsid exited with exit code 0
  3858. 2015-02-26 10:23:04,542 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5ad23e7d
  3859. 2015-02-26 10:23:04,548 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3860. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3861. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3862. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3863. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3864. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3865. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3866. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located column family: 'p'
  3867. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3868. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located column family: 'f'
  3869. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3870. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located column family: 'sc'
  3871. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3872. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Located super column family
  3873. 2015-02-26 10:23:04,549 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3874. 2015-02-26 10:23:04,551 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3875. 2015-02-26 10:23:04,551 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3876. 2015-02-26 10:23:04,551 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3877. 2015-02-26 10:23:04,559 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3878. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3879. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3880. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3881. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3882. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3883. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3884. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - Located column family: 'p'
  3885. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3886. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - Located column family: 'f'
  3887. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3888. 2015-02-26 10:23:04,559 DEBUG store.CassandraMapping - Located column family: 'sc'
  3889. 2015-02-26 10:23:04,560 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3890. 2015-02-26 10:23:04,560 DEBUG store.CassandraMapping - Located super column family
  3891. 2015-02-26 10:23:04,560 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3892. 2015-02-26 10:23:04,562 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3893. 2015-02-26 10:23:04,562 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3894. 2015-02-26 10:23:04,562 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3895. 2015-02-26 10:23:04,564 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  3896. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  3897. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  3898. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  3899. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  3900. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  3901. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  3902. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located column family: 'p'
  3903. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3904. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located column family: 'f'
  3905. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3906. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located column family: 'sc'
  3907. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  3908. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Located super column family
  3909. 2015-02-26 10:23:04,565 DEBUG store.CassandraMapping - Added super column family: 'sc'
  3910. 2015-02-26 10:23:04,567 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  3911. 2015-02-26 10:23:04,567 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  3912. 2015-02-26 10:23:04,567 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  3913. 2015-02-26 10:23:04,567 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  3914. 2015-02-26 10:23:04,568 INFO mapred.MapTask - Processing split: org.apache.gora.mapreduce.GoraInputSplit@7a5837bb
  3915. 2015-02-26 10:23:04,568 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  3916. 2015-02-26 10:23:04,572 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  3917. 2015-02-26 10:23:04,572 DEBUG mapred.Counters - Adding BYTES_READ
  3918. 2015-02-26 10:23:04,574 INFO mapreduce.GoraRecordReader - gora.buffer.read.limit = 10000
  3919. 2015-02-26 10:23:04,584 INFO mapred.MapTask - io.sort.mb = 100
  3920. 2015-02-26 10:23:04,637 INFO mapred.MapTask - data buffer = 79691776/99614720
  3921. 2015-02-26 10:23:04,637 INFO mapred.MapTask - record buffer = 262144/327680
  3922. 2015-02-26 10:23:04,642 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  3923. 2015-02-26 10:23:04,642 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  3924. 2015-02-26 10:23:04,642 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  3925. 2015-02-26 10:23:04,642 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  3926. 2015-02-26 10:23:04,642 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  3927. 2015-02-26 10:23:04,661 INFO plugin.PluginRepository - Plugins: looking in: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins
  3928. 2015-02-26 10:23:04,662 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-xml/plugin.xml
  3929. 2015-02-26 10:23:04,667 DEBUG plugin.PluginRepository - plugin: id=lib-xml name=XML Libraries version=1.0 provider=org.apache.nutch.xmlclass=null
  3930. 2015-02-26 10:23:04,668 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-sftp/plugin.xml
  3931. 2015-02-26 10:23:04,671 DEBUG plugin.PluginRepository - plugin: id=protocol-sftp name=Sftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  3932. 2015-02-26 10:23:04,671 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.sftp.Sftp
  3933. 2015-02-26 10:23:04,673 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-automaton/plugin.xml
  3934. 2015-02-26 10:23:04,675 DEBUG plugin.PluginRepository - plugin: id=urlfilter-automaton name=Automaton URL Filter version=1.0.0 provider=nutch.orgclass=null
  3935. 2015-02-26 10:23:04,675 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.automaton.AutomatonURLFilter
  3936. 2015-02-26 10:23:04,676 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-tika/plugin.xml
  3937. 2015-02-26 10:23:04,679 DEBUG plugin.PluginRepository - plugin: id=parse-tika name=Tika Parser Plug-in version=1.0.0 provider=nutch.orgclass=null
  3938. 2015-02-26 10:23:04,681 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.tika.TikaParser
  3939. 2015-02-26 10:23:04,685 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-elastic/plugin.xml
  3940. 2015-02-26 10:23:04,688 DEBUG plugin.PluginRepository - plugin: id=indexer-elastic name=ElasticIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  3941. 2015-02-26 10:23:04,688 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
  3942. 2015-02-26 10:23:04,691 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/microformats-reltag/plugin.xml
  3943. 2015-02-26 10:23:04,693 DEBUG plugin.PluginRepository - plugin: id=microformats-reltag name=Rel-Tag microformat Parser/Indexer/Querier version=1.0.0 provider=nutch.orgclass=null
  3944. 2015-02-26 10:23:04,693 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.microformats.reltag.RelTagParser
  3945. 2015-02-26 10:23:04,694 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.microformats.reltag.RelTagIndexingFilter
  3946. 2015-02-26 10:23:04,694 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-html/plugin.xml
  3947. 2015-02-26 10:23:04,696 DEBUG plugin.PluginRepository - plugin: id=parse-html name=Html Parse Plug-in version=1.0.0 provider=nutch.orgclass=null
  3948. 2015-02-26 10:23:04,697 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.html.HtmlParser
  3949. 2015-02-26 10:23:04,697 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/subcollection/plugin.xml
  3950. 2015-02-26 10:23:04,700 DEBUG plugin.PluginRepository - plugin: id=subcollection name=Subcollection indexing and query filter version=1.0.0 provider=apache.orgclass=null
  3951. 2015-02-26 10:23:04,700 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.subcollection.SubcollectionIndexingFilter
  3952. 2015-02-26 10:23:04,700 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-metadata/plugin.xml
  3953. 2015-02-26 10:23:04,703 DEBUG plugin.PluginRepository - plugin: id=index-metadata name=Metadata Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  3954. 2015-02-26 10:23:04,703 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.metadata.MetadataIndexer
  3955. 2015-02-26 10:23:04,703 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-httpclient/plugin.xml
  3956. 2015-02-26 10:23:04,706 DEBUG plugin.PluginRepository - plugin: id=protocol-httpclient name=Http / Https Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  3957. 2015-02-26 10:23:04,706 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  3958. 2015-02-26 10:23:04,706 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.httpclient.Http
  3959. 2015-02-26 10:23:04,707 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/nutch-extensionpoints/plugin.xml
  3960. 2015-02-26 10:23:04,709 DEBUG plugin.PluginRepository - plugin: id=nutch-extensionpoints name=the nutch core extension points version=2.0.0 provider=nutch.orgclass=null
  3961. 2015-02-26 10:23:04,711 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-pass/plugin.xml
  3962. 2015-02-26 10:23:04,713 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-pass name=Pass-through URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  3963. 2015-02-26 10:23:04,714 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer
  3964. 2015-02-26 10:23:04,714 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-regex/plugin.xml
  3965. 2015-02-26 10:23:04,716 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-regex name=Regex URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  3966. 2015-02-26 10:23:04,717 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
  3967. 2015-02-26 10:23:04,717 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-ftp/plugin.xml
  3968. 2015-02-26 10:23:04,719 DEBUG plugin.PluginRepository - plugin: id=protocol-ftp name=Ftp Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  3969. 2015-02-26 10:23:04,720 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.ftp.Ftp
  3970. 2015-02-26 10:23:04,720 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-nekohtml/plugin.xml
  3971. 2015-02-26 10:23:04,722 DEBUG plugin.PluginRepository - plugin: id=lib-nekohtml name=CyberNeko HTML Parser version=1.9.19 provider=net.sourceforge.nekohtmlclass=null
  3972. 2015-02-26 10:23:04,723 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-http/plugin.xml
  3973. 2015-02-26 10:23:04,725 DEBUG plugin.PluginRepository - plugin: id=protocol-http name=Http Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  3974. 2015-02-26 10:23:04,726 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  3975. 2015-02-26 10:23:04,726 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http
  3976. 2015-02-26 10:23:04,726 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/indexer-solr/plugin.xml
  3977. 2015-02-26 10:23:04,729 DEBUG plugin.PluginRepository - plugin: id=indexer-solr name=SOLRIndexWriter version=1.0.0 provider=nutch.apache.orgclass=null
  3978. 2015-02-26 10:23:04,729 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexWriter class=org.apache.nutch.indexwriter.solr.SolrIndexWriter
  3979. 2015-02-26 10:23:04,731 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/language-identifier/plugin.xml
  3980. 2015-02-26 10:23:04,733 DEBUG plugin.PluginRepository - plugin: id=language-identifier name=Language Identification Parser/Filter version=1.0.0 provider=nutch.orgclass=null
  3981. 2015-02-26 10:23:04,734 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.analysis.lang.HTMLLanguageParser
  3982. 2015-02-26 10:23:04,734 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.analysis.lang.LanguageIndexingFilter
  3983. 2015-02-26 10:23:04,734 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-basic/plugin.xml
  3984. 2015-02-26 10:23:04,736 DEBUG plugin.PluginRepository - plugin: id=index-basic name=Basic Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  3985. 2015-02-26 10:23:04,737 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.basic.BasicIndexingFilter
  3986. 2015-02-26 10:23:04,737 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-link/plugin.xml
  3987. 2015-02-26 10:23:04,739 DEBUG plugin.PluginRepository - plugin: id=scoring-link name=Link Analysis Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  3988. 2015-02-26 10:23:04,739 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.link.LinkAnalysisScoringFilter
  3989. 2015-02-26 10:23:04,740 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-http/plugin.xml
  3990. 2015-02-26 10:23:04,742 DEBUG plugin.PluginRepository - plugin: id=lib-http name=HTTP Framework version=1.0 provider=org.apache.nutchclass=null
  3991. 2015-02-26 10:23:04,742 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/lib-regex-filter/plugin.xml
  3992. 2015-02-26 10:23:04,744 DEBUG plugin.PluginRepository - plugin: id=lib-regex-filter name=Regex URL Filter Framework version=1.0 provider=org.apache.nutchclass=null
  3993. 2015-02-26 10:23:04,745 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-prefix/plugin.xml
  3994. 2015-02-26 10:23:04,747 DEBUG plugin.PluginRepository - plugin: id=urlfilter-prefix name=Prefix URL Filter version=1.0.0 provider=nutch.orgclass=null
  3995. 2015-02-26 10:23:04,747 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.prefix.PrefixURLFilter
  3996. 2015-02-26 10:23:04,748 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlnormalizer-basic/plugin.xml
  3997. 2015-02-26 10:23:04,750 DEBUG plugin.PluginRepository - plugin: id=urlnormalizer-basic name=Basic URL Normalizer version=1.0.0 provider=nutch.orgclass=null
  3998. 2015-02-26 10:23:04,750 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLNormalizer class=org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
  3999. 2015-02-26 10:23:04,751 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/tld/plugin.xml
  4000. 2015-02-26 10:23:04,753 DEBUG plugin.PluginRepository - plugin: id=tld name=Top Level Domain Plugin version=1.0.0 provider=nutch.orgclass=null
  4001. 2015-02-26 10:23:04,753 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.tld.TLDIndexingFilter
  4002. 2015-02-26 10:23:04,754 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.tld.TLDScoringFilter
  4003. 2015-02-26 10:23:04,754 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-anchor/plugin.xml
  4004. 2015-02-26 10:23:04,756 DEBUG plugin.PluginRepository - plugin: id=index-anchor name=Anchor Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  4005. 2015-02-26 10:23:04,756 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.anchor.AnchorIndexingFilter
  4006. 2015-02-26 10:23:04,757 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-suffix/plugin.xml
  4007. 2015-02-26 10:23:04,759 DEBUG plugin.PluginRepository - plugin: id=urlfilter-suffix name=Suffix URL Filter version=1.0.0 provider=nutch.orgclass=null
  4008. 2015-02-26 10:23:04,759 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.suffix.SuffixURLFilter
  4009. 2015-02-26 10:23:04,760 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-regex/plugin.xml
  4010. 2015-02-26 10:23:04,763 DEBUG plugin.PluginRepository - plugin: id=urlfilter-regex name=Regex URL Filter version=1.0.0 provider=nutch.orgclass=null
  4011. 2015-02-26 10:23:04,763 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.regex.RegexURLFilter
  4012. 2015-02-26 10:23:04,764 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-metatags/plugin.xml
  4013. 2015-02-26 10:23:04,766 DEBUG plugin.PluginRepository - plugin: id=parse-metatags name=MetaTags version=1.0 provider=digitalpebble.comclass=null
  4014. 2015-02-26 10:23:04,767 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.metatags.MetaTagsParser
  4015. 2015-02-26 10:23:04,767 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-domain/plugin.xml
  4016. 2015-02-26 10:23:04,769 DEBUG plugin.PluginRepository - plugin: id=urlfilter-domain name=Domain URL Filter version=1.0.0 provider=nutch.orgclass=null
  4017. 2015-02-26 10:23:04,770 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.domain.DomainURLFilter
  4018. 2015-02-26 10:23:04,770 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/scoring-opic/plugin.xml
  4019. 2015-02-26 10:23:04,772 DEBUG plugin.PluginRepository - plugin: id=scoring-opic name=OPIC Scoring Plug-in version=1.0.0 provider=nutch.orgclass=null
  4020. 2015-02-26 10:23:04,773 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.scoring.ScoringFilter class=org.apache.nutch.scoring.opic.OPICScoringFilter
  4021. 2015-02-26 10:23:04,773 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/index-more/plugin.xml
  4022. 2015-02-26 10:23:04,776 DEBUG plugin.PluginRepository - plugin: id=index-more name=More Indexing Filter version=1.0.0 provider=nutch.orgclass=null
  4023. 2015-02-26 10:23:04,776 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.more.MoreIndexingFilter
  4024. 2015-02-26 10:23:04,776 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/creativecommons/plugin.xml
  4025. 2015-02-26 10:23:04,779 DEBUG plugin.PluginRepository - plugin: id=creativecommons name=Creative Commons Plugins version=1.0.0 provider=nutch.orgclass=null
  4026. 2015-02-26 10:23:04,779 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.creativecommons.nutch.CCParseFilter
  4027. 2015-02-26 10:23:04,779 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.indexer.IndexingFilter class=org.creativecommons.nutch.CCIndexingFilter
  4028. 2015-02-26 10:23:04,779 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/protocol-file/plugin.xml
  4029. 2015-02-26 10:23:04,782 DEBUG plugin.PluginRepository - plugin: id=protocol-file name=File Protocol Plug-in version=1.0.0 provider=nutch.orgclass=null
  4030. 2015-02-26 10:23:04,782 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.file.File
  4031. 2015-02-26 10:23:04,782 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/parse-js/plugin.xml
  4032. 2015-02-26 10:23:04,785 DEBUG plugin.PluginRepository - plugin: id=parse-js name=JavaScript Parser version=1.0.0 provider=nutch.orgclass=null
  4033. 2015-02-26 10:23:04,785 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.js.JSParseFilter
  4034. 2015-02-26 10:23:04,786 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.parse.ParseFilter class=org.apache.nutch.parse.js.JSParseFilter
  4035. 2015-02-26 10:23:04,786 DEBUG plugin.PluginRepository - parsing: /hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/plugins/urlfilter-validator/plugin.xml
  4036. 2015-02-26 10:23:04,788 DEBUG plugin.PluginRepository - plugin: id=urlfilter-validator name=URL Validator version=1.0.0 provider=nutch.orgclass=null
  4037. 2015-02-26 10:23:04,788 DEBUG plugin.PluginRepository - impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.urlfilter.validator.UrlValidator
  4038. 2015-02-26 10:23:04,789 DEBUG plugin.PluginRepository - not including: urlfilter-suffix
  4039. 2015-02-26 10:23:04,789 DEBUG plugin.PluginRepository - not including: language-identifier
  4040. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: protocol-httpclient
  4041. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: scoring-link
  4042. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: indexer-elastic
  4043. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: creativecommons
  4044. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: index-more
  4045. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: lib-http
  4046. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: lib-xml
  4047. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: subcollection
  4048. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: urlfilter-prefix
  4049. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: lib-regex-filter
  4050. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: nutch-extensionpoints
  4051. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: urlfilter-automaton
  4052. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: urlfilter-validator
  4053. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: protocol-sftp
  4054. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: protocol-ftp
  4055. 2015-02-26 10:23:04,790 DEBUG plugin.PluginRepository - not including: lib-nekohtml
  4056. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - not including: parse-js
  4057. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - not including: protocol-file
  4058. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - not including: urlfilter-domain
  4059. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - not including: microformats-reltag
  4060. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - not including: tld
  4061. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexingFilter
  4062. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexCleaningFilter
  4063. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.Parser
  4064. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.parse.ParseFilter
  4065. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.protocol.Protocol
  4066. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLFilter
  4067. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.net.URLNormalizer
  4068. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.scoring.ScoringFilter
  4069. 2015-02-26 10:23:04,791 DEBUG plugin.PluginRepository - Adding extension point org.apache.nutch.indexer.IndexWriter
  4070. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
  4071. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Registered Plugins:
  4072. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
  4073. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Basic URL Normalizer (urlnormalizer-basic)
  4074. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
  4075. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
  4076. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - SOLRIndexWriter (indexer-solr)
  4077. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - HTTP Framework (lib-http)
  4078. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
  4079. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Pass-through URL Normalizer (urlnormalizer-pass)
  4080. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Http Protocol Plug-in (protocol-http)
  4081. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Regex URL Normalizer (urlnormalizer-regex)
  4082. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
  4083. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Tika Parser Plug-in (parse-tika)
  4084. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
  4085. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
  4086. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
  4087. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - MetaTags (parse-metatags)
  4088. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Metadata Indexing Filter (index-metadata)
  4089. 2015-02-26 10:23:04,792 INFO plugin.PluginRepository - Registered Extension-Points:
  4090. 2015-02-26 10:23:04,793 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
  4091. 2015-02-26 10:23:04,793 INFO plugin.PluginRepository - Nutch Index Cleaning Filter (org.apache.nutch.indexer.IndexCleaningFilter)
  4092. 2015-02-26 10:23:04,793 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
  4093. 2015-02-26 10:23:04,793 INFO plugin.PluginRepository - Parse Filter (org.apache.nutch.parse.ParseFilter)
  4094. 2015-02-26 10:23:04,793 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
  4095. 2015-02-26 10:23:04,793 INFO plugin.PluginRepository - Nutch Index Writer (org.apache.nutch.indexer.IndexWriter)
  4096. 2015-02-26 10:23:04,793 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
  4097. 2015-02-26 10:23:04,793 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
  4098. 2015-02-26 10:23:04,793 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
  4099. 2015-02-26 10:23:04,793 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local46914250_0001.xml, instantiating a new object cache
  4100. 2015-02-26 10:23:04,818 INFO conf.Configuration - found resource regex-normalize.xml at file:/hwnet/dtg_devel/web/beta-katon/applications/apache-nutch-2.3/runtime/local/conf/regex-normalize.xml
  4101. 2015-02-26 10:23:04,827 DEBUG util.ObjectCache - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, file:/tmp/hadoop-katonj/mapred/local/localRunner/job_local46914250_0001.xml, instantiating a new object cache
  4102. 2015-02-26 10:23:04,827 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
  4103. 2015-02-26 10:23:04,827 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
  4104. 2015-02-26 10:23:04,827 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
  4105. 2015-02-26 10:23:04,919 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  4106. 2015-02-26 10:23:04,975 DEBUG client.HThriftClient - keyspace reseting from null to webpage
  4107. 2015-02-26 10:23:04,995 DEBUG crawl.GeneratorJob - -shouldFetch rejected 'http://nutch.apache.org/', fetchTime=1427556159599, curTime=1424964182724
  4108. 2015-02-26 10:23:04,997 INFO mapred.MapTask - Starting flush of map output
  4109. 2015-02-26 10:23:05,014 INFO compress.CodecPool - Got brand-new compressor
  4110. 2015-02-26 10:23:05,017 INFO mapred.Task - Task:attempt_local46914250_0001_m_000000_0 is done. And is in the process of commiting
  4111. 2015-02-26 10:23:05,021 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4112. 2015-02-26 10:23:05,021 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  4113. 2015-02-26 10:23:05,021 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  4114. 2015-02-26 10:23:05,021 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  4115. 2015-02-26 10:23:05,021 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  4116. 2015-02-26 10:23:05,021 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  4117. 2015-02-26 10:23:05,021 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  4118. 2015-02-26 10:23:05,023 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4119. 2015-02-26 10:23:05,024 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  4120. 2015-02-26 10:23:05,024 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4121. 2015-02-26 10:23:05,024 INFO mapred.LocalJobRunner -
  4122. 2015-02-26 10:23:05,024 INFO mapred.Task - Task 'attempt_local46914250_0001_m_000000_0' done.
  4123. 2015-02-26 10:23:05,024 INFO mapred.LocalJobRunner - Finishing task: attempt_local46914250_0001_m_000000_0
  4124. 2015-02-26 10:23:05,025 INFO mapred.LocalJobRunner - Map task executor complete.
  4125. 2015-02-26 10:23:05,031 DEBUG mapred.SortedRanges - currentIndex 0 0:0
  4126. 2015-02-26 10:23:05,031 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4127. 2015-02-26 10:23:05,031 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  4128. 2015-02-26 10:23:05,031 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  4129. 2015-02-26 10:23:05,031 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  4130. 2015-02-26 10:23:05,031 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  4131. 2015-02-26 10:23:05,031 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  4132. 2015-02-26 10:23:05,031 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  4133. 2015-02-26 10:23:05,033 DEBUG mapred.TaskRunner - mapred.local.dir for child : /tmp/hadoop-katonj/mapred/local/taskTracker/katonj/jobcache/job_local46914250_0001/attempt_local46914250_0001_r_000000_0
  4134. 2015-02-26 10:23:05,038 DEBUG mapred.Task - using new api for output committer
  4135. 2015-02-26 10:23:05,038 INFO mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3c3708e6
  4136. 2015-02-26 10:23:05,039 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4137. 2015-02-26 10:23:05,039 INFO mapred.LocalJobRunner -
  4138. 2015-02-26 10:23:05,045 INFO mapred.Merger - Merging 1 sorted segments
  4139. 2015-02-26 10:23:05,051 INFO compress.CodecPool - Got brand-new decompressor
  4140. 2015-02-26 10:23:05,052 INFO mapred.Merger - Down to the last merge-pass, with 0 segments left of total size: 0 bytes
  4141. 2015-02-26 10:23:05,053 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4142. 2015-02-26 10:23:05,053 INFO mapred.LocalJobRunner -
  4143. 2015-02-26 10:23:05,058 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  4144. 2015-02-26 10:23:05,058 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  4145. 2015-02-26 10:23:05,061 DEBUG store.CassandraMappingManager - persistentClassName=org.apache.nutch.storage.WebPage -> keyspaceName=webpage
  4146. 2015-02-26 10:23:05,061 DEBUG store.CassandraMapping - Located Cassandra Keyspace
  4147. 2015-02-26 10:23:05,061 DEBUG store.CassandraMapping - Located Cassandra Keyspace name: 'webpage'
  4148. 2015-02-26 10:23:05,061 DEBUG store.CassandraMapping - Located Cassandra Keyspace cluster: 'Test Cluster'
  4149. 2015-02-26 10:23:05,061 DEBUG store.CassandraMapping - Located Cassandra Keyspace host: 'localhost:9160'
  4150. 2015-02-26 10:23:05,061 DEBUG store.CassandraMapping - setting Keyspace replication strategy to org.apache.cassandra.locator.SimpleStrategy
  4151. 2015-02-26 10:23:05,061 DEBUG store.CassandraMapping - setting Keyspace replication factor to 1
  4152. 2015-02-26 10:23:05,061 DEBUG store.CassandraMapping - Located column family: 'p'
  4153. 2015-02-26 10:23:05,061 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  4154. 2015-02-26 10:23:05,062 DEBUG store.CassandraMapping - Located column family: 'f'
  4155. 2015-02-26 10:23:05,062 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  4156. 2015-02-26 10:23:05,062 DEBUG store.CassandraMapping - Located column family: 'sc'
  4157. 2015-02-26 10:23:05,062 DEBUG store.CassandraMapping - Located gc_grace_seconds: '0'
  4158. 2015-02-26 10:23:05,062 DEBUG store.CassandraMapping - Located super column family
  4159. 2015-02-26 10:23:05,062 DEBUG store.CassandraMapping - Added super column family: 'sc'
  4160. 2015-02-26 10:23:05,064 DEBUG store.CassandraClient - The comparator type of f column family is BytesType.
  4161. 2015-02-26 10:23:05,064 DEBUG store.CassandraClient - The comparator type of p column family is BytesType.
  4162. 2015-02-26 10:23:05,064 DEBUG store.CassandraClient - The comparator type of sc column family is BytesType.
  4163. 2015-02-26 10:23:05,064 INFO mapreduce.GoraRecordWriter - gora.buffer.write.limit = 10000
  4164. 2015-02-26 10:23:05,071 DEBUG store.CassandraStore - close
  4165. 2015-02-26 10:23:05,071 INFO mapred.Task - Task:attempt_local46914250_0001_r_000000_0 is done. And is in the process of commiting
  4166. 2015-02-26 10:23:05,072 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4167. 2015-02-26 10:23:05,072 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  4168. 2015-02-26 10:23:05,072 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  4169. 2015-02-26 10:23:05,072 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  4170. 2015-02-26 10:23:05,072 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  4171. 2015-02-26 10:23:05,072 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  4172. 2015-02-26 10:23:05,072 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  4173. 2015-02-26 10:23:05,073 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  4174. 2015-02-26 10:23:05,073 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4175. 2015-02-26 10:23:05,073 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4176. 2015-02-26 10:23:05,073 INFO mapred.LocalJobRunner - reduce > reduce
  4177. 2015-02-26 10:23:05,074 INFO mapred.Task - Task 'attempt_local46914250_0001_r_000000_0' done.
  4178. 2015-02-26 10:23:05,075 WARN mapred.FileOutputCommitter - Output path is null in cleanup
  4179. 2015-02-26 10:23:05,448 INFO mapred.JobClient - map 100% reduce 100%
  4180. 2015-02-26 10:23:05,450 INFO mapred.JobClient - Job complete: job_local46914250_0001
  4181. 2015-02-26 10:23:05,450 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  4182. 2015-02-26 10:23:05,450 DEBUG mapred.Counters - Adding BYTES_READ
  4183. 2015-02-26 10:23:05,450 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4184. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  4185. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  4186. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4187. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  4188. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  4189. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  4190. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  4191. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  4192. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  4193. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  4194. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  4195. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  4196. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  4197. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  4198. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  4199. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4200. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  4201. 2015-02-26 10:23:05,451 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  4202. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  4203. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding BYTES_READ
  4204. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4205. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  4206. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  4207. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  4208. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  4209. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  4210. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  4211. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  4212. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  4213. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  4214. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  4215. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  4216. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  4217. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  4218. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  4219. 2015-02-26 10:23:05,452 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  4220. 2015-02-26 10:23:05,453 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  4221. 2015-02-26 10:23:05,453 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  4222. 2015-02-26 10:23:05,453 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  4223. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Counters: 20
  4224. 2015-02-26 10:23:05,453 INFO mapred.JobClient - File Output Format Counters
  4225. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Bytes Written=0
  4226. 2015-02-26 10:23:05,453 INFO mapred.JobClient - File Input Format Counters
  4227. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Bytes Read=0
  4228. 2015-02-26 10:23:05,453 INFO mapred.JobClient - FileSystemCounters
  4229. 2015-02-26 10:23:05,453 INFO mapred.JobClient - FILE_BYTES_READ=1217262
  4230. 2015-02-26 10:23:05,453 INFO mapred.JobClient - FILE_BYTES_WRITTEN=1393214
  4231. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Map-Reduce Framework
  4232. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Reduce input groups=0
  4233. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Map output materialized bytes=14
  4234. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Combine output records=0
  4235. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Map input records=1
  4236. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Reduce shuffle bytes=0
  4237. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Physical memory (bytes) snapshot=0
  4238. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Reduce output records=0
  4239. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Spilled Records=0
  4240. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Map output bytes=0
  4241. 2015-02-26 10:23:05,453 INFO mapred.JobClient - Total committed heap usage (bytes)=1516240896
  4242. 2015-02-26 10:23:05,454 INFO mapred.JobClient - CPU time spent (ms)=0
  4243. 2015-02-26 10:23:05,454 INFO mapred.JobClient - Virtual memory (bytes) snapshot=0
  4244. 2015-02-26 10:23:05,454 INFO mapred.JobClient - SPLIT_RAW_BYTES=729
  4245. 2015-02-26 10:23:05,454 INFO mapred.JobClient - Map output records=0
  4246. 2015-02-26 10:23:05,454 INFO mapred.JobClient - Combine input records=0
  4247. 2015-02-26 10:23:05,454 INFO mapred.JobClient - Reduce input records=0
  4248. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  4249. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Adding BYTES_READ
  4250. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4251. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  4252. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  4253. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4254. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  4255. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  4256. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  4257. 2015-02-26 10:23:05,455 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  4258. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  4259. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  4260. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  4261. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  4262. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  4263. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  4264. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  4265. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  4266. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4267. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  4268. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  4269. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  4270. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding BYTES_READ
  4271. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4272. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  4273. 2015-02-26 10:23:05,456 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  4274. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  4275. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  4276. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  4277. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  4278. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  4279. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  4280. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  4281. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  4282. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  4283. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  4284. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  4285. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  4286. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  4287. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  4288. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  4289. 2015-02-26 10:23:05,457 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  4290. 2015-02-26 10:23:05,458 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  4291. 2015-02-26 10:23:05,458 DEBUG mapred.Counters - Adding BYTES_READ
  4292. 2015-02-26 10:23:05,458 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4293. 2015-02-26 10:23:05,458 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  4294. 2015-02-26 10:23:05,458 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  4295. 2015-02-26 10:23:05,458 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4296. 2015-02-26 10:23:05,458 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  4297. 2015-02-26 10:23:05,458 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  4298. 2015-02-26 10:23:05,458 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  4299. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  4300. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  4301. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  4302. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  4303. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  4304. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  4305. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  4306. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  4307. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  4308. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Creating group FileSystemCounters with nothing
  4309. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding FILE_BYTES_READ
  4310. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding FILE_BYTES_WRITTEN
  4311. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with bundle
  4312. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding BYTES_READ
  4313. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
  4314. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding MAP_OUTPUT_MATERIALIZED_BYTES
  4315. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding COMBINE_OUTPUT_RECORDS
  4316. 2015-02-26 10:23:05,459 DEBUG mapred.Counters - Adding MAP_INPUT_RECORDS
  4317. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding PHYSICAL_MEMORY_BYTES
  4318. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding SPILLED_RECORDS
  4319. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding MAP_OUTPUT_BYTES
  4320. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding CPU_MILLISECONDS
  4321. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding COMMITTED_HEAP_BYTES
  4322. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding VIRTUAL_MEMORY_BYTES
  4323. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding COMBINE_INPUT_RECORDS
  4324. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding MAP_OUTPUT_RECORDS
  4325. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding SPLIT_RAW_BYTES
  4326. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Creating group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with bundle
  4327. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding BYTES_WRITTEN
  4328. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding REDUCE_INPUT_GROUPS
  4329. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding REDUCE_SHUFFLE_BYTES
  4330. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding REDUCE_OUTPUT_RECORDS
  4331. 2015-02-26 10:23:05,460 DEBUG mapred.Counters - Adding REDUCE_INPUT_RECORDS
  4332. 2015-02-26 10:23:05,464 INFO crawl.GeneratorJob - GeneratorJob: finished at 2015-02-26 10:23:05, time elapsed: 00:00:02
  4333. 2015-02-26 10:23:05,464 INFO crawl.GeneratorJob - GeneratorJob: generated batch id: 1424964181-12478 containing 0 URLs
  4334. 2015-02-26 10:23:05,469 DEBUG fs.FileSystem - Starting clear of FileSystem cache with 1 elements.
  4335. 2015-02-26 10:23:05,469 DEBUG fs.FileSystem - Removing filesystem for file:///
  4336. 2015-02-26 10:23:05,469 DEBUG fs.FileSystem - Removing filesystem for file:///
  4337. 2015-02-26 10:23:05,469 DEBUG fs.FileSystem - Done clearing cache
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!