API tools faq
paste
Advertisement
faneron

Master DataNode LOG

faneron
Sep 13th, 2022
2,113
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Apache Log 32.62 KB | Source Code | 0 0
raw download clone embed print report
  1. 'stevel' on 2022-07-29T12:32Z
  2. STARTUP_MSG:   java = 1.8.0_342
  3. ************************************************************/
  4. 2022-09-13 11:16:11,067 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
  5. 2022-09-13 11:16:11,377 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/home/ml/dfs/datanode
  6. 2022-09-13 11:16:11,453 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
  7. 2022-09-13 11:16:11,511 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
  8. 2022-09-13 11:16:11,511 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
  9. 2022-09-13 11:16:11,657 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
  10. 2022-09-13 11:16:11,671 INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576
  11. 2022-09-13 11:16:11,675 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is DataScience
  12. 2022-09-13 11:16:11,675 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
  13. 2022-09-13 11:16:11,679 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting DataNode with maxLockedMemory = 0
  14. 2022-09-13 11:16:11,694 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:9866
  15. 2022-09-13 11:16:11,695 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwidth is 104857600 bytes/s
  16. 2022-09-13 11:16:11,695 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for balancing is 100
  17. 2022-09-13 11:16:11,727 INFO org.eclipse.jetty.util.log: Logging initialized @1153ms to org.eclipse.jetty.util.log.Slf4jLog
  18. 2022-09-13 11:16:11,810 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/ml/hadoop-http-auth-signature-secret
  19. 2022-09-13 11:16:11,816 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.datanode is not defined
  20. 2022-09-13 11:16:11,823 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
  21. 2022-09-13 11:16:11,824 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
  22. 2022-09-13 11:16:11,824 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
  23. 2022-09-13 11:16:11,824 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
  24. 2022-09-13 11:16:11,890 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 42419
  25. 2022-09-13 11:16:11,891 INFO org.eclipse.jetty.server.Server: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
  26. 2022-09-13 11:16:11,909 INFO org.eclipse.jetty.server.session: DefaultSessionIdManager workerName=node0
  27. 2022-09-13 11:16:11,909 INFO org.eclipse.jetty.server.session: No SessionScavenger set, using defaults2022-09-13 11:16:11,910 INFO org.eclipse.jetty.server.session: node0 Scavenging every 600000ms
  28. 2022-09-13 11:16:11,920 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2049a9c1{logs,/logs,file:///home/ml/hadoop/logs/,AVAILABLE}
  29. 2022-09-13 11:16:11,921 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@7c51f34b{static,/static,file:///home/ml/hadoop/share/hadoop/hdfs/webapps/static/,AVAILABLE}
  30. 2022-09-13 11:16:11,976 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@53f6fd09{datanode,/,file:///home/ml/hadoop/share/hadoop/hdfs/webapps/datanode/,AVAILABLE}{file:/home/ml/hadoop/share/hadoop/hdfs/webapps/datanode}
  31. 2022-09-13 11:16:11,982 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@3e74829{HTTP/1.1, (http/1.1)}{localhost:42419}
  32. 2022-09-13 11:16:11,982 INFO org.eclipse.jetty.server.Server: Started @1408ms
  33. 2022-09-13 11:16:12,056 WARN org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Got null for restCsrfPreventionFilter - will not do any filtering.
  34. 2022-09-13 11:16:12,128 INFO org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:9864
  35. 2022-09-13 11:16:12,132 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
  36. 2022-09-13 11:16:12,133 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = ml
  37. 2022-09-13 11:16:12,133 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup
  38. 2022-09-13 11:16:12,182 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
  39. 2022-09-13 11:16:12,197 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9867
  40. 2022-09-13 11:16:12,362 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:9867
  41. 2022-09-13 11:16:12,384 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
  42. 2022-09-13 11:16:12,393 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
  43. 2022-09-13 11:16:12,402 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to DataScience/127.0.1.1:9000 starting to offer service
  44. 2022-09-13 11:16:12,407 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
  45. 2022-09-13 11:16:12,407 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9867: starting
  46. 2022-09-13 11:16:12,677 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to DataScience/127.0.1.1:9000
  47. 2022-09-13 11:16:12,679 INFO org.apache.hadoop.hdfs.server.common.Storage: Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1)
  48. 2022-09-13 11:16:12,689 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/ml/dfs/datanode/in_use.lock acquired by nodename 12583@DataScience
  49. 2022-09-13 11:16:12,715 INFO org.apache.hadoop.hdfs.server.common.Storage: Analyzing storage directories for bpid BP-1514539440-127.0.1.1-1662735586830
  50. 2022-09-13 11:16:12,715 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled for /home/ml/dfs/datanode/current/BP-1514539440-127.0.1.1-1662735586830
  51. 2022-09-13 11:16:12,724 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage: nsid=167770591;bpid=BP-1514539440-127.0.1.1-1662735586830;lv=-57;nsInfo=lv=-66;cid=CID-2488a887-bdf1-4c9e-8b5b-3792e28228b3;nsid=167770591;c=1662735586830;bpid=BP-1514539440-127.0.1.1-1662735586830;dnuuid=cebcadab-ea70-4a99-b46c-62ea9082769e
  52. 2022-09-13 11:16:12,740 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: The datanode lock is a read write lock
  53. 2022-09-13 11:16:12,828 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-31f50cfa-a0b1-4cc6-b667-5bbdaf796ac2
  54. 2022-09-13 11:16:12,828 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - [DISK]file:/home/ml/dfs/datanode, StorageType: DISK
  55. 2022-09-13 11:16:12,834 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader: Initializing cache loader: MemoryMappableBlockLoader.
  56. 2022-09-13 11:16:12,836 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean
  57. 2022-09-13 11:16:12,842 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1514539440-127.0.1.1-1662735586830
  58. 2022-09-13 11:16:12,843 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1514539440-127.0.1.1-1662735586830 on volume /home/ml/dfs/datanode...
  59. 2022-09-13 11:16:12,853 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Cached dfsUsed found for /home/ml/dfs/datanode/current/BP-1514539440-127.0.1.1-1662735586830/current: 32768
  60. 2022-09-13 11:16:12,870 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1514539440-127.0.1.1-1662735586830 on /home/ml/dfs/datanode: 26ms
  61. 2022-09-13 11:16:12,870 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-1514539440-127.0.1.1-1662735586830: 28ms
  62. 2022-09-13 11:16:12,872 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-1514539440-127.0.1.1-1662735586830 on volume /home/ml/dfs/datanode...
  63. 2022-09-13 11:16:12,872 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice: Replica Cache file: /home/ml/dfs/datanode/current/BP-1514539440-127.0.1.1-1662735586830/current/replicas doesn't exist
  64. 2022-09-13 11:16:12,874 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1514539440-127.0.1.1-1662735586830 on volume /home/ml/dfs/datanode: 2ms
  65. 2022-09-13 11:16:12,874 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to add all replicas to map for block pool BP-1514539440-127.0.1.1-1662735586830: 3ms
  66. 2022-09-13 11:16:12,875 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for /home/ml/dfs/datanode
  67. 2022-09-13 11:16:12,885 INFO org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker: Scheduled health check for volume /home/ml/dfs/datanode
  68. 2022-09-13 11:16:12,899 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/home/ml/dfs/datanode, DS-31f50cfa-a0b1-4cc6-b667-5bbdaf796ac2): no suitable block pools found to scan.  Waiting 1457882218 ms.
  69. 2022-09-13 11:16:12,903 WARN org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value above 1000 ms/sec. Assuming default value of -1
  70. 2022-09-13 11:16:12,904 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting in 13165142ms with interval of 21600000ms and throttle limit of -1ms/s
  71. 2022-09-13 11:16:12,911 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1514539440-127.0.1.1-1662735586830 (Datanode Uuid cebcadab-ea70-4a99-b46c-62ea9082769e) service to DataScience/127.0.1.1:9000 beginning handshake with NN
  72. 2022-09-13 11:16:13,040 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1514539440-127.0.1.1-1662735586830 (Datanode Uuid cebcadab-ea70-4a99-b46c-62ea9082769e) service to DataScience/127.0.1.1:9000 successfully registered with NN
  73. 2022-09-13 11:16:13,040 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode DataScience/127.0.1.1:9000 using BLOCKREPORT_INTERVAL of 21600000msecs CACHEREPORT_INTERVAL of 10000msecs Initial delay: 0msecs; heartBeatInterval=3000
  74. 2022-09-13 11:16:13,170 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Successfully sent block report 0x6781f639d6e11e3b with lease ID 0x8d7a159c7ac2b265 to namenode: DataScience/127.0.1.1:9000,  containing 1 storage report(s), of which we sent 1. The reports had 0 total blocks and used 1 RPC(s). This took 3 msecs to generate and 48 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.
  75. 2022-09-13 11:16:13,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Got finalize command for block pool BP-1514539440-127.0.1.1-1662735586830
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!