API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Mar 15th, 2016
132
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 11.22 KB | None | 0 0
raw download clone embed print report
  1. Python 2.7.11 |Anaconda 2.3.0 (64-bit)| (default, Dec 6 2015, 18:08:32)
  2. [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
  3. Type "help", "copyright", "credits" or "license" for more information.
  4. Anaconda is brought to you by Continuum Analytics.
  5. Please check out: http://continuum.io/thanks and https://anaconda.org
  6. 16/03/15 17:43:56 DEBUG PythonGatewayServer: Started PythonGatewayServer on port 42693
  7. 16/03/15 17:43:56 DEBUG PythonGatewayServer: Communicating GatewayServer port to Python driver at 127.0.0.1:52297
  8. 16/03/15 17:43:56 INFO SparkContext: Running Spark version 1.6.1
  9. 16/03/15 17:43:56 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
  10. 16/03/15 17:43:56 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
  11. 16/03/15 17:43:56 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[GetGroups], about=, type=DEFAULT, always=false, sampleName=Ops)
  12. 16/03/15 17:43:56 DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
  13. 16/03/15 17:43:57 DEBUG KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
  14. 16/03/15 17:43:57 DEBUG Groups: Creating new Groups object
  15. 16/03/15 17:43:57 DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
  16. 16/03/15 17:43:57 DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
  17. 16/03/15 17:43:57 DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
  18. 16/03/15 17:43:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  19. 16/03/15 17:43:57 DEBUG PerformanceAdvisory: Falling back to shell based
  20. 16/03/15 17:43:57 DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
  21. 16/03/15 17:43:57 DEBUG Shell: Failed to detect a valid hadoop home directory
  22. java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
  23. at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:302)
  24. at org.apache.hadoop.util.Shell.<clinit>(Shell.java:327)
  25. at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
  26. at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)
  27. at org.apache.hadoop.security.Groups.<init>(Groups.java:86)
  28. at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
  29. at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
  30. at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:271)
  31. at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:248)
  32. at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:763)
  33. at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:748)
  34. at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:621)
  35. at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2160)
  36. at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2160)
  37. at scala.Option.getOrElse(Option.scala:120)
  38. at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2160)
  39. at org.apache.spark.SparkContext.<init>(SparkContext.scala:322)
  40. at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
  41. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  42. at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  43. at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  44. at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  45. at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
  46. at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
  47. at py4j.Gateway.invoke(Gateway.java:214)
  48. at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
  49. at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
  50. at py4j.GatewayConnection.run(GatewayConnection.java:209)
  51. at java.lang.Thread.run(Thread.java:745)
  52. 16/03/15 17:43:57 DEBUG Shell: setsid exited with exit code 0
  53. 16/03/15 17:43:57 DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
  54. 16/03/15 17:43:57 DEBUG UserGroupInformation: hadoop login
  55. 16/03/15 17:43:57 DEBUG UserGroupInformation: hadoop login commit
  56. 16/03/15 17:43:57 DEBUG UserGroupInformation: using local user:UnixPrincipal: casimiro
  57. 16/03/15 17:43:57 DEBUG UserGroupInformation: Using user: "UnixPrincipal: casimiro" with name casimiro
  58. 16/03/15 17:43:57 DEBUG UserGroupInformation: User entry: "casimiro"
  59. 16/03/15 17:43:57 DEBUG UserGroupInformation: UGI loginUser:casimiro (auth:SIMPLE)
  60. 16/03/15 17:43:57 INFO SecurityManager: Changing view acls to: casimiro
  61. 16/03/15 17:43:57 INFO SecurityManager: Changing modify acls to: casimiro
  62. 16/03/15 17:43:57 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(casimiro); users with modify permissions: Set(casimiro)
  63. 16/03/15 17:43:57 DEBUG SSLOptions: No SSL protocol specified
  64. 16/03/15 17:43:57 DEBUG SSLOptions: No SSL protocol specified
  65. 16/03/15 17:43:57 DEBUG SSLOptions: No SSL protocol specified
  66. 16/03/15 17:43:57 DEBUG SecurityManager: SSLConfiguration for file server: SSLOptions{enabled=false, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
  67. 16/03/15 17:43:57 DEBUG SecurityManager: SSLConfiguration for Akka: SSLOptions{enabled=false, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
  68. 16/03/15 17:43:57 DEBUG InternalLoggerFactory: Using SLF4J as the default logging framework
  69. 16/03/15 17:43:57 DEBUG PlatformDependent0: java.nio.Buffer.address: available
  70. 16/03/15 17:43:57 DEBUG PlatformDependent0: sun.misc.Unsafe.theUnsafe: available
  71. 16/03/15 17:43:57 DEBUG PlatformDependent0: sun.misc.Unsafe.copyMemory: available
  72. 16/03/15 17:43:57 DEBUG PlatformDependent0: java.nio.Bits.unaligned: true
  73. 16/03/15 17:43:57 DEBUG PlatformDependent: Java version: 7
  74. 16/03/15 17:43:57 DEBUG PlatformDependent: -Dio.netty.noUnsafe: false
  75. 16/03/15 17:43:57 DEBUG PlatformDependent: sun.misc.Unsafe: available
  76. 16/03/15 17:43:57 DEBUG PlatformDependent: -Dio.netty.noJavassist: false
  77. 16/03/15 17:43:57 DEBUG PlatformDependent: Javassist: unavailable
  78. 16/03/15 17:43:57 DEBUG PlatformDependent: You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
  79. 16/03/15 17:43:57 DEBUG PlatformDependent: -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
  80. 16/03/15 17:43:57 DEBUG PlatformDependent: -Dio.netty.bitMode: 64 (sun.arch.data.model)
  81. 16/03/15 17:43:57 DEBUG PlatformDependent: -Dio.netty.noPreferDirect: false
  82. 16/03/15 17:43:57 DEBUG MultithreadEventLoopGroup: -Dio.netty.eventLoopThreads: 8
  83. 16/03/15 17:43:57 DEBUG NioEventLoop: -Dio.netty.noKeySetOptimization: false
  84. 16/03/15 17:43:57 DEBUG NioEventLoop: -Dio.netty.selectorAutoRebuildThreshold: 512
  85. 16/03/15 17:43:57 TRACE NioEventLoop: Instrumented an optimized java.util.Set into: sun.nio.ch.EPollSelectorImpl@7e4d38a7
  86. 16/03/15 17:43:57 TRACE NioEventLoop: Instrumented an optimized java.util.Set into: sun.nio.ch.EPollSelectorImpl@7fc29cb8
  87. 16/03/15 17:43:57 TRACE NioEventLoop: Instrumented an optimized java.util.Set into: sun.nio.ch.EPollSelectorImpl@41bf358f
  88. 16/03/15 17:43:57 TRACE NioEventLoop: Instrumented an optimized java.util.Set into: sun.nio.ch.EPollSelectorImpl@70556101
  89. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.numHeapArenas: 8
  90. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.numDirectArenas: 8
  91. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.pageSize: 8192
  92. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.maxOrder: 11
  93. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.chunkSize: 16777216
  94. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.tinyCacheSize: 512
  95. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.smallCacheSize: 256
  96. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.normalCacheSize: 64
  97. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.maxCachedBufferCapacity: 32768
  98. 16/03/15 17:43:57 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.cacheTrimInterval: 8192
  99. 16/03/15 17:43:57 TRACE NioEventLoop: Instrumented an optimized java.util.Set into: sun.nio.ch.EPollSelectorImpl@6721bed0
  100. 16/03/15 17:43:57 TRACE NioEventLoop: Instrumented an optimized java.util.Set into: sun.nio.ch.EPollSelectorImpl@58767e95
  101. 16/03/15 17:43:57 TRACE NioEventLoop: Instrumented an optimized java.util.Set into: sun.nio.ch.EPollSelectorImpl@4b709592
  102. 16/03/15 17:43:57 TRACE NioEventLoop: Instrumented an optimized java.util.Set into: sun.nio.ch.EPollSelectorImpl@4c6bcaef
  103. 16/03/15 17:43:57 DEBUG ThreadLocalRandom: -Dio.netty.initialSeedUniquifier: 0xaa2a5e6b866a8f72 (took 0 ms)
  104. 16/03/15 17:43:57 DEBUG ByteBufUtil: -Dio.netty.allocator.type: unpooled
  105. 16/03/15 17:43:57 DEBUG ByteBufUtil: -Dio.netty.threadLocalDirectBufferSize: 65536
  106. 16/03/15 17:43:57 DEBUG NetUtil: Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%1)
  107. 16/03/15 17:43:57 DEBUG NetUtil: /proc/sys/net/core/somaxconn: 128
  108. 16/03/15 17:43:57 DEBUG TransportServer: Shuffle server started on port :48942
  109. 16/03/15 17:43:57 INFO Utils: Successfully started service 'sparkDriver' on port 48942.
  110. 16/03/15 17:43:57 DEBUG AkkaUtils: In createActorSystem, requireCookie is: off
  111. 16/03/15 17:43:58 INFO Slf4jLogger: Slf4jLogger started
  112. 16/03/15 17:43:58 INFO Remoting: Starting remoting
  113. 16/03/15 17:43:58 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@127.0.0.1:37983]
  114. 16/03/15 17:43:58 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 37983.
  115. 16/03/15 17:43:58 DEBUG SparkEnv: Using serializer: class org.apache.spark.serializer.JavaSerializer
  116. 16/03/15 17:43:58 INFO SparkEnv: Registering MapOutputTracker
  117. 16/03/15 17:43:58 INFO SparkEnv: Registering BlockManagerMaster
  118. 16/03/15 17:43:58 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-fd60b578-425e-468e-a5b8-243f75eb041b
  119. 16/03/15 17:43:58 INFO MemoryStore: MemoryStore started with capacity 511.5 MB
  120. 16/03/15 17:43:58 INFO SparkEnv: Registering OutputCommitCoordinator
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!