Error:

15:09:56 [SEVERE] java.lang.OutOfMemoryError: unable to create new native thread
15:09:56 [SEVERE]       at java.lang.Thread.start0(Native Method)
15:09:56 [SEVERE]       at java.lang.Thread.start(Thread.java:657)
15:09:56 [SEVERE]       at net.minecraft.server.NetworkManager.d(NetworkManager.java:244)
15:09:56 [SEVERE]       at net.minecraft.server.NetLoginHandler.a(NetLoginHandler.java:153)
15:09:56 [SEVERE]       at net.minecraft.server.Packet254GetInfo.handle(SourceFile:16)
15:09:56 [SEVERE]       at net.minecraft.server.NetworkManager.b(NetworkManager.java:226)
15:09:56 [SEVERE]       at net.minecraft.server.NetLoginHandler.a(NetLoginHandler.java:47)
15:09:56 [SEVERE]       at net.minecraft.server.NetworkListenThread.a(NetworkListenThread.java:61)
15:09:56 [SEVERE]       at net.minecraft.server.MinecraftServer.w(MinecraftServer.java:536)
15:09:56 [SEVERE]       at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:434)
15:09:56 [SEVERE]       at net.minecraft.server.ThreadServerApplication.run(SourceFile:465)

Start Script:

java -Xmx5G -jar craftbukkit-1.1-R1.jar

Java:

java version "1.6.0_23"
OpenJDK Runtime Environment (IcedTea6 1.11pre) (6b23~pre11-0ubuntu1.11.10.1)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)

Craftbukkit:

This server is running Craftbukkit version git-Bukkit-1.1-R1-b1818jnks (MC: 1.1) (Implementing API version 1.1-R1)

free -m output after the error:

             total       used       free     shared    buffers     cached
Mem:          7981       3979       4001          0         49       2177
-/+ buffers/cache:       1752       6228
Swap:         8171          0       8171