We had a strange behaviour when increasing the max heap size of a web application, we got OutOfMemoryError: unable to create new native thread.

We tracked down the problem and wrote some tests:

public class ThreadTest {

public static void main(String[] pArgs) throws Exception {

try {
while (true) {
new TestThread().start();
}
} catch ( OutOfMemoryError e ) {
System.out.println(TestThread.CREATE_COUNT);
System.exit(-1);
}
}

static class TestThread extends Thread {
private static int CREATE_COUNT = 0;

public TestThread() {
CREATE_COUNT++;
}

public void run() {
try {
sleep(Integer.MAX_VALUE);
} catch (InterruptedException e) {
}
}
}

}


Testing with different max heap sizes shows strange result:

for i in 32 64 128 256 512 768 1024; do echo $j $i `java -Xmx${i}M ThreadTest`; done

"max heap size in MB" "max thread count"
32 3597
64 3532
128 3404
256 3153
512 2647
768 2140
1024 1635

on a 32bit SuSE Linux 9.2 with Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)

on our Server (64bit SuSE SLES9.0,Java HotSpot(TM) Server VM (build 1.5.0_04-b05, mixed mode)):

32 4982
64 4920
128 4792
256 4534
512 4032
768 3530
1024 3024
1280 2520
1536 2014
1792 1506
2048 999
2304 498

2304 was the max. heap we can start the vm with

If we reduce the Thread stack size to 128kb (-Xss128k) the results on the server are

32 19148
64 18907
128 18423
256 17444
512 15516
768 13564
1024 11629
1280 9691
1536 7748
1792 5804
2048 3869
2304 1932

Is this a expected behaviour and so the only way to have many threads and a big heap is to change the thread stack size to a lower value?

Thanx for any reply

Armin

 

 

yes, this is limitation of virtual memory space per process in 32 bit enviroment. 64 bit OS/JVM has much bigger values.

PS Executing thousand of threads on one CPU quite ineffective and useless, so these values are mostly enough.

 

 

The purpose of my posting was to find a explanation of the behaviour of having less threads available with more maximum heap. I cannot see why this is a limitation of virtual memory space?

Having only 500 Threads available with a 4 processor server with 4GB memory is not enough for us (only 2GB is available for the java vm)

 

 

This is hardware limitation. We can't just override it. Memory allocated for managed heap inside JVM can't be used as thread stack space. So increasing one number will reduce the other.

(Managed Heap native heap thread stack size * number of threads) couldn't exceed 2 GB on 32bit x86 systems.

 

 

Ok, I understand that on 32bit systems. We have the same behaviour with out 64bit server (sles9 linux with 64bit jdk1.5). Are the similar limitations

 

 

Sorry, I checked the currently installed vm on the server. It is a 32bit version. Thanx for your reply, it clears our problem...

 

 

Well you must not be surprised by this...
JVM manages memory in a specific way and heap is not a place to get memory for threads stack.

With 32-bit platform you are limited by OS inability to address more then 4Gb memory fragment for JVM. It's NOT 2Gb as many other resources says and there is NO direct limitation in JVM code.

I can say this for sure b/c i checking it by myself and have JVM 1.5.0 update 06 running on Linux 2.6.9 smp with the following options:


-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=10001
-Djava.awt.headless=true

-server
-Xcomp
-Xconcurrentio
-Xnoclassgc

#-XX:+UseBiasedLocking #Only works with 6.0 - 5.0 crashes with this
-XX:+UseSuspendResumeThreadLists
-XX:+UseLWPSynchronization
-XX:ParallelGCThreads=8
-XX:SurvivorRatio=8
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=15
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:GCHeapFreeLimit=10
-XX:+UseLargePages
-Xms2400m
-Xmx2400m
-Xmn1500m
-Xss128k
-XX:MaxPermSize=256m
-XX:PermSize=128m

It took days to figure out how to adjust my linux kernel, sysctl and bash settings. I had (of course!) recompile standard RedHat kernel. Finally it works fine and my only problem is a small swap partition i need to expand in the nearest future.

Here is a brief list of what to check in your system settings before you'll be able to run JVM with the options listed above:

ulimit--------------
data seg size (kbytes, -d) unlimited
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 206129
stack size (kbytes, -s) 2048 #This ALSO affects -Xss !!!
max user processes (-u) 16379
virtual memory (kbytes, -v) unlimited


/etc/sysctl.com--------------------
vm.nr_hugepages=1790
kernel.shmmax=3758096384
kernel.shmall=3758096384
#don't forget to do:
# mkdir /mnt/hugepages
# mount -t hugetlbfs nodev /mnt/hugepages
# chmod 777 /mnt/hugepages

/ect/fstab

---

none /dev/shm tmpfs defaults,size=4G 0 0

Validation:

cat /proc/meminfo | grep Huge
HugePages_Total: 1790
HugePages_Free: 462
Hugepagesize: 2048 kB

cat /proc/mounts
none /dev/shm tmpfs rw 0 0
nodev /mnt/hugepages hugetlbfs rw 0 0

uname -a
Linux host.name 2.6.9-5.0.5.ELsmp #1 SMP Fri Apr 8 14:29:47 EDT 2005 i686 i686 i386 GNU/Linux

getconf GNU_LIBPTHREAD_VERSION
NPTL 2.3.4

Please notice that with a high throwput your server JVM may crash wtih OOEM b/c of PermGen size even when it 256M.

If you are interested to get more details and list of source articles then just email me topgun_at_4youall.com

 

 

When i try this with 1.4.2_07 on Linux 2.4 kernel, after creating 1423 threads the JVM goes into a defunct state with the following error:
Fatal: Stack size too small. Use 'java -Xss' to increase default stack size

This is independant of the Xss and Xmx settings which do not seem to make any difference.

Any clues on what is happening here?

Thanks

 

 

It's quite obvious - 128k is enough for stack in my configuration as i do not use TLAB and the code itself designed to use thread local carefully. IMO your system just step over the limit of 128k for statck size.
In order to fix this you may need to increase -Xss128k to be like -Xss512k