apache httpclient 4.x - java.net.SocketException: Too many open files

Question

I have a java app which runs just fine (on Ubuntu 10.04) for few hours until it hits "java.net.SocketException: Too many open files". The code for Sender.java can be found here

Is it because I create a new instance of HttpPut and HttpPost for each thread? I'm using apache-commons HTTPClient 4.

Here's the exception log:

java.net.SocketException: Too many open files
    at java.net.Socket.createImpl(Socket.java:414)
    at java.net.Socket.connect(Socket.java:544)
    at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:123)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:133)
    at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
    at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:108)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
    at com.marketplace.io.Sender.doBasicHttpPost(Sender.java:434)
    at com.marketplace.io.Sender.appVisualExists(Sender.java:223)
    at com.marketplace.io.Sender.addVisualToCollection(Sender.java:350)
    at com.marketplace.service.ImageThread.run(ImageThread.java:136)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:636)

Answer 1

"java.net.SocketException: Too many files open"can be seen any Java Server application e.g. Tomcat, Weblogic, WebSphere etc, with client connecting and disconnecting frequently.

Please note that socket connections are treated like files and they use file descriptor, which is a limited resource.

Different operating system has different limits on number of file handles they can manage.

In short, this error is coming because clients are connecting and disconnecting frequently.If you want to handle it on your side, you have two options :

1) Increase number of open file handles or file descriptors per process.

In UNIX based operating system e.g. Ubuntu or Solaris, you can use command ulimit -a to find out how many open file handles per process is allowed.

$ ulimit -a
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 256
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 2048
virtual memory        (kbytes, -v) unlimited

You can see that, open files (-n) 256, which means only 256 open file handles per process is allowed. If your Java program, remember Tomcat, weblogic or any other application server are Java programs and they run on JVM, exceeds this limit, it will throw java.net.SocketException: Too many files open error.

You can change this limit by using ulimit -n to a larger number e.g. 4096, but do it with advise of UNIX system administrator and if you have separate UNIX support team, than better escalate to them.

2) Reduce timeout for TIME_WAIT state in your operating system

In UNIX based systems, you can see current configuration in /proc/sys/net/ipv4/tcp_fin_timeout file.

In Windows based system, you can see this information in windows registry. You can change the TCPTIME_WAIT timeout in Windows by following below steps :

1) Open Windows Registry Editor, by typing regedit in run command window
2) Find the key HKEY_LOCAL_MACHINESystemCurrentControlSetServicescpipParameters
3) Add a new key value pair TcpTimedWaitDelay asa decimal and set the desired timeout in seconds (60-240)
4) Restart your windows machine.