上周五回到家，大概是晚上六点的样子，接到项目现场的电话，“服务器不太正常，在压力测试的时候，用户登录不了，登录页面报了NoRouteToHostException的错误，服务器日志上报了httpClient 地址正在使用的错误等等”，现场的同志都不是技术开发出生，他们的关注点不在这两个异常上面（否则没我啥事了），当时我也没条件远程这个事情（家里没网），然后商量时间后，定在星期天上午一起解决这个问题（无法登录的问题是由于压力测试造成的，周六周天没多少访问量，不会出现用户登录不了的问题）。不过这个时候感觉是socket这个东西出问题了，因为其他项目遇到过socket不释放连接（没close），当socket数过大的时候，服务端就建立不了socket了。
星期六下午有空查了下NoRouteToHostException这个异常，前面两个链接中有这么一篇文章http://performtest163.blog.163.com/blog/static/14007696420115295119462，其中描述的问题跟我们项目现场的问题简直太像了，都是用压力测试，都是报了同样的错误，好吧，这时候我心里更有数点了，看来我之前想到了服务端，忽略了客户端，因为一个链接的建立，不仅要服务器端口，客户端也需要分配本地端口，现在客户段分配不了端口了，自然就不能建立连接，问题就爆出来了。
问题解决，修改linux下配置文件/etc/sysctl.conf，增加下面两行：
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10000 61000其中tcp_tw_recycle作用是加速属于TIME_WAIT状态的TCP连接的回收，默认值是0，即关闭，
tcp_tw_reuse端口重用的意思，默认值0，也是关闭，这里也一起打开，
然后再扩大了本地端口分配的范围。
做了这翻调整之后，效果相当明显。调整之前出路TIME_WAIT的连接数：
调整之后，这种连接数，不同数量级的减少。
注意：
1、不要随意调整net.ipv4.tcp_fin_timeout（单位是秒）这个参数，这个网络环境有比较大的关系，网络好，估计可以适当调小点，我的建议是不调整。
我调整之后，随后报了下面的错误：
[14-5-18 12:12:07:802 CST] 00000134 webapp E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet 错误]-[portal]: com.ibm.wsspi.webcontainer.ClosedConnectionException: OutputStream encountered error during write
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.write(WCCByteBufferOutputStream.java:106)
at com.ibm.ws.webcontainer.srt.SRTOutputStream.write(SRTOutputStream.java:97)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.writeOut(BufferedServletOutputStream.java:592)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.write(BufferedServletOutputStream.java:376)
at com.ibm.ws.io.WSByteArrayOutputStream.writeTo(WSByteArrayOutputStream.java:168)
at com.ibm.ws.cache.servlet.CacheProxyResponse.flushOutput(CacheProxyResponse.java:112)
at com.ibm.ws.cache.servlet.ESISupport.handleESIPostProcessing(ESISupport.java:162)
at com.ibm.ws.cache.servlet.CacheHook.postProcess(CacheHook.java:661)
at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:587)
at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:250)
at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:259)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at com.ibm.wps.engine.ExtensionFilter$ExtendedFilterChain.doFilter(ExtensionFilter.java:91)
at com.ibm.wps.engine.ExtensionFilter.doFilter(ExtensionFilter.java:178)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.engine.ExtendedLocaleFilter.doFilter(ExtendedLocaleFilter.java:113)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.resolver.friendly.servlet.FriendlySelectionFilter.doFilter(FriendlySelectionFilter.java:191)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.mappingurl.impl.URLAnalyzer.doFilter(URLAnalyzer.java:412)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.engine.VirtualPortalFilter.doFilter(VirtualPortalFilter.java:88)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.state.filter.StateCleanup.doFilter(StateCleanup.java:94)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3935)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:931)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Caused by: java.io.IOException: Async IO operation failed (1), reason: RC: 32 断开的管道
at com.ibm.io.async.AsyncLibrary$IOExceptionCache.<init>(AsyncLibrary.java:891)
at com.ibm.io.async.AsyncLibrary$IOExceptionCache.get(AsyncLibrary.java:904)
at com.ibm.io.async.AsyncLibrary.getIOException(AsyncLibrary.java:918)
at com.ibm.io.async.AbstractAsyncChannel.multiIO(AbstractAsyncChannel.java:473)
at com.ibm.io.async.AsyncSocketChannelHelper.write(AsyncSocketChannelHelper.java:478)
at com.ibm.io.async.AsyncSocketChannelHelper.write(AsyncSocketChannelHelper.java:396)
at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.writeAIO(AioSocketIOChannel.java:267)
at com.ibm.ws.tcp.channel.impl.AioTCPWriteRequestContextImpl.processAsyncWriteRequest(AioTCPWriteRequestContextImpl.java:53)
at com.ibm.ws.tcp.channel.impl.TCPWriteRequestContextImpl.writeInternal(TCPWriteRequestContextImpl.java:382)
at com.ibm.ws.tcp.channel.impl.TCPWriteRequestContextImpl.write(TCPWriteRequestContextImpl.java:353)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.asynchWrite(HttpServiceContextImpl.java:2442)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.sendOutgoing(HttpServiceContextImpl.java:2229)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl.sendResponseBody(HttpInboundServiceContextImpl.java:866)
at com.ibm.ws.webcontainer.channel.WCChannelLink.writeBufferAsynch(WCChannelLink.java:551)
at com.ibm.ws.webcontainer.channel.WCChannelLink.writeBufferResponse(WCChannelLink.java:528)
at com.ibm.ws.webcontainer.channel.WCChannelLink.writeBuffer(WCChannelLink.java:472)
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.flushWriteBuffer(WCCByteBufferOutputStream.java:406)
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.checkWriteArray(WCCByteBufferOutputStream.java:378)
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.write(WCCByteBufferOutputStream.java:111)
at com.ibm.ws.webcontainer.srt.SRTOutputStream.write(SRTOutputStream.java:97)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.writeOut(BufferedServletOutputStream.java:592)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.write(BufferedServletOutputStream.java:376)
at com.ibm.ws.io.WSByteArrayOutputStream.writeTo(WSByteArrayOutputStream.java:168)
at com.ibm.ws.cache.servlet.CacheProxyResponse.flushOutput(CacheProxyResponse.java:112)
at com.ibm.ws.cache.servlet.ESISupport.handleESIPostProcessing(ESISupport.java:162)
at com.ibm.ws.cache.servlet.CacheHook.postProcess(CacheHook.java:661)
at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:587)
at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:250)
at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:259)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at com.ibm.wps.engine.ExtensionFilter$ExtendedFilterChain.doFilter(ExtensionFilter.java:91)
at com.ibm.wps.engine.ExtensionFilter.doFilter(ExtensionFilter.java:178)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.engine.ExtendedLocaleFilter.doFilter(ExtendedLocaleFilter.java:113)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.resolver.friendly.servlet.FriendlySelectionFilter.doFilter(FriendlySelectionFilter.java:191)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.mappingurl.impl.URLAnalyzer.doFilter(URLAnalyzer.java:412)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.engine.VirtualPortalFilter.doFilter(VirtualPortalFilter.java:88)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.wps.state.filter.StateCleanup.doFilter(StateCleanup.java:94)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3935)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:931)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:78)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
... 1 more
我推测跟我调的这个参数有关系，还原了，就没这个问题了。
2、谨慎调整这些参数，调整之后，多压力测试几次，看调整是否恰当。
参考链接
http://willvvv.iteye.com/blog/1558321
http://performtest163.blog.163.com/blog/static/14007696420115295119462
