Solve "PermGen space OutOfMemoryError"

原创
2015/04/13 15:57
阅读数 203

What is PermGen used for?

Before answer this question, we should first know that the permanent generation is used for storing meta data and constants. We can think of it as a place for ClassLoader(s) and String internals.


What does that mean if a PermGen space OOM occurs?

Apparently, the permgen OOM indicates that all the permgen space has been exhausted by your application.


Why there would be a PermGen space OOM?

There might be several reasons for this error.

1. Your application loads quite a number of libraries(jars) but the up limit of permanent generation space is less that requirement.

Enlarge the permgen space via the VM options can solve it.

For JDKs lower than Java 8, use:

-XX:PermSize=<perm-size>[k|m|g] -XX:MaxPermSize=<perm-size>[k|m|g]

For JDKs equal or greater than Java 8, use:

-XX:MaxMetaspaceSize=<metaspace-size>[k|m|g]

2. ClassLoader leak when undeploy and then redeploy.

So what is a ClassLoader leak? In fact, the JVM allows to unloading ClassLoader(s) when GC if a ClassLoader is possible for garbage collection. A ClassLoader is a gc candidate if there is no more path from this ClassLoader to GC roots. A path route through a weak reference will not prevent GC, only soft reference and strong reference will.

As long as your application has such a offender path, your ClassLoader will not be GCed. This is a big problem if your application runs in a web application container, as generally the container allocate a dedicated ClassLoader(e.g., a WebappClassLoader in Tomcat) for each deployed application. Imaging that you deployed a ClassLoader leaked application to the container, then undeploy it. Yes, all permgen data loaded by that application's ClassLoader will not be GCed, consumes spaces forever. And if you undeploy and redeploy several times, you will quickly meet the PermGen space OOM.

How to solve this kind of PermGen space OOM? Well...much complicated than the previous situation. In general, steps listed as following:

  1. get a heap dump of the ClassLoader leaked application

  2. open the heap dump file via a certain tool(e.g., the built-in jhat tool provided by each JDK, or Eclipse Memory Analyzer Tool)

  3. locate the instance of your application's ClassLoader(this varies among different application container or servers)

  4. find paths to gc roots from that ClassLoader object without weak references

  5. keep an eye on the found paths to locate the unwanted reference

  6. remove that(those) references

In next section, details will be explained for each step.


Kill the ClassLoader Leak!

I assumed that you have the knowledge of the built-in tools provided by the JDK, if not please BaiDu or Google them by yourself.

Step 1. Get a heap dump

There are several approaches to get a heap dump file.

Using the built-in jmap tool, more details just type jmap -h:

jmap -dump:format=b,file=<path-to-dump-file> <pid>

Or using the built-in jvisualvm:

Right click on the running Java process(connect to remote JVM if needed) > 'Heap Dump'

Or using the Eclipse Memory Analyzer Tool(MAT):

'File' > 'Acquire Heap Dump'

Note that the MAT is limited to get heap dump for local JVMs only.

Besides, you may want to get a heap dump when JVM OOM occurred, then add these VM options:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=

/path/to/dir/or/path/to/file

Step 2. Open the heap dump

Using the built-in jhat tool, more details just type jhat -h:

jhat -J-mx512m <file>

The -J-mx512m demand for a 512m sized heap when running the jhat tool itself. Once the jhat is ready, you could access it via http://localhost:7000, the 7000 is the default port if you didn't specify it.

Or using the jvisualvm:

'File' > 'Load...' > change 'Files of type' to 'Heap Dumps(*.hprof)' > 'Open'

Or using the Eclipse MAT:

'File' > 'Open Heap Dump...' > 'Open'

Step 3. Locate your application's ClassLoader

For killing this ClassLoader leak problem, I recommend to use a more modern tool e.g., Eclipse MAT, YouKit(learned from others but I have no experience of it) as they will be more intuitive to find and locate things for big projects. The built-in jhat tool can solve the ClassLoader leak in theory but if you're a newbie for this problem you really need a modern tool to simplify the whole process. For demonstrating this process, I will use Eclipse MAT. And once you've experience for ClassLoader leak problem under a real circumstance, it will be more easier if you'd like to turn to other tools including the built-in jhat.

As I mentioned in ClassLoader leak when undeploy and then redeploy, you mission is to locate the leaked application's ClassLoader. Each application server(container)'s web application ClassLoader might be different. In the following example, that ClassLoader is named org.apache.catalina.loader.WebappClassLoader as I was using the Tomcat server. Look up the documentation of your application server(container), and adjust the name here according to that doc.

When opened a heap dump file, the Eclipse MAT(for simplicity, MAT will be used for short) will ask you whether you want to have a 'Leak Suspects Report'. That might help with the OOM problem caused by normal un-GCed objects, but for our ClassLoader leak problem just click Cancel as generally it gives not much help.

Click 'Open Query Browser' button  in the top of the just opened tab, then 'Java Basics' > 'Class Loader Explorer'.

The shown 'Java Basics / Class Loader Explorer' dialog will as you to fill in a pattern to search. As I know the application's ClassLoader is named org.apache.catalina.loader.WebappClassLoader, so type in .*WebappClassLoader. It's OK to leave it empty as the pattern is just a way to shorten the searched result.

'Finish' the dialog brings you to a result list like:

The Tomcat server default to have several applications under its webapps folder so we have multiple WebappClassLoader listed above. Go through the list and your application's context name.

Your application server(container) might have an indicator to point out whether your application is running or whether it is eligible for garbage collected from the server(container)'s perspective. Tomcat server has a flag 'started' to point whether a application is running.

Step 4. find paths from gc roots to the leaked ClassLoader

Once you find the leaked ClassLoader, right click that instance of WebappClassLoader, choose 'ClassLoader' > 'Path To GC Roots' > 'exclude all phantom/weak/soft etc. references'.

Before stepping further, you may wonder why we exclude those references? I think you should know that the phantom reference and weak reference will not prevent GC even if they exists. For the soft reference, the Sun(Oracle) said that once the space is not enough and only soft references are left, they will be GCed before OOM.

The listed paths will be like:

Step 5. keep an eye on the found paths to locate the unwanted reference

Note that the bottom domainTb is still not a GC root if you see the details in the Inspector tab.

The MAT didn't expanded all under that domainTb, that trick have been once cheated me and I immediately think of there is no problem here... At this place, expand the rest nodes manually and you will see the ManagementFactory is loaded by the system ClassLoader and it has a reference via the path:

ManagementFactory

  > platformMBeanServer

    > mbsInterceptor

      > repository

Step 6. remove that(those) references

The MBean is a general root cause for ClassLoader leak problem, as third-party libraries might register a certain MBean. Those libraries work well in the standard application server, but for the application container NOT. They should be unregistered before undeploy to let hot-redeploy possible. To fix ClassLoader leak in our example, we need to unregister that MBean when undeploy.

Another general root cause is your application started Threads but didn't stop them when undeploy. The Thread class is always loaded by the Java system class loader, and if they are not stopped then apparently they cannot be GCed.


Let the JVM unloading unused ClassLoader(s) for you!

It's possible for asking JVM to unload classes when GC. That means if you forget to ask to do this, no permgen space will be reclaimed and the unloading magic will not happen.

For JDKs greater than Java 5, add these VM options:

-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC

This means you must use the CMS collector otherwise it will not effect. These options do work and if the leaked ClassLoader and classes are not reclaimed, continue to fix your application.

For JDKs lower than or equal to Java 5, add these VM options:

-XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC


Thanks for reading!

Hopefully this blog helps for your PermGen space OOM! At this very end, I kindly ask you to link to this blog when forwarding to your personal SNS.


References

http://frankkieviet.blogspot.com/2006/10/classloader-leaks-dreaded-permgen-space.html

http://frankkieviet.blogspot.com/2006/10/how-to-fix-dreaded-permgen-space.html

http://java.jiderhamn.se/2011/12/11/classloader-leaks-i-how-to-find-classloader-leaks-with-eclipse-memory-analyser-mat/

http://java.jiderhamn.se/2012/01/01/classloader-leaks-ii-find-and-work-around-unwanted-references/

http://java.jiderhamn.se/2012/01/15/classloader-leaks-iii-die-thread-die/

http://java.jiderhamn.se/2012/01/29/classloader-leaks-iv-threadlocal-dangers-and-why-threadglobal-may-have-been-a-more-appropriate-name/


展开阅读全文
打赏
1
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
1
分享
返回顶部
顶部