一次永久代泄漏（perm泄漏）排查

写在前面

网上关于perm区泄露的文章比较少，特别是对于动态类加载方面问题的分析比较少，在此记录下。

perm区问题一般两种解决方案：

  • 启动时perm区问题，一般修改PermSize,MaxPermSize参数即可
  • 运行时动态生成类加载问题，这种问题比较难搞，需要关心动态加载了哪些类

周末早晨收到几台机器告警(fullgc告警)(perm大于90%告警)，为快速解决问题，先把几台机器重启解决，留下了一台禁用端口保留现场进行问题分析。平时上线发版比较频繁，发版后jvm回收，如果较长时间没有发版可能会造成泄露，收到告警。

吃完早饭后登上机器进行排查。

排查过程

登入机器，查看内存使用高的进程：

top

pid:15298

既然是perm区问题，查看永久代情况：

jmap -permstat pid > 15298dump.permstat

 

  1. class_loader classes bytes parent_loader alive? type

  2.  

  3. <bootstrap> 3630 21866152 null live <internal>

  4. 0x0000000705e4df00 1 2008 0x00000006c002e450 dead sun/reflect/[email protected]

  5. 0x00000007282e1040 4 23480 0x00000006c002e450 dead com/facebook/swift/codec/internal/compiler/[email protected]

通过awk统计type类型，查看加载了哪种类型的类：

awk '{ arr[$6]+=$3 } END { for (key in arr) printf("%s\t%s\n", key, arr[key]) }' 15298dump.permstat | sort -k2,2

 

  1. ava/net/[email protected] 0

  2. java/util/[email protected] 0

  3. type 0

  4. sun/reflect/[email protected] 11361872

  5. sun/reflect/misc/[email protected] 134528

  6. <internal> 21866152

  7. com/alibaba/fastjson/util/[email protected] 4785232

  8. N/A 5146

  9. sun/misc/[email protected] 5423216

  10. com/facebook/swift/codec/internal/compiler/[email protected] 5488288

  11. sun/misc/[email protected] 615216

  12. org/eclipse/jetty/webapp/[email protected] 738107040

WebAppClassLoader加载最多，达到了738107040，近738m。

知道了加载的类最多，怎么分析具体加载了哪种类呢？

之前一般通过Java -verbose查看启动类加载过程。

对于运行时貌似有两种方式：

  • 基于ClassFilter写一个拦截器，dump特定的类，使用SA的jar($JAVA_HOME/lib/sa-jdi.jar)编译好类，在编译好的类目录下调用下面的命令进行dump
  • 问了(政威老师)，准备试试arthas，https://alibaba.github.io/arthas/classloader.html

机器上安装好arthas-boot.jar，并启动：

java -jar arthas-boot.jar

选择进程，执行类后台导出，便后续分析：

classloader -a >> &

打开导出文件：

 

  1. hash:null, BootstrapClassLoader

  2. [B

  3. [C

  4. [D

  5. [F

  6. [I

  7. [J

  8. [Lcom.sun.activation.registries.MimeTypeFile;

  9. [Lcom.sun.imageio.plugins.jpeg.DHTMarkerSegment$Htable;

  10. [Lcom.sun.imageio.plugins.jpeg.ImageTypeProducer;

  11. [Lcom.sun.imageio.plugins.jpeg.JPEGImageReader$CallBackLock$State;

  12. [Lcom.sun.imageio.plugins.jpeg.JPEGImageWriter$CallBackLock$State;

  13. [Lcom.sun.imageio.plugins.jpeg.SOFMarkerSegment$ComponentSpec;

  14. [Lcom.sun.imageio.plugins.jpeg.SOSMarkerSegment$ScanComponentSpec;

  15. [Lcom.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry;

  16. [Lcom.sun.jmx.mbeanserver.MXBeanMapping;

  17. [Lcom.sun.org.apache.xalan.internal.utils.FeatureManager$Feature;

  18. [Lcom.sun.org.apache.xalan.internal.utils.FeaturePropertyBase$State;

  19. [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$Limit;

  20. [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$NameMap;

  21. [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$State;

  22. [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityPropertyManager$Property;

  23. [Lcom.sun.org.apache.xerces.internal.impl.XMLEntityManager$CharacterBuffer;

  24. [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMLeaf;

  25. [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMNode;

  26. [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMStateSet;

  27. [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.ContentModelValidator;

  28. [Lcom.sun.org.apache.xerces.internal.impl.dv.DatatypeValidator;

  29. [Lcom.sun.org.apache.xerces.internal.impl.dv.XSSimpleType;

  30. [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.AbstractDateTimeDV$DateTimeData;

  31. [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.TypeValidator;

  32. [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.XSSimpleTypeDecl;

  33. [Lcom.sun.org.apache.xerces.internal.impl.xpath.regex.RegularExpression;

  34. [Lcom.sun.org.apache.xerces.internal.impl.xs.SchemaGrammar;

  35. [Lcom.sun.org.apache.xerces.internal.impl.xs.SubstitutionGroupHandler$OneSubGroup;

  36. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAnnotationImpl;

  37. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAttributeUseImpl;

  38. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSComplexTypeDecl;

  39. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSElementDecl;

  40. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSGroupDecl;

  41. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSNotationDecl;

  42. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSParticleDecl;

  43. [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.IdentityConstraint;

  44. [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.XPathMatcher;

  45. [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMLeaf;

  46. [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMValidator;

  47. [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.DefaultNode;

  48. [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.NodeImpl;

  49. [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.OneAttr;

  50. [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.XSDocumentInfo;

  51. [Lcom.sun.org.apache.xerces.internal.impl.xs.util.SimpleLocator;

  52. [Lcom.sun.org.apache.xerces.internal.impl.xs.util.XInt;

  53. [Lcom.sun.org.apache.xerces.internal.util.Status;

  54. [Lcom.sun.org.apache.xerces.internal.util.SymbolHash$Entry;

  55. [Lcom.sun.org.apache.xerces.internal.util.SymbolTable$Entry;

  56. [Lcom.sun.org.apache.xerces.internal.util.XMLAttributesImpl$Attribute;

  57. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$Limit;

  58. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$NameMap;

  59. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$State;

  60. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$Property;

  61. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$State;

  62. [Lcom.sun.org.apache.xerces.internal.xni.QName;

  63. [Lcom.sun.org.apache.xerces.internal.xni.XMLLocator;

  64. [Lcom.sun.org.apache.xerces.internal.xni.grammars.Grammar;

  65. [Lcom.sun.org.apache.xerces.internal.xni.grammars.XSGrammar;

  66. [Lcom.sun.org.apache.xerces.internal.xs.ShortList;

  67. [Lcom.sun.org.apache.xerces.internal.xs.XSAnnotation;

  68. [Lcom.sun.org.apache.xerces.internal.xs.XSAttributeUse;

  69. [Lcom.sun.org.apache.xerces.internal.xs.XSComplexTypeDefinition;

  70. [Lcom.sun.org.apache.xerces.internal.xs.XSElementDeclaration;

  71. [Lcom.sun.org.apache.xerces.internal.xs.XSIDCDefinition;

  72. [Lcom.sun.org.apache.xerces.internal.xs.XSModelGroupDefinition;

  73. [Lcom.sun.org.apache.xerces.internal.xs.XSNamespaceItem;

  74. [Lcom.sun.org.apache.xerces.internal.xs.XSNotationDeclaration;

  75. [Lcom.sun.org.apache.xerces.internal.xs.XSObject;

  76. [Lcom.sun.org.apache.xerces.internal.xs.XSParticle;

  77. [Lcom.sun.org.apache.xerces.internal.xs.XSSimpleTypeDefinition;

  78. [Lcom.sun.org.apache.xerces.internal.xs.XSTerm;

  79. [Lcom.sun.org.apache.xerces.internal.xs.XSTypeDefinition;

  80. [Lcom.sun.org.apache.xerces.internal.xs.datatypes.XSDateTime;

  81. [Lcom.sun.org.apache.xml.internal.dtm.DTM;

  82. [Lcom.sun.org.apache.xml.internal.dtm.DTMAxisTraverser;

  83. [Lcom.sun.org.apache.xml.internal.dtm.DTMIterator;

  84. [Lcom.sun.org.apache.xml.internal.dtm.ref.ExpandedNameTable$HashEntry;

  85. [Lcom.sun.org.apache.xml.internal.dtm.ref.ExtendedType;

  86. [Lcom.sun.org.apache.xpath.internal.Expression;

  87. [Lcom.sun.org.apache.xpath.internal.ExpressionNode;

  88. [Lcom.sun.org.apache.xpath.internal.XPathVisitable;

  89. [Lcom.sun.org.apache.xpath.internal.axes.LocPathIterator;

  90. [Lcom.sun.org.apache.xpath.internal.axes.PathComponent;

  91. [Lcom.sun.org.apache.xpath.internal.axes.PredicatedNodeTest;

  92. [Lcom.sun.org.apache.xpath.internal.axes.SubContextList;

  93. [Lcom.sun.org.apache.xpath.internal.objects.XObject;

  94. [Lcom.sun.org.apache.xpath.internal.patterns.NodeTest;

  95. [Lcom.sun.xml.internal.ws.org.objectweb.asm.Item;

  96. [Lcom.sun.xml.internal.ws.org.objectweb.asm.Type;

  97. [Ljava.awt.AWTKeyStroke;

  98. [Ljava.awt.Dimension;

  99. [Ljava.awt.Queue;

  100. [Ljava.awt.event.ActionListener;

  101. [Ljava.awt.event.ComponentListener;

  102. [Ljava.awt.event.FocusListener;

  103. [Ljava.awt.event.HierarchyBoundsListener;

  104. ......

统计最多的类：

 

  1. #!/usr/bin/python

  2.  

  3. from collections import Counter

  4.  

  5. package_name_count_dic = {}

  6. for s in open('14.txt'):

  7. full_qualified_name = s.strip()

  8. if not full_qualified_name:

  9. continue

  10. class_name_index = full_qualified_name.rfind('.')

  11. if class_name_index >= 0:

  12. package_name, class_name = full_qualified_name[:class_name_index], full_qualified_name[class_name_index + 1:]

  13. else:

  14. package_name = class_name = full_qualified_name

  15. if package_name in package_name_count_dic:

  16. package_name_count_dic[package_name] += 1

  17. else:

  18. package_name_count_dic[package_name] = 1

  19. k = Counter(package_name_count_dic)

  20. high = k.most_common(5)

  21. for p, c in high:

  22. print p, c

最多的几个类：

一次永久代泄漏（perm泄漏）排查

真凶：ma.glasnost.orika.generated。

查看代码中谁使用了orika类库。

一次永久代泄漏（perm泄漏）排查

发现是闪购同学，由于我们的系统目前对接多方，闪购同学还在我们系统做代码开发，在codereview上存在一些问题。

 

  1. @Component

  2. public class SGMapperFactory implements FactoryBean<MapperFactory> {

  3. @Override

  4. public MapperFactory getObject() {

  5. return new DefaultMapperFactory.Builder().build();

  6. }

  7.  

  8. @Override

  9. public Class<?> getObjectType() {

  10. return MapperFactory.class;

  11. }

  12.  

  13. @Override

  14. public boolean isSingleton() {

  15. return true;

  16. }

  17. }

每次调用getObject都会新创建DefaultMapperFactory对象。MapperGenerator 每次会动态产生类。

至此问题排查，推动闪购同学改动。

后记

看代码是3月份代码写的，为何到现在才发现问题呢？

首先是平时发版比较频繁，jvm发版后问题解决。 最近发版比较少，同时这些是扩容机器，在最近的几次发版中并没有发版，所以造成类加载持续一段时间，最后造成永久代泄露。 为尽早发现问题需要在流程上进行控制，比如增加codereview细致程度，在灰度发版后对发版机器进行引流压测，尽早发现问题，解决问题。

写在前面

网上关于perm区泄露的文章比较少，特别是对于动态类加载方面问题的分析比较少，在此记录下。

perm区问题一般两种解决方案：

  • 启动时perm区问题，一般修改PermSize,MaxPermSize参数即可
  • 运行时动态生成类加载问题，这种问题比较难搞，需要关心动态加载了哪些类

周末早晨收到几台机器告警(fullgc告警)(perm大于90%告警)，为快速解决问题，先把几台机器重启解决，留下了一台禁用端口保留现场进行问题分析。平时上线发版比较频繁，发版后jvm回收，如果较长时间没有发版可能会造成泄露，收到告警。

吃完早饭后登上机器进行排查。

排查过程

登入机器，查看内存使用高的进程：

top

pid:15298

既然是perm区问题，查看永久代情况：

jmap -permstat pid > 15298dump.permstat

 

  1. class_loader classes bytes parent_loader alive? type

  2.  

  3. <bootstrap> 3630 21866152 null live <internal>

  4. 0x0000000705e4df00 1 2008 0x00000006c002e450 dead sun/reflect/[email protected]

  5. 0x00000007282e1040 4 23480 0x00000006c002e450 dead com/facebook/swift/codec/internal/compiler/[email protected]

通过awk统计type类型，查看加载了哪种类型的类：

awk '{ arr[$6]+=$3 } END { for (key in arr) printf("%s\t%s\n", key, arr[key]) }' 15298dump.permstat | sort -k2,2

 

  1. ava/net/[email protected] 0

  2. java/util/[email protected] 0

  3. type 0

  4. sun/reflect/[email protected] 11361872

  5. sun/reflect/misc/[email protected] 134528

  6. <internal> 21866152

  7. com/alibaba/fastjson/util/[email protected] 4785232

  8. N/A 5146

  9. sun/misc/[email protected] 5423216

  10. com/facebook/swift/codec/internal/compiler/[email protected] 5488288

  11. sun/misc/[email protected] 615216

  12. org/eclipse/jetty/webapp/[email protected] 738107040

WebAppClassLoader加载最多，达到了738107040，近738m。

知道了加载的类最多，怎么分析具体加载了哪种类呢？

之前一般通过Java -verbose查看启动类加载过程。

对于运行时貌似有两种方式：

  • 基于ClassFilter写一个拦截器，dump特定的类，使用SA的jar($JAVA_HOME/lib/sa-jdi.jar)编译好类，在编译好的类目录下调用下面的命令进行dump
  • 问了(政威老师)，准备试试arthas，https://alibaba.github.io/arthas/classloader.html

机器上安装好arthas-boot.jar，并启动：

java -jar arthas-boot.jar

选择进程，执行类后台导出，便后续分析：

classloader -a >> &

打开导出文件：

 

  1. hash:null, BootstrapClassLoader

  2. [B

  3. [C

  4. [D

  5. [F

  6. [I

  7. [J

  8. [Lcom.sun.activation.registries.MimeTypeFile;

  9. [Lcom.sun.imageio.plugins.jpeg.DHTMarkerSegment$Htable;

  10. [Lcom.sun.imageio.plugins.jpeg.ImageTypeProducer;

  11. [Lcom.sun.imageio.plugins.jpeg.JPEGImageReader$CallBackLock$State;

  12. [Lcom.sun.imageio.plugins.jpeg.JPEGImageWriter$CallBackLock$State;

  13. [Lcom.sun.imageio.plugins.jpeg.SOFMarkerSegment$ComponentSpec;

  14. [Lcom.sun.imageio.plugins.jpeg.SOSMarkerSegment$ScanComponentSpec;

  15. [Lcom.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry;

  16. [Lcom.sun.jmx.mbeanserver.MXBeanMapping;

  17. [Lcom.sun.org.apache.xalan.internal.utils.FeatureManager$Feature;

  18. [Lcom.sun.org.apache.xalan.internal.utils.FeaturePropertyBase$State;

  19. [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$Limit;

  20. [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$NameMap;

  21. [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$State;

  22. [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityPropertyManager$Property;

  23. [Lcom.sun.org.apache.xerces.internal.impl.XMLEntityManager$CharacterBuffer;

  24. [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMLeaf;

  25. [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMNode;

  26. [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMStateSet;

  27. [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.ContentModelValidator;

  28. [Lcom.sun.org.apache.xerces.internal.impl.dv.DatatypeValidator;

  29. [Lcom.sun.org.apache.xerces.internal.impl.dv.XSSimpleType;

  30. [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.AbstractDateTimeDV$DateTimeData;

  31. [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.TypeValidator;

  32. [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.XSSimpleTypeDecl;

  33. [Lcom.sun.org.apache.xerces.internal.impl.xpath.regex.RegularExpression;

  34. [Lcom.sun.org.apache.xerces.internal.impl.xs.SchemaGrammar;

  35. [Lcom.sun.org.apache.xerces.internal.impl.xs.SubstitutionGroupHandler$OneSubGroup;

  36. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAnnotationImpl;

  37. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAttributeUseImpl;

  38. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSComplexTypeDecl;

  39. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSElementDecl;

  40. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSGroupDecl;

  41. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSNotationDecl;

  42. [Lcom.sun.org.apache.xerces.internal.impl.xs.XSParticleDecl;

  43. [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.IdentityConstraint;

  44. [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.XPathMatcher;

  45. [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMLeaf;

  46. [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMValidator;

  47. [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.DefaultNode;

  48. [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.NodeImpl;

  49. [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.OneAttr;

  50. [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.XSDocumentInfo;

  51. [Lcom.sun.org.apache.xerces.internal.impl.xs.util.SimpleLocator;

  52. [Lcom.sun.org.apache.xerces.internal.impl.xs.util.XInt;

  53. [Lcom.sun.org.apache.xerces.internal.util.Status;

  54. [Lcom.sun.org.apache.xerces.internal.util.SymbolHash$Entry;

  55. [Lcom.sun.org.apache.xerces.internal.util.SymbolTable$Entry;

  56. [Lcom.sun.org.apache.xerces.internal.util.XMLAttributesImpl$Attribute;

  57. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$Limit;

  58. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$NameMap;

  59. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$State;

  60. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$Property;

  61. [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$State;

  62. [Lcom.sun.org.apache.xerces.internal.xni.QName;

  63. [Lcom.sun.org.apache.xerces.internal.xni.XMLLocator;

  64. [Lcom.sun.org.apache.xerces.internal.xni.grammars.Grammar;

  65. [Lcom.sun.org.apache.xerces.internal.xni.grammars.XSGrammar;

  66. [Lcom.sun.org.apache.xerces.internal.xs.ShortList;

  67. [Lcom.sun.org.apache.xerces.internal.xs.XSAnnotation;

  68. [Lcom.sun.org.apache.xerces.internal.xs.XSAttributeUse;

  69. [Lcom.sun.org.apache.xerces.internal.xs.XSComplexTypeDefinition;

  70. [Lcom.sun.org.apache.xerces.internal.xs.XSElementDeclaration;

  71. [Lcom.sun.org.apache.xerces.internal.xs.XSIDCDefinition;

  72. [Lcom.sun.org.apache.xerces.internal.xs.XSModelGroupDefinition;

  73. [Lcom.sun.org.apache.xerces.internal.xs.XSNamespaceItem;

  74. [Lcom.sun.org.apache.xerces.internal.xs.XSNotationDeclaration;

  75. [Lcom.sun.org.apache.xerces.internal.xs.XSObject;

  76. [Lcom.sun.org.apache.xerces.internal.xs.XSParticle;

  77. [Lcom.sun.org.apache.xerces.internal.xs.XSSimpleTypeDefinition;

  78. [Lcom.sun.org.apache.xerces.internal.xs.XSTerm;

  79. [Lcom.sun.org.apache.xerces.internal.xs.XSTypeDefinition;

  80. [Lcom.sun.org.apache.xerces.internal.xs.datatypes.XSDateTime;

  81. [Lcom.sun.org.apache.xml.internal.dtm.DTM;

  82. [Lcom.sun.org.apache.xml.internal.dtm.DTMAxisTraverser;

  83. [Lcom.sun.org.apache.xml.internal.dtm.DTMIterator;

  84. [Lcom.sun.org.apache.xml.internal.dtm.ref.ExpandedNameTable$HashEntry;

  85. [Lcom.sun.org.apache.xml.internal.dtm.ref.ExtendedType;

  86. [Lcom.sun.org.apache.xpath.internal.Expression;

  87. [Lcom.sun.org.apache.xpath.internal.ExpressionNode;

  88. [Lcom.sun.org.apache.xpath.internal.XPathVisitable;

  89. [Lcom.sun.org.apache.xpath.internal.axes.LocPathIterator;

  90. [Lcom.sun.org.apache.xpath.internal.axes.PathComponent;

  91. [Lcom.sun.org.apache.xpath.internal.axes.PredicatedNodeTest;

  92. [Lcom.sun.org.apache.xpath.internal.axes.SubContextList;

  93. [Lcom.sun.org.apache.xpath.internal.objects.XObject;

  94. [Lcom.sun.org.apache.xpath.internal.patterns.NodeTest;

  95. [Lcom.sun.xml.internal.ws.org.objectweb.asm.Item;

  96. [Lcom.sun.xml.internal.ws.org.objectweb.asm.Type;

  97. [Ljava.awt.AWTKeyStroke;

  98. [Ljava.awt.Dimension;

  99. [Ljava.awt.Queue;

  100. [Ljava.awt.event.ActionListener;

  101. [Ljava.awt.event.ComponentListener;

  102. [Ljava.awt.event.FocusListener;

  103. [Ljava.awt.event.HierarchyBoundsListener;

  104. ......

统计最多的类：

 

  1. #!/usr/bin/python

  2.  

  3. from collections import Counter

  4.  

  5. package_name_count_dic = {}

  6. for s in open('14.txt'):

  7. full_qualified_name = s.strip()

  8. if not full_qualified_name:

  9. continue

  10. class_name_index = full_qualified_name.rfind('.')

  11. if class_name_index >= 0:

  12. package_name, class_name = full_qualified_name[:class_name_index], full_qualified_name[class_name_index + 1:]

  13. else:

  14. package_name = class_name = full_qualified_name

  15. if package_name in package_name_count_dic:

  16. package_name_count_dic[package_name] += 1

  17. else:

  18. package_name_count_dic[package_name] = 1

  19. k = Counter(package_name_count_dic)

  20. high = k.most_common(5)

  21. for p, c in high:

  22. print p, c

最多的几个类：

一次永久代泄漏（perm泄漏）排查

真凶：ma.glasnost.orika.generated。

查看代码中谁使用了orika类库。

一次永久代泄漏（perm泄漏）排查

发现是闪购同学，由于我们的系统目前对接多方，闪购同学还在我们系统做代码开发，在codereview上存在一些问题。

 

  1. @Component

  2. public class SGMapperFactory implements FactoryBean<MapperFactory> {

  3. @Override

  4. public MapperFactory getObject() {

  5. return new DefaultMapperFactory.Builder().build();

  6. }

  7.  

  8. @Override

  9. public Class<?> getObjectType() {

  10. return MapperFactory.class;

  11. }

  12.  

  13. @Override

  14. public boolean isSingleton() {

  15. return true;

  16. }

  17. }

每次调用getObject都会新创建DefaultMapperFactory对象。MapperGenerator 每次会动态产生类。

至此问题排查，推动闪购同学改动。

后记

看代码是3月份代码写的，为何到现在才发现问题呢？

首先是平时发版比较频繁，jvm发版后问题解决。 最近发版比较少，同时这些是扩容机器，在最近的几次发版中并没有发版，所以造成类加载持续一段时间，最后造成永久代泄露。 为尽早发现问题需要在流程上进行控制，比如增加codereview细致程度，在灰度发版后对发版机器进行引流压测，尽早发现问题，解决问题。

相关文章：

  • 2021-09-27
  • 2021-11-29
  • 2021-04-30
  • 2021-07-08
  • 2021-06-06
  • 2021-12-28
  • 2021-07-01
  • 2022-12-23
猜你喜欢
  • 2021-10-25
  • 2021-04-14
  • 2021-06-03
  • 2021-05-19
相关资源
相似解决方案
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode