【发布时间】:2012-04-04 16:03:52
【问题描述】:
我在一个 Solaris 容器中的 Sun JVM 上运行了几个需要大量内存的 Java 服务器进程。最近我添加了更多进程,可能已经耗尽了总可用内存。 Java 进程开始在负载下随机崩溃。在每种情况下,它都是 JVM 中线程 GCTaskThread 中的 SIGSEGV。在某一时刻,甚至top 也死于 SIGSEGV。
当内存不足时,Solaris 是否会使用 SIGSEGV 杀死进程?
我希望在随机位置得到 OutOfMemoryErrors 而不是 SIGSEGV。此外,我一直认为 JVM 中的段错误表示错误(除非涉及 JNI)。我需要知道我是否遇到了额外的 JVM 错误。
这是 Solaris 10 和 32 位 JRE 1.6u20。
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xfeb6297c, pid=26468, tid=6
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode solaris-sparc )
# Problematic frame:
# V [libjvm.so+0x76297c]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x0003bc00): GCTaskThread [stack: 0x8b300000,0x8b380000] [id=6]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x8b828bfb
Registers:
O0=0xf824d1d0 O1=0xdcdc43e8 O2=0xf824d1d3 O3=0x00000001
O4=0xfee2d800 O5=0x00000003 O6=0x8b37f718 O7=0xfe546d18
G1=0xfe55a854 G2=0x90d7f7b4 G3=0xfffffff4 G4=0x00000000
G5=0xfee2dd5f G6=0x00000000 G7=0xfeed2200 Y=0x00000000
PC=0xfeb6297c nPC=0xfeb62980
Top of Stack: (sp=0x8b37f718)
0x8b37f718: 8bc00000 00033158 fee3f000 fbc00000
0x8b37f728: 70000000 1c000000 8b3a2000 00033200
0x8b37f738: fee3b800 00486bfb 00000011 00000003
0x8b37f748: f824d1d3 dbc00000 8b37f778 fe55a854
0x8b37f758: ed204000 00001000 00000000 00000032
0x8b37f768: 8737f7f0 0003c3a8 8b37f7ec 0191c568
0x8b37f778: 0003c3a8 00000010 00000003 00000003
0x8b37f788: 00000009 8b37f84c 00000004 8b37f84c
Instructions: (pc=0xfeb6297c)
0xfeb6296c: ee 06 21 b8 b3 36 60 09 b4 10 20 11 ec 05 e0 4c
0xfeb6297c: f4 2d 80 19 81 c7 e0 08 81 e8 20 00 9d e3 bf a0
Stack: [0x8b300000,0x8b380000], sp=0x8b37f718, free space=1fdfeb6297ck
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x76297c]
V [libjvm.so+0x15a85c]
V [libjvm.so+0x486acc]
V [libjvm.so+0x169314]
V [libjvm.so+0x16bda0]
V [libjvm.so+0x24c2a4]
V [libjvm.so+0x72a1a0]
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x027bf800 JavaThread "pool-3-thread-31" [_thread_blocked, id=1171, stack(0x90d00000,0x90d80000)]
0x028df000 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=1169, stack(0x91400000,0x91480000)]
0x028dec00 JavaThread "AWT-Shutdown" [_thread_blocked, id=1168, stack(0x91700000,0x91780000)]
0x00dd4800 JavaThread "Product-producer-thread-7" [_thread_blocked, id=1165, stack(0x90b00000,0x90b80000)]
0x03a18400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" [_thread_in_native, id=580, stack(0x85100000,0x85180000)]
0x00f83c00 JavaThread "BaseComTimerThread-9" daemon [_thread_blocked, id=287, stack(0x88d80000,0x88e00000)]
0x014d5000 JavaThread "PACCommand_7" [_thread_blocked, id=105, stack(0x86500000,0x86580000)]
0x0109b800 JavaThread "PACCommand_6" [_thread_blocked, id=104, stack(0x86600000,0x86680000)]
0x0109b000 JavaThread "PACCommand_5" [_thread_blocked, id=103, stack(0x86700000,0x86780000)]
0x00d75800 JavaThread "PACCommand_4" [_thread_blocked, id=102, stack(0x86800000,0x86880000)]
0x0106ac00 JavaThread "PACCommand_3" [_thread_blocked, id=101, stack(0x86900000,0x86980000)]
0x01529000 JavaThread "PACCommand_2" [_thread_blocked, id=100, stack(0x86a00000,0x86a80000)]
0x0131e800 JavaThread "PACCommand_1" [_thread_blocked, id=99, stack(0x86b00000,0x86b80000)]
0x01163400 JavaThread "BaseComSingleThread-1" daemon [_thread_blocked, id=91, stack(0x86d00000,0x86d80000)]
0x002f6800 JavaThread "FailoverExecutorThread-0" daemon [_thread_blocked, id=90, stack(0x87500000,0x87580000)]
0x00dd6800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=89, stack(0x86f00000,0x86f80000)]
0x00b8fc00 JavaThread "NonGUI_5" [_thread_blocked, id=87, stack(0x86e00000,0x86e80000)]
0x0091d000 JavaThread "NonGUI_4" [_thread_blocked, id=86, stack(0x87000000,0x87080000)]
0x0191e800 JavaThread "NonGUI_3" [_thread_blocked, id=82, stack(0x87100000,0x87180000)]
0x01fa0400 JavaThread "NonGUI_2" [_thread_blocked, id=81, stack(0x87200000,0x87280000)]
0x018f6800 JavaThread "NonGUI_1" [_thread_blocked, id=80, stack(0x87300000,0x87380000)]
0x00ef0400 JavaThread "Timer legends" [_thread_blocked, id=79, stack(0x87400000,0x87480000)]
0x01179400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=75, stack(0x87600000,0x87680000)]
0x009bd800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=73, stack(0x87700000,0x87780000)]
0x00931800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=71, stack(0x88980000,0x88a00000)]
0x00a6e400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=69, stack(0x88a80000,0x88b00000)]
0x0113b400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=67, stack(0x88b80000,0x88c00000)]
0x00c72000 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=65, stack(0x87900000,0x87980000)]
0x001bbc00 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=63, stack(0x87a00000,0x87a80000)]
0x00f45c00 JavaThread "Access Layer MDT Update Thread #0" daemon [_thread_blocked, id=60, stack(0x89900000,0x89980000)]
0x00a70c00 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=59, stack(0x87800000,0x87880000)]
0x00a68800 JavaThread "AWT-XAWT" daemon [_thread_in_native, id=53, stack(0x87b00000,0x87b80000)]
0x008ecc00 JavaThread "ProductTimer" [_thread_blocked, id=52, stack(0x87d00000,0x87d80000)]
0x00030400 JavaThread "DestroyJavaVM" [_thread_blocked, id=2, stack(0xfe300000,0xfe380000)]
0x00e34c00 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" [_thread_in_native, id=48, stack(0x87e00000,0x87e80000)]
0x00e32800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=47, stack(0x87f00000,0x87f80000)]
0x00f68400 JavaThread "Proxy.cleaner" daemon [_thread_blocked, id=45, stack(0x88000000,0x88080000)]
0x00f67400 JavaThread "Thread-10" [_thread_blocked, id=44, stack(0x88100000,0x88180000)]
0x00f66400 JavaThread "Timer-1" [_thread_blocked, id=43, stack(0x88200000,0x88280000)]
0x00f64c00 JavaThread "CORBA cleaner" daemon [_thread_blocked, id=42, stack(0x88300000,0x88380000)]
0x00f64400 JavaThread "Timer-0" [_thread_blocked, id=41, stack(0x88400000,0x88480000)]
0x0117e800 JavaThread "BaseComExecutorThread-4" [_thread_blocked, id=40, stack(0x88500000,0x88580000)]
0x01fbd400 JavaThread "BaseComExecutorThread-3" [_thread_blocked, id=39, stack(0x88600000,0x88680000)]
0x011ad400 JavaThread "BaseComExecutorThread-2" [_thread_blocked, id=38, stack(0x88700000,0x88780000)]
0x011acc00 JavaThread "Product.scheduler.thread" [_thread_blocked, id=37, stack(0x88800000,0x88880000)]
0x00a9f800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=35, stack(0x88e80000,0x88f00000)]
0x00e95000 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" [_thread_in_native, id=34, stack(0x89400000,0x89480000)]
0x00c14400 JavaThread "BaseComExecutorThread-1" [_thread_blocked, id=32, stack(0x89500000,0x89580000)]
0x00e92000 JavaThread "ConfigCacheCleaner4" daemon [_thread_blocked, id=31, stack(0x89600000,0x89680000)]
0x00f4e400 JavaThread "ConfigCacheCleaner3" daemon [_thread_blocked, id=30, stack(0x89700000,0x89780000)]
0x00cb6000 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=29, stack(0x89800000,0x89880000)]
0x00ba0800 JavaThread "BaseComTimerThread-1" daemon [_thread_blocked, id=25, stack(0x89a00000,0x89a80000)]
0x00133000 JavaThread "TaskRouterThread-1" daemon [_thread_blocked, id=24, stack(0x89b00000,0x89b80000)]
0x00f3c000 JavaThread "NINJO_CACHE_1333544439197_ScratchConfigCache1.3_2.data" daemon [_thread_blocked, id=23, stack(0x89c00000,0x89c80000)]
0x00c00400 JavaThread "NINJO_CACHE_1333544439197_ScratchConfigCache1.3_1.data" daemon [_thread_blocked, id=22, stack(0x89d00000,0x89d80000)]
0x00ed6c00 JavaThread "NINJO_CACHE_1333544439197_ScratchConfigCache1.3.data" daemon [_thread_blocked, id=21, stack(0x89e00000,0x89e80000)]
0x00ba3c00 JavaThread "com.product.fwk.common.cache.impl.EHCacheManager@b02928" daemon [_thread_blocked, id=20, stack(0x89f00000,0x89f80000)]
0x00aa3400 JavaThread "com.product.fwk.common.cache.impl.EHCacheManager@b02928" daemon [_thread_blocked, id=19, stack(0x8a000000,0x8a080000)]
0x00ab5400 JavaThread "ConfigCacheCleaner2" daemon [_thread_blocked, id=18, stack(0x8a100000,0x8a180000)]
0x00c0d800 JavaThread "ConfigCacheCleaner1" daemon [_thread_blocked, id=17, stack(0x8a200000,0x8a280000)]
0x00e40000 JavaThread "ORB-Thread" [_thread_blocked, id=16, stack(0x8a300000,0x8a380000)]
0x00e3e800 JavaThread "ORBacus:Server:StarterThread" [_thread_in_native, id=15, stack(0x8a400000,0x8a480000)]
0x00127c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=13, stack(0x8aa80000,0x8ab00000)]
0x00126000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=12, stack(0x8ab80000,0x8ac00000)]
0x00123800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=11, stack(0x8ac80000,0x8ad00000)]
0x00122000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=10, stack(0x8ad80000,0x8ae00000)]
0x0010d000 JavaThread "Finalizer" daemon [_thread_blocked, id=9, stack(0x8ae80000,0x8af00000)]
0x0010b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=8, stack(0x8af80000,0x8b000000)]
Other Threads:
0x00109400 VMThread [stack: 0x8b080000,0x8b100000] [id=7]
0x0012ac00 WatcherThread [stack: 0x8a980000,0x8aa00000] [id=14]
=>0x0003bc00 (exited) GCTaskThread [stack: 0x8b300000,0x8b380000] [id=6]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x0002f578] Threads_lock - owner thread: 0x00109400
[0x0002f988] Heap_lock - owner thread: 0x00dd4800
Heap
PSYoungGen total 457280K, used 83928K [0xdbc00000, 0xfbc00000, 0xfbc00000)
eden space 391808K, 21% used [0xdbc00000,0xe0df6178,0xf3aa0000)
from space 65472K, 0% used [0xf3aa0000,0xf3aa0000,0xf7a90000)
to space 61632K, 4% used [0xf7fd0000,0xf8254130,0xfbc00000)
PSOldGen total 1048576K, used 331538K [0x9bc00000, 0xdbc00000, 0xdbc00000)
object space 1048576K, 31% used [0x9bc00000,0xaffc4b68,0xdbc00000)
PSPermGen total 49152K, used 45797K [0x8bc00000, 0x8ec00000, 0x9bc00000)
object space 49152K, 93% used [0x8bc00000,0x8e8b9690,0x8ec00000)
Dynamic libraries:
0x00010000 /opt/product/release_1.6/server/jre/bin/java
0xff3a0000 /lib/libthread.so.1
0xff370000 /opt/product/release_1.6/server/jre/bin/../lib/sparc/jli/libjli.so
0xff350000 /lib/libdl.so.1
0xff200000 /lib/libc.so.1
0xff390000 /platform/SUNW,SPARC-Enterprise/lib/libc_psr.so.1
0xfe400000 /opt/product/release_1.6/server/jre/lib/sparc/server/libjvm.so
0xff1d0000 /lib/libsocket.so.1
0xff1f0000 /usr/lib/libsched.so.1
0xff1b0000 /lib/libm.so.1
0xff180000 /usr/lib/libCrun.so.1
0xff160000 /lib/libdoor.so.1
0xff080000 /lib/libnsl.so.1
0xfef80000 /lib/libm.so.2
0xff050000 /lib/libscf.so.1
0xff140000 /lib/libuutil.so.1
0xff030000 /lib/libgen.so.1
0xfef50000 /lib/libmd.so.1
0xfef30000 /lib/libmp.so.2
0xfeea0000 /opt/product/release_1.6/server/jre/lib/sparc/libverify.so
0xfee60000 /opt/product/release_1.6/server/jre/lib/sparc/libjava.so
0xfe3e0000 /opt/product/release_1.6/server/jre/lib/sparc/native_threads/libhpi.so
0xfe2e0000 /lib/nss_files.so.1
0xfe2b0000 /usr/lib/nss_vas4.so.1
0xfe280000 /opt/quest/lib/libvtsmartcache.so.1
0xfe180000 /opt/quest/lib/libvtutil.so.1
0xfe230000 /opt/quest/lib/libvtcacheipc.so.1
0xfe120000 /lib/libresolv.so.2
0xfe100000 /opt/product/release_1.6/server/jre/lib/sparc/libzip.so
0xfe0b0000 /usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3
0x8ae50000 /opt/product/release_1.6/server/jre/lib/sparc/libnet.so
0x89200000 /opt/product/release_1.6/server/jre/lib/sparc/libawt.so
0x8ad20000 /opt/product/release_1.6/server/jre/lib/sparc/xawt/libmawt.so
0x8ae10000 /usr/openwin//lib/libXext.so.0
0x89080000 /usr/openwin//lib/libX11.so.4
0x8ac50000 /usr/openwin//lib/libXtst.so.1
0x8ac20000 /usr/openwin//lib/libXi.so.5
0x88f80000 /opt/product/release_1.6/server/jre/lib/sparc/libfontmanager.so
0x8b110000 /opt/product/release_1.6/server/jre/lib/sparc/libnio.so
0x8ab60000 /lib/librt.so.1
0x8ab40000 /lib/libaio.so.1
0x8af10000 /usr/lib/libsendfile.so.1
0x887d0000 /opt/product/release_1.6/server/jre/lib/sparc/libmanagement.so
0x887b0000 /usr/openwin/lib/locale/common/xlibi18n.so.2
0x886d0000 /usr/X11/lib/libXcursor.so.1
0x886a0000 /usr/openwin/sfw/lib/libXrender.so.1
0x8a850000 /opt/product/release_1.6/server/jre/lib/sparc/libdcpr.so
VM Arguments:
jvm_args: -Xmx1536M -Xms1536M -XX:MaxPermSize=256m -XX:-ReduceInitialCardMarks -Djava.ext.dirs=/opt/product/release_1.6/server/lib/ext -Djava.endorsed.dirs=/opt/product/release_1.6/server/lib/boot -Dorg.omg.CORBA.ORBClass=com.ooc.CORBA.ORB -Dorg.omg.CORBA.ORBSingletonClass=com.ooc.CORBA.ORBSingleton -Dcorba.port=23801 -Dserver.name=ProductServer -Dserver.basedir=/data/product -Dproduct.log.path=../../log -Dproduct.i18n.locale.language=en
java_command: /opt/product/release_1.6/server/lib/product/ProductMain.jar /opt/product/release_1.6/server/config.properties
Launcher Type: SUN_STANDARD
Environment Variables:
CLASSPATH=/opt/local/opkg/share/oracle/product/10.2.0/jdbc/lib/classes12.zip
PATH=/usr/bin:/usr/sbin:/usr/local/bin:/opt/local/opkg/bin:/opt/local/opkg/sbin:/home/ext/user/bin:/opt/local/opkg/share/oracle/product/10.2.0/bin:/usr/bin:/usr/sbin:/usr/local/bin:/opt/local/opkg/bin:/opt/local/opkg/sbin:/home/users/user/bin:/usr/bin:/home/users/user/bin
LD_LIBRARY_PATH=/opt/product/release_1.6/server/jre/lib/sparc/server:/opt/product/release_1.6/server/jre/lib/sparc:/opt/product/release_1.6/server/jre/../lib/sparc:/opt/local/opkg/share/oracle/product/10.2.0/lib32:/opt/local/opkg/share/oracle/product/10.2.0/lib:/opt/local/opkg/share/oracle/product/10.2.0/jdbc/lib32
SHELL=/bin/bash
DISPLAY=localhost:63.0
Signal Handlers:
SIGSEGV: [libjvm.so+0x84dd5c], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0x84dd5c], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGXFSZ: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x72c934], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x72c934], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIG39: [libjvm.so+0x730360], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIG40: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
--------------- S Y S T E M ---------------
OS: Solaris 10 10/09 s10s_u8wos_08a SPARC
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 16 September 2009
uname:SunOS 5.10 Generic_142909-17 sun4u (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 8192, AS infinity
load average:7.79 3.58 3.31
CPU:total 4 has_v8, has_v9, popc, has_vis1, has_vis2, is_ultra3
Memory: 8k page, physical 15728640k(1399920k free)
vm_info: Java HotSpot(TM) Server VM (16.3-b01) for solaris-sparc JRE (1.6.0_20-b02), built on Apr 12 2010 14:27:21 by "" with Workshop 5.8
time: Wed Apr 4 13:50:18 2012
elapsed time: 2982 seconds
【问题讨论】:
-
由于您使用 SIGSEGV 编写了甚至 top failed ,因此不太可能是 jvm 问题。一种可能是您以某种方式更改了共享库依赖项。 hs_err_
.log 文件中有什么内容? -
在我之前做过的一些测试中,JVM实际上是使用SIGSEGV信号来判断是否有NullPointerException。根据您关于 SIGSEGV 的“top”也失败的说法,我最好的猜测是这些程序只是假设 malloc(或其某些变体)永远不会失败;他们得到一个 NULL 指针,这会导致一个 SIGSEGV。可能 Java 也在做同样的事情,因为它只发生在 GCTaskThread 中?
-
@rm5248: 是的,许多 handled SIGSEGV 用于内存管理,参见例如this question。内存分配失败时的规范错误消息是“java.lang.OutOfMemoryError: requested ... bytes for .... Out of swap space?”,所以我猜分配实际上是检查过的。
-
每当 GC 运行时,它都需要一些本机内存来记录其数据结构,如果可用的 JVM 本机内存耗尽,malloc/calloc 可能会失败并导致 SIGSEGV。
标签: java garbage-collection solaris segmentation-fault jvm-crash