背景：先前在生产库上配置了oracle监控，每5分钟尝试连接一次数据库，若连接失败则通过nagios+fetion自动报警,此配置参考文章：http://www.it165.net/database/html/201207/2750.html
早晨收到报警信息后，登陆数据库执行ps -ef查看oracle的后台进程都在，使用conn /as sysdba的方式登陆数据库，提示连接到空闲的实例，使用easy connect 方式连接则报oracle实例无法分配内存，从报错提示上看，就像oracle数据库实例未打开的状态！分析alert日志不断出现如下错误信息： Process J002 died, see its trace file kkjcre1p: unable to spawn jobq slave process Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_cjq0_18577.trc:
trace文件摘要信息如下： *** 2012-07-26 10:20:31.068 Process J002 is dead (pid=13857 req_ver=1136 cur_ver=1136 state=KSOSP_SPAWNED).
*** 2012-07-26 10:20:32.069 Process J002 is dead (pid=13876 req_ver=1594 cur_ver=1594 state=KSOSP_SPAWNED).
google查询一番后，发现大部分描述和oracle的进程数设置有关,又或者是内存不足引起！于是在oracle 10g环境下测试，线上数据库环境为11.2.0.3
一：设置processes初始化参数值为20，重启数据库后，已经占用19个进程
01.SQL> select count(*) from v$process;
02.03. COUNT(*)
04.----------
05. 19
06.07.SQL> show parameter process;
08.09.NAME TYPE VALUE
10.------------------------------------ ----------- ------------------------------
11.aq_tm_processes integer 0
12.db_writer_processes integer 1
13.gcs_server_processes integer 0
14.job_queue_processes integer 10
15.log_archive_max_processes integer 2
16.processes integer 20
新的会话连接，则报连接到空闲的实例，alert日志则出现相应的报错
01.[root@db1 ~]# su - oracle
02.[oracle@db1 ~]$ sqlplus /nolog
03.SQL*Plus: Release 10.2.0.1.0 - Production on Wed Jul 4 13:50:22 2012
04.Copyright (c) 1982, 2005, Oracle. All rights reserved.
05.06.SQL> conn /as sysdba
07.Connected to an idle instance.
08. www.it165.net
09.[oracle@db1 dbs]$ tail -f /u01/app/oracle/admin/db1/bdump/alert_db1.log
10.Wed Jul 4 13:52:23 2012
11.ksvcreate: Process(q000) creation failed
12.Wed Jul 4 13:52:35 2012
13.Process q001 died, see its trace file
14.Wed Jul 4 13:52:35 2012
15.ksvcreate: Process(q001) creation failed
16.Wed Jul 4 13:52:37 2012
17.Process m000 died, see its trace file
18.Wed Jul 4 13:52:37 2012
19.ksvcreate: Process(m000) creation failed
refer:
上午刚到公司就接到客服电话，某系统看不到最近一小时以来的统计数据。看了下统计分析程序进程日志，报大量连接数据库失败错误，tnsping能通，怀疑数据库出了问题。 通过sqlplus本地登录数据库, sqlplus user/user，报“超过最大进程数”错误；以sysdba登录，提示连接到connect a idle instance，难道oracle没启动？检查oracle进程，后台进程都在，ps -elf | grep ora，进程总数有500多个，记得参数processes配置的300，正常情况下进程不会超过100个。故障有可能是进程超过数据库限制导致。 查看alert.log，在相应时间段发现大量： Process J002 died, see its trace file Mon May 9 07:30:15 2011 kkjcre1p: unable to spawn jobq slave process Mon May 9 07:30:15 2011 Errors in file /export/home/oracle/product/10.2.0/admin/abcdb/bdump/abcdb_cjq0_1052.trc 这个错误也证明了：因为数据库进程超过限制，造成后台进程j002无法派生新进程而僵死，导致数据库服务中断。 解决该问题需要找到什么应用产生了如此多的数据库进程。检查故障期间执行的应用，最后发现是昨晚应用升级，定义错了一个表的结构，凌晨开始向该表装载数据时报错，但程序有无限制的重试机制，不断产生新的数据库连接，最终超过processes限制。 重启数据库（无法登录数据库，只有先kill掉后台进程再重启），重新定义表后，故障消除。
二：于是想到监控oracle的进程和会话数来进一步确定问题 1：首先要对用户进行显示授权，否则后面创建存储过程编译将会报错
1.SQL> grant select on V_$SESSION to hr;
2.Grant succeeded.
3.4.SQL> grant select on V_$PROCESS to hr;
5.Grant succeeded.
2：建表，用来存储结果
1.SQL> create table session_monitor(time timestamp,session_count number,process_count number);
2.Table created.
3：创建存储过程，将数据插入表
01.SQL> create or replace procedure proc_session
02. 2 is
03. 3 v_session number(8);
04. 4 v_process number(8);
05. 5 begin
06. 6 select count(*) into v_session from v$session;
07. 7 select count(*) into v_process from v$process;
08. 8 insert into session_monitor values (sysdate,v_session,v_process);
09. 9 commit;
10. 10 end proc_session;
11.Procedure created.
4：创建任务
1.SQL> var job number;
2.SQL> begin
3. 2 sys.dbms_job.submit(job => :job,
4. 3 what => \'proc_session();\',
5. 4 next_date => sysdate,
6. 5 interval => \'sysdate+2/1440\');
7. 6* end;
8.PL/SQL procedure successfully completed.
5：测试效果
01.SQL> exec proc_session;
02.PL/SQL procedure successfully completed.
03.04.SQL> alter session set nls_date_format=\'YYYY-MM-DD-HH24:MI:SS\';
05.Session altered.
06.07.SQL> select * from session_monitor;
08.09.TIME SESSION_COUNT PROCESS_COUNT
10.---------------------------------------- ------------- -------------
11.26-JUL-12 03.02.12.000000 PM 140 155
12.26-JUL-12 03.02.14.000000 PM 141 157
13.14.SQL> select job,next_date from user_jobs where what=\'proc_session();\';
15.16. JOB NEXT_DATE
17.---------- -------------------
18. 145 2012-07-26-15:04:14
19.20.SQL> select * from session_monitor;
21.22.TIME SESSION_COUNT PROCESS_COUNT
23.---------------------------------------- ------------- -------------
24.26-JUL-12 03.04.14.000000 PM 87 94
25.26-JUL-12 03.02.12.000000 PM 140 155
26.26-JUL-12 03.02.14.000000 PM 141 157
6：如果要删除任务，则运行下列的命令，145代表user_jobs视图中的job列
1.SQL> begin
2. 2 dbms_job.remove(145);
3. 3 end;
4.PL/SQL procedure successfully completed.
三：使用nagios+fetion，定时去监控会话和进程数 1：创建监控脚本，该脚本放任务计划中运行，每2分钟自动执行
01.[root@server240 libexec]# cat session_oracle.sh
02.#!/bin/sh
03.rm -rf /tmp/session_oracle.log
04.export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db1
05./u01/app/oracle/product/11.2.0/db1/bin/sqlplus hr/hr@192.168.1.240:1521/orcl <<EOF
06.set echo off
07.set feedback off
08.spool /tmp/session_oracle.log
09.alter session set nls_date_format=\'YYYY-MM-DD:HH24:MI:SS\';
10.select session_count from (select * from session_monitor order by time desc ) where rownum=1;
11.select process_count from (select * from session_monitor order by time desc ) where rownum=1;
12.spool off
13.set echo on
14.set feedback on
2：创建第二脚本，用来处理前面监控脚本的日志输出，将结果返回给监控服务器
01.[root@server240 ~]# cat /tmp/session_oracle.log
02.SQL> alter session set nls_date_format=\'YYYY-MM-DD:HH24:MI:SS\';
03.SQL> select session_count from (select * from session_monitor order by time desc ) where rownum=1;
04.SESSION_COUNT
05.-------------
06. 138
07.SQL> select process_count from (select * from session_monitor order by time desc ) where rownum=1;
08.PROCESS_COUNT
09.-------------
10. 153
11.SQL> spool off
12.13.[root@server240 libexec]# cat check_oracle_session.sh
14.#!/bin/sh
15.STATE_OK=0
16.STATE_CRITICAL=2
17.18.if [ -f /tmp/session_oracle.log ];then
19. SESSION=$(grep -A 2 \'SESSION_COUNT\' /tmp/session_oracle.log |tail -1|sed \'s/[ ][ ]*//g\')
20. PROCESS=$(grep -A 2 \'PROCESS_COUNT\' /tmp/session_oracle.log |tail -1|sed \'s/[ ][ ]*//g\')
21. else
22. echo "something wrong,please check monitor script"
23. exit $STATE_CRITICAL
24.fi
25.26.if [ $SESSION -gt 500 ] || [ $PROCESS -gt 500 ];then
27. echo "Current session is $SESSION,process is $PROCESS "
28. exit $STATE_CRITICAL
29. else
30. echo "Current session is $SESSION,process is $PROCESS "
31. exit $STATE_OK
32.fi
四：实际效果 [root@server198 ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.240 -c check_oracle_session Current session is 138,process is 153
五：后期观察结果，估计和内存问题相关 [oracle@server240 ~]$ sar -r |grep 10:20 10:20:02 AM 3481492 21195164 85.89 675584 13661448 3683012 413552 10.10 [oracle@server240 ~]$ sar -r |grep 04:40 04:40:01 PM 2076748 22599908 91.58 734088 14581728 4048864 47700 1.16