Tuesday 4 October 2011

Trace Core file by STACKX script

I posted "How to debug core dump for a Process ID?". I felt curious what I can do with core dump file from oracle's process. 
Thank You Roger Snowden. He suggested me stackx script. STACKX User Guide [ID 362791.1] 
Description: http://i.ixnp.com/images/v6.59/t.gif
 
stackx is a shell script utility for extracting diagnostic stack trace information from core file on UNIX/LINUX platforms.

I downloaded and tested it(just test on 10g with real core dump file). 
$ ls stackx1_2.tar 
stackx1_2.tar 

$ 
tar xf stackx1_2.tar 

$ 
ls stackx 
stackx 

$ 
./stackx 
Wrong number of parameters. 
Usage: stackx [debugger] <corefilename> 


$ 
./stackx /oracle/product/admin/orcl/cdump/core_29158/core.29158 
###################################################################### 
stackx core / stack trace extraction utility 
version 1.2 (c) 2005, 2006, 2007 Oracle Corp. all rights reserved. 
Reporting: Mon Aug 15 00:49:24 ICT 2011 
Machine : Linux corcl01 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:32:02 EDT 2006 x86_64 x86_64 x86_64 GNU/Linux 
CPU Bits : 32 
Core file: /oracle/product/admin/orcl/cdump/core_29158/core.29158 
Core date: Oct 16 2010 
Tool(s) : /usr/bin/gdb 
Program : oracle 
Core extraction section follows 
###################################################################### 
For: /oracle/product/10.2.0/db_1/bin/oracle executable file found in core, 
a core extraction will be attempted... 
<debugging session> 
(no debugging symbols found) 
Using host libthread_db library "/lib64/tls/libthread_db.so.1". 
Core was generated by `ora_j002_orcl1'. 
Program terminated with signal 6, Aborted. 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libskgxp10.so...(no debugging symbols found)...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libskgxp10.so 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libhasgen10.so...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libhasgen10.so 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libskgxn2.so...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libskgxn2.so 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libocr10.so...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libocr10.so 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libocrb10.so...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libocrb10.so 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libocrutl10.so...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libocrutl10.so 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libjox10.so...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libjox10.so 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libclsra10.so...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libclsra10.so 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libdbcfg10.so...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libdbcfg10.so 
Reading symbols from /oracle/product/10.2.0/db_1/lib/libnnz10.so...done. 
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libnnz10.so 
Reading symbols from /usr/lib64/libaio.so.1...done. 
Loaded symbols for /usr/lib64/libaio.so.1 
Reading symbols from /lib64/libdl.so.2...done. 
Loaded symbols for /lib64/libdl.so.2 
Reading symbols from /lib64/tls/libm.so.6...done. 
Loaded symbols for /lib64/tls/libm.so.6 
Reading symbols from /lib64/tls/libpthread.so.0...done. 
Loaded symbols for /lib64/tls/libpthread.so.0 
Reading symbols from /lib64/libnsl.so.1...done. 
Loaded symbols for /lib64/libnsl.so.1 
Reading symbols from /lib64/tls/libc.so.6...done. 
Loaded symbols for /lib64/tls/libc.so.6 
Reading symbols from /lib64/ld-linux-x86-64.so.2...done. 
Loaded symbols for /lib64/ld-linux-x86-64.so.2 
Reading symbols from /opt/oracle/extapi/64/asm/orcl/1/libasm.so...done. 
Loaded symbols for /opt/oracle/extapi/64/asm/orcl/1/libasm.so 
#0 0x000000328642e829 in kill () from /lib64/tls/libc.so.6 
(gdb) (gdb) <stack> 

Thread 1 (process 29158): 
#0 0x000000328642e829 in kill () from /lib64/tls/libc.so.6 
#1 0x00000000022e21e4 in slcra () 
#2 0x0000000000745aab in ssexhd () 
#3 <signal handler called> 
#4 0x0000000002323312 in kpnfch () 
#5 0x0000000002390911 in opifch2 () 
#6 0x000000000238e700 in opifch () 
#7 0x00000000023bc6ce in opipls () 
#8 0x000000000074c338 in opiodr () 
#9 0x0000000000751aea in rpidrus () 
#10 0x0000000003c69316 in skgmstack () 
#11 0x0000000000751e34 in rpidru () 
#12 0x0000000000750f2c in rpiswu2 () 
#13 0x00000000007507bd in rpidrv () 
#14 0x00000000031cbe56 in psddr0 () 
#15 0x00000000031cbbdb in psdnal () 
#16 0x0000000004573411 in pevm_BFTCHC () 
#17 0x0000000004529f05 in pfrinstr_FTCHC () 
#18 0x00000000045248c9 in pfrrun_no_tool () 
#19 0x0000000004523312 in pfrrun () 
#20 0x000000000456086d in plsql_run () 
#21 0x000000000450c7bc in peicnt () 
#22 0x0000000003fc1d50 in kkxexe () 
#23 0x00000000023b3f93 in opiexe () 
#24 0x0000000002325c2c in kpoal8 () 
#25 0x000000000074c338 in opiodr () 
#26 0x0000000002359999 in kpoodr () 
#27 0x00000000038b315e in upirtrc () 
#28 0x000000000382a2a3 in kpurcsc () 
#29 0x00000000037d3266 in kpuexecv8 () 
#30 0x00000000037d0a67 in kpuexec () 
#31 0x00000000038858ef in OCIStmtExecute () 
#32 0x0000000001cd8e65 in jslvec_execcb () 
#33 0x0000000001cd1792 in jslvswu () 
#34 0x0000000001cc84a2 in jslve_execute0 () 
#35 0x0000000001cc7936 in jslve_execute () 
#36 0x0000000000750f2c in rpiswu2 () 
#37 0x0000000001c297cf in kkjex1e () 
#38 0x0000000001c28f17 in kkjsexe () 
#39 0x0000000001c284c4 in kkjrdp () 
#40 0x0000000002380a18 in opirip () 
#41 0x0000000000746086 in opidrv () 
#42 0x000000000074450e in sou2o () 
#43 0x000000000070a565 in opimai_real () 
#44 0x000000000070a41c in main () 
#0 0x000000328642e829 in kill () from /lib64/tls/libc.so.6 
#1 0x00000000022e21e4 in slcra () 
#2 0x0000000000745aab in ssexhd () 
#3 <signal handler called> 
#4 0x0000000002323312 in kpnfch () 
#5 0x0000000002390911 in opifch2 () 
#6 0x000000000238e700 in opifch () 
#7 0x00000000023bc6ce in opipls () 
#8 0x000000000074c338 in opiodr () 
#9 0x0000000000751aea in rpidrus () 
#10 0x0000000003c69316 in skgmstack () 
#11 0x0000000000751e34 in rpidru () 
#12 0x0000000000750f2c in rpiswu2 () 
#13 0x00000000007507bd in rpidrv () 
#14 0x00000000031cbe56 in psddr0 () 
#15 0x00000000031cbbdb in psdnal () 
#16 0x0000000004573411 in pevm_BFTCHC () 
#17 0x0000000004529f05 in pfrinstr_FTCHC () 
#18 0x00000000045248c9 in pfrrun_no_tool () 
#19 0x0000000004523312 in pfrrun () 
#20 0x000000000456086d in plsql_run () 
#21 0x000000000450c7bc in peicnt () 
#22 0x0000000003fc1d50 in kkxexe () 
#23 0x00000000023b3f93 in opiexe () 
#24 0x0000000002325c2c in kpoal8 () 
#25 0x000000000074c338 in opiodr () 
#26 0x0000000002359999 in kpoodr () 
#27 0x00000000038b315e in upirtrc () 
#28 0x000000000382a2a3 in kpurcsc () 
#29 0x00000000037d3266 in kpuexecv8 () 
#30 0x00000000037d0a67 in kpuexec () 
#31 0x00000000038858ef in OCIStmtExecute () 
#32 0x0000000001cd8e65 in jslvec_execcb () 
#33 0x0000000001cd1792 in jslvswu () 
#34 0x0000000001cc84a2 in jslve_execute0 () 
#35 0x0000000001cc7936 in jslve_execute () 
#36 0x0000000000750f2c in rpiswu2 () 
#37 0x0000000001c297cf in kkjex1e () 
#38 0x0000000001c28f17 in kkjsexe () 
#39 0x0000000001c284c4 in kkjrdp () 
#40 0x0000000002380a18 in opirip () 
#41 0x0000000000746086 in opidrv () 
#42 0x000000000074450e in sou2o () 
#43 0x000000000070a565 in opimai_real () 
#44 0x000000000070a41c in main () 
</stack> 
<registers> 
rax 0x0 0 
rbx 0x1 1 
rcx 0xffffffffffffffff -1 
rdx 0x71e6 29158 
rsi 0x6 6 
rdi 0x71e6 29158 
rbp 0x2a9715aa40 0x2a9715aa40 
rsp 0x2a9715a2d8 0x2a9715a2d8 
r8 0x0 0 
r9 0x0 0 
r10 0x2a9715a201 182923403777 
r11 0x206 518 
r12 0xbdb71760 3182892896 
r13 0x7 7 
r14 0x7b 123 
r15 0x2a9715ac40 182923406400 
rip 0x328642e829 0x328642e829 <kill+9> 
eflags 0x206 518 
cs 0x33 51 
ss 0x2b 43 
ds 0x0 0 
es 0x0 0 
fs 0x0 0 
gs 0x0 0 
st0 0 (raw 0x00000000000000000000) 
st1 0 (raw 0x00000000000000000000) 
st2 0 (raw 0x00000000000000000000) 
st3 0 (raw 0x00000000000000000000) 
st4 0 (raw 0x00000000000000000000) 
st5 0 (raw 0x00000000000000000000) 
st6 0 (raw 0x00000000000000000000) 
st7 0 (raw 0x00000000000000000000) 
fctrl 0x37f 895 
fstat 0x0 0 
ftag 0xffff 65535 
fiseg 0x0 0 
fioff 0x0 0 
foseg 0x0 0 
fooff 0x0 0 
fop 0x0 0 
xmm0 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
xmm1 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
xmm2 {f = {0x0, 0x0, 0x0, 0x0}} {f = {4.34402524e-44, 3.36311631e-44, 0, 0}} 
xmm3 {f = {0x0, 0x0, 0x0, 0x0}} {f = {3.50324616e-44, 8.40779079e-45, 0, 0}} 
xmm4 {f = {0x0, 0x0, 0xc, 0x0}} {f = {6.40969056e-10, 1.14785479e-08, 12.3875132, 2.05823494e-06}} 
xmm5 {f = {0x304d, 0x0, 0x31, 0x0}} {f = {12365.0557, 0.00067597907, 49.5529366, 7.21062543e-10}} 
xmm6 {f = {0x0, 0x0, 0x0, 0x0}} {f = {1.35631564e-19, 1.3571458e-19, 4.12584313e-08, 6.40982378e-10}} 
xmm7 {f = {0x0, 0xa03420, 0x81314c00, 0x0}} {f = {1.35701254e-19, 10499104, 1.07574262e+10, 1.7984663e-19}} 
xmm8 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
xmm9 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
xmm10 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
xmm11 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
xmm12 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
xmm13 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
xmm14 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
xmm15 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}} 
mxcsr 0x1f80 8064 
</registers> 
</debugging session> 
###################################################################### 
stackx done.
Note: 
Core stack trace extraction is supported on the following platforms: 
- IBM AIX 
- Intel Linux (RedHat and SuSE) 
- HP-UX (10.* and 11.*) 
- HP Tru64 
- Sun Solaris (2.5 - 2.10) 

Debuggers recognized and discovered by stackx are, in order of preferential priority: 
- pstack (Solaris only) 
- gdb 
- dbx 
- adb 
- sdb 

It's easy with this script. 

No comments: