Native crash (Tombstone) in android AOSP

hello Guys!!! hope you all are doing well.
Toady I am going to discuss about tombstone a native crash in Android framework. I am starting with my favourite question What..

What is tombstone?
Tombstone is a crash occurred in native c/c++ code of AOSP.

Why crash occurs in Native c/c++ Code?

When a process’ native code runs into a fatal error, the Android OS sends a fatal signal to the process, then dumps the process and finally terminates it.

Scenario

1. App have some native code and that code is generating some exception.
2. Background service or App is interacting with native side and stuck somewhere
3. Framework itself interacting with native side and get crashed

Visible Symptoms:

1. A foreground application disappears unexpectedly, without "ForceClose" or "ANR" dialog. 
2. If tombstone happened in system server process, the system server is restarted. Black screen is visible for couple of seconds; 
3. No visible symptoms if the application is in background.

Invisible Symptoms:

1. The “debuggerd” process creates a “/data/tombstones/tombstone_0X” (X: 0-9) file.
2. Dumps the crashing thread’s backtrace in the “main” log. The log tag is “DEBUG”, the keyword is “Build fingerprint”.
3. For “userdebug” and “eng” builds, a full core dump file can also be created in the “/sdcard/app_dump” folder.
4. Debuggerd will keep up to 10 tombstones, cycling through the numbers 00 to 09 and overwriting existing tombstones as necessary.

The tombstone informs about-

1. Crashed process and PIDs 
2. Terminated signal and fault address 
3. CPU registers 
4. Call Stack 
5. Stack content of each call
6. Build fingerprint

Tombstone Example

Main log:

10-17 16:10:09.085 10167 10167 I DEBUG : Build fingerprint: 'verizon/spyder_vzw/cdma_spyder:2.3.5/6.5.1_51/7:userdebug/test-keys'

10-17 16:10:09.085 10167 10167 I DEBUG : pid: 29911, tid: 29936 >>> com.android.browser <<<

10-17 16:10:09.085 10167 10167 I DEBUG : signal 7 (SIGBUS), code 128 (?), fault addr 00000000

/data/tombstones/tombstone_0X:

Build fingerprint: 'verizon/spyder_vzw/cdma_spyder:2.3.5/6.5.1_51/7:userdebug/test-keys'

pid: 29911, tid: 29936 >>> com.android.browser <<<

signal 7 (SIGBUS)*, code 128 (?), fault addr 00000000 tls: 452cdf00

r0 0000dd58 r1 012e5bbc r2 00000000 r3 00000000

r4 012e5b5c r5 afd42584 r6 00000003 r7 012e5b8c

r8 0000017f r9 ffffffff 10 00000002 fp ffffffff

ip 012e5b62 sp 452bcea8 lr 012e5b68 pc afd2f374 cpsr 80000030

d0 207373616c63202c d1 3b09094431090969

d2 0a7972697078656d d3 4d30330909090967

d4 40dcb000002c6e50 d5 0000000049000000

d6 000000050006e580 d7 0125fe8042c80000

d8 0000000000000000 d9 0000000000000000

d10 0000000000000000 d11 0000000000000000

*Unix Signal

SIGHUP	1	Exit Hang up
SIGINT	2	Exit Interrupt
SIGQUIT	3	Core Quit
SIGILL	4	Core Illegal Instruction
SIGTRAP	5	Core Trace/Breakpoint Trap
SIGABRT	6	Core Abort
SIGEMT	7	Core Emulation Trap
SIGFPE	8	Core Arithmetic Exception
SIGKILL	9	Exit Killed
SIGBUS	10	Core Bus Error
SIGSEGV	11	Core Segmentation Fault
SIGSYS	12	Core Bad System Call
SIGPIPE	13	Exit Broken Pipe

How to collect necessary logs

Tombstone file in /data/tombstones

adb pull /data/tombstones .

Full Coredump file in /sdcard/app_dump

adb pull /sdcard/app_dump .

AP logs (Both of the following approaches are OK)

Manually collected logs (not available in “user” build)

adb shell dmesg >kernel_log.txt

adb shell su ^^ /proc/kmsg >> kernel_log.txt

adb logcat –b main –b system –b radio –b events >logcat_logs.txt

Aplogd collected logs (not available in “user” build)

adb pull /sdcard/logger .

adb pull /data/logger .

Bugreport

adb bugreport >bugreport.txt

How to analyse tombstone

Necessary preconditions:

Get all AP logs (main/system/event/kernel/radio log) and Bugreport . Use the commands mention above to get all the logs from the phone.

1. Download symbol files , Arm, addr2line and gdb tool 
2. Install Android NDK http://developer.android.com/sdk/ndk/index.html ndk-stack, addr2line, gdb tools can be found in NDK  Or 
3. Find them from AOSP <working-dir> prebuilt/linux-86/toolchain folder 
4. Get tombstone and full core dump file if it’s generated 
5. Download the symbols.tar from jenkins/local build( if tombstone occurred in local build), for the corresponding build on which tombstone happened. 
6. Decode each line in the backtrace using addr2line tool. 

addr2line -C -f -e <path of symbols/system/lib>/<library_name> <address from backtraces>

Working Example

backtrace:

#00 pc 00044320 /system/lib/libc.so (tgkill+12)

#01 pc 00041f21 /system/lib/libc.so (pthread_kill+32)

#02 pc 0001ba4f /system/lib/libc.so (raise+10)

#03 pc 00018bf1 /system/lib/libc.so (__libc_android_abort+34)

#04 pc 000167b0 /system/lib/libc.so (abort+4)

#05 pc 0001a663 /system/lib/libc.so (__libc_fatal+16)

#06 pc 0001a67b /system/lib/libc.so (__fortify_chk_fail+18)

#07 pc 00016b4c /system/lib/libc.so (__memcpy_chk_fail+16)

#08 pc 00000849 /data/app/com.exe.jazz-1/lib/arm/libhellojava_jni.so (Java_com_exe_jazz_Hellojava_crash+36)

#09 pc 000ea8a9 /system/lib/libart.so (art_quick_generic_jni_trampoline+40)

#10 pc 000e61b1 /system/lib/libart.so (art_quick_invoke_stub_internal+64)

#11 pc 003eb4e3 /system/lib/libart.so (art_quick_invoke_stub+170)

#12 pc 007fc9ec [stack]

stack:

ff8827dc 00000000

ff8827e0 00000000

ff8827e4 00000000

Command:

<AOSP-working-dir>/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin$ ./arm-linux-androideabi-addr2line -C -f -e
<working-dir>/out/target/product/falcon/symbol/lib/libhellojava_jni.so 00000849

Output:

Java_com_sks_softsolution_Hellojava_crash

??:?

For more detail of addr2line

<aosp-working-dir>/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin$ arm-linux-androideabi-addr2line -help

Usage: arm-linux-androideabi-addr2line [option(s)] [addr(s)]
Convert addresses into line number/file name pairs.
If no addresses are specified on the command line, they will be read from stdin
The options are:

• @<file> Read options from <file> 
• -a --addresses Show addresses 
• -b --target=<bfdname> Set the binary file format 
• -e --exe=<executable> Set the input file name (default is a.out) 
• -i --inlines Unwind inlined functions 
• -j --section=<name> Read section-relative offsets instead of addresses 
• -p --pretty-print Make the output easier to read for humans 
• -s --basenames Strip directory names 
• -f --functions Show function names 
• -C --demangle[=style] Demangle function names 
• -h --help Display this information 
• -v --version Display the program's version

Happy Coding !!!
Thanks
Saurabh

Note:- Please proved your comment and suggestion for my post. It will energize me.