Native crash (Tombstone) in android AOSP

hello Guys!!! hope you all are doing well.
Toady I am going to discuss about tombstone a native crash in Android framework. I am starting with my favourite question What..

What is tombstone?
Tombstone is a crash occurred in native c/c++ code of AOSP.

Why crash occurs in Native c/c++ Code?

When a process’ native code runs into a fatal error, the Android OS sends a fatal signal to the process, then dumps the process and finally terminates it.

Scenario

1. App have some native code and that code is generating some exception.
2. Background service or App is interacting with native side and stuck somewhere
3. Framework itself interacting with native side and get crashed

Visible Symptoms:

1. A foreground application disappears unexpectedly, without "ForceClose" or "ANR" dialog. 
2. If tombstone happened in system server process, the system server is restarted. Black screen is visible for couple of seconds; 
3. No visible symptoms if the application is in background.

Invisible Symptoms:

1. The “debuggerd” process creates a “/data/tombstones/tombstone_0X” (X: 0-9) file.
2. Dumps the crashing thread’s backtrace in the “main” log. The log tag is “DEBUG”, the keyword is “Build fingerprint”.
3. For “userdebug” and “eng” builds, a full core dump file can also be created in the “/sdcard/app_dump” folder.
4. Debuggerd will keep up to 10 tombstones, cycling through the numbers 00 to 09 and overwriting existing tombstones as necessary.

The tombstone informs about-

1. Crashed process and PIDs 
2. Terminated signal and fault address 
3. CPU registers 
4. Call Stack 
5. Stack content of each call
6. Build fingerprint

Tombstone Example

Main log:

10-17 16:10:09.085 10167 10167 I DEBUG : Build fingerprint: 'verizon/spyder_vzw/cdma_spyder:2.3.5/6.5.1_51/7:userdebug/test-keys'

10-17 16:10:09.085 10167 10167 I DEBUG : pid: 29911, tid: 29936 >>> com.android.browser <<<

10-17 16:10:09.085 10167 10167 I DEBUG : signal 7 (SIGBUS), code 128 (?), fault addr 00000000

/data/tombstones/tombstone_0X:

Build fingerprint: 'verizon/spyder_vzw/cdma_spyder:2.3.5/6.5.1_51/7:userdebug/test-keys'

pid: 29911, tid: 29936 >>> com.android.browser <<<

signal 7 (SIGBUS)*, code 128 (?), fault addr 00000000 tls: 452cdf00

r0 0000dd58 r1 012e5bbc r2 00000000 r3 00000000

r4 012e5b5c r5 afd42584 r6 00000003 r7 012e5b8c

r8 0000017f r9 ffffffff 10 00000002 fp ffffffff

ip 012e5b62 sp 452bcea8 lr 012e5b68 pc afd2f374 cpsr 80000030

d0 207373616c63202c d1 3b09094431090969

d2 0a7972697078656d d3 4d30330909090967

d4 40dcb000002c6e50 d5 0000000049000000

d6 000000050006e580 d7 0125fe8042c80000

d8 0000000000000000 d9 0000000000000000

d10 0000000000000000 d11 0000000000000000

*Unix Signal

SIGHUP	1	Exit Hang up
SIGINT	2	Exit Interrupt
SIGQUIT	3	Core Quit
SIGILL	4	Core Illegal Instruction
SIGTRAP	5	Core Trace/Breakpoint Trap
SIGABRT	6	Core Abort
SIGEMT	7	Core Emulation Trap
SIGFPE	8	Core Arithmetic Exception
SIGKILL	9	Exit Killed
SIGBUS	10	Core Bus Error
SIGSEGV	11	Core Segmentation Fault
SIGSYS	12	Core Bad System Call
SIGPIPE	13	Exit Broken Pipe

How to collect necessary logs

Tombstone file in /data/tombstones

adb pull /data/tombstones .

Full Coredump file in /sdcard/app_dump

adb pull /sdcard/app_dump .

AP logs (Both of the following approaches are OK)

Manually collected logs (not available in “user” build)

adb shell dmesg >kernel_log.txt

adb shell su ^^ /proc/kmsg >> kernel_log.txt

adb logcat –b main –b system –b radio –b events >logcat_logs.txt

Aplogd collected logs (not available in “user” build)

adb pull /sdcard/logger .

adb pull /data/logger .

Bugreport

adb bugreport >bugreport.txt

How to analyse tombstone

Necessary preconditions:

Get all AP logs (main/system/event/kernel/radio log) and Bugreport . Use the commands mention above to get all the logs from the phone.

1. Download symbol files , Arm, addr2line and gdb tool 
2. Install Android NDK http://developer.android.com/sdk/ndk/index.html ndk-stack, addr2line, gdb tools can be found in NDK  Or 
3. Find them from AOSP <working-dir> prebuilt/linux-86/toolchain folder 
4. Get tombstone and full core dump file if it’s generated 
5. Download the symbols.tar from jenkins/local build( if tombstone occurred in local build), for the corresponding build on which tombstone happened. 
6. Decode each line in the backtrace using addr2line tool. 

addr2line -C -f -e <path of symbols/system/lib>/<library_name> <address from backtraces>

Working Example

backtrace:

#00 pc 00044320 /system/lib/libc.so (tgkill+12)

#01 pc 00041f21 /system/lib/libc.so (pthread_kill+32)

#02 pc 0001ba4f /system/lib/libc.so (raise+10)

#03 pc 00018bf1 /system/lib/libc.so (__libc_android_abort+34)

#04 pc 000167b0 /system/lib/libc.so (abort+4)

#05 pc 0001a663 /system/lib/libc.so (__libc_fatal+16)

#06 pc 0001a67b /system/lib/libc.so (__fortify_chk_fail+18)

#07 pc 00016b4c /system/lib/libc.so (__memcpy_chk_fail+16)

#08 pc 00000849 /data/app/com.exe.jazz-1/lib/arm/libhellojava_jni.so (Java_com_exe_jazz_Hellojava_crash+36)

#09 pc 000ea8a9 /system/lib/libart.so (art_quick_generic_jni_trampoline+40)

#10 pc 000e61b1 /system/lib/libart.so (art_quick_invoke_stub_internal+64)

#11 pc 003eb4e3 /system/lib/libart.so (art_quick_invoke_stub+170)

#12 pc 007fc9ec [stack]

stack:

ff8827dc 00000000

ff8827e0 00000000

ff8827e4 00000000

Command:

<AOSP-working-dir>/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin$ ./arm-linux-androideabi-addr2line -C -f -e
<working-dir>/out/target/product/falcon/symbol/lib/libhellojava_jni.so 00000849

Output:

Java_com_sks_softsolution_Hellojava_crash

??:?

For more detail of addr2line

<aosp-working-dir>/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin$ arm-linux-androideabi-addr2line -help

Usage: arm-linux-androideabi-addr2line [option(s)] [addr(s)]
Convert addresses into line number/file name pairs.
If no addresses are specified on the command line, they will be read from stdin
The options are:

• @<file> Read options from <file> 
• -a --addresses Show addresses 
• -b --target=<bfdname> Set the binary file format 
• -e --exe=<executable> Set the input file name (default is a.out) 
• -i --inlines Unwind inlined functions 
• -j --section=<name> Read section-relative offsets instead of addresses 
• -p --pretty-print Make the output easier to read for humans 
• -s --basenames Strip directory names 
• -f --functions Show function names 
• -C --demangle[=style] Demangle function names 
• -h --help Display this information 
• -v --version Display the program's version

Happy Coding !!!
Thanks
Saurabh

Note:- Please proved your comment and suggestion for my post. It will energize me.

Building and Flashing AOSP for Device ( Nexus 5 hammerhead )

Hello Guys hope you are doing well!!!.
How to build AOSP for real device let say Nexus 5 (Hammerhead). In my previous blog, I had given info for AOSP download and build for Emulator. But building AOSP for a real device and flashing it is quite different. Today I am going to discuss it step by step.

Pre requisites:
Before proceeding for Downloading and building

1. GIT has to be installed
2. Make sure the following packages are installed in the system
- gcc-multilib                         --> sudo apt-get install gcc-multilib
- bison                                    --> sudo apt-get install bison
- flex                                        --> sudo apt-get install flex
- lib32z1 and lib32z1-dev         --> sudo apt-get install lib32z1 lib32z1-dev

Downloading and building:

Step 1. Choose Right Source code
Based on your available Test device and its binary file (device driver) , Download the respective source code. Please follow this link 

https://source.android.com/source/build-numbers.html#source-code-tags-and-builds

Open above link search for your device and then find the corresponding AOSP branch code. Let say for Nexus 5 (hammerhead) I have chosen Marshmallow (android-6.0.1_r77).

Step 2 Download the AOSP code
Please follow my previous blog for download.

Step 3. Download the binaries file
Open following link

https://developers.google.com/android/drivers#hammerheadm4b30z

and search for your device binaries. In my case I have Nexus 5 (hammerhead) and I have selected android-6.0.1_r77 this AOSP branch . So code Name is M4B30Z.
After successful download of device driver binary , extract it (say /u dir) and go to root folder of your AOSP code base. cd ../<AOSP code>/
Run this scripts from there.

• sh /u/extract-broadcom-hammerhead.sh
• sh /u/extract-qcom-hammerhead.sh
• sh /u/extract-lge-hammerhead.sh

run each command one by one  and hit enter slowly and at the end you have to enter  I Accept for T&C.  After successful this operation  check your <aosp code> folder there you find one new folder named as vendor. This folder contain all binaries for your device.

Step 4. Build the source code
Use following command for building. For more detail on build command follow my previous blog.
> source build/envsetup.sh
>lunch
> select your lunch argument. For me it is 19 (aosp_hammerhead-userdebug).
>make -j4 (or -j16 based on your RAM and CPU core)
Once the build is complete, the system.img file can be found at out/target/product/hammerhead

Step 5. Flash Image into device
After building successful system image, We can flash now. Connect your Test device via USB cable and give following command in your terminal

1. <aosp-working-dir> ./adb reboot bootloader 
2. <aosp-working-dir> ./fastboot flash boot /path_of_your_boot_image 
3. <aosp-working-dir> ./fastboot flash system /path_of_your_system.img 
4. <aosp-working-dir> ./fastboot flash userdata /path_of_your_userdata.img 
5. <aosp-working-dir> ./fastboot flash recovery /path_of_your_recovery.img 
6. <aosp-working-dir> .fastboot reboot 

Happy Coding !!!
Thanks
Saurabh

Note:- Please proved your comment and suggestion for my post. It will energize me.

Bound services in Android, Types and uses

A bound service is the server in a client-server interface. It allows components (such as activities) to bind to the service, send requests, receive responses, and perform interprocess communication (IPC).
A bound service typically lives only while it serves another application component and does not run in the background indefinitely.
Before Going further detail of Bound Services. Let Start with a basic Question
“What is Service in android and its types.”
As per Android developer site, Service is one of the basic application component and Its duty is to perform long-running operations in the background. Service has no UI.
Type of services in Android

1. Scheduled

• JobScheduler API(introduced in Android 5.0 API level 21) used to schedule a service for execution at the appropriate times. 
• We can use the JobScheduler by registering jobs and specifying their requirements for network and timing. 
• The system then gracefully schedules the jobs 

For more detail on JobScheduler visit my previous blogs on it.
2. Started

• A service is started when any application component (activity, Broadcast etc.) calls startService() method. 
• Started service can run in the background indefinitely, even if the component that started it is destroyed. 
• Generally a started service performs a single operation and does not return a result to the caller. 

As for example, download or upload a file over the network. When the operation is complete, the service should stop itself.
3. Bound

• A service is bound when any application component binds to it by calling bindService() method 
• A bound service offers a client-server interface that allows components to interact with the service 
• A bound service runs only as long as another application component is bound to it 
• Multiple components can bind to the service at once, but when all of them unbind, the service is destroyed

Important Point:

• A service runs in the main thread of its hosting process. 
• The service by default does not create its own thread and does not run in a separate process unless you specify it. 
• For CPU-intensive work(MP3 playback or network related work), you should create a new thread within the service to complete that work. 
• By using a separate thread, you can reduce the risk of Application Not Responding (ANR) errors. 

Today Here I am going to discuss bound service only. I will wrote a separate blog for Started Service.
Implementation Steps

1. A bound service is an implementation of the Service class that allows other applications to bind to it and interact with it. 
2. To provide binding for a service, you must implement the onBind() callback method. 
3. clients use IBinder object (return by onBind() method) to interact with the service. 
4. A client binds to a service by calling bindService(). 
5. Client must provide an implementation of ServiceConnection, which monitors the connection with the service. 
6. The bindService() method returns immediately without a value 
7. Android system creates the connection between the client and service and it calls onServiceConnected() on the ServiceConnection, to deliver the IBinder that the client can use to communicate with the service. 
8. Multiple clients can call bindService() simultaneously. But Android system calls our service's onBind() method to retrieve the IBinder only once when the first client binds. 
9. After that Android system delivers the same IBinder to any additional clients that bind, without calling onBind() again. 
10. When the last client unbinds from the service, the system destroys the service, unless the service was also started by startService().

Creating a bound service

• For creating a bound service We must have to provide an IBinder that provides the programming interface 
• clients use this interface to interact with the service. 
• There are three ways you can define the interface:

1. Extending the Binder class 
2. Using a Messenger 
3. Using AIDL

Extending the Binder class 

• If our service is private to our own application and runs in the same process as the client 
• In that case we create our interface by extending the Binder class and returning an instance of it from onBind() 
• The client receives the Binder and can use it to directly access public methods available in either the Binder implementation or the Service. 

This is the preferred technique when our service is merely a background worker for our own application.

Using a Messenger

• If we need our interface to work across different processes 
• The service defines a Handler that responds to different types of Message objects. 
• This Handler is the basis for a Messenger that can then share an IBinder with the client, allowing the client to send commands to the service using Message objects. 
• The client can define a Messenger of its own, so the service can send messages back

This is the simplest way to perform interprocess communication (IPC), because the Messenger queues all requests into a single thread so that We don't have to design our service to be thread-safe.

Using AIDL

• Android Interface Definition Language (AIDL) decomposes objects into primitives that the operating system can understand and marshals them across processes to perform IPC. 
• For using AIDL our service must be thread-safe and capable of multi-threading.

Diff b/w AIDL and Messenger
Messenger is actually based on AIDL as its underlying structure.
The main Difference is :-
Messenger creates a queue of all the client requests in a single thread, so the service receives requests one at a time
But AIDL handle multiple requests simultaneously

For more detail on AIDL check my previous blog on AIDL.

Example Code of Binder.....(Coming soon...)

Example Code of Messenger.....(coming soon..)

AIDL and its uses in Android

Hello Guys !!! Hope You all are doing well. Today I am going to discuss AIDL and its uses in Android Application. 

AIDL (Android Interface Definition Language) is similar to other IDLs.

It allows us to define the programming interface that both the client and service agree upon in order to communicate with each other using interprocess communication (IPC).

On Android, one process cannot normally access the memory of another process. So to talk, they need to decompose their objects into primitives that the operating system can understand, and marshall the objects across that boundary for you. The code to do that marshalling is tedious to write, so Android handles it for us with AIDL.

# When to use AIDL

1. Clients from different applications want to access your service for IPC.
2. Your service want to handle multithreading for IPC(any method defined in service can be executed simultaneously by more than one application).
3. If you want to share data and control something in another application.
4. You want to create some new functionalities and distribute them as a library.

# AIDL Implementation Steps

In order to allow for one application to call into another, we typically have to:

1. Define the AIDL interface
2. Implement the Stub for the remote service
3. Expose the remote service to the local client

# The data types supported by AIDL

• All primitive types (Supported by Java such as int, long, char, boolean, and so on)
• String
• CharSequence
• List(must be one of the supported data types or parcelables)
• Generic List such as List<String> supported
• Map (must be one of the supported data types or parcelables )
• Generic maps, such as Map<String,Integer> are not supported

# Defining the AIDL

• AIDL syntax is very similar to regular Java interface , Define AIDL interface in an .aidl
• Then save it in the source code (in the src/ directory) of both the application hosting the service and any other application that binds to the service.
• When you build each application that contains the .aidl file, the Android SDK tools generate an IBinder interface based on the .aidl file and save it in the project's gen/ directory.

# Steps for creating a bounded service using AIDL

1. Create the .aidl file

This file defines the programming interface with method signatures.

2. Implement the interface

The Android SDK tools generate an interface in the Java programming language, based on your .aidl file. This interface has an inner abstract class named Stub that extends Binder and implements methods from your AIDL interface. You must extend the Stub class and implement the methods.

3. Expose the interface to clients

Implement a Service and override onBind() to return your implementation of the Stub class.

Note:

1.   Any changes that you make to your AIDL interface after your first App release must remain backward compatible in order to avoid breaking other applications that use your service.
2. That is, because your .aidl file must be copied to other applications in order for them to access your service's interface, you must maintain support for the original interface.

Example Code on Github 

In above example, I am using two application Client App and Server App.  Client App will get number from user and send it to server.
Server App will receive number from client and perform addition and send result back to client App.

In my example code App is server App and app2 is client App.
Hope you people enjoy it. Flow Diagram for AIDL.

Happy Coding !!!

Thanks
Saurabh

Note:- Please proved your comment and suggestion for my post. It will energize me.

Understanding of Android Build system

Hello Guys !!! Hope you all doing well. Today I am going to discuss AOSP build system.

Android Build system is a part of the AOSP source code. How to download and build AOSP you can follow my previous blog link.

• Android Build system used to compile the Android system, Android SDK and related package and documents.
• Android Build system is mainly contain Make files, Shell scripts and Python scripts. The most important one is Make files.
• From Android 6 (M, API 23) AOSP master platform builds with ninja instead ofGNU make and in Android 7 (N, API 24, 25) , ninja and kati become default build system.
• The build/core makefiles and Android.mk files are read by kati which will interpret them and write out a$OUT_DIR/build_$(TARGET_file, and then the execution of the build rules will be handled by ninja.

Building with kati and ninja provides some immediate benefits:

1. Faster start-up time for new builds. Running kati and then ninja results in builds starting about 25% faster than GNU make.
2. Much faster incremental builds. kati caches the mtimes of files and directories it read, what environment variables it used, etc., and only re-reads all the makefiles if something has changed. This includes globs like $(call all-files-under). Incremental builds with a warm disk cache will start within 5 seconds.
3. Better build command output. Ninja buffers the output of each command and prints it without mixing output from multiple commands, meaning error messages will be printed as a group after a build failure, and not mixed with other concurrent build output. Ninja also prints a configurable status line that defaults to printing the percentage of build commands that have been run, the number that have been run, and the number that need to run, giving a reasonable approximation of the progress through the build.
4. Incremental builds should be more reliable. Ninja considers outputs whose command line has changed to be dirty, which will catch changes to global cflags, etc. and rebuild anything that has changed.
5. The use of ninja should be transparent, it is wrapped by the top level Makefile, so all make-based commands should continue to work.

For more detail you can follow following link

https://android.googlesource.

https://github.com/google/

https://android.googlesource.

https://ninja-build.org/

https://android-review.

https://groups.google.com/

https://www.xda-developers.

The build system uses some pre-set environment variables and a series of 'make' files in order to build an Android system and prepare it for deployment to a platform.

Step by step detail of AOSP build System

1. Types of tools used by AOSP build system

• During the build we use 'make' to control the build steps themselves.
• A host toolchain (compiler, linker and other tools) and libraries will be used to build programs and tools that will run on the host.
• A different toolchain is used to compile the C and C++ code that will wind up on the target (which is an embedded board, device or the emulator).
• The kernel is compiled as a standalone binary (it does not use a program loader or link to any outside libraries).
• Native programs (e.g. init or toolbox), daemons or libraries will link against bionic or other system libraries.
• A Java compiler and a bunch of java-related tools to build most of the application framework, system services and Android applications themselves.
• Finally, tools are used to package the applications and resource files, and to create the filesystem images that can be installed on a device or used with the Emulator.

2. Setup "what to build"

• In order to decide what to build, and how to build it, the build system requires that some variables be set.
• Different products, with different packages and options can be built from the same source tree.
• The variables to control this can be set via a file with declarations of 'make' variables, or can be specified in the environment.
• A device vendor can create definition files that describe what is to be included on a particular board or for a particular product. The definition file is called: buildspec.mk, and it is located in the top-level source directory.
• Another method of specifying options is to set environment variables. To set up your build environment, you need to load the variables and function in build/envsetup.sh. your can set shell environment, like this

$ . build/envsetup.sh

or

$ source build/envsetup.sh

You can type 'help' (or 'hmm') at this point to see some utility functions that are available to make it easier to work with the source.

3. Building the system

Once We have things set up, you actually build the system with the 'make' command. Before issuing the make command we have to put lunch command in our terminal.

• lunch command provide a list of available build types.
• It will show both non-official or non-standard builds for your device
• It includes special debug versions and also allows you to build for Android Emulator

After lunch command, We have to use 'make' command for final AOSP building.

• $ make -j4 // Based on your CPU core and RAM you can give -j8, -j16 etc
• $ make -j4 showcommands // It will show build process used by AOSP

4. Make targets

List of different make targets we can use to build different parts of the system

• make sdk- build the tools that are part of an SDK (adb, fastboot, etc.)
• make snod- build the system image from the current software binaries
• make services
• make runtime
• make all- make everything, whether it is included in the product definition or not
• make clean- remove all built files (prepare for a new build). Same as rm -rf out/<configuration>/
• make modules- shows a list of submodules that can be built (List of all LOCAL_MODULE definitions)
• make update-api- To update the framework and native libs during AOSP build

5. Helper macros and functions

There are some helper macros and functions that are installed when we set envsetup.sh. Following are most used one :-

• m- execute 'make' from the top of the tree (even if your current directory is somewhere else)
• mm- Builds all of the modules in the current directory, but not their dependencies
• mmm <dir1>- Builds all of the modules in the supplied directories, but not their dependencies
• mma- Builds all of the modules in the current directory, and their dependencies.
• mmma <dir>- Builds all of the modules in the supplied directories, and their dependencies
• mm -B : Rebuild the module even it hasn’t any changes.
• cgrep <pattern>- grep on all local C/C++ files
• jgrep <pattern>- grep on all local Java files
• resgrep <pattern>- grep on all local res/*.xml files
• godir <filename>- go to the directory containing a file

6. Syncing the changes directly

If you make changes in some file and you want to update this changes in your connected device or emulator you can do it easily. Example:-

we can do our change on the PhoneStatusBar.java file and run the following commands to update the device:

• $ mmm frameworks/base/packages/
• $ adb sync // Sync the updated module to the device
• $ adb shell stop // Stopping the shell
• $ adb shell start // Starting the shell again

And all that will take less than 30 seconds!

Hope Guys!!! You enjoyed this articles.

Happy Coding!!!

Thanks

Saurabh