asterisk-debug.md

# Asterisk Debug

This document describes how to debug and trouble shoot Asterisk and its modules, especially for crashes with segmentation fault.

## Build Asterisk with Debug Information

GDB can be used to analyze stacks when a crash occurs. In order to let gdb provide sufficient stack trace, asterisk and its modules must have debug information not being stripped.

Here comes procedures to build asterisk with as much debug information as possible.

### Enable package gdb

Run ```make menuconfig``` and enable ```Development → gdb```

```
~/git/iopsyswrt$ grep gdb .config
CONFIG_PACKAGE_gdb=y
```

gdb will be installed and running on the target directly.

### Append libbfd as the dependency

This package is requred when enabling more debug options for Asterisk.

```
diff --git a/net/asterisk/Makefile b/net/asterisk/Makefile
index 824b9e7..ab85f07 100644
--- a/net/asterisk/Makefile
+++ b/net/asterisk/Makefile
@@ -463,7 +463,7 @@ $(call Package/$(PKG_NAME)/Default)
   TITLE:=Complete open source PBX, v$(PKG_VERSION)
   MENU:=1
   DEPENDS:=+libstdcpp +jansson +libcap +libedit +libopenssl +libsqlite3 +libuuid +libxml2 +zlib \
-          +libncurses +libpopt +libxslt +ubus +libpicoevent
+          +libncurses +libpopt +libxslt +ubus +libpicoevent +libbfd
 endef
 
 define Package/$(PKG_NAME)/description

```

### Build An Entire Firmware as Usual First

Note that this step is absolutely mandatory. Otherwise the size of the firmware might be too big to fit the target when strip option is disabled in the later steps.

### Disable Strip Option

The debug information of all executable and shared object files on the target are stripped. That causes gdb not able to show useful stack information in most scenarios. So asterisk and its modules must have debug information not being stripped.

 
Run ```make menuconfig``` and check ```Global build settings → Binary stripping method → none```

```
~/git/iopsyswrt$ grep -i strip .config
# CONFIG_KERNEL_PROC_STRIPPED is not set
# Stripping options
CONFIG_NO_STRIP=y
# CONFIG_USE_STRIP is not set
# CONFIG_USE_SSTRIP is not set
# CONFIG_STRIP_KERNEL_EXPORTS is not set
```

### Enable Debug Options in Asterisk

Enter the directory of package asterisk in iopsyswrt/build_dir and run ```make menuselect```. 
Go to ```Compiler Flags```. Enable the followings and press S to save the settings.

```
   [*] DONT_OPTIMIZE
   [*] COMPILE_DOUBLE
   [*] DEBUG_THREADS
   [ ] DEBUG_FD_LEAKS
   [*] BETTER_BACKTRACES
   [ ] LOTS_OF_SPANS
   [*] MALLOC_DEBUG
   [*] DEBUG_CHAOS
   XXX BUILD_NATIVE
       --- Extended ---
   [*] REF_DEBUG
   [*] AO2_DEBUG
   [ ] REBUILD_PARSERS
   [ ] LOW_MEMORY
   [*] DISABLE_INLINE
   [*] OPTIONAL_API
   XXX USE_HOARD_ALLOCATOR
   [ ] RADIO_RELAX
   [ ] G711_NEW_ALGORITHM
   < > G711_REDUCED_BRANCHING
   < > TEST_CODING_TABLES
   < > TEST_TANDEM_TRANSCODING
   (*) ADDRESS_SANITIZER
   (*) THREAD_SANITIZER
   (*) LEAK_SANITIZER
```

Note that you can even enable more flags.


### Rebuild asterisk and asterisk-chan-voicemngr

```
make package/asterisk/compile V=s

```

You shall see the entire package is rebuilt since the compiler flags have been changed.

Use utility ```file``` to check asterisk and .so files in ```build_dir/target-<target-name>/asterisk-18.11.2/ipkg-<target-name>/asterisk/``` are not stripped. 

Rebuild  asterisk-chan-voicemngr. But {clean,compile} is needed. Otherwise the package won’t be rebuilt. 

```
make package/asterisk-chan-voicemngr/{clean,compile} V=s

```

### Rebuild toolchain

```
make package/toolchain/{clean,compile}

```
The purpose is to generate a libc.so which has debug information not being stripped. This is very useful to get a full stack trace in GDB.


### Build An Entire Firmware as Usual Again

Asterisk has many .ipk files in bin/packages/ folder and this makes difficult to manually install them on the target. So the most convenient way is to rebuild the entire firmware image which contains Asterisk with debug information.

Note that all other packages do have debug information stripped.
### Upgrade the Firmware to the DUT

Upgrade the firmware to the DUT as usual.

### Modify the Init Startup Script and Restart Asterisk

* Add command line argument “-vvvvvg” to enable debug mode
* Enable core dump

Restart asterisk after modification of ```/etc/init.d/asterisk```.

```
openwrt-telephony$ git diff
diff --git a/net/asterisk/files/asterisk.init b/net/asterisk/files/asterisk.init
index a169304..125fd9c 100755
--- a/net/asterisk/files/asterisk.init
+++ b/net/asterisk/files/asterisk.init
@@ -71,10 +71,10 @@ start_service() {
        config_get_bool log_stdout general log_stdout 1
 
        procd_open_instance
-       procd_set_param command $PROG -f -C /var/etc/asterisk/asterisk.conf
+       procd_set_param command $PROG -f -vvvvvg -C  /var/etc/asterisk/asterisk.conf
        procd_set_param respawn 6 5 3 # add extra time for reload config
        procd_set_param term_timeout 20
-       #procd_set_param limits core="unlimited" # uncomment this for debug
+       procd_set_param limits core="unlimited" # uncomment this for debug
        # Forward stderr and stdout to logd
        procd_set_param stderr $log_stderr
        procd_set_param stdout $log_stdout
```

### Analysis when a Crash Occurs

When asterisk crashes, run the following commands to analyze.

```
root@eagle-44d437883010:~# service asterisk stop
root@eagle-44d437883010:~# gdb -se asterisk -c /tmp/asterisk*.core
(gdb) bt full
#0  __cp_begin () at src/thread/arm/syscall_cp.s:23
No locals.
#1  0xb6ee81dc in __syscall_cp_c (nr=168, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=0, y=2122088, z=-1233931155)
    at src/thread/pthread_cancel.c:33
        self = 0x1cf9ab
        r = <optimized out>
        st = <optimized out>
#2  0x00227e20 in ?? ()
(gdb)
(gdb) thread apply all
Please specify a command at the end of 'thread apply all'
(gdb) thread apply all bt

Thread 45 (LWP 14354):
#0  __cp_begin () at src/thread/arm/syscall_cp.s:23
#1  0xb6ee81dc in __syscall_cp_c (nr=162, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=0, y=0, z=0) at src/thread/pthread_cancel.c:33
#2  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 44 (LWP 14341):
#0  __cp_begin () at src/thread/arm/syscall_cp.s:23
#1  0xb6ee81dc in __syscall_cp_c (nr=142, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=0, y=1024, z=-1252840276) at src/thread/pthread_cancel.c:33
#2  0xb5532c00 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
...
```

**Note that using gdbserver attaching to a running asterisk may interfere the behavior of the process, e.g. some ubus call could be hanging. So gdbserver is NOT recommended. The effective way is to run asterisk normally and run gdb from the target directly when a crash occurs.**

### External Links

[Get a Backtrace of Asterisk](https://wiki.asterisk.org/wiki/pages/viewpage.action?pageId=5243139)

[Asterisk Debugging](https://wiki.asterisk.org/wiki/display/AST/Debugging)