Skip to content
Snippets Groups Projects
asterisk-debug.md 6.91 KiB
Newer Older
  • Learn to ignore specific revisions
  • Yalu Zhang's avatar
    Yalu Zhang committed
    # Asterisk Debug
    
    
    This document describes how to debug and trouble shoot Asterisk and its modules, especially for crashes with segmentation fault.
    
    ## Build Asterisk with Debug Information
    
    GDB can be used to analyze stacks when a crash occurs. In order to let gdb provide sufficient stack trace, asterisk and its modules must have debug information not being stripped.
    
    Here comes procedures to build asterisk with as much debug information as possible.
    
    ### Enable package gdb
    
    Run ```make menuconfig``` and enable ```Development → gdb```
    
    ```
    ~/git/iopsyswrt$ grep gdb .config
    CONFIG_PACKAGE_gdb=y
    ```
    
    gdb will be installed and running on the target directly.
    
    
    ### Append libbfd as the dependency
    
    This package is requred when enabling more debug options for Asterisk.
    
    diff --git a/net/asterisk/Makefile b/net/asterisk/Makefile
    index 824b9e7..ab85f07 100644
    --- a/net/asterisk/Makefile
    +++ b/net/asterisk/Makefile
    @@ -463,7 +463,7 @@ $(call Package/$(PKG_NAME)/Default)
       TITLE:=Complete open source PBX, v$(PKG_VERSION)
       MENU:=1
       DEPENDS:=+libstdcpp +jansson +libcap +libedit +libopenssl +libsqlite3 +libuuid +libxml2 +zlib \
    -          +libncurses +libpopt +libxslt +ubus +libpicoevent
    +          +libncurses +libpopt +libxslt +ubus +libpicoevent +libbfd
     endef
     
     define Package/$(PKG_NAME)/description
    
    
    ### Build An Entire Firmware as Usual First
    
    Note that this step is absolutely mandatory. Otherwise the size of the firmware might be too big to fit the target when strip option is disabled in the later steps.
    
    ### Disable Strip Option
    
    The debug information of all executable and shared object files on the target are stripped. That causes gdb not able to show useful stack information in most scenarios. So asterisk and its modules must have debug information not being stripped.
    
     
    Run ```make menuconfig``` and check ```Global build settings → Binary stripping method → none```
    
    ```
    ~/git/iopsyswrt$ grep -i strip .config
    # CONFIG_KERNEL_PROC_STRIPPED is not set
    # Stripping options
    CONFIG_NO_STRIP=y
    # CONFIG_USE_STRIP is not set
    # CONFIG_USE_SSTRIP is not set
    # CONFIG_STRIP_KERNEL_EXPORTS is not set
    ```
    
    ### Enable Debug Options in Asterisk
    
    Enter the directory of package asterisk in iopsyswrt/build_dir and run ```make menuselect```. 
    Go to ```Compiler Flags```. Enable the followings and press S to save the settings.
    
    ```
       [*] DONT_OPTIMIZE
       [*] COMPILE_DOUBLE
       [*] DEBUG_THREADS
       [ ] DEBUG_FD_LEAKS
       [*] BETTER_BACKTRACES
       [ ] LOTS_OF_SPANS
       [*] MALLOC_DEBUG
       [*] DEBUG_CHAOS
       XXX BUILD_NATIVE
           --- Extended ---
       [*] REF_DEBUG
       [*] AO2_DEBUG
       [ ] REBUILD_PARSERS
       [ ] LOW_MEMORY
       [*] DISABLE_INLINE
       [*] OPTIONAL_API
       XXX USE_HOARD_ALLOCATOR
       [ ] RADIO_RELAX
       [ ] G711_NEW_ALGORITHM
       < > G711_REDUCED_BRANCHING
       < > TEST_CODING_TABLES
       < > TEST_TANDEM_TRANSCODING
       (*) ADDRESS_SANITIZER
       (*) THREAD_SANITIZER
       (*) LEAK_SANITIZER
    ```
    
    Note that you can even enable more flags.
    
    
    ### Rebuild asterisk and asterisk-chan-voicemngr
    
    ```
    make package/asterisk/compile V=s
    
    ```
    
    You shall see the entire package is rebuilt since the compiler flags have been changed.
    
    Use utility ```file``` to check asterisk and .so files in ```build_dir/target-<target-name>/asterisk-18.11.2/ipkg-<target-name>/asterisk/``` are not stripped. 
    
    Rebuild  asterisk-chan-voicemngr. But {clean,compile} is needed. Otherwise the package won’t be rebuilt. 
    
    ```
    make package/asterisk-chan-voicemngr/{clean,compile} V=s
    
    ```
    
    ### Rebuild toolchain
    
    ```
    make package/toolchain/{clean,compile}
    
    ```
    The purpose is to generate a libc.so which has debug information not being stripped. This is very useful to get a full stack trace in GDB.
    
    
    ### Build An Entire Firmware as Usual Again
    
    Asterisk has many .ipk files in bin/packages/ folder and this makes difficult to manually install them on the target. So the most convenient way is to rebuild the entire firmware image which contains Asterisk with debug information.
    
    Note that all other packages do have debug information stripped.
    ### Upgrade the Firmware to the DUT
    
    Upgrade the firmware to the DUT as usual.
    
    ### Modify the Init Startup Script and Restart Asterisk
    
    * Add command line argument “-vvvvvg” to enable debug mode
    * Enable core dump
    
    Restart asterisk after modification of ```/etc/init.d/asterisk```.
    
    ```
    openwrt-telephony$ git diff
    diff --git a/net/asterisk/files/asterisk.init b/net/asterisk/files/asterisk.init
    index a169304..125fd9c 100755
    --- a/net/asterisk/files/asterisk.init
    +++ b/net/asterisk/files/asterisk.init
    @@ -71,10 +71,10 @@ start_service() {
            config_get_bool log_stdout general log_stdout 1
     
            procd_open_instance
    -       procd_set_param command $PROG -f -C /var/etc/asterisk/asterisk.conf
    +       procd_set_param command $PROG -f -vvvvvg -C  /var/etc/asterisk/asterisk.conf
            procd_set_param respawn 6 5 3 # add extra time for reload config
            procd_set_param term_timeout 20
    -       #procd_set_param limits core="unlimited" # uncomment this for debug
    +       procd_set_param limits core="unlimited" # uncomment this for debug
            # Forward stderr and stdout to logd
            procd_set_param stderr $log_stderr
            procd_set_param stdout $log_stdout
    ```
    
    ### Analysis when a Crash Occurs
    
    When asterisk crashes, run the following commands to analyze.
    
    ```
    root@eagle-44d437883010:~# service asterisk stop
    root@eagle-44d437883010:~# gdb -se asterisk -c /tmp/asterisk*.core
    (gdb) bt full
    #0  __cp_begin () at src/thread/arm/syscall_cp.s:23
    No locals.
    #1  0xb6ee81dc in __syscall_cp_c (nr=168, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=0, y=2122088, z=-1233931155)
        at src/thread/pthread_cancel.c:33
            self = 0x1cf9ab
            r = <optimized out>
            st = <optimized out>
    #2  0x00227e20 in ?? ()
    (gdb)
    (gdb) thread apply all
    Please specify a command at the end of 'thread apply all'
    (gdb) thread apply all bt
    
    Thread 45 (LWP 14354):
    #0  __cp_begin () at src/thread/arm/syscall_cp.s:23
    #1  0xb6ee81dc in __syscall_cp_c (nr=162, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=0, y=0, z=0) at src/thread/pthread_cancel.c:33
    #2  0x00000000 in ?? ()
    Backtrace stopped: previous frame identical to this frame (corrupt stack?)
    
    Thread 44 (LWP 14341):
    #0  __cp_begin () at src/thread/arm/syscall_cp.s:23
    #1  0xb6ee81dc in __syscall_cp_c (nr=142, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=0, y=1024, z=-1252840276) at src/thread/pthread_cancel.c:33
    #2  0xb5532c00 in ?? ()
    Backtrace stopped: previous frame identical to this frame (corrupt stack?)
    ...
    ```
    
    **Note that using gdbserver attaching to a running asterisk may interfere the behavior of the process, e.g. some ubus call could be hanging. So gdbserver is NOT recommended. The effective way is to run asterisk normally and run gdb from the target directly when a crash occurs.**
    
    ### External Links
    
    [Get a Backtrace of Asterisk](https://wiki.asterisk.org/wiki/pages/viewpage.action?pageId=5243139)
    
    [Asterisk Debugging](https://wiki.asterisk.org/wiki/display/AST/Debugging)