Newer
Older
This document describes how to debug and trouble shoot Asterisk and its modules, especially for crashes with segmentation fault.
## Build Asterisk with Debug Information
GDB can be used to analyze stacks when a crash occurs. In order to let gdb provide sufficient stack trace, asterisk and its modules must have debug information not being stripped.
Here comes procedures to build asterisk with as much debug information as possible.
### Enable package gdb
Run ```make menuconfig``` and enable ```Development → gdb```
```
~/git/iopsyswrt$ grep gdb .config
CONFIG_PACKAGE_gdb=y
```
gdb will be installed and running on the target directly.
### Append libbfd as the dependency
This package is requred when enabling more debug options for Asterisk.
diff --git a/net/asterisk/Makefile b/net/asterisk/Makefile
index 824b9e7..ab85f07 100644
--- a/net/asterisk/Makefile
+++ b/net/asterisk/Makefile
@@ -463,7 +463,7 @@ $(call Package/$(PKG_NAME)/Default)
TITLE:=Complete open source PBX, v$(PKG_VERSION)
MENU:=1
DEPENDS:=+libstdcpp +jansson +libcap +libedit +libopenssl +libsqlite3 +libuuid +libxml2 +zlib \
- +libncurses +libpopt +libxslt +ubus +libpicoevent
+ +libncurses +libpopt +libxslt +ubus +libpicoevent +libbfd
endef
define Package/$(PKG_NAME)/description
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
### Build An Entire Firmware as Usual First
Note that this step is absolutely mandatory. Otherwise the size of the firmware might be too big to fit the target when strip option is disabled in the later steps.
### Disable Strip Option
The debug information of all executable and shared object files on the target are stripped. That causes gdb not able to show useful stack information in most scenarios. So asterisk and its modules must have debug information not being stripped.
Run ```make menuconfig``` and check ```Global build settings → Binary stripping method → none```
```
~/git/iopsyswrt$ grep -i strip .config
# CONFIG_KERNEL_PROC_STRIPPED is not set
# Stripping options
CONFIG_NO_STRIP=y
# CONFIG_USE_STRIP is not set
# CONFIG_USE_SSTRIP is not set
# CONFIG_STRIP_KERNEL_EXPORTS is not set
```
### Enable Debug Options in Asterisk
Enter the directory of package asterisk in iopsyswrt/build_dir and run ```make menuselect```.
Go to ```Compiler Flags```. Enable the followings and press S to save the settings.
```
[*] DONT_OPTIMIZE
[*] COMPILE_DOUBLE
[*] DEBUG_THREADS
[ ] DEBUG_FD_LEAKS
[*] BETTER_BACKTRACES
[ ] LOTS_OF_SPANS
[*] MALLOC_DEBUG
[*] DEBUG_CHAOS
XXX BUILD_NATIVE
--- Extended ---
[*] REF_DEBUG
[*] AO2_DEBUG
[ ] REBUILD_PARSERS
[ ] LOW_MEMORY
[*] DISABLE_INLINE
[*] OPTIONAL_API
XXX USE_HOARD_ALLOCATOR
[ ] RADIO_RELAX
[ ] G711_NEW_ALGORITHM
< > G711_REDUCED_BRANCHING
< > TEST_CODING_TABLES
< > TEST_TANDEM_TRANSCODING
(*) ADDRESS_SANITIZER
(*) THREAD_SANITIZER
(*) LEAK_SANITIZER
```
Note that you can even enable more flags.
### Rebuild asterisk and asterisk-chan-voicemngr
```
make package/asterisk/compile V=s
```
You shall see the entire package is rebuilt since the compiler flags have been changed.
Use utility ```file``` to check asterisk and .so files in ```build_dir/target-<target-name>/asterisk-18.11.2/ipkg-<target-name>/asterisk/``` are not stripped.
Rebuild asterisk-chan-voicemngr. But {clean,compile} is needed. Otherwise the package won’t be rebuilt.
```
make package/asterisk-chan-voicemngr/{clean,compile} V=s
```
### Rebuild toolchain
```
make package/toolchain/{clean,compile}
```
The purpose is to generate a libc.so which has debug information not being stripped. This is very useful to get a full stack trace in GDB.
### Build An Entire Firmware as Usual Again
Asterisk has many .ipk files in bin/packages/ folder and this makes difficult to manually install them on the target. So the most convenient way is to rebuild the entire firmware image which contains Asterisk with debug information.
Note that all other packages do have debug information stripped.
### Upgrade the Firmware to the DUT
Upgrade the firmware to the DUT as usual.
### Modify the Init Startup Script and Restart Asterisk
* Add command line argument “-vvvvvg” to enable debug mode
* Enable core dump
Restart asterisk after modification of ```/etc/init.d/asterisk```.
```
openwrt-telephony$ git diff
diff --git a/net/asterisk/files/asterisk.init b/net/asterisk/files/asterisk.init
index a169304..125fd9c 100755
--- a/net/asterisk/files/asterisk.init
+++ b/net/asterisk/files/asterisk.init
@@ -71,10 +71,10 @@ start_service() {
config_get_bool log_stdout general log_stdout 1
procd_open_instance
- procd_set_param command $PROG -f -C /var/etc/asterisk/asterisk.conf
+ procd_set_param command $PROG -f -vvvvvg -C /var/etc/asterisk/asterisk.conf
procd_set_param respawn 6 5 3 # add extra time for reload config
procd_set_param term_timeout 20
- #procd_set_param limits core="unlimited" # uncomment this for debug
+ procd_set_param limits core="unlimited" # uncomment this for debug
# Forward stderr and stdout to logd
procd_set_param stderr $log_stderr
procd_set_param stdout $log_stdout
```
### Analysis when a Crash Occurs
When asterisk crashes, run the following commands to analyze.
```
root@eagle-44d437883010:~# service asterisk stop
root@eagle-44d437883010:~# gdb -se asterisk -c /tmp/asterisk*.core
(gdb) bt full
#0 __cp_begin () at src/thread/arm/syscall_cp.s:23
No locals.
#1 0xb6ee81dc in __syscall_cp_c (nr=168, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=0, y=2122088, z=-1233931155)
at src/thread/pthread_cancel.c:33
self = 0x1cf9ab
r = <optimized out>
st = <optimized out>
#2 0x00227e20 in ?? ()
(gdb)
(gdb) thread apply all
Please specify a command at the end of 'thread apply all'
(gdb) thread apply all bt
Thread 45 (LWP 14354):
#0 __cp_begin () at src/thread/arm/syscall_cp.s:23
#1 0xb6ee81dc in __syscall_cp_c (nr=162, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=0, y=0, z=0) at src/thread/pthread_cancel.c:33
#2 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 44 (LWP 14341):
#0 __cp_begin () at src/thread/arm/syscall_cp.s:23
#1 0xb6ee81dc in __syscall_cp_c (nr=142, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=0, y=1024, z=-1252840276) at src/thread/pthread_cancel.c:33
#2 0xb5532c00 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
...
```
**Note that using gdbserver attaching to a running asterisk may interfere the behavior of the process, e.g. some ubus call could be hanging. So gdbserver is NOT recommended. The effective way is to run asterisk normally and run gdb from the target directly when a crash occurs.**
### External Links
[Get a Backtrace of Asterisk](https://wiki.asterisk.org/wiki/pages/viewpage.action?pageId=5243139)
[Asterisk Debugging](https://wiki.asterisk.org/wiki/display/AST/Debugging)