Instead of searching for the asterisk binary and the modules in the
filesystem, we now get their locations, along with libdir, from
the coredump itself...
For the binary, we can use `gdb -c <coredump> ... "info proc exe"`.
gdb can print this even without having the executable and symbols.
Once we have the binary, we can get the location of the modules with
`gdb ... "print ast_config_AST_MODULE_DIR`
If there was no result then either it's not an asterisk coredump
or there were no symbols loaded. Either way, it's not usable.
For libdir, we now run "strings" on the note0 section of the
coredump (which has the shared library -> memory address xref) and
search for "libasteriskssl|libasteriskpj", then take the dirname.
Since we're now getting everything from the coredump, it has to be
correct as long as we're not crossing namespace boundaries like
running asterisk in a docker container but trying to run
ast_coredumper from the host using a shared file system (which you
shouldn't be doing).
There is still a case for using --asterisk-bin and/or --libdir: If
you've updated asterisk since the coredump was taken, the binary,
libraries and modules won't match the coredump which will render it
useless. If you can restore or rebuild the original files that
match the coredump and place them in a temporary directory, you can
use --asterisk-bin, --libdir, and a new --moddir option to point to
them and they'll be correctly captured in a tarball created
with --tarball-coredumps. If you also use --tarball-config, you can
use a new --etcdir option to point to what normally would be the
/etc/asterisk directory.
Also addressed many "shellcheck" findings.
Resolves: #445
(cherry picked from commit aec2453688)
When OUTPUTDIR is set to another directory and the
--delete-results-after is set, the resulting txt files are
not deleted.
ASTERISK-29794 #close
Change-Id: I1c0071f6809a1e3f5cfc455d6eb08378bc0d7286
Correct typos of the following word families:
standard
increase
comments
valgrind
promiscuous
editing
libtonezone
storage
aggressive
whitespace
russellbryant
consecutive
peternixon
ASTERISK-29714
Change-Id: I9cafbf41b579c9c0c84c81719d2c4f900beec245
The search for a running asterisk when --running is used
has been greatly simplified and in the event it doesn't
work, you can now specify a pid to use on the command
line with --pid.
The search for asterisk modules when --tarball-coredumps
is used has been enhanced to have a better chance of finding
them and in the event it doesn't work, you can now specify
--libdir on the command line to indicate the library directory
where they were installed.
The DATEFORMAT variable was renamed to DATEOPTS and is now
passed to the 'date' utility rather than running DATEFORMAT
as a command.
The coredump and output files are now renamed with DATEOPTS.
This can be disabled by specifying --no-rename.
Several confusing and conflicting options were removed:
--append-coredumps
--conffile
--no-default-search
--tarball-uniqueid
The script was re-structured to make it easier for follow.
Change-Id: I674be64bdde3ef310b6a551d4911c3b600ffee59
If you run ast_coredumper --tarball-coredumps in the same directory
as the actual coredump, tar can fail because the link to the
actual coredump becomes recursive. The resulting tarball will
have everything _except_ the coredump (which is usually what
you need)
There's also an issue that the directory name in the tarball
is the same as the coredump so if you extract the tarball the
directory it creates will overwrite the coredump.
So:
* Made the link to the coredump use the absolute path to the
file instead of a relative one. This prevents the recursive
link and allows tar to add the coredump.
* The tarballed directory is now named <coredump>.output instead
of just <coredump> so if you expand the tarball it won't
overwrite the coredump.
Change-Id: I8b3eeb26e09a577c702ff966924bb0a2f9a759ea
This patch makes it so ast_coredumper now outputs the following information to
a *-info.txt file when processing a core file:
asterisk version and "built by" string
BUILD_OPTS
system start, and last reloaded date/time
taskprocessor list
equivalent of "bridge show all"
equivalent of "core show channels verbose"
Also a slight modification was made when trying to obtain the pid(s) of a
running Asterisk. If it fails to retrieve any it now reports an error.
Change-Id: I54f35c19ab69b8f8dc78cc933c3fb7c99cef346b
In order to get a dump of the running process, we need to find the
pid of the main asterisk process. This can be tricky if there are
also instances of "asterisk -r" running or if an alternate location
for asterisk.conf was specified on the command line with the -C
option that also specified an alternation location for the pid file.
So now...
1. We find the asterisk executable with "which" or the --asterisk-bin
command line option.
2. If there's only 1 process with an executable path that matches,
we use that pid. If not...
3. We try "<asterisk-bin> -rx 'core show settings'" and parse the
output to find the pidfile, then read that for the pid. If that
didn't work...
4. We get a list of all the pids matching <asterisk-bin> and look
in /proc/<pid>/cmdline for a -C argument and retry the "core show
settings" using the same -C option. We can't parse the output
of "ps" to get the -C path because it may contain spaces. The
contents of /proc/<pid>/cmdline are delimited by NULLs. For BSDs
we may have to mount /proc first. :(
ASTERISK-28221
Reported by: Andrew Nagy
Change-Id: I8aa1f3f912f949df2b5348908803c636bde1d57c
The OUTPUTDIR variable in ast_debug_tools.conf.sample is now set
to "/tmp" instead of "/some/directory".
Variables set on the command line or that are already in the
environment now take predecence over variables set in the config files.
ASTERISK-27846
Reported by: Ted G
Change-Id: Ie8baec52d531886bf5849ec1d59bb59dc87ad387
* Fix --tarball-config so the option doesn't cause an error.
* Allow for missing /etc/os-release.
* Add a sleep between tarballing the coredump and removing the
output directory to allow the filesystem to settle.
Change-Id: I73e03b13087978bcc7f6bc9f45753990f82d9d77
The OUTPUTDIR environment variable can now be set either in the
environment itself or in ast_debug_tools.conf. If set, it's used
for all work products instead of /tmp.
Also added the --tarball-config option that includes the contents
of /etc/asterisk when either --tarball-coredumps or --tarball-results
are used.
Change-Id: I66b2553319df61caea5b313d084f51978f730b4c
Adds an extra option, --asterisk-bin=<path> to ast_coredumper. If
provided, the binary given to gdb will be the parameter, rather than
asterisk from the PATH.
ASTERISK-27380 #close
Change-Id: I25f5b91eb75059b0fb2f142e468c26b283b0a9f3
The --tarball-coredump option now creates a gzipped tarball of
coredumps processed, their results txt files and copies of
/etc/os-release, /usr/sbin/asterisk, /usr/lib(64)/libasterisk* and
/usr/lib(64)/asterisk as those files are needed to properly examine
the coredump. The file will be named
/tmp/asterisk.<timestamp>.coredumps.tar.gz or
/tmp/asterisk-<uniqueid>.coredumps.tar.gz if --tarball-uniqueid was
specified.
Added dumps of *_siginfo to the top of the txt files so you can
tell what signal was invoked.
Change-Id: Ib9ee6d83592d4b1bc90cb3419a05376a88d1ded9
ast_loggrabber gathers log files from customizable search patterns,
optionally converts POSIX timestamps to a readable format and
tarballs the results.
Also a few tweaks were made to ast_coredumper.
Change-Id: I8bfe1468ada24c1344ce4abab7b002a59a659495
(cherry picked from commit c709152878)
This utility allows easy manipulation of asterisk coredumps.
* Configurable search paths and patterns for existing coredumps
* Can generate a consistent coredump from the running instance
* Can dump the lock_infos table from a coredump
* Dumps backtraces to separate files...
- thread apply 1 bt full -> <coredump>.thread1.txt
- thread apply all bt -> <coredump>.brief.txt
- thread apply all bt full -> <coredump>.full.txt
- lock_infos table -> <coredump>.locks.txt
* Can tarball corefiles and optionally delete them after processing
* Can tarball results files and optionally delete them after processing
* Converts ':' in coredump and results file names '-' to facilitate
uploading. Jira for instance, won't accept file names with colons
in them.
Tested on Fedora24+, Ubuntu14+, Debian6+, CentOS6+ and FreeBSD9+[1].
[1] For *BSDs, the "devel/gdb" package might have to be installed to
get a recent gdb. The utility will check all instances of gdb
it finds in $PATH and if one isn't found that can run python, it
prints a friendly error.
Change-Id: I935d37ab9db85ef923f32b05579897f0893d33cd
(cherry picked from commit cb47b45560)
