Commit Graph

15 Commits

Author SHA1 Message Date
George Joseph
ecb46511cc ast_coredumper: Refactor to better find things
The search for a running asterisk when --running is used
has been greatly simplified and in the event it doesn't
work, you can now specify a pid to use on the command
line with --pid.

The search for asterisk modules when --tarball-coredumps
is used has been enhanced to have a better chance of finding
them and in the event it doesn't work, you can now specify
--libdir on the command line to indicate the library directory
where they were installed.

The DATEFORMAT variable was renamed to DATEOPTS and is now
passed to the 'date' utility rather than running DATEFORMAT
as a command.

The coredump and output files are now renamed with DATEOPTS.
This can be disabled by specifying --no-rename.

Several confusing and conflicting options were removed:
--append-coredumps
--conffile
--no-default-search
--tarball-uniqueid

The script was re-structured to make it easier for follow.

Change-Id: I674be64bdde3ef310b6a551d4911c3b600ffee59
2021-10-29 09:48:14 -05:00
George Joseph
a14c5301e7 ast_coredumper: Fix issues with naming
If you run ast_coredumper --tarball-coredumps in the same directory
as the actual coredump, tar can fail because the link to the
actual coredump becomes recursive.  The resulting tarball will
have everything _except_ the coredump (which is usually what
you need)

There's also an issue that the directory name in the tarball
is the same as the coredump so if you extract the tarball the
directory it creates will overwrite the coredump.

So:

 * Made the link to the coredump use the absolute path to the
   file instead of a relative one.  This prevents the recursive
   link and allows tar to add the coredump.

 * The tarballed directory is now named <coredump>.output instead
   of just <coredump> so if you expand the tarball it won't
   overwrite the coredump.

Change-Id: I8b3eeb26e09a577c702ff966924bb0a2f9a759ea
2020-08-31 18:08:57 -05:00
Kevin Harwell
ebddff3453 ast_coredumper: add Asterisk information dump
This patch makes it so ast_coredumper now outputs the following information to
a *-info.txt file when processing a core file:

  asterisk version and "built by" string
  BUILD_OPTS
  system start, and last reloaded date/time
  taskprocessor list
  equivalent of "bridge show all"
  equivalent of "core show channels verbose"

Also a slight modification was made when trying to obtain the pid(s) of a
running Asterisk. If it fails to retrieve any it now reports an error.

Change-Id: I54f35c19ab69b8f8dc78cc933c3fb7c99cef346b
2020-03-26 08:28:28 -05:00
George Joseph
3efe5061d5 ast_coredumper: Refactor the pid determination process
In order to get a dump of the running process, we need to find the
pid of the main asterisk process.  This can be tricky if there are
also instances of "asterisk -r" running or if an alternate location
for asterisk.conf was specified on the command line with the -C
option that also specified an alternation location for the pid file.

So now...

1. We find the asterisk executable with "which" or the --asterisk-bin
   command line option.
2. If there's only 1 process with an executable path that matches,
   we use that pid.  If not...
3. We try "<asterisk-bin> -rx 'core show settings'" and parse the
   output to find the pidfile, then read that for the pid.  If that
   didn't work...
4. We get a list of all the pids matching <asterisk-bin> and look
   in /proc/<pid>/cmdline for a -C argument and retry the "core show
   settings" using the same -C option.  We can't parse the output
   of "ps" to get the -C path because it may contain spaces.  The
   contents of /proc/<pid>/cmdline are delimited by NULLs.  For BSDs
   we may have to mount /proc first. :(

ASTERISK-28221
Reported by: Andrew Nagy

Change-Id: I8aa1f3f912f949df2b5348908803c636bde1d57c
2018-12-24 14:17:32 -05:00
Jenkins2
f9cc33c821 Merge "ast_coredumper: Remove .gdbinit file on exit" into 16 2018-10-04 10:00:21 -05:00
Sean Bright
b2f96fdcfe ast_coredumper: Remove .gdbinit file on exit
Change-Id: I1297de78628773ca368e687c6f148bf74857cae9
2018-10-03 17:03:05 -05:00
George Joseph
d027d193b8 ast_coredumper: Don't use "declare -n"
Change-Id: I7ddfed4cd6549a0cd458e4d5cf9ac95d784de6cb
2018-10-03 15:30:13 -05:00
George Joseph
2bf26ce5ac ast_coredumper: Fix output directory and variable precedence
The OUTPUTDIR variable in ast_debug_tools.conf.sample is now set
to "/tmp" instead of "/some/directory".

Variables set on the command line or that are already in the
environment now take predecence over variables set in the config files.

ASTERISK-27846
Reported by: Ted G

Change-Id: Ie8baec52d531886bf5849ec1d59bb59dc87ad387
2018-05-24 12:59:37 -06:00
George Joseph
7e2128c8e6 ast_coredumper: Minor fixes
* Fix --tarball-config so the option doesn't cause an error.

* Allow for missing /etc/os-release.

* Add a sleep between tarballing the coredump and removing the
  output directory to allow the filesystem to settle.

Change-Id: I73e03b13087978bcc7f6bc9f45753990f82d9d77
2018-02-23 15:01:18 -06:00
George Joseph
cf1cb3345e ast_coredumper: Add ability to use directory other than /tmp
The OUTPUTDIR environment variable can now be set either in the
environment itself or in ast_debug_tools.conf.  If set, it's used
for all work products instead of /tmp.

Also added the --tarball-config option that includes the contents
of /etc/asterisk when either --tarball-coredumps or --tarball-results
are used.

Change-Id: I66b2553319df61caea5b313d084f51978f730b4c
2017-11-15 10:43:44 -05:00
Joshua Colp
d05391ed55 Merge "ast_coredumper: allow setting asterisk binary explicitly" 2017-10-31 06:48:16 -05:00
Tzafrir Cohen
022de525be ast_coredumper: allow setting asterisk binary explicitly
Adds an extra option, --asterisk-bin=<path> to ast_coredumper. If
provided, the binary given to gdb will be the parameter, rather than
asterisk from the PATH.

ASTERISK-27380 #close

Change-Id: I25f5b91eb75059b0fb2f142e468c26b283b0a9f3
2017-10-30 16:20:18 +02:00
George Joseph
4aec70690d ast_coredumper: Add gzipping of binaries and display of signal info
The --tarball-coredump option now creates a gzipped tarball of
coredumps processed, their results txt files and copies of
/etc/os-release, /usr/sbin/asterisk, /usr/lib(64)/libasterisk* and
/usr/lib(64)/asterisk as those files are needed to properly examine
the coredump.  The file will be named
/tmp/asterisk.<timestamp>.coredumps.tar.gz or
/tmp/asterisk-<uniqueid>.coredumps.tar.gz if --tarball-uniqueid was
specified.

Added dumps of *_siginfo to the top of the txt files so you can
tell what signal was invoked.

Change-Id: Ib9ee6d83592d4b1bc90cb3419a05376a88d1ded9
2017-10-25 12:28:21 -05:00
George Joseph
d16b3a9917 debug_utilities: Create ast_loggrabber
ast_loggrabber gathers log files from customizable search patterns,
optionally converts POSIX timestamps to a readable format and
tarballs the results.

Also a few tweaks were made to ast_coredumper.

Change-Id: I8bfe1468ada24c1344ce4abab7b002a59a659495
(cherry picked from commit c709152878)
2017-01-20 11:20:22 -06:00
George Joseph
0d53c91fba debug_utilities: Create the ast_coredumper utility
This utility allows easy manipulation of asterisk coredumps.

* Configurable search paths and patterns for existing coredumps
* Can generate a consistent coredump from the running instance
* Can dump the lock_infos table from a coredump
* Dumps backtraces to separate files...
  - thread apply 1 bt full -> <coredump>.thread1.txt
  - thread apply all bt -> <coredump>.brief.txt
  - thread apply all bt full -> <coredump>.full.txt
  - lock_infos table -> <coredump>.locks.txt
* Can tarball corefiles and optionally delete them after processing
* Can tarball results files and optionally delete them after processing
* Converts ':' in coredump and results file names '-' to facilitate
  uploading.  Jira for instance, won't accept file names with colons
  in them.

Tested on Fedora24+, Ubuntu14+, Debian6+, CentOS6+ and FreeBSD9+[1].

[1] For *BSDs, the "devel/gdb" package might have to be installed to
get a recent gdb.  The utility will check all instances of gdb
it finds in $PATH and if one isn't found that can run python, it
prints a friendly error.

Change-Id: I935d37ab9db85ef923f32b05579897f0893d33cd
(cherry picked from commit cb47b45560)
2017-01-11 12:11:45 -06:00