Ran into an issue with a firewall on my CentOS7 system. In an attempt to diagnose the problem I wanted to gather as much info as possible.
A great tool to do so is sysdig https://github.com/draios/sysdig Sysdig can be considered the tcpdump for linux systems, providing a data capture of all low level system operations. In the past I had only run sysdig as a process, capturing events as I replicate an issue.
In this current case, my issue was occurring on reboot and was related to the firewall service. In order for sysdig to capture the event data for that slice of time, I had to enable it to run and capture at boot. To do so I created the following systemd service located in /etc/systemd/system/sysdig.service
[Unit]
Description=sysdig Service
After=network-pre.target
[Service]
Type=simple
User=root
ExecStart=/usr/bin/sysdig -w /var/log/sysdig.dump
Restart=on-abort
[Install]
WantedBy=multi-user.target
Resource: http://www.freedesktop.org/software/systemd/man/
The advantages to having a sysdig service
To troubleshoot on boot I can run:
sudo systemctl enable sysdig
After reboot I can stop sysdig and disable it from running on my next boot
systemctl stop sysdig
systemctl disable sysdig
The dump file is saved to /var/log
sysdig -r /var/log/sysdig.dump
This is just a starting point and can be taken much further; such as:
1. reading in a sysdig.conf file to allow for setting up a capture filter
2. versioning dump files
3. scheduling to prevent enormous captures on reboot
Code added to github: https://github.com/mdevans1/scripts/blob/master/sysdig.service