0c4d4ace76
A few issues where discovered with crashkernel on Arista platforms. 1) platforms using `docker_inram=on` would end up OOM in kdump environment. This happens because the same initramfs is used by SONiC and the crashkernel. With `docker_inram=on` the `dockerfs.tar.gz` is extracted in a `tmpfs` created for the occasion. Since `dockerfs.tar.gz` weights more than 1.5G, it doesn't fit into the kdump environment and ends up OOM. This OOM event can in turn trigger a panic. 2) Arista platforms with `secureboot` enabled would fail to load the crashkernel because the kernel parameter would be discarded on boot. This happens because the `boot0` in secureboot mode is strict about kernel parameter injection. 3) The secureboot path allowlist would remove kernel crash reports. 4) The kdump service would fail on Arista products since `/boot/` is empty in `secureboot` **- How I did it** 1) To prevent an OOM event in the crashkernel the fix is to avoid the codepaths in `union-mount` that create tmpfs and populate them. Some more codepath specific to Arista devices are also skipped to make the kdump process faster. This relies on detecting that the initramfs is starting in a kdump environment and skipping some initialization. The `/usr/sbin/kdump-config` tool appends a few kernel cmdline arguments when loading the crashkernel. The most unique one is `systemd.unit=kdump-tools.service` which is used in a few initramfs hooks to set `in_kdump`. 2) To allow `kdump` to work in `secureboot` environment the cmdline generation in boot0 was slightly modified. The codepath to load kernel parameters changed by SONiC is now running for booting in secure mode. It was altered to prevent an append only behavior which would grow the `kernel-cmdline` at every reboot. This ever growing behavior would lead `kexec` to fail to load the kernel due to a too long cmdline. 3) To get the kernel crash under /var/crash this path has to be added to `allowlist_paths` 4) The `/host/image-XXX/boot` folder is now populated in `secureboot` mode but not used. **- How to verify it** Regular boot: - enable kdump - enable docker_inram=on via kernel-params - reboot - generate a crash `echo c > /proc/sysrq-trigger` - before: witness OOM events on the console - after: crash kernel works and crash available under /var/crash Secure boot: - enable kdump - reboot - generate a crash `echo c > /proc/sysrq-trigger` - before: witness no kdump - after: crash kernel works and crash available under /var/crash Co-authored-by: Boyang Yu <byu@arista.com>
83 lines
1.7 KiB
Bash
83 lines
1.7 KiB
Bash
#!/bin/sh
|
|
|
|
PREREQS="arista-convertfs"
|
|
|
|
prereqs() { echo "$PREREQS"; }
|
|
|
|
case $1 in
|
|
prereqs)
|
|
prereqs
|
|
exit 0
|
|
;;
|
|
esac
|
|
|
|
info() { printf "%04.2f: $@\n" "$(cut -f1 -d' ' /proc/uptime)"; }
|
|
err() { info "Error: $@"; }
|
|
warn() { info "Warning: $@"; }
|
|
|
|
set -e
|
|
|
|
root_mnt='/mnt/arista-firmware'
|
|
root_dev=''
|
|
aboot_flag=''
|
|
|
|
# Alway run cleanup before exit
|
|
cleanup() {
|
|
if grep -q "$root_mnt" /proc/mounts; then
|
|
umount "$root_mnt"
|
|
fi
|
|
[ -e "$root_mnt" ] && rmdir "$root_mnt"
|
|
}
|
|
trap cleanup EXIT
|
|
|
|
# Extract kernel parameters
|
|
set -- $(cat /proc/cmdline)
|
|
for x in "$@"; do
|
|
case "$x" in
|
|
Aboot=*)
|
|
aboot_flag="${x#Aboot=}"
|
|
;;
|
|
loop=*)
|
|
x1="${x#loop=}"
|
|
image_dir="${x1%/*}"
|
|
;;
|
|
SONIC_BOOT_TYPE=warm*|SONIC_BOOT_TYPE=fast*)
|
|
# Skip this script for warm-reboot and fast-reboot
|
|
exit 0
|
|
;;
|
|
systemd.unit=kdump-tools.service)
|
|
# In kdump environment, skip hooks
|
|
exit 0
|
|
;;
|
|
esac
|
|
done
|
|
|
|
[ -z "$aboot_flag" ] && exit 0
|
|
|
|
root_dev="$ROOT"
|
|
if [ -z "$root_dev" ]; then
|
|
err "Error: root device name is not provided"
|
|
exit 1
|
|
fi
|
|
|
|
mkdir -p "$root_mnt"
|
|
mount -t ext4 "$root_dev" "$root_mnt"
|
|
|
|
get_sorted_hooks() {
|
|
echo $(find "$1" -name '[0-9][0-9]-*' -type f | sort)
|
|
}
|
|
|
|
if [ -d "$root_mnt/$image_dir/platform/hooks/boot1" ]; then
|
|
for hook in $(get_sorted_hooks "$root_mnt/$image_dir/platform/hooks/boot1"); do
|
|
if [ ! -z "$hook" ]; then
|
|
cp "$hook" /tmp/
|
|
hook="/tmp/`basename $hook`"
|
|
info "Running hook $(basename $hook)"
|
|
. "$hook"
|
|
fi
|
|
done
|
|
fi
|
|
|
|
umount "$root_mnt"
|
|
rmdir "$root_mnt"
|