4da5099919
Fix #13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.
129 lines
4.4 KiB
Django/Jinja
129 lines
4.4 KiB
Django/Jinja
{% set prefix = DEFAULT_CONTAINER_REGISTRY %}
|
|
{% from "dockers/dockerfile-macros.j2" import install_debian_packages, install_python_wheels, copy_files %}
|
|
{% if CONFIGURED_ARCH == "armhf" and (MULTIARCH_QEMU_ENVIRON == "y" or CROSS_BUILD_ENVIRON == "y") %}
|
|
FROM {{ prefix }}multiarch/debian-debootstrap:armhf-bullseye
|
|
{% elif CONFIGURED_ARCH == "arm64" and (MULTIARCH_QEMU_ENVIRON == "y" or CROSS_BUILD_ENVIRON == "y") %}
|
|
FROM {{ prefix }}multiarch/debian-debootstrap:arm64-bullseye
|
|
{% else %}
|
|
FROM {{ prefix }}{{DOCKER_BASE_ARCH}}/debian:bullseye
|
|
{% endif %}
|
|
|
|
# Clean documentation in FROM image
|
|
RUN find /usr/share/doc -depth \( -type f -o -type l \) ! -name copyright | xargs rm || true
|
|
|
|
# Clean doc directories that are empty or only contain empty directories
|
|
RUN while [ -n "$(find /usr/share/doc -depth -type d -empty -print -exec rmdir {} +)" ]; do :; done && \
|
|
rm -rf \
|
|
/usr/share/man/* \
|
|
/usr/share/groff/* \
|
|
/usr/share/info/* \
|
|
/usr/share/lintian/* \
|
|
/usr/share/linda/* \
|
|
/var/cache/man/* \
|
|
/usr/share/locale/*
|
|
|
|
# Make apt-get non-interactive
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
# Configure data sources for apt/dpkg
|
|
COPY ["dpkg_01_drop", "/etc/dpkg/dpkg.cfg.d/01_drop"]
|
|
COPY ["sources.list.{{ CONFIGURED_ARCH }}", "/etc/apt/sources.list"]
|
|
COPY ["no_install_recommend_suggest", "/etc/apt/apt.conf.d"]
|
|
COPY ["no-check-valid-until", "/etc/apt/apt.conf.d"]
|
|
COPY ["apt-multiple-retries", "/etc/apt/apt.conf.d"]
|
|
|
|
# Update apt cache and
|
|
# pre-install fundamental packages
|
|
RUN apt-get update && \
|
|
apt-get -y install \
|
|
curl \
|
|
less \
|
|
perl \
|
|
procps \
|
|
python3 \
|
|
python3-distutils \
|
|
python3-pip \
|
|
python-is-python3 \
|
|
vim-tiny \
|
|
# Install redis-tools
|
|
redis-tools \
|
|
# common dependencies
|
|
libdaemon0 \
|
|
libdbus-1-3 \
|
|
libjansson4 \
|
|
# ip and ifconfig utility missing in docker for arm arch
|
|
iproute2 \
|
|
net-tools \
|
|
# for processing/handling json files in bash environment
|
|
jq \
|
|
# for sairedis zmq rpc channel
|
|
libzmq5 \
|
|
libwrap0
|
|
|
|
# default rsyslog version is 8.2110.0 which has a bug on log rate limit,
|
|
# use backport version 8.2206.0-1~bpo11+1
|
|
RUN apt-get -t bullseye-backports -y install rsyslog
|
|
|
|
# Upgrade pip via PyPI and uninstall the Debian version
|
|
RUN pip3 install --upgrade pip
|
|
RUN apt-get purge -y python3-pip
|
|
|
|
# setuptools and wheel are necessary for installing some Python wheel packages
|
|
RUN pip3 install --no-cache-dir setuptools==49.6.00
|
|
RUN pip3 install --no-cache-dir wheel==0.35.1
|
|
|
|
# For templating
|
|
RUN pip3 install j2cli
|
|
|
|
# Install supervisor
|
|
RUN pip3 install supervisor==4.2.1
|
|
|
|
# Add support for supervisord to handle startup dependencies
|
|
RUN pip3 install supervisord-dependent-startup==1.4.0
|
|
|
|
RUN mkdir -p /var/log/supervisor /etc/supervisor/conf.d
|
|
|
|
# Install gcc, libc6-dev and python3-dev for compiling python-lzf
|
|
RUN apt-get -y install build-essential libc6-dev python3-dev
|
|
|
|
# Install python-lzf
|
|
RUN pip3 install 'python-lzf==0.2.4'
|
|
|
|
# Install rdbtools
|
|
RUN pip3 install 'rdbtools==0.1.15'
|
|
|
|
# Uninstall gcc, libc6-dev and python3-dev for compiling python-lzf
|
|
RUN apt-get -y purge build-essential libc6-dev python3-dev
|
|
|
|
# Uninstall unused dependencies
|
|
RUN apt autoremove -y --purge
|
|
|
|
RUN apt-get -y purge \
|
|
exim4 \
|
|
exim4-base \
|
|
exim4-config \
|
|
exim4-daemon-light
|
|
|
|
{% if docker_base_bullseye_debs.strip() -%}
|
|
# Copy locally-built Debian package dependencies
|
|
{{ copy_files("debs/", docker_base_bullseye_debs.split(' '), "/debs/") }}
|
|
|
|
# Install built Debian packages and implicitly install their dependencies
|
|
{{ install_debian_packages(docker_base_bullseye_debs.split(' ')) }}
|
|
{%- endif %}
|
|
|
|
# Clean up apt
|
|
# Remove /var/lib/apt/lists/*, could be obsoleted for derived images
|
|
RUN apt-get clean -y && \
|
|
apt-get autoclean -y && \
|
|
apt-get autoremove -y && \
|
|
rm -rf /var/lib/apt/lists/* /tmp/* ~/.cache
|
|
|
|
COPY ["etc/rsyslog.d/*", "/etc/rsyslog.d/"]
|
|
COPY ["root/.vimrc", "/root/.vimrc"]
|
|
|
|
RUN ln /usr/bin/vim.tiny /usr/bin/vim
|
|
|
|
COPY ["etc/supervisor/supervisord.conf", "/etc/supervisor/"]
|
|
COPY ["etc/supervisor/containercfgd.conf", "/etc/supervisor/conf.d/"]
|