matthew/sonic-buildimage
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

First cut image update for kubernetes support. (#5421)

Browse Source 
* First cut image update for kubernetes support.
With this,
    1)  dockers dhcp_relay, lldp, pmon, radv, snmp, telemetry are enabled
        for kube management
        init_cfg.json configure set_owner as kube for these

    2)  Each docker's start.sh updated to call container_startup.py to register going up
          As part of this call, it registers the current owner as local/kube and its version
          The images are built with its version ingrained into image during build

    3)  Update all docker's bash script to call 'container start/stop/wait' instead of 'docker start/stop/wait'.
         For all locally managed containers, it calls docker commands, hence no change for locally managed.
        
    4)  Introduced a new ctrmgrd service, that helps with transition between owners as  kube & local and carry over any labels update from STATE-DB to API server

    5)  hostcfgd updated to handle owner change

    6) Reboot scripts are updatd to tag kube running images as local, so upon reboot they run the same image.

   7) Added kube_commands.py to handle all updates with Kubernetes API serrver -- dedicated for k8s interaction only.
This commit is contained in:
Renuka Manavalan 2020-12-22 08:01:33 -08:00 committed by GitHub
parent c146eeaaa6
commit ba02209141
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
48 changed files with 4917 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
dockers/docker-dhcp-relay/Dockerfile.j2
View File

@ -2,11 +2,15 @@
FROM docker-config-engine-buster
ARG docker_container_name
ARG image_version
RUN [ -f /etc/rsyslog.conf ] && sed -ri "s/%syslogtag%/$docker_container_name#%syslogtag%/;" /etc/rsyslog.conf
# Make apt-get non-interactive
ENV DEBIAN_FRONTEND=noninteractive
# Pass the image_version to container
ENV IMAGE_VERSION=$image_version
# Update apt's cache of available packages
RUN apt-get update

10
dockers/docker-dhcp-relay/start.sh
View File

@ -1,5 +1,15 @@
#!/usr/bin/env bash
if [ "${RUNTIME_OWNER}" == "" ]; then
RUNTIME_OWNER="kube"
fi
CTR_SCRIPT="/usr/share/sonic/scripts/container_startup.py"
if test -f ${CTR_SCRIPT}
then
${CTR_SCRIPT} -f dhcp_relay -o ${RUNTIME_OWNER} -v ${IMAGE_VERSION}
fi
# If our supervisor config has entries in the "isc-dhcp-relay" group...
if [ $(supervisorctl status | grep -c "^isc-dhcp-relay:") -gt 0 ]; then
# Wait for all interfaces to come up and be assigned IPv4 addresses before

4
dockers/docker-lldp/Dockerfile.j2
View File

@ -2,11 +2,15 @@
FROM docker-config-engine-buster
ARG docker_container_name
ARG image_version
RUN [ -f /etc/rsyslog.conf ] && sed -ri "s/%syslogtag%/$docker_container_name#%syslogtag%/;" /etc/rsyslog.conf
# Make apt-get non-interactive
ENV DEBIAN_FRONTEND=noninteractive
# Pass the image_version to container
ENV IMAGE_VERSION=$image_version
# Update apt's cache of available packages
RUN apt-get update

10
dockers/docker-lldp/start.sh
View File

@ -1,5 +1,15 @@
#!/usr/bin/env bash
if [ "${RUNTIME_OWNER}" == "" ]; then
RUNTIME_OWNER="kube"
fi
CTR_SCRIPT="/usr/share/sonic/scripts/container_startup.py"
if test -f ${CTR_SCRIPT}
then
${CTR_SCRIPT} -f lldp -o ${RUNTIME_OWNER} -v ${IMAGE_VERSION}
fi
sonic-cfggen -d -t /usr/share/sonic/templates/lldpd.conf.j2 > /etc/lldpd.conf
mkdir -p /var/sonic

4
dockers/docker-platform-monitor/Dockerfile.j2
View File

@ -2,11 +2,15 @@
FROM docker-config-engine-buster
ARG docker_container_name
ARG image_version
RUN [ -f /etc/rsyslog.conf ] && sed -ri "s/%syslogtag%/$docker_container_name#%syslogtag%/;" /etc/rsyslog.conf
# Make apt-get non-interactive
ENV DEBIAN_FRONTEND=noninteractive
# Pass the image_version to container
ENV IMAGE_VERSION=$image_version
# Install required packages
RUN apt-get update && \
apt-get install -y \

10
dockers/docker-platform-monitor/start.sh
View File

@ -2,6 +2,16 @@
declare -r EXIT_SUCCESS="0"
if [ "${RUNTIME_OWNER}" == "" ]; then
RUNTIME_OWNER="kube"
fi
CTR_SCRIPT="/usr/share/sonic/scripts/container_startup.py"
if test -f ${CTR_SCRIPT}
then
${CTR_SCRIPT} -f pmon -o ${RUNTIME_OWNER} -v ${IMAGE_VERSION}
fi
mkdir -p /var/sonic
echo "# Config files managed by sonic-config-engine" > /var/sonic/config_status

5
dockers/docker-router-advertiser/Dockerfile.j2
View File

@ -2,11 +2,15 @@
FROM docker-config-engine-buster
ARG docker_container_name
ARG image_version
RUN [ -f /etc/rsyslog.conf ] && sed -ri "s/%syslogtag%/$docker_container_name#%syslogtag%/;" /etc/rsyslog.conf
# Make apt-get non-interactive
ENV DEBIAN_FRONTEND=noninteractive
# Pass the image_version to container
ENV IMAGE_VERSION=$image_version
# Update apt's cache of available packages
RUN apt-get update
@ -27,6 +31,7 @@ RUN apt-get clean -y && \
apt-get autoremove -y && \
rm -rf /debs
COPY ["start.sh", "/usr/bin/"]
COPY ["docker-init.sh", "/usr/bin/"]
COPY ["radvd.conf.j2", "wait_for_link.sh.j2", "docker-router-advertiser.supervisord.conf.j2", "/usr/share/sonic/templates/"]
COPY ["files/supervisor-proc-exit-listener", "/usr/bin"]

13
dockers/docker-router-advertiser/docker-router-advertiser.supervisord.conf.j2
View File

@ -27,6 +27,17 @@ stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
[program:start]
command=/usr/bin/start.sh
priority=1
autostart=true
autorestart=false
startsecs=0
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=rsyslogd:running
{# Router advertiser should only run on ToR (T0) devices which have #}
{# at least one VLAN interface which has an IPv6 address asigned #}
{%- set vlan_v6 = namespace(count=0) -%}
@ -51,7 +62,7 @@ startsecs=0
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=rsyslogd:running
dependent_startup_wait_for=start:exited
[program:radvd]
command=/usr/sbin/radvd -n

12
dockers/docker-router-advertiser/start.sh Executable file
View File

@ -0,0 +1,12 @@
#! /bin/bash
if [ "${RUNTIME_OWNER}" == "" ]; then
RUNTIME_OWNER="kube"
fi
CTR_SCRIPT="/usr/share/sonic/scripts/container_startup.py"
if test -f ${CTR_SCRIPT}
then
${CTR_SCRIPT} -f radv -o ${RUNTIME_OWNER} -v ${IMAGE_VERSION}
fi

4
dockers/docker-snmp/Dockerfile.j2
View File

@ -2,6 +2,7 @@
FROM docker-config-engine-buster
ARG docker_container_name
ARG image_version
RUN [ -f /etc/rsyslog.conf ] && sed -ri "s/%syslogtag%/$docker_container_name#%syslogtag%/;" /etc/rsyslog.conf
# Enable -O for all Python calls
@ -10,6 +11,9 @@ ENV PYTHONOPTIMIZE 1
# Make apt-get non-interactive
ENV DEBIAN_FRONTEND=noninteractive
# Pass the image_version to container
ENV IMAGE_VERSION=$image_version
# Update apt's cache of available packages
# Install make/gcc which is required for installing hiredis
RUN apt-get update && \

11
dockers/docker-snmp/start.sh
View File

@ -1,5 +1,16 @@
#!/usr/bin/env bash
if [ "${RUNTIME_OWNER}" == "" ]; then
RUNTIME_OWNER="kube"
fi
CTR_SCRIPT="/usr/share/sonic/scripts/container_startup.py"
if test -f ${CTR_SCRIPT}
then
${CTR_SCRIPT} -f snmp -o ${RUNTIME_OWNER} -v ${IMAGE_VERSION}
fi
mkdir -p /etc/ssw /etc/snmp
SONIC_CFGGEN_ARGS=" \

4
dockers/docker-sonic-telemetry/Dockerfile.j2
View File

@ -2,11 +2,15 @@
FROM docker-config-engine-buster
ARG docker_container_name
ARG image_version
RUN [ -f /etc/rsyslog.conf ] && sed -ri "s/%syslogtag%/$docker_container_name#%syslogtag%/;" /etc/rsyslog.conf
## Make apt-get non-interactive
ENV DEBIAN_FRONTEND=noninteractive
# Pass the image_version to container
ENV IMAGE_VERSION=$image_version
RUN apt-get update
{% if docker_sonic_telemetry_debs.strip() -%}

10
dockers/docker-sonic-telemetry/start.sh
View File

@ -1,4 +1,14 @@
#!/usr/bin/env bash
if [ "${RUNTIME_OWNER}" == "" ]; then
RUNTIME_OWNER="kube"
fi
CTR_SCRIPT="/usr/share/sonic/scripts/container_startup.py"
if test -f ${CTR_SCRIPT}
then
${CTR_SCRIPT} -f telemetry -o ${RUNTIME_OWNER} -v ${IMAGE_VERSION}
fi
mkdir -p /var/sonic
echo "# Config files managed by sonic-config-engine" > /var/sonic/config_status

20
files/build_templates/docker_image_ctl.j2
View File

@ -210,13 +210,14 @@ start() {
DOCKERMOUNT=`getMountPoint "$DOCKERCHECK"`
{%- endif %}
if [ x"$DOCKERMOUNT" == x"$MOUNTPATH" ]; then
preStartAction
{%- if docker_container_name == "database" %}
echo "Starting existing ${DOCKERNAME} container"
docker start ${DOCKERNAME}
{%- else %}
echo "Starting existing ${DOCKERNAME} container with HWSKU $HWSKU"
/usr/local/bin/container start ${DOCKERNAME}
{%- endif %}
preStartAction
docker start ${DOCKERNAME}
postStartAction
exit $?
fi
@ -343,6 +344,7 @@ start() {
{%- endif %}
docker create {{docker_image_run_opt}} \
--net=$NET \
-e RUNTIME_OWNER=local \
--uts=host \{# W/A: this should be set per-docker, for those dockers which really need host's UTS namespace #}
{%- if install_debug_image == "y" %}
-v /src:/src:ro -v /debug:/debug:rw \
@ -400,21 +402,31 @@ start() {
}
preStartAction
{%- if docker_container_name == "database" %}
docker start $DOCKERNAME
{%- else %}
/usr/local/bin/container start ${DOCKERNAME}
{%- endif %}
postStartAction
}
wait() {
{%- if docker_container_name == "database" %}
docker wait $DOCKERNAME
{%- else %}
/usr/local/bin/container wait $DOCKERNAME
{%- endif %}
}
stop() {
{%- if docker_container_name == "database" %}
docker stop $DOCKERNAME
{%- if docker_container_name == "database" %}
if [ "$DEV" ]; then
ip netns delete "$NET_NS"
fi
{%- endif %}
{%- else %}
/usr/local/bin/container stop $DOCKERNAME
{%- endif %}
}
DOCKERNAME={{docker_container_name}}

4
files/build_templates/init_cfg.json.j2
View File

@ -44,6 +44,10 @@
"has_global_scope": {% if feature + '.service' in installer_services.split(' ') %}true{% else %}false{% endif %},
"has_per_asic_scope": {% if feature + '@.service' in installer_services.split(' ') %}true{% else %}false{% endif %},
"auto_restart": "{{autorestart}}",
{%- if include_kubernetes == "y" %}
{%- if feature in ["dhcp_relay", "lldp", "pmon", "radv", "snmp", "telemetry"] %}
"set_owner": "kube", {% else %}
"set_owner": "local", {% endif %} {% endif %}
"high_mem_alert": "disabled"
}{% if not loop.last %},{% endif -%}
{% endfor %}

21
files/build_templates/kube_cni.10-flannel.conflist Normal file
View File

@ -0,0 +1,21 @@
{
"_comment": "This is a dummy file to simulate flannel so that node status will show as Ready",
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}

49
files/build_templates/sonic_debian_extension.j2
View File

@ -37,6 +37,7 @@ FILESYSTEM_ROOT_USR="$FILESYSTEM_ROOT/usr"
FILESYSTEM_ROOT_USR_LIB_SYSTEMD_SYSTEM="$FILESYSTEM_ROOT/usr/lib/systemd/system"
FILESYSTEM_ROOT_USR_SHARE="$FILESYSTEM_ROOT_USR/share"
FILESYSTEM_ROOT_USR_SHARE_SONIC="$FILESYSTEM_ROOT_USR_SHARE/sonic"
FILESYSTEM_ROOT_USR_SHARE_SONIC_SCRIPTS="$FILESYSTEM_ROOT_USR_SHARE_SONIC/scripts"
FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES="$FILESYSTEM_ROOT_USR_SHARE_SONIC/templates"
FILESYSTEM_ROOT_ETC="$FILESYSTEM_ROOT/etc"
FILESYSTEM_ROOT_ETC_SONIC="$FILESYSTEM_ROOT_ETC/sonic"
@ -395,10 +396,36 @@ sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install watchd
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip2 install futures==3.3.0
{% if include_kubernetes == "y" %}
# Copy kubelet service files
# Keep it disabled until join, else it continuously restart and as well spew too many
# non-required log lines wasting syslog resources.
# Install remote Container mgmt package
# Required even if include_kubernetes != y, as it contains the
# the container wrapper for docker start/stop/wait commands.
#
SONIC_CTRMGMT_WHEEL_NAME=$(basename {{sonic_ctrmgmt_py3_wheel_path}})
sudo cp {{sonic_ctrmgmt_py3_wheel_path}} $FILESYSTEM_ROOT/$SONIC_CTRMGMT_WHEEL_NAME
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install $SONIC_CTRMGMT_WHEEL_NAME
sudo rm -rf $FILESYSTEM_ROOT/$SONIC_CTRMGMT_WHEEL_NAME
# Copy remote container mangement files
# File called from each container upon start/stop to record the state
sudo mkdir -p ${FILESYSTEM_ROOT_USR_SHARE_SONIC_SCRIPTS}
sudo cp ${files_path}/container_startup.py ${FILESYSTEM_ROOT_USR_SHARE_SONIC_SCRIPTS}/
sudo chmod a+x ${FILESYSTEM_ROOT_USR_SHARE_SONIC_SCRIPTS}/container_startup.py
# Config file used by container mgmt scripts/service
sudo cp ${files_path}/remote_ctr.config.json ${FILESYSTEM_ROOT_ETC_SONIC}/
# Remote container management service files
sudo cp ${files_path}/ctrmgrd.service ${FILESYSTEM_ROOT_USR_LIB_SYSTEMD_SYSTEM}/
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable ctrmgrd.service
# kubelet service is controlled by ctrmgrd daemon.
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl disable kubelet.service
{% else %}
# container script for docker commands, which is required as
# all docker commands are replaced with container commands.
# So just copy that file only.
#
sudo cp ${files_path}/container $FILESYSTEM_ROOT/usr/local/bin/
{% endif %}
# Copy the buffer configuration template
@ -585,6 +612,22 @@ else
fi
sudo umount /proc || true
sudo rm $FILESYSTEM_ROOT/etc/init.d/docker
sudo bash -c "echo { > $FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES/ctr_image_names.json"
{% for entry in feature_vs_image_names.split(' ') -%}
{% if entry|length %}
{% set lst = entry.split(':') %}
{% set lst1 = lst[1].split('.') %}
sudo bash -c "echo -n -e \"\x22{{ lst[0] }}\x22 : \x22{{ lst1[0] }}\x22\" >> $FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES/ctr_image_names.json"
{% if not loop.last %}
sudo bash -c "echo \",\" >> $FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES/ctr_image_names.json"
{% else %}
sudo bash -c "echo \"\" >> $FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES/ctr_image_names.json"
{% endif %}
{% endif %}
{% endfor %}
sudo bash -c "echo } >> $FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES/ctr_image_names.json"
{% for script in installer_start_scripts.split(' ') -%}
sudo cp {{script}} $FILESYSTEM_ROOT/usr/bin/
{% endfor %}

1
rules/docker-config-engine-buster.mk
View File

@ -12,6 +12,7 @@ $(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SONIC_CONFIG_ENGINE_PY2)
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SONIC_CONFIG_ENGINE_PY3)
$(DOCKER_CONFIG_ENGINE_BUSTER)_LOAD_DOCKERS += $(DOCKER_BASE_BUSTER)
$(DOCKER_CONFIG_ENGINE_BUSTER)_FILES += $(SWSS_VARS_TEMPLATE)
$(DOCKER_CONFIG_ENGINE_BUSTER)_FILES += $($(SONIC_CTRMGRD)_CONTAINER_SCRIPT)
$(DOCKER_CONFIG_ENGINE_BUSTER)_DBG_DEPENDS = $($(DOCKER_BASE_BUSTER)_DBG_DEPENDS)
$(DOCKER_CONFIG_ENGINE_BUSTER)_DBG_IMAGE_PACKAGES = $($(DOCKER_BASE_BUSTER)_DBG_IMAGE_PACKAGES)

2
rules/docker-dhcp-relay.mk
View File

@ -7,6 +7,7 @@ DOCKER_DHCP_RELAY_DBG = $(DOCKER_DHCP_RELAY_STEM)-$(DBG_IMAGE_MARK).gz
$(DOCKER_DHCP_RELAY)_PATH = $(DOCKERS_PATH)/$(DOCKER_DHCP_RELAY_STEM)
$(DOCKER_DHCP_RELAY)_DEPENDS += $(ISC_DHCP_RELAY) $(SONIC_DHCPMON)
$(DOCKER_DHCP_RELAY)_DBG_DEPENDS = $($(DOCKER_CONFIG_ENGINE_BUSTER)_DBG_DEPENDS)
$(DOCKER_DHCP_RELAY)_DBG_DEPENDS += $(ISC_DHCP_RELAY_DBG)
@ -23,4 +24,5 @@ SONIC_INSTALL_DOCKER_DBG_IMAGES += $(DOCKER_DHCP_RELAY_DBG)
$(DOCKER_DHCP_RELAY)_CONTAINER_NAME = dhcp_relay
$(DOCKER_DHCP_RELAY)_RUN_OPT += --privileged -t
$(DOCKER_DHCP_RELAY)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro
$(DOCKER_DHCP_RELAY)_RUN_OPT += -v /usr/share/sonic/scripts:/usr/share/sonic/scripts:ro
$(DOCKER_DHCP_RELAY)_FILES += $(SUPERVISOR_PROC_EXIT_LISTENER_SCRIPT)

1
rules/docker-lldp.mk
View File

@ -25,6 +25,7 @@ SONIC_INSTALL_DOCKER_DBG_IMAGES += $(DOCKER_LLDP_DBG)
$(DOCKER_LLDP)_CONTAINER_NAME = lldp
$(DOCKER_LLDP)_RUN_OPT += --privileged -t
$(DOCKER_LLDP)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro
$(DOCKER_LLDP)_RUN_OPT += -v /usr/share/sonic/scripts:/usr/share/sonic/scripts:ro
$(DOCKER_LLDP)_BASE_IMAGE_FILES += lldpctl:/usr/bin/lldpctl
$(DOCKER_LLDP)_BASE_IMAGE_FILES += lldpcli:/usr/bin/lldpcli

1
rules/docker-platform-monitor.mk
View File

@ -45,6 +45,7 @@ SONIC_INSTALL_DOCKER_DBG_IMAGES += $(DOCKER_PLATFORM_MONITOR_DBG)
$(DOCKER_PLATFORM_MONITOR)_CONTAINER_NAME = pmon
$(DOCKER_PLATFORM_MONITOR)_RUN_OPT += --privileged -t
$(DOCKER_PLATFORM_MONITOR)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro
$(DOCKER_PLATFORM_MONITOR)_RUN_OPT += -v /usr/share/sonic/scripts:/usr/share/sonic/scripts:ro
$(DOCKER_PLATFORM_MONITOR)_RUN_OPT += -v /var/run/platform_cache:/var/run/platform_cache:ro
$(DOCKER_PLATFORM_MONITOR)_RUN_OPT += -v /usr/share/sonic/device/pddf:/usr/share/sonic/device/pddf:ro

1
rules/docker-router-advertiser.mk
View File

@ -22,4 +22,5 @@ SONIC_INSTALL_DOCKER_DBG_IMAGES += $(DOCKER_ROUTER_ADVERTISER_DBG)
$(DOCKER_ROUTER_ADVERTISER)_CONTAINER_NAME = radv
$(DOCKER_ROUTER_ADVERTISER)_RUN_OPT += --privileged -t
$(DOCKER_ROUTER_ADVERTISER)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro
$(DOCKER_ROUTER_ADVERTISER)_RUN_OPT += -v /usr/share/sonic/scripts:/usr/share/sonic/scripts:ro
$(DOCKER_ROUTER_ADVERTISER)_FILES += $(SUPERVISOR_PROC_EXIT_LISTENER_SCRIPT)

1
rules/docker-snmp.mk
View File

@ -26,5 +26,6 @@ SONIC_INSTALL_DOCKER_DBG_IMAGES += $(DOCKER_SNMP_DBG)
$(DOCKER_SNMP)_CONTAINER_NAME = snmp
$(DOCKER_SNMP)_RUN_OPT += --privileged -t
$(DOCKER_SNMP)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro
$(DOCKER_SNMP)_RUN_OPT += -v /usr/share/sonic/scripts:/usr/share/sonic/scripts:ro
$(DOCKER_SNMP)_FILES += $(SUPERVISOR_PROC_EXIT_LISTENER_SCRIPT)
$(DOCKER_SNMP)_BASE_IMAGE_FILES += monit_snmp:/etc/monit/conf.d

1
rules/docker-telemetry.mk
View File

@ -26,6 +26,7 @@ endif
$(DOCKER_TELEMETRY)_CONTAINER_NAME = telemetry
$(DOCKER_TELEMETRY)_RUN_OPT += --privileged -t
$(DOCKER_TELEMETRY)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro
$(DOCKER_TELEMETRY)_RUN_OPT += -v /usr/share/sonic/scripts:/usr/share/sonic/scripts:ro
$(DOCKER_TELEMETRY)_RUN_OPT += -v /var/run/dbus:/var/run/dbus:rw
$(DOCKER_TELEMETRY)_FILES += $(SUPERVISOR_PROC_EXIT_LISTENER_SCRIPT)

8
rules/sonic-ctrmgrd.dep Normal file
View File

@ -0,0 +1,8 @@
SPATH := $($(SONIC_CTRMGRD)_SRC_PATH)
DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/sonic-ctrmgrd.mk rules/sonic-ctrmgrd.dep
DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST)
DEP_FILES += $(shell git ls-files $(SPATH))
$(SONIC_CTRMGRD)_CACHE_MODE := GIT_CONTENT_SHA
$(SONIC_CTRMGRD)_DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST)
$(SONIC_CTRMGRD)_DEP_FILES := $(DEP_FILES)

31
rules/sonic-ctrmgrd.mk Normal file
View File

@ -0,0 +1,31 @@
# sonic-ctrmgrd package
SONIC_CTRMGRD = sonic_ctrmgrd-1.0.0-py3-none-any.whl
$(SONIC_CTRMGRD)_SRC_PATH = $(SRC_PATH)/sonic-ctrmgrd
$(SONIC_CTRMGRD)_FILES_PATH = $($(SONIC_CTRMGRD)_SRC_PATH)/ctrmgr
$(SONIC_CTRMGRD)_PYTHON_VERSION = 3
$(SONIC_CTRMGRD)_DEBS_DEPENDS += $(PYTHON3_SWSSCOMMON)
$(SONIC_CTRMGRD)_DEPENDS += $(SONIC_PY_COMMON_PY3)
$(SONIC_CTRMGRD)_CONTAINER_SCRIPT = container
$($(SONIC_CTRMGRD)_CONTAINER_SCRIPT)_PATH = $($(SONIC_CTRMGRD)_FILES_PATH)
$(SONIC_CTRMGRD)_STARTUP_SCRIPT = container_startup.py
$($(SONIC_CTRMGRD)_STARTUP_SCRIPT)_PATH = $($(SONIC_CTRMGRD)_FILES_PATH)
$(SONIC_CTRMGRD)_CFG_JSON = remote_ctr.config.json
$($(SONIC_CTRMGRD)_CFG_JSON)_PATH = $($(SONIC_CTRMGRD)_FILES_PATH)
$(SONIC_CTRMGRD)_SERVICE = ctrmgrd.service
$($(SONIC_CTRMGRD)_SERVICE)_PATH = $($(SONIC_CTRMGRD)_FILES_PATH)
SONIC_PYTHON_WHEELS += $(SONIC_CTRMGRD)
$(SONIC_CTRMGRD)_FILES = $($(SONIC_CTRMGRD)_CONTAINER_SCRIPT)
$(SONIC_CTRMGRD)_FILES += $($(SONIC_CTRMGRD)_STARTUP_SCRIPT)
$(SONIC_CTRMGRD)_FILES += $($(SONIC_CTRMGRD)_CFG_JSON)
$(SONIC_CTRMGRD)_FILES += $($(SONIC_CTRMGRD)_SERVICE)
SONIC_COPY_FILES += $($(SONIC_CTRMGRD)_FILES)

5
slave.mk
View File

@ -750,6 +750,7 @@ $(addprefix $(TARGET_PATH)/, $(DOCKER_IMAGES)) : $(TARGET_PATH)/%.gz : .platform
--build-arg docker_container_name=$($*.gz_CONTAINER_NAME) \
--build-arg frr_user_uid=$(FRR_USER_UID) \
--build-arg frr_user_gid=$(FRR_USER_GID) \
--build-arg image_version=$(SONIC_IMAGE_VERSION) \
--label Tag=$(SONIC_IMAGE_VERSION) \
-t $* $($*.gz_PATH) $(LOG)
scripts/collect_docker_version_files.sh $* $(TARGET_PATH)
@ -865,6 +866,8 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PLATFORM_API_PY2)) \
$(if $(findstring y,$(PDDF_SUPPORT)),$(addprefix $(PYTHON_WHEELS_PATH)/,$(PDDF_PLATFORM_API_BASE_PY2))) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MODELS_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CTRMGRD)) \
$(addprefix $(FILES_PATH)/,$($(SONIC_CTRMGRD)_FILES)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MGMT_PY2)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MGMT_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SYSTEM_HEALTH)) \
@ -911,6 +914,7 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
export redis_dump_load_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(REDIS_DUMP_LOAD_PY3))"
export install_debug_image="$(INSTALL_DEBUG_TOOLS)"
export sonic_yang_models_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MODELS_PY3))"
export sonic_ctrmgmt_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CTRMGRD))"
export sonic_yang_mgmt_py2_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MGMT_PY2))"
export sonic_yang_mgmt_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MGMT_PY3))"
export multi_instance="false"
@ -959,6 +963,7 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
# Exported variables are used by sonic_debian_extension.sh
export installer_start_scripts="$(foreach docker, $($*_DOCKERS),$(addsuffix .sh, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME)))"
export feature_vs_image_names="$(foreach docker, $($*_DOCKERS), $(addsuffix :, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME):$(docker:-dbg.gz=.gz)))"
# Marks template services with an "@" according to systemd convention
# If the $($docker)_TEMPLATE) variable is set, the service will be treated as a template

12
src/sonic-ctrmgrd/.gitignore vendored Normal file
View File

@ -0,0 +1,12 @@
.eggs/
build/
dist/
*.egg-info/
ctrmgr/*.pyc
tests/*.pyc
tests/__pycache__/
.idea
.coverage
ctrmgr/__pycache__/
venv
tests/.coverage*

3
src/sonic-ctrmgrd/README.rst Normal file
View File

@ -0,0 +1,3 @@
"
This Package contains modules required for remote container management.
"

0
src/sonic-ctrmgrd/ctrmgr/__init__.py Normal file
View File

409
src/sonic-ctrmgrd/ctrmgr/container Executable file
View File

@ -0,0 +1,409 @@
#!/usr/bin/env python3
import argparse
import os
import inspect
import json
import syslog
import time
import datetime
import docker
from swsscommon import swsscommon
CTR_STATE_SCR_PATH = '/usr/share/sonic/scripts/container_startup.py'
state_db = None
# DB field names
FEATURE_TABLE = "FEATURE"
SET_OWNER = "set_owner"
NO_FALLBACK = "no_fallback_to_local"
CURRENT_OWNER = "current_owner"
UPD_TIMESTAMP = "update_time"
CONTAINER_ID = "container_id"
REMOTE_STATE = "remote_state"
VERSION = "container_version"
SYSTEM_STATE = "system_state"
KUBE_LABEL_TABLE = "KUBE_LABELS"
KUBE_LABEL_SET_KEY = "SET"
# Get seconds to wait for remote docker to start.
# If not, revert to local
#
SONIC_CTR_CONFIG = "/etc/sonic/remote_ctr.config.json"
SONIC_CTR_CONFIG_PEND_SECS = "revert_to_local_on_wait_seconds"
DEFAULT_PEND_SECS = ( 5 * 60 )
WAIT_POLL_SECS = 2
remote_ctr_enabled = False
def debug_msg(m):
msg = "{}: {}".format(inspect.stack()[1][3], m)
# print(msg)
syslog.syslog(syslog.LOG_DEBUG, msg)
def init():
""" Get DB connections """
global state_db, cfg_db, remote_ctr_enabled
cfg_db = swsscommon.DBConnector("CONFIG_DB", 0)
state_db = swsscommon.DBConnector("STATE_DB", 0)
remote_ctr_enabled = os.path.exists(CTR_STATE_SCR_PATH)
def get_config_data(fld, dflt):
""" Read entry from kube config file """
if os.path.exists(SONIC_CTR_CONFIG):
with open(SONIC_CTR_CONFIG, "r") as s:
d = json.load(s)
if fld in d:
return d[fld]
return dflt
def read_data(is_config, feature, fields):
""" Read data from DB for desired fields using given defaults"""
ret = []
db = cfg_db if is_config else state_db
tbl = swsscommon.Table(db, FEATURE_TABLE)
data = dict(tbl.get(feature)[1])
for (field, default) in fields:
val = data.get(field, default)
ret += [val]
debug_msg("config:{} feature:{} fields:{} val:{}".format(
is_config, feature, str(fields), str(ret)))
return tuple(ret)
def read_config(feature):
""" Read requried feature config """
set_owner, no_fallback = read_data(True, feature,
[(SET_OWNER, "local"), (NO_FALLBACK, False)])
return (set_owner, not no_fallback)
def read_state(feature):
""" Read requried feature state """
return read_data(False, feature,
[(CURRENT_OWNER, "none"), (REMOTE_STATE, "none"), (CONTAINER_ID, "")])
def docker_action(action, feature):
""" Execute docker action """
try:
client = docker.from_env()
container = client.containers.get(feature)
getattr(container, action)()
syslog.syslog(syslog.LOG_INFO, "docker cmd: {} for {}".format(action, feature))
return 0
except (docker.errors.NotFound, docker.errors.APIError) as err:
syslog.syslog(syslog.LOG_ERR, "docker cmd: {} for {} failed with {}".
format(action, feature, str(err)))
return -1
def set_label(feature, create):
""" Set/drop label as required
Update is done in state-db.
ctrmgrd sets it with kube API server as required
"""
if remote_ctr_enabled:
tbl = swsscommon.Table(state_db, KUBE_LABEL_TABLE)
fld = "{}_enabled".format(feature)
# redundant set (data already exist) can still raise subscriber
# notification. So check & set.
# Redundant delete (data doesn't exist) does not raise any
# subscriber notification. So no need to pre-check for delete.
#
tbl.set(KUBE_LABEL_SET_KEY, [(fld, "true" if create else "false")])
def update_data(feature, data):
if remote_ctr_enabled:
debug_msg("feature:{} data:{}".format(feature, str(data)))
tbl = swsscommon.Table(state_db, FEATURE_TABLE)
tbl.set(feature, list(data.items()))
def container_id(feature):
"""
Return the container ID for the feature.
if current_owner is local, use feature name as the start/stop
of local image is synchronous.
Else get it from FEATURE table in STATE-DB
:param feature: Name of the feature to start.
"""
init()
tbl = swsscommon.Table(state_db, "FEATURE")
data = dict(tbl.get(feature)[1])
if (data.get(CURRENT_OWNER, "").lower() == "local"):
return feature
else:
return data.get(CONTAINER_ID, feature)
def container_start(feature):
"""
Starts a container for given feature.
Starts from local image and/or trigger kubernetes to deploy the image
for this feature. Marks the feature state up in STATE-DB FEATURE table.
If feature's set_owner is local, invoke docker start.
If feature's set_owner is kube, it creates a node label that
would trigger kubernetes to start the container. With kube as
owner, if fallback is enabled and remote_state==none, it starts
the local image using docker, which will run until kube
deployment occurs.
:param feature: Name of the feature to start.
"""
START_LOCAL = 1
START_KUBE = 2
ret = 0
debug_msg("BEGIN")
init()
set_owner, fallback = read_config(feature)
_, remote_state, _ = read_state(feature)
debug_msg("{}: set_owner:{} fallback:{} remote_state:{}".format(
feature, set_owner, fallback, remote_state))
data = {
SYSTEM_STATE: "up",
UPD_TIMESTAMP: str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
}
start_val = 0
if set_owner == "local":
start_val = START_LOCAL
else:
start_val = START_KUBE
if fallback and (remote_state == "none"):
start_val |= START_LOCAL
if start_val == START_LOCAL:
# Implies *only* local.
# Ensure label is not there, to block kube deployment.
set_label(feature, False)
data[REMOTE_STATE] = "none"
if (start_val & START_LOCAL):
data[CURRENT_OWNER] = "local"
data[CONTAINER_ID] = feature
update_data(feature, data)
if (start_val & START_LOCAL):
ret = docker_action("start", feature)
if (start_val & START_KUBE):
set_label(feature, True)
debug_msg("END")
return ret
def container_stop(feature):
"""
Stops the running container for this feature.
Instruct/ensure kube terminates, by removing label, unless
an kube upgrade is happening.
Gets the container ID for this feature and call docker stop.
Marks the feature state down in STATE-DB FEATURE table.
:param feature: Name of the feature to stop.
"""
debug_msg("BEGIN")
init()
set_owner, _ = read_config(feature)
current_owner, remote_state, _ = read_state(feature)
docker_id = container_id(feature)
remove_label = (remote_state != "pending") or (set_owner == "local")
debug_msg("{}: set_owner:{} current_owner:{} remote_state:{} docker_id:{}".format(
feature, set_owner, current_owner, remote_state, docker_id))
if remove_label:
set_label(feature, False)
if docker_id:
docker_action("stop", docker_id)
else:
syslog.syslog(
syslog.LOG_ERR if current_owner != "none" else syslog.LOG_INFO,
"docker stop skipped as no docker-id for {}".format(feature))
# Container could get killed or crashed. In either case
# it does not have opportunity to mark itself down.
# Even during normal termination, with SIGTERM received
# container process may not have enough window of time to
# mark itself down and has the potential to get aborted.
#
# systemctl ensures that it handles only one instance for
# a feature at anytime and however the feature container
# exits, upon stop/kill/crash, systemctl-stop process
# is assured to get called. So mark the feature down here.
#
data = {
CURRENT_OWNER: "none",
UPD_TIMESTAMP: str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")),
CONTAINER_ID: "",
VERSION: "",
SYSTEM_STATE: "down"
}
if remote_state == "running":
data[REMOTE_STATE] = "stopped"
update_data(feature, data)
debug_msg("END")
def container_kill(feature):
"""
Kills the running container for this feature.
Instruct/ensure kube terminates, by removing label.
:param feature: Name of the feature to kill.
"""
debug_msg("BEGIN")
init()
set_owner, _ = read_config(feature)
current_owner, remote_state, _ = read_state(feature)
docker_id = container_id(feature)
remove_label = (set_owner != "local") or (current_owner != "local")
debug_msg("{}: set_owner:{} current_owner:{} remote_state:{} docker_id:{}".format(
feature, set_owner, current_owner, remote_state, docker_id))
if remove_label:
set_label(feature, False)
if docker_id:
docker_action("kill", docker_id)
else:
syslog.syslog(
syslog.LOG_ERR if current_owner != "none" else syslog.LOG_INFO,
"docker stop skipped as no docker-id for {}".format(feature))
debug_msg("END")
def container_wait(feature):
"""
Waits on the running container for this feature.
Get the container-id and call docker wait.
If docker-id can't be obtained for a configurable fail-duration
the wait clears the feature's remote-state in STATE-DB FEATURE
table and exit.
:param feature: Name of the feature to wait.
"""
debug_msg("BEGIN")
init()
set_owner, fallback = read_config(feature)
current_owner, remote_state, _ = read_state(feature)
docker_id = container_id(feature)
pend_wait_secs = 0
if not docker_id and fallback:
pend_wait_secs = get_config_data(
SONIC_CTR_CONFIG_PEND_SECS, DEFAULT_PEND_SECS)
debug_msg("{}: set_owner:{} ct_owner:{} state:{} id:{} pend={}".format(
feature, set_owner, current_owner, remote_state, docker_id,
pend_wait_secs))
while not docker_id:
if fallback:
pend_wait_secs = pend_wait_secs - WAIT_POLL_SECS
if pend_wait_secs < 0:
break
time.sleep(WAIT_POLL_SECS)
current_owner, remote_state, docker_id = read_state(feature)
debug_msg("wait_loop: {} = {} {} {}".format(feature, current_owner, remote_state, docker_id))
if (remote_state == "pending"):
update_data(feature, {REMOTE_STATE: "ready"})
if not docker_id:
# Clear remote state and exit.
# systemd would restart and fallback to local
update_data(feature, { REMOTE_STATE: "none" })
debug_msg("{}: Exiting to fallback as remote is *not* starting".
format(feature))
else:
debug_msg("END -- transitioning to docker wait")
docker_action("wait", docker_id)
def main():
parser=argparse.ArgumentParser(description="container commands for start/stop/wait/kill/id")
parser.add_argument("action", choices=["start", "stop", "wait", "kill", "id"])
parser.add_argument("name")
args = parser.parse_args()
if args.action == "start":
container_start(args.name)
elif args.action == "stop":
container_stop(args.name)
elif args.action == "kill":
container_kill(args.name)
elif args.action == "wait":
container_wait(args.name)
elif args.action == "id":
id = container_id(args.name)
print(id)
if __name__ == "__main__":
main()

291
src/sonic-ctrmgrd/ctrmgr/container_startup.py Executable file
View File

@ -0,0 +1,291 @@
#!/usr/bin/env python3
import argparse
import datetime
import inspect
import json
import subprocess
import syslog
import time
from swsscommon import swsscommon
# DB field names
SET_OWNER = "set_owner"
CURRENT_OWNER = "current_owner"
UPD_TIMESTAMP = "update_time"
DOCKER_ID = "container_id"
REMOTE_STATE = "remote_state"
VERSION = "container_version"
SYSTEM_STATE = "system_state"
KUBE_LABEL_TABLE = "KUBE_LABELS"
KUBE_LABEL_SET_KEY = "SET"
UNIT_TESTING = 0
def debug_msg(m):
msg = "{}: {}".format(inspect.stack()[1][3], m)
print(msg)
syslog.syslog(syslog.LOG_DEBUG, msg)
def _get_version_key(feature, version):
# Coin label for version control
return "{}_{}_enabled".format(feature, version)
def read_data(feature):
state_data = {
CURRENT_OWNER: "none",
UPD_TIMESTAMP: "",
DOCKER_ID: "",
REMOTE_STATE: "none",
VERSION: "0.0.0",
SYSTEM_STATE: ""
}
set_owner = "local"
# read owner from config-db and current state data from state-db.
db = swsscommon.DBConnector("CONFIG_DB", 0)
tbl = swsscommon.Table(db, 'FEATURE')
data = dict(tbl.get(feature)[1])
if (SET_OWNER in data):
set_owner = data[SET_OWNER]
state_db = swsscommon.DBConnector("STATE_DB", 0)
tbl = swsscommon.Table(state_db, 'FEATURE')
state_data.update(dict(tbl.get(feature)[1]))
return (state_db, set_owner, state_data)
def read_fields(state_db, feature, fields):
# Read directly from STATE-DB, given fields
# for given feature.
# Fields is a list of tuples (<field name>, <default val>)
#
tbl = swsscommon.Table(state_db, 'FEATURE')
ret = []
# tbl.get for non-existing feature would return
# [False, {} ]
#
data = dict(tbl.get(feature)[1])
for (field, default) in fields:
val = data[field] if field in data else default
ret += [val]
return tuple(ret)
def check_version_blocked(state_db, feature, version):
# Ensure this version is *not* blocked explicitly.
#
tbl = swsscommon.Table(state_db, KUBE_LABEL_TABLE)
labels = dict(tbl.get(KUBE_LABEL_SET_KEY)[1])
key = _get_version_key(feature, version)
return (key in labels) and (labels[key].lower() == "false")
def drop_label(state_db, feature, version):
# Mark given feature version as dropped in labels.
# Update is done in state-db.
# ctrmgrd sets it with kube API server per reaschability
tbl = swsscommon.Table(state_db, KUBE_LABEL_TABLE)
name = _get_version_key(feature, version)
tbl.set(KUBE_LABEL_SET_KEY, [ (name, "false")])
def update_data(state_db, feature, data):
# Update STATE-DB entry for this feature with given data
#
debug_msg("{}: {}".format(feature, str(data)))
tbl = swsscommon.Table(state_db, "FEATURE")
tbl.set(feature, list(data.items()))
def get_docker_id():
# Read the container-id
# Note: This script runs inside the context of container
#
cmd = 'cat /proc/self/cgroup | grep -e ":memory:" | rev | cut -f1 -d\'/\' | rev'
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
output = proc.communicate()[0].decode("utf-8")
return output.strip()[:12]
def instance_higher(feature, ct_version, version):
# Compares given version against current version in STATE-DB.
# Return True if this version is higher than current.
#
ret = False
if ct_version:
ct = ct_version.split('.')
nxt = version.split('.')
for cs, ns in zip(ct, nxt):
c = int(cs)
n = int(ns)
if n < c:
break
elif n > c:
ret = True
break
else:
# Empty version implies no one is running.
ret = True
debug_msg("compare version: new:{} current:{} res={}".format(
version, ct_version, ret))
return ret
def is_active(feature, system_state):
# Check current system state of the feature
if system_state == "up":
return True
else:
syslog.syslog(syslog.LOG_ERR, "Found inactive for {}".format(feature))
return False
def update_state(state_db, feature, owner=None, version=None):
"""
sets owner, version & container-id for this feature in state-db.
If owner is local, update label to block remote deploying same version or
if kube, sets state to "running".
"""
data = {
CURRENT_OWNER: owner,
DOCKER_ID: get_docker_id() if owner != "local" else feature,
UPD_TIMESTAMP: str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")),
"hello": "world",
VERSION: version
}
if (owner == "local"):
# Disable deployment of this version as available locally
drop_label(state_db, feature, version)
else:
data[REMOTE_STATE] = "running"
debug_msg("{} up data:{}".format(feature, str(data)))
update_data(state_db, feature, data)
def do_freeze(feat, m):
# Exiting will kick off the container to run.
# So sleep forever with periodic logs.
#
while True:
syslog.syslog(syslog.LOG_ERR, "Blocking .... feat:{} docker_id:{} msg:{}".format(
feat, get_docker_id(), m))
if UNIT_TESTING:
break
time.sleep(60)
def container_up(feature, owner, version):
"""
This is called by container upon post start.
The container will run its application, only upon this call
complete.
This call does the basic check for if this starting-container can be allowed
to run based on current state, and owner & version of this starting
container.
If allowed to proceed, this info is recorded in state-db and return
to enable container start the main application. Else it proceeds to
sleep forever, blocking the container from starting the main application.
"""
debug_msg("BEGIN")
(state_db, set_owner, state_data) = read_data(feature)
debug_msg("args: feature={}, owner={}, version={} DB: set_owner={} state_data={}".format(
feature, owner, version, set_owner, json.dumps(state_data, indent=4)))
if owner == "local":
update_state(state_db, feature, owner, version)
else:
if (set_owner == "local"):
do_freeze(feature, "bail out as set_owner is local")
return
if not is_active(feature, state_data[SYSTEM_STATE]):
do_freeze(feature, "bail out as system state not active")
return
if check_version_blocked(state_db, feature, version):
do_freeze(feature, "This version is marked disabled. Exiting ...")
return
if not instance_higher(feature, state_data[VERSION], version):
# TODO: May Remove label <feature_name>_<version>_enabled
# Else kubelet will continue to re-deploy every 5 mins, until
# master removes the lable to un-deploy.
#
do_freeze(feature, "bail out as current deploy version {} is not higher".
format(version))
return
update_data(state_db, feature, { VERSION: version })
mode = state_data[REMOTE_STATE]
if mode in ("none", "running", "stopped"):
update_data(state_db, feature, { REMOTE_STATE: "pending" })
mode = "pending"
else:
debug_msg("{}: Skip remote_state({}) update".format(feature, mode))
i = 0
while (mode != "ready"):
if (i % 10) == 0:
debug_msg("{}: remote_state={}. Waiting to go ready".format(feature, mode))
i += 1
time.sleep(2)
mode, db_version = read_fields(state_db,
feature, [(REMOTE_STATE, "none"), (VERSION, "")])
if version != db_version:
# looks like another instance has overwritten. Exit for now.
# If this happens to be higher version, next deploy by kube will fix
# This is a very rare window of opportunity, for this version to be higher.
#
do_freeze(feature,
"bail out as current deploy version={} is different than {}. re-deploy higher one".
format(version, db_version))
return
if UNIT_TESTING:
return
update_state(state_db, feature, owner, version)
debug_msg("END")
def main():
parser = argparse.ArgumentParser(description="container_startup <feature> kube/local [<version>]")
parser.add_argument("-f", "--feature", required=True)
parser.add_argument("-o", "--owner", choices=["local", "kube"], required=True)
parser.add_argument("-v", "--version", required=True)
args = parser.parse_args()
container_up(args.feature, args.owner, args.version)
if __name__ == "__main__":
main()

147
src/sonic-ctrmgrd/ctrmgr/ctrmgr_tools.py Executable file
View File

@ -0,0 +1,147 @@
#! /usr/bin/env python3
import argparse
import json
import os
import syslog
import docker
from swsscommon import swsscommon
CTR_NAMES_FILE = "/usr/share/sonic/templates/ctr_image_names.json"
LATEST_TAG = "latest"
# DB fields
CT_OWNER = 'current_owner'
CT_ID = 'container_id'
SYSTEM_STATE = 'system_state'
def _get_local_image_name(feature):
d = {}
if os.path.exists(CTR_NAMES_FILE):
with open(CTR_NAMES_FILE, "r") as s:
d = json.load(s)
return d[feature] if feature in d else None
def _remove_container(client, feature):
try:
# Remove stopped container instance, if any for this feature
container = client.containers.get(feature)
container.remove()
syslog.syslog(syslog.LOG_INFO, "Removed container for {}".
format(feature))
return 0
except Exception as err:
syslog.syslog(syslog.LOG_INFO, "No container to remove for {} err={}".
format(feature, str(err)))
return -1
def _tag_container_image(feature, container_id, image_name, image_tag):
client = docker.from_env()
try:
container = client.containers.get(container_id)
# Tag this image for given name & tag
container.image.tag(image_name, image_tag, force=True)
syslog.syslog(syslog.LOG_INFO,
"Tagged image for {} with container-id={} to {}:{}".
format(feature, container_id, image_name, image_tag))
ret = _remove_container(client, feature)
return ret
except Exception as err:
syslog.syslog(syslog.LOG_ERR, "Image tag: container:{} {}:{} failed with {}".
format(container_id, image_name, image_tag, str(err)))
return -1
def tag_feature(feature=None, image_name=None, image_tag=None):
ret = 0
state_db = swsscommon.DBConnector("STATE_DB", 0)
tbl = swsscommon.Table(state_db, 'FEATURE')
keys = tbl.getKeys()
for k in keys:
if (not feature) or (k == feature):
d = dict(tbl.get(k)[1])
owner = d.get(CT_OWNER, "")
id = d.get(CT_ID, "")
if not image_name:
image_name = _get_local_image_name(k)
if not image_tag:
image_tag = LATEST_TAG
if id and (owner == "kube") and image_name:
ret = _tag_container_image(k, id, image_name, image_tag)
else:
syslog.syslog(syslog.LOG_ERR,
"Skip to tag feature={} image={} tag={}".format(
k, str(image_name), str(image_tag)))
ret = -1
image_name=None
return ret
def func_tag_feature(args):
return tag_feature(args.feature, args.image_name, args.image_tag)
def func_tag_all(args):
return tag_feature()
def func_kill_all(args):
client = docker.from_env()
state_db = swsscommon.DBConnector("STATE_DB", 0)
tbl = swsscommon.Table(state_db, 'FEATURE')
keys = tbl.getKeys()
for k in keys:
data = dict(tbl.get(k)[1])
is_up = data.get(SYSTEM_STATE, "").lower() == "up"
id = data.get(CT_ID, "") if is_up else ""
if id:
try:
container = client.containers.get(id)
container.kill()
syslog.syslog(syslog.LOG_INFO,
"Killed container for {} with id={}".format(k, id))
except Exception as err:
syslog.syslog(syslog.LOG_ERR,
"kill: feature={} id={} unable to get container".
format(k, id))
return -1
return 0
def main():
parser = argparse.ArgumentParser(description="Remote container mgmt tools")
subparsers = parser.add_subparsers(title="ctrmgr_tools")
parser_tag = subparsers.add_parser('tag')
parser_tag.add_argument("-f", "--feature", required=True)
parser_tag.add_argument("-n", "--image-name")
parser_tag.add_argument("-t", "--image-tag")
parser_tag.set_defaults(func=func_tag_feature)
parser_tag_all = subparsers.add_parser('tag-all')
parser_tag_all.set_defaults(func=func_tag_all)
parser_kill_all = subparsers.add_parser('kill-all')
parser_kill_all.set_defaults(func=func_kill_all)
args = parser.parse_args()
if len(args.__dict__) < 1:
parser.print_help()
return -1
ret = args.func(args)
return ret
if __name__ == "__main__":
main()

598
src/sonic-ctrmgrd/ctrmgr/ctrmgrd.py Executable file
View File

@ -0,0 +1,598 @@
#!/usr/bin/env python3
import datetime
import inspect
import json
import os
import sys
import syslog
from collections import defaultdict
from swsscommon import swsscommon
from sonic_py_common import device_info
sys.path.append(os.path.dirname(os.path.realpath(__file__)))
import kube_commands
UNIT_TESTING = 0
UNIT_TESTING_ACTIVE = 0
# Kube config file
SONIC_CTR_CONFIG = "/etc/sonic/remote_ctr.config.json"
CONFIG_DB_NAME = "CONFIG_DB"
STATE_DB_NAME = "STATE_DB"
# DB SERVER
SERVER_TABLE = "KUBERNETES_MASTER"
SERVER_KEY = "SERVER"
CFG_SER_IP = "ip"
CFG_SER_PORT = "port"
CFG_SER_DISABLE = "disable"
CFG_SER_INSECURE = "insecure"
ST_SER_IP = "ip"
ST_SER_PORT = "port"
ST_SER_CONNECTED = "connected"
ST_SER_UPDATE_TS = "update_time"
# DB FEATURE
FEATURE_TABLE = "FEATURE"
CFG_FEAT_OWNER = "set_owner"
CFG_FEAT_NO_FALLBACK = "no_fallback_to_local"
CFG_FEAT_FAIL_DETECTION = "remote_fail_detection"
ST_FEAT_OWNER = "current_owner"
ST_FEAT_UPDATE_TS = "update_time"
ST_FEAT_CTR_ID = "container_id"
ST_FEAT_CTR_VER = "container_version"
ST_FEAT_REMOTE_STATE = "remote_state"
ST_FEAT_SYS_STATE = "system_state"
KUBE_LABEL_TABLE = "KUBE_LABELS"
KUBE_LABEL_SET_KEY = "SET"
remote_connected = False
dflt_cfg_ser = {
CFG_SER_IP: "",
CFG_SER_PORT: "6443",
CFG_SER_DISABLE: "false",
CFG_SER_INSECURE: "false"
}
dflt_st_ser = {
ST_SER_IP: "",
ST_SER_PORT: "",
ST_SER_CONNECTED: "",
ST_SER_UPDATE_TS: ""
}
dflt_cfg_feat= {
CFG_FEAT_OWNER: "local",
CFG_FEAT_NO_FALLBACK: "false",
CFG_FEAT_FAIL_DETECTION: "300"
}
dflt_st_feat= {
ST_FEAT_OWNER: "none",
ST_FEAT_UPDATE_TS: "",
ST_FEAT_CTR_ID: "",
ST_FEAT_CTR_VER: "",
ST_FEAT_REMOTE_STATE: "none",
ST_FEAT_SYS_STATE: ""
}
JOIN_LATENCY = "join_latency_on_boot_seconds"
JOIN_RETRY = "retry_join_interval_seconds"
LABEL_RETRY = "retry_labels_update_seconds"
remote_ctr_config = {
JOIN_LATENCY: 10,
JOIN_RETRY: 10,
LABEL_RETRY: 2
}
def log_debug(m):
msg = "{}: {}".format(inspect.stack()[1][3], m)
print(msg)
syslog.syslog(syslog.LOG_DEBUG, msg)
def log_error(m):
syslog.syslog(syslog.LOG_ERR, msg)
def log_info(m):
syslog.syslog(syslog.LOG_INFO, msg)
def ts_now():
return str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
def is_systemd_active(feat):
if not UNIT_TESTING:
status = os.system('systemctl is-active --quiet {}'.format(feat))
else:
status = UNIT_TESTING_ACTIVE
log_debug("system status for {}: {}".format(feat, str(status)))
return status == 0
def restart_systemd_service(server, feat, owner):
log_debug("Restart service {} to owner:{}".format(feat, owner))
if not UNIT_TESTING:
status = os.system("systemctl restart {}".format(feat))
else:
server.mod_db_entry(STATE_DB_NAME,
FEATURE_TABLE, feat, {"restart": "true"})
status = 0
if status != 0:
syslog.syslog(syslog.LOG_ERR,
"Failed to restart {} to switch to {}".format(feat, owner))
else:
syslog.syslog(syslog.LOG_INFO,
"Restarted {} to switch to {}".format(feat, owner))
return status
def init():
if os.path.exists(SONIC_CTR_CONFIG):
with open(SONIC_CTR_CONFIG, "r") as s:
d = json.load(s)
remote_ctr_config.update(d)
class MainServer:
""" Implements main io-loop of the application
Accept handler registration per db/table.
Call handler on update
"""
SELECT_TIMEOUT = 1000
def __init__(self):
""" Constructor """
self.db_connectors = {}
self.selector = swsscommon.Select()
self.callbacks = defaultdict(lambda: defaultdict(list)) # db -> table -> handlers[]
self.timer_handlers = defaultdict(list)
self.subscribers = set()
def register_db(self, db_name):
""" Get DB connector, if not there """
if db_name not in self.db_connectors:
self.db_connectors[db_name] = swsscommon.DBConnector(db_name, 0)
def register_timer(self, ts, handler):
""" Register timer based handler.
The handler will be called on/after give timestamp, ts
"""
self.timer_handlers[ts].append(handler)
def register_handler(self, db_name, table_name, handler):
"""
Handler registration for any update in given table
in given db. The handler will be called for any update
to the table in this db.
"""
self.register_db(db_name)
if table_name not in self.callbacks[db_name]:
conn = self.db_connectors[db_name]
subscriber = swsscommon.SubscriberStateTable(conn, table_name)
self.subscribers.add(subscriber)
self.selector.addSelectable(subscriber)
self.callbacks[db_name][table_name].append(handler)
def get_db_entry(self, db_name, table_name, key):
""" Return empty dict if key not present """
conn = self.db_connectors[db_name]
tbl = swsscommon.Table(conn, table_name)
return dict(tbl.get(key)[1])
def mod_db_entry(self, db_name, table_name, key, data):
""" Modify entry for given table|key with given dict type data """
conn = self.db_connectors[db_name]
tbl = swsscommon.Table(conn, table_name)
print("mod_db_entry: db={} tbl={} key={} data={}".format(db_name, table_name, key, str(data)))
tbl.set(key, list(data.items()))
def set_db_entry(self, db_name, table_name, key, data):
""" Set given data as complete data, which includes
removing any fields that are in DB but not in data
"""
conn = self.db_connectors[db_name]
tbl = swsscommon.Table(conn, table_name)
ct_data = dict(tbl.get(key)[1])
for k in ct_data:
if k not in data:
# Drop fields that are not in data
tbl.hdel(key, k)
tbl.set(key, list(data.items()))
def run(self):
""" Main loop """
while True:
timeout = MainServer.SELECT_TIMEOUT
ct_ts = datetime.datetime.now()
while self.timer_handlers:
k = sorted(self.timer_handlers.keys())[0]
if k <= ct_ts:
lst = self.timer_handlers[k]
del self.timer_handlers[k]
for fn in lst:
fn()
else:
timeout = (k - ct_ts).seconds
break
state, _ = self.selector.select(timeout)
if state == self.selector.TIMEOUT:
continue
elif state == self.selector.ERROR:
if not UNIT_TESTING:
raise Exception("Received error from select")
else:
print("Skipped Exception; Received error from select")
return
for subscriber in self.subscribers:
key, op, fvs = subscriber.pop()
if not key:
continue
log_debug("Received message : '%s'" % str((key, op, fvs)))
for callback in (self.callbacks
[subscriber.getDbConnector().getDbName()]
[subscriber.getTableName()]):
callback(key, op, dict(fvs))
def set_node_labels(server):
labels = {}
version_info = (device_info.get_sonic_version_info() if not UNIT_TESTING
else { "build_version": "20201230.111"})
dev_data = server.get_db_entry(CONFIG_DB_NAME, 'DEVICE_METADATA',
'localhost')
dep_type = dev_data['type'] if 'type' in dev_data else "unknown"
labels["sonic_version"] = version_info['build_version']
labels["hwsku"] = device_info.get_hwsku() if not UNIT_TESTING else "mock"
labels["deployment_type"] = dep_type
server.mod_db_entry(STATE_DB_NAME,
KUBE_LABEL_TABLE, KUBE_LABEL_SET_KEY, labels)
def _update_entry(ct, upd):
# Helper function, to update with case lowered.
ret = dict(ct)
for (k, v) in upd.items():
ret[k.lower()] = v.lower()
return ret
#
# SERVER-config changes:
# Act if IP or disable changes.
# If disabled or IP removed:
# reset connection
# else:
# join
# Update state-DB appropriately
#
class RemoteServerHandler:
def __init__(self, server):
""" Register self for updates """
self.server = server
server.register_handler(
CONFIG_DB_NAME, SERVER_TABLE, self.on_config_update)
self.cfg_server = _update_entry(dflt_cfg_ser, server.get_db_entry(
CONFIG_DB_NAME, SERVER_TABLE, SERVER_KEY))
log_debug("startup config: {}".format(str(self.cfg_server)))
server.register_db(STATE_DB_NAME)
self.st_server = _update_entry(dflt_st_ser, server.get_db_entry(
STATE_DB_NAME, SERVER_TABLE, SERVER_KEY))
self.start_time = datetime.datetime.now()
if not self.st_server[ST_FEAT_UPDATE_TS]:
# This is upon system start. Sleep 10m before join
self.start_time += datetime.timedelta(
seconds=remote_ctr_config[JOIN_LATENCY])
server.register_timer(self.start_time, self.handle_update)
self.pending = True
log_debug("Pause to join {} seconds @ {}".format(
remote_ctr_config[JOIN_LATENCY], self.start_time))
else:
self.pending = False
self.handle_update()
def on_config_update(self, key, op, data):
""" On config update """
if key != SERVER_KEY:
return
cfg_data = _update_entry(dflt_cfg_ser, data)
if self.cfg_server == cfg_data:
log_debug("No change in server config")
return
log_debug("Received config update: {}".format(str(data)))
self.cfg_server = cfg_data
if self.pending:
tnow = datetime.datetime.now()
if tnow < self.start_time:
# Pausing for initial latency since reboot or last retry
due_secs = (self.start_time - tnow).seconds
else:
due_secs = 0
log_debug("Pending to start in {} seconds at {}".format(
due_secs, self.start_time))
return
self.handle_update()
def handle_update(self):
# Called upon start and upon any config-update only.
# Called by timer / main thread.
# Main thread calls only if timer is not running.
#
self.pending = False
ip = self.cfg_server[CFG_SER_IP]
disable = self.cfg_server[CFG_SER_DISABLE] != "false"
pre_state = dict(self.st_server)
log_debug("server: handle_update: disable={} ip={}".format(disable, ip))
if disable or not ip:
self.do_reset()
else:
self.do_join(ip, self.cfg_server[ST_SER_PORT],
self.cfg_server[CFG_SER_INSECURE])
if pre_state != self.st_server:
self.st_server[ST_SER_UPDATE_TS] = ts_now()
self.server.set_db_entry(STATE_DB_NAME,
SERVER_TABLE, SERVER_KEY, self.st_server)
log_debug("Update server state={}".format(str(self.st_server)))
def do_reset(self):
global remote_connected
kube_commands.kube_reset_master(True)
self.st_server[ST_SER_CONNECTED] = "false"
log_debug("kube_reset_master called")
remote_connected = False
def do_join(self, ip, port, insecure):
global remote_connected
(ret, out, err) = kube_commands.kube_join_master(
ip, port, insecure)
if ret == 0:
self.st_server[ST_SER_CONNECTED] = "true"
self.st_server[ST_SER_IP] = ip
self.st_server[ST_SER_PORT] = port
remote_connected = True
set_node_labels(self.server)
log_debug("kube_join_master succeeded")
else:
# Ensure state reflects the current status
self.st_server[ST_SER_CONNECTED] = "false"
remote_connected = False
# Join failed. Retry after an interval
self.start_time = datetime.datetime.now()
self.start_time += datetime.timedelta(
seconds=remote_ctr_config[JOIN_RETRY])
self.server.register_timer(self.start_time, self.handle_update)
self.pending = True
log_debug("kube_join_master failed retry after {} seconds @{}".
format(remote_ctr_config[JOIN_RETRY], self.start_time))
#
# Feature changes
#
# Handle Set_owner change:
# restart service and/or label add/drop
#
# Handle remote_state change:
# When pending, trigger restart
#
class FeatureTransitionHandler:
def __init__(self, server):
self.server = server
self.cfg_data = defaultdict(lambda: defaultdict(str))
self.st_data = defaultdict(lambda: defaultdict(str))
server.register_handler(
CONFIG_DB_NAME, FEATURE_TABLE, self.on_config_update)
server.register_handler(
STATE_DB_NAME, FEATURE_TABLE, self.on_state_update)
return
def handle_update(self, feat, set_owner, ct_owner, remote_state):
# Called upon startup once for every feature in config & state DBs.
# There after only called upon changes in either that requires action
#
if not is_systemd_active(feat):
# Nothing todo, if system state is down
return
label_add = set_owner == "kube"
service_restart = False
if set_owner == "local":
if ct_owner != "local":
service_restart = True
else:
if (ct_owner != "none") and (remote_state == "pending"):
service_restart = True
log_debug(
"feat={} set={} ct={} state={} restart={} label_add={}".format(
feat, set_owner, ct_owner, remote_state, service_restart,
label_add))
# read labels and add/drop if different
self.server.mod_db_entry(STATE_DB_NAME, KUBE_LABEL_TABLE, KUBE_LABEL_SET_KEY,
{ "{}_enabled".format(feat): ("true" if label_add else "false") })
# service_restart
if service_restart:
restart_systemd_service(self.server, feat, set_owner)
def on_config_update(self, key, op, data):
# Hint/Note:
# If the key don't pre-exist:
# This attempt to read will create the key for given
# field with empty string as value and return empty string
init = key not in self.cfg_data
old_set_owner = self.cfg_data[key][CFG_FEAT_OWNER]
self.cfg_data[key] = _update_entry(dflt_cfg_feat, data)
set_owner = self.cfg_data[key][CFG_FEAT_OWNER]
if (not init) and (old_set_owner == set_owner):
# No change, bail out
log_debug("No change in feat={} set_owner={}. Bail out.".format(
key, set_owner))
return
if key in self.st_data:
log_debug("{} init={} old_set_owner={} owner={}".format(key, init, old_set_owner, set_owner))
self.handle_update(key, set_owner,
self.st_data[key][ST_FEAT_OWNER],
self.st_data[key][ST_FEAT_REMOTE_STATE])
else:
# Handle it upon data from state-db, which must come
# if systemd is up
log_debug("Yet to get state info key={}. Bail out.".format(key))
return
def on_state_update(self, key, op, data):
# Hint/Note:
# If the key don't pre-exist:
# This attempt to read will create the key for given
# field with empty string as value and return empty string
init = key not in self.st_data
old_remote_state = self.st_data[key][ST_FEAT_REMOTE_STATE]
self.st_data[key] = _update_entry(dflt_st_feat, data)
remote_state = self.st_data[key][ST_FEAT_REMOTE_STATE]
if (not init) and (
(old_remote_state == remote_state) or (remote_state != "pending")):
# no change or nothing to do.
return
if key in self.cfg_data:
log_debug("{} init={} old_remote_state={} remote_state={}".format(key, init, old_remote_state, remote_state))
self.handle_update(key, self.cfg_data[key][CFG_FEAT_OWNER],
self.st_data[key][ST_FEAT_OWNER],
remote_state)
return
#
# Label re-sync
# When labels applied at remote are pending, start monitoring
# API server reachability and trigger the label handler to
# apply the labels
#
# Non-empty KUBE_LABELS|PENDING, monitor API Server reachability
#
class LabelsPendingHandler:
def __init__(self, server):
server.register_handler(STATE_DB_NAME, KUBE_LABEL_TABLE, self.on_update)
self.server = server
self.pending = False
self.set_labels = {}
return
def on_update(self, key, op, data):
# For any update sync with kube API server.
# Don't optimize, as API server could differ with DB's contents
# in many ways.
#
if (key == KUBE_LABEL_SET_KEY):
self.set_labels = dict(data)
else:
return
if remote_connected and not self.pending:
self.update_node_labels()
else:
log_debug("Skip label update: connected:{} pending:{}".
format(remote_connected, self.pending))
def update_node_labels(self):
# Called on config update by main thread upon init or config-change or
# it was not connected during last config update
# NOTE: remote-server-handler forces a config update notification upon
# join.
# Or it could be called by timer thread, if last upate to API server
# failed.
#
self.pending = False
ret = kube_commands.kube_write_labels(self.set_labels)
if (ret != 0):
self.pending = True
pause = remote_ctr_config[LABEL_RETRY]
ts = (datetime.datetime.now() + datetime.timedelta(seconds=pause))
self.server.register_timer(ts, self.update_node_labels)
log_debug("ret={} set={} pending={}".format(ret,
str(self.set_labels), self.pending))
return
def main():
init()
server = MainServer()
RemoteServerHandler(server)
FeatureTransitionHandler(server)
LabelsPendingHandler(server)
server.run()
print("ctrmgrd.py main called")
return 0
if __name__ == '__main__':
if os.geteuid() != 0:
exit("Please run as root. Exiting ...")
main()

14
src/sonic-ctrmgrd/ctrmgr/ctrmgrd.service Normal file
View File

@ -0,0 +1,14 @@
[Unit]
Description=Container Manager watcher daemon
Requires=updategraph.service
After=updategraph.service
[Service]
Type=simple
ExecStart=/usr/local/bin/ctrmgrd.py
Restart=always
RestartSec=30
[Install]
WantedBy=multi-user.target

391
src/sonic-ctrmgrd/ctrmgr/kube_commands.py Executable file
View File

@ -0,0 +1,391 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import argparse
import fcntl
import inspect
import json
import os
import shutil
import ssl
import subprocess
import sys
import syslog
import tempfile
import urllib.request
from urllib.parse import urlparse
import yaml
from sonic_py_common import device_info
KUBE_ADMIN_CONF = "/etc/sonic/kube_admin.conf"
KUBELET_YAML = "/var/lib/kubelet/config.yaml"
SERVER_ADMIN_URL = "https://{}/admin.conf"
LOCK_FILE = "/var/lock/kube_join.lock"
# kubectl --kubeconfig <KUBE_ADMIN_CONF> label nodes
# <device_info.get_hostname()> <label to be added>
def log_debug(m):
msg = "{}: {}".format(inspect.stack()[1][3], m)
print(msg)
syslog.syslog(syslog.LOG_DEBUG, msg)
def log_error(m):
msg = "{}: {}".format(inspect.stack()[1][3], m)
print(msg)
syslog.syslog(syslog.LOG_ERR, m)
def to_str(s):
if isinstance(s, str):
return s
if isinstance(s, bytes):
return s.decode('utf-8')
return str(s)
def _run_command(cmd, timeout=5):
""" Run shell command and return exit code, along with stdout. """
ret = 0
try:
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
(o, e) = proc.communicate(timeout)
output = to_str(o)
err = to_str(e)
if err:
ret = -1
else:
ret = proc.returncode
except subprocess.TimeoutExpired as error:
proc.kill()
output = ""
err = str(error)
ret = -1
log_debug("cmd:{}\nret={}".format(cmd, ret))
if output:
log_debug("out:{}".format(output))
if err:
log_debug("err:{}".format(err))
return (ret, output.strip(), err.strip())
def kube_read_labels():
""" Read current labels on node and return as dict. """
KUBECTL_GET_CMD = "kubectl --kubeconfig {} get nodes --show-labels |\
grep {} | tr -s ' ' | cut -f6 -d' '"
labels = {}
ret, out, _ = _run_command(KUBECTL_GET_CMD.format(
KUBE_ADMIN_CONF, device_info.get_hostname()))
if ret == 0:
lst = out.split(",")
for label in lst:
tmp = label.split("=")
labels[tmp[0]] = tmp[1]
# log_debug("{} kube labels {} ret={}".format(
# "Applied" if ret == 0 else "Failed to apply",
# json.dumps(labels, indent=4), ret))
return (ret, labels)
def kube_write_labels(set_labels):
""" Set given set_labels.
"""
KUBECTL_SET_CMD = "kubectl --kubeconfig {} label --overwrite nodes {} {}"
ret, node_labels = kube_read_labels()
if ret != 0:
log_debug("Read before set failed. Hence skipping set {}".
format(str(set_labels)))
return ret
del_label_str = ""
add_label_str = ""
for (name, val) in set_labels.items():
skip = False
if name in node_labels:
if val != node_labels[name]:
# label value can't be modified. Remove it first
# and then add
del_label_str += "{}- ".format(name)
else:
# Already exists with same value.
skip = True
if not skip:
# Add label
add_label_str += "{}={} ".format(name, val)
if add_label_str:
# First remove if any
if del_label_str:
(ret, _, _) = _run_command(KUBECTL_SET_CMD.format(
KUBE_ADMIN_CONF, device_info.get_hostname(), del_label_str.strip()))
(ret, _, _) = _run_command(KUBECTL_SET_CMD.format(
KUBE_ADMIN_CONF, device_info.get_hostname(), add_label_str.strip()))
log_debug("{} kube labels {} ret={}".format(
"Applied" if ret == 0 else "Failed to apply", add_label_str, ret))
else:
log_debug("Given labels are in sync with node labels. Hence no-op")
return ret
def func_get_labels(args):
""" args parser default function for get labels"""
ret, node_labels = kube_read_labels()
if ret != 0:
log_debug("Labels read failed.")
return ret
print(json.dumps(node_labels, indent=4))
return 0
def is_connected(server=""):
""" Check if we are currently connected """
if (os.path.exists(KUBELET_YAML) and os.path.exists(KUBE_ADMIN_CONF)):
with open(KUBE_ADMIN_CONF, 'r') as s:
d = yaml.load(s, yaml.SafeLoader)
d = d['clusters'] if 'clusters' in d else []
d = d[0] if len(d) > 0 else {}
d = d['cluster'] if 'cluster' in d else {}
d = d['server'] if 'server' in d else ""
if d:
o = urlparse(d)
if o.hostname:
return not server or server == o.hostname
return False
def func_is_connected(args):
""" Get connected state """
connected = is_connected()
print("Currently {} to Kube master".format(
"connected" if connected else "not connected"))
return 0 if connected else 1
def _take_lock():
""" Take a lock to block concurrent calls """
lock_fd = None
try:
lock_fd = open(LOCK_FILE, "w")
fcntl.lockf(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
log_debug("Lock taken {}".format(LOCK_FILE))
except IOError as e:
lock_fd = None
log_error("Lock {} failed: {}".format(LOCK_FILE, str(e)))
return lock_fd
def _download_file(server, port, insecure):
""" Download file from Kube master to assist join as node. """
if insecure:
r = urllib.request.urlopen(SERVER_ADMIN_URL.format(server),
context=ssl._create_unverified_context())
else:
r = urllib.request.urlopen(SERVER_ADMIN_URL.format(server))
(h, fname) = tempfile.mkstemp(suffix="_kube_join")
data = r.read()
os.write(h, data)
os.close(h)
# Ensure the admin.conf has given VIP as server-IP.
update_file = "{}.upd".format(fname)
cmd = r'sed "s/server:.*:{}/server: https:\/\/{}:{}/" {} > {}'.format(
str(port), server, str(port), fname, update_file)
(ret, _, err) = _run_command(cmd)
print("sed command: ret={}".format(ret))
if ret != 0:
log_error("sed update of downloaded file failed with ret={}".
format(ret))
print("sed command failed: ret={}".format(ret))
return ret
shutil.copyfile(update_file, KUBE_ADMIN_CONF)
_run_command("rm -f {} {}".format(fname, update_file))
log_debug("{} downloaded".format(KUBE_ADMIN_CONF))
return ret
def _troubleshoot_tips():
""" log troubleshoot tips which could be handy,
when in trouble with join
"""
msg = """
if join fails, check the following
a. Ensure both master & node run same or compatible k8s versions
b. Check if this node already exists in master
Use 'sudo kubectl --kubeconfig=${KUBE_ADMIN_CONF} get nodes' to list nodes at master.
If yes, delete it, as the node is attempting a new join.
'kubectl --kubeconfig=${KUBE_ADMIN_CONF} drain <node name> --ignore-daemonsets'
'kubectl --kubeconfig=${KUBE_ADMIN_CONF} delete node <node name>'
c. In Master check if all system pods are running good.
'kubectl get pods --namespace kube-system'
If any not running properly, say READY column has 0/1, decribe pod for more detail.
'kubectl --namespace kube-system describe pod <pod name>'
For additional details, look into pod's logs.
@ node: /var/log/pods/<podname>/...
@ master: 'kubectl logs -n kube-system <pod name>'
"""
(h, fname) = tempfile.mkstemp(suffix="kube_hints_")
os.write(h, str.encode(msg))
os.close(h)
log_error("Refer file {} for troubleshooting tips".format(fname))
def _do_reset(pending_join = False):
# Drain & delete self from cluster. If not, the next join would fail
#
if os.path.exists(KUBE_ADMIN_CONF):
_run_command(
"kubectl --kubeconfig {} --request-timeout 20s drain {} --ignore-daemonsets".
format(KUBE_ADMIN_CONF, device_info.get_hostname()))
_run_command("kubectl --kubeconfig {} --request-timeout 20s delete node {}".
format(KUBE_ADMIN_CONF, device_info.get_hostname()))
_run_command("kubeadm reset -f", 10)
_run_command("rm -rf /etc/cni/net.d")
if not pending_join:
_run_command("rm -f {}".format(KUBE_ADMIN_CONF))
_run_command("systemctl stop kubelet")
def _do_join(server, port, insecure):
KUBEADM_JOIN_CMD = "kubeadm join --discovery-file {} --node-name {}"
err = ""
out = ""
try:
ret = _download_file(server, port, insecure)
print("_download ret={}".format(ret))
if ret == 0:
_do_reset(True)
_run_command("modprobe br_netfilter")
(ret, _, _) = _run_command("systemctl start kubelet")
if ret == 0:
(ret, out, err) = _run_command(KUBEADM_JOIN_CMD.format(
KUBE_ADMIN_CONF, device_info.get_hostname()), timeout=60)
print("ret = {}".format(ret))
except IOError as e:
err = "Download failed: {}".format(str(e))
ret = -1
out = ""
_troubleshoot_tips()
if (ret != 0):
log_error(err)
return (ret, out, err)
def kube_join_master(server, port, insecure, force=False):
""" The main function that initiates join to master """
out = ""
err = ""
ret = 0
log_debug("join: server:{} port:{} insecure:{} force:{}".
format(server, port, insecure, force))
lock_fd = _take_lock()
if not lock_fd:
log_error("Lock {} is active; Bail out".format(LOCK_FILE))
return (-1, "", "")
if ((not force) and is_connected(server)):
_run_command("systemctl start kubelet")
err = "Master {} is already connected. "
err += "Reset or join with force".format(server)
else:
(ret, out, err) = _do_join(server, port, insecure)
log_debug("join: ret={} out:{} err:{}".format(ret, out, err))
return (ret, out, err)
def kube_reset_master(force):
err = ""
ret = 0
lock_fd = _take_lock()
if not lock_fd:
log_error("Lock {} is active; Bail out".format(LOCK_FILE))
return (-1, "")
if not force:
if not is_connected():
err = "Currently not connected to master. "
err += "Use force reset if needed"
log_debug("Not connected ... bailing out")
ret = -1
if ret == 0:
_do_reset()
else:
_run_command("systemctl stop kubelet")
return (ret, err)
def main():
syslog.openlog("kube_commands")
parser=argparse.ArgumentParser(description=
"get-labels")
subparsers = parser.add_subparsers(title='actions')
parser_get_labels = subparsers.add_parser("get-labels",
help="Get current labels on node")
parser_get_labels.set_defaults(func=func_get_labels)
parser_is_connected = subparsers.add_parser("connected",
help="Get connnected status")
parser_is_connected.set_defaults(func=func_is_connected)
if len(sys.argv) < 2:
parser.print_help()
return -1
args = parser.parse_args()
ret = args.func(args)
syslog.closelog()
return ret
if __name__ == "__main__":
if os.geteuid() != 0:
exit("Please run as root. Exiting ...")
main()
sys.exit(0)

7
src/sonic-ctrmgrd/ctrmgr/remote_ctr.config.json Normal file
View File

@ -0,0 +1,7 @@
{
"join_latency_on_boot_seconds": 300,
"retry_join_interval_seconds": 30,
"retry_labels_update_seconds": 5,
"revert_to_local_on_wait_seconds": 60
}

2
src/sonic-ctrmgrd/pytest.ini Normal file
View File

@ -0,0 +1,2 @@
[pytest]
addopts = --cov=ctrmgr --cov-report html --cov-report term --cov-report xml

5
src/sonic-ctrmgrd/setup.cfg Normal file
View File

@ -0,0 +1,5 @@
[aliases]
test=pytest
[tool:pytest]
addopts = --verbose
python_files = tests/*.py

39
src/sonic-ctrmgrd/setup.py Normal file
View File

@ -0,0 +1,39 @@
from setuptools import setup, find_packages
setup_requirements = ['pytest-runner']
test_requirements = ['pytest>=3']
# read me
with open('README.rst') as readme_file:
readme = readme_file.read()
setup(
author="sonic-dev",
author_email='remanava@microsoft.com',
python_requires='>=3.6',
classifiers=[
'Development Status :: 2 - Pre-Alpha',
'Intended Audience :: Developers',
'License :: OSI Approved :: GNU General Public License v3 (GPLv3)',
'Natural Language :: English',
'Programming Language :: Python :: 3.6',
],
description="Package contains remote container mgmt modules",
url='https://github.com/Azure/sonic-buildimage',
tests_require=[
'pytest',
'pytest-cov',
],
install_requires=['netaddr', 'pyyaml'],
license="GNU General Public License v3",
long_description=readme + '\n\n',
include_package_data=True,
name='sonic_ctrmgrd',
py_modules=[],
packages=find_packages(),
setup_requires=setup_requirements,
version='1.0.0',
scripts=['ctrmgr/container', 'ctrmgr/ctrmgr_tools.py', 'ctrmgr/kube_commands.py', 'ctrmgr/ctrmgrd.py'],
zip_safe=False,
)

0
src/sonic-ctrmgrd/tests/__init__.py Normal file
View File

666
src/sonic-ctrmgrd/tests/common_test.py Executable file
View File

@ -0,0 +1,666 @@
import copy
import importlib
import json
import os
import subprocess
import sys
import time
CONFIG_DB_NO = 4
STATE_DB_NO = 6
FEATURE_TABLE = "FEATURE"
KUBE_LABEL_TABLE = "KUBE_LABELS"
KUBE_LABEL_SET_KEY = "SET"
SERVER_TABLE = "KUBERNETES_MASTER"
SERVER_KEY = "SERVER"
CFG_SER_IP = "ip"
CFG_SER_PORT = "port"
CFG_SER_INSECURE = "insecure"
CFG_SER_DISABLE = "disable"
PRE = "pre_test"
UPD = "update_test"
POST = "post_test"
ACTIONS = "container_actions"
RETVAL = "return_value"
ARGS = "args"
NO_INIT = "no_init"
DESCR = "description"
TRIGGER_THROW = "throw_on_call"
TRIGGER_GET_THROW = "throw_on_get"
TRIGGER_RM_THROW = "throw_on_rm"
ACTIVE = "active"
CONNECTED = "remote_connected"
KUBE_CMD = "kube_commands"
KUBE_JOIN = "join"
KUBE_RESET = "reset"
KUBE_WR = "write"
KUBE_RETURN = "kube_return"
IMAGE_TAG = "image_tag"
FAIL_LOCK = "fail_lock"
DO_JOIN = "do_join"
# subproc key words
PROC_CMD = "subproc_cmd"
PROC_RUN = "skip_mock"
PROC_FAIL = "proc_fail"
PROC_THROW = "proc_throw"
PROC_OUT = "subproc_output"
PROC_ERR = "subproc_error"
PROC_KILLED = "procs_killed"
# container_start test cases
# test case 0 -- container start local
# test case 1 -- container start kube with fallback
# test case 2 -- container start kube with fallback but remote_state != none
#
sample_test_data = {
0: {
PRE: {
CONFIG_DB_NO: {
FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
STATE_DB_NO: {
FEATURE_TABLE: {
"snmp": {
"remote_state": "xxx"
}
}
}
},
UPD: {
CONFIG_DB_NO: {
FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
STATE_DB_NO: {
FEATURE_TABLE: {
"snmp": {
"remote_state": "xxx"
}
}
}
},
POST: {
CONFIG_DB_NO: {
FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
STATE_DB_NO: {
FEATURE_TABLE: {
"snmp": {
"remote_state": "none",
"system_state": "up",
"current_owner": "local",
"container_id": "snmp"
}
},
KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "false"
}
}
}
},
ACTIONS: {
"snmp": [ "start" ]
}
}
}
current_test_name = None
current_test_no = None
current_test_data = None
tables_returned = {}
mock_containers = {}
selector_returned = None
subscribers_returned = {}
kube_actions = {}
def do_start_test(tname, tno, ctdata):
global current_test_name, current_test_no, current_test_data
global tables_returned, mock_containers, selector_returned
global subscribers_returned, kube_actions
current_test_name = tname
current_test_no = tno
current_test_data = ctdata
tables_returned = {}
mock_containers = {}
selector_returned = None
subscribers_returned = {}
kube_actions = {}
mock_procs_init()
print("Starting test case {} number={}".format(tname, tno))
class mock_image:
def __init__(self, actions):
self.actions = actions
def tag(self, image_name, image_tag, force):
print("mock_image: tag name={} tag={} force={}".format(image_name, image_tag, force))
d = {}
d[IMAGE_TAG] = {
"image_name": image_name,
"image_tag": image_tag,
"force": force
}
self.actions.append(d)
class mock_container:
def __init__(self, name):
self.actions = []
self.name = name
self.image = mock_image(self.actions)
def start(self):
self.actions.append("start")
def stop(self):
self.actions.append("stop")
def kill(self):
self.actions.append("kill")
def wait(self):
self.actions.append("wait")
def remove(self):
if self.name == TRIGGER_RM_THROW:
raise Exception("container {} can't be removed".format(self.name))
else:
self.actions.append("remove")
def kill(self):
self.actions.append("kill")
class mock_container_list:
def get(self, feature):
if feature == TRIGGER_GET_THROW:
raise Exception("container not found")
else:
if not feature in mock_containers:
mock_containers[feature] = mock_container(feature)
return mock_containers[feature]
class dock_client:
def __init__(self):
self.containers = mock_container_list()
def docker_from_env_side_effect():
return dock_client()
def check_mock_containers():
global current_test_data
expected = current_test_data.get(ACTIONS, {})
found = {}
for k in mock_containers:
found[k] = mock_containers[k].actions
if expected != found:
print("Failed test={} no={}".format(current_test_name, current_test_no))
print("expected: {}".format(json.dumps(expected, indent=4)))
print("found: {}".format(json.dumps(found, indent=4)))
return -1
return 0
def check_subset(d_sub, d_all):
if type(d_sub) != type(d_all):
return -1
if not type(d_sub) is dict:
ret = 0 if d_sub == d_all else -2
return ret
for (k, v) in d_sub.items():
if not k in d_all:
return -3
ret = check_subset(v, d_all[k])
if ret != 0:
return ret
return 0
def recursive_update(d, t):
assert (type(t) is dict)
for k in t.keys():
if type(t[k]) is not dict:
d.update(t)
return
if k not in d:
d[k] = {}
recursive_update(d[k], t[k])
class Table:
def __init__(self, db, tbl):
self.db = db
self.tbl = tbl
self.data = copy.deepcopy(self.get_val(current_test_data[PRE], [db, tbl]))
# print("Table:init: db={} tbl={} data={}".format(db, tbl, json.dumps(self.data, indent=4)))
def update(self):
t = copy.deepcopy(self.get_val(current_test_data.get(UPD, {}),
[self.db, self.tbl]))
if t:
recursive_update(self.data, t)
return True
return False
def get_val(self, d, keys):
for k in keys:
d = d[k] if k in d else {}
return d
def getKeys(self):
return list(self.data.keys())
def get(self, key):
ret = copy.deepcopy(self.data.get(key, {}))
return (True, ret)
def set(self, key, items):
if key not in self.data:
self.data[key] = {}
d = self.data[key]
for (k, v) in items:
d[k] = v
def check(self):
expected = self.get_val(current_test_data, [POST, self.db, self.tbl])
ret = check_subset(expected, self.data)
if ret != 0:
print("Failed test={} no={} ret={}".format(
current_test_name, current_test_no, ret))
print("Found: {}".format(json.dumps(self.data, indent=4)))
print("expect: {}".format(json.dumps(expected, indent=4)))
return ret
return 0
db_conns = {"CONFIG_DB": CONFIG_DB_NO, "STATE_DB": STATE_DB_NO}
def conn_side_effect(arg, _):
return db_conns[arg]
def table_side_effect(db, tbl):
if not db in tables_returned:
tables_returned[db] = {}
if not tbl in tables_returned[db]:
tables_returned[db][tbl] = Table(db, tbl)
return tables_returned[db][tbl]
def check_tables_returned():
for d in tables_returned:
for t in tables_returned[d]:
ret = tables_returned[d][t].check()
if ret != 0:
return ret
return 0
class mock_selector:
SLEEP_SECS = 0
TIMEOUT = 1
ERROR = 2
MAX_CNT = 4
def __init__(self):
self.select_state = 0
self.select_cnt = 0
# print("Mock Selector constructed")
def addSelectable(self, subs):
return 0
def select(self, timeout):
# Toggle between good & timeout
#
state = self.select_state
if self.select_state == 0:
if self.SLEEP_SECS:
time.sleep(self.SLEEP_SECS)
self.select_state = self.TIMEOUT
else:
self.select_state = 0
self.select_cnt += 1
if self.select_cnt > self.MAX_CNT:
state = self.ERROR
print("select return ({}, None)".format(state))
return (state, None)
class mock_db_conn:
def __init__(self, db):
self.db_name = None
for (k, v) in db_conns.items():
if v == db:
self.db_name = k
assert self.db_name != None
def getDbName(self):
return self.db_name
class mock_subscriber:
def __init__(self, db, tbl):
self.state = PRE
self.db = db
self.tbl = tbl
self.dbconn = mock_db_conn(db)
def pop(self):
# Handles only one key; Good enough for our requirements.
mock_tbl = table_side_effect(self.db, self.tbl)
keys = []
if self.state == PRE:
self.state = UPD
keys = list(mock_tbl.data.keys())
elif self.state == UPD:
if mock_tbl.update():
keys = list(mock_tbl.data.keys())
self.state = POST
if keys:
key = keys[0]
return (key, "", mock_tbl.get(key)[1])
else:
return ("", "", {})
def getDbConnector(self):
return self.dbconn
def getTableName(self):
return self.tbl
def subscriber_side_effect(db, tbl):
global subscribers_returned
key = "db_{}_tbl_{}".format(db, tbl)
if not key in subscribers_returned:
subscribers_returned[key] = mock_subscriber(db, tbl)
return subscribers_returned[key]
def select_side_effect():
global selector_returned
if not selector_returned:
selector_returned = mock_selector()
return selector_returned
def set_mock(mock_table, mock_conn, mock_docker=None):
mock_conn.side_effect = conn_side_effect
mock_table.side_effect = table_side_effect
if mock_docker != None:
mock_docker.side_effect = docker_from_env_side_effect
def set_mock_sel(mock_sel, mock_subs):
mock_sel.side_effect = select_side_effect
mock_subs.side_effect = subscriber_side_effect
kube_return = 0
def kube_labels_side_effect(labels):
global kube_actions, kube_return
if not KUBE_CMD in kube_actions:
kube_actions[KUBE_CMD] = {}
if not kube_return:
if not KUBE_WR in kube_actions[KUBE_CMD]:
kube_actions[KUBE_CMD][KUBE_WR] = {}
kube_actions[KUBE_CMD][KUBE_WR].update(labels)
kube_return = 1
return 0
else:
kube_return = 0
return 1
def kube_join_side_effect(ip, port, insecure):
global kube_actions, kube_return
if not KUBE_CMD in kube_actions:
kube_actions[KUBE_CMD] = {}
if not kube_return:
if not KUBE_JOIN in kube_actions[KUBE_CMD]:
kube_actions[KUBE_CMD][KUBE_JOIN] = {}
kube_actions[KUBE_CMD][KUBE_JOIN][CFG_SER_IP] = ip
kube_actions[KUBE_CMD][KUBE_JOIN][CFG_SER_PORT] = port
kube_actions[KUBE_CMD][KUBE_JOIN][CFG_SER_INSECURE] = insecure
kube_return = 1
return (0, "joined", "no error")
else:
kube_return = 0
return (1, "not joined", "error")
def kube_reset_side_effect(flag):
global kube_actions
if not KUBE_CMD in kube_actions:
kube_actions[KUBE_CMD] = {}
if not KUBE_RESET in kube_actions[KUBE_CMD]:
kube_actions[KUBE_CMD][KUBE_RESET] = {}
kube_actions[KUBE_CMD][KUBE_RESET]["flag"] = (
"true" if flag else "false")
return 0
def check_kube_actions():
global kube_actions
ret = 0
expected = {}
expected[KUBE_CMD] = current_test_data.get(KUBE_CMD, {})
if expected[KUBE_CMD]:
ret = check_subset(expected, kube_actions)
if ret != 0:
print("Failed test={} no={} ret={}".format(
current_test_name, current_test_no, ret))
print("Found: {}".format(json.dumps(kube_actions, indent=4)))
print("expect: {}".format(json.dumps(expected, indent=4)))
return -1
return 0
def set_mock_kube(kube_labels, kube_join, kube_reset):
kube_labels.side_effect = kube_labels_side_effect
kube_join.side_effect = kube_join_side_effect
kube_reset.side_effect = kube_reset_side_effect
def str_comp(needle, hay):
nlen = len(needle)
hlen = len(hay)
if needle and (needle[-1] == '*'):
nlen -= 1
hlen = nlen
return (nlen == hlen) and (needle[0:nlen] == hay[0:nlen])
def check_mock_proc(index, tag):
lst_run = current_test_data.get(tag, [])
return lst_run[index] if len(lst_run) > index else False
class mock_proc:
def __init__(self, cmd, index):
self.index = index
lst_cmd = current_test_data.get(PROC_CMD, [])
assert len(lst_cmd) > index
expect = lst_cmd[index]
self.cmd = cmd
self.returncode = 0
if not str_comp(expect, cmd):
print("PROC_CMD != cmd")
print("PROC_CMD={}".format(expect))
print("MIS cmd={}".format(cmd))
assert False
self.skip_mock = check_mock_proc(index, PROC_RUN)
self.fail_mock = check_mock_proc(index, PROC_FAIL)
self.trigger_throw = check_mock_proc(index, PROC_THROW)
def do_run(self, tout):
print("do_run: {}".format(self.cmd))
self.returncode = os.system(self.cmd)
return ("", "")
def communicate(self, timeout):
if self.trigger_throw:
raise IOError()
if self.fail_mock:
self.returncode = -1
return ("", "{} failed".format(self.cmd))
if self.skip_mock:
return self.do_run(timeout)
do_throw = current_test_data.get(TRIGGER_THROW, False)
if do_throw:
self.returncode = -1
raise subprocess.TimeoutExpired(self.cmd, timeout)
out_lst = current_test_data.get(PROC_OUT, None)
err_lst = current_test_data.get(PROC_ERR, None)
if out_lst:
assert (len(out_lst) > self.index)
out = out_lst[self.index]
else:
out = ""
if err_lst:
assert (len(err_lst) > self.index)
err = err_lst[self.index]
else:
err = ""
self.returncode = 0
return (out, err)
def kill(self):
global procs_killed
procs_killed += 1
procs_killed = 0
procs_index = 0
def mock_procs_init():
global procs_killed, procs_index
procs_killed = 0
procs_index = 0
def mock_subproc_side_effect(cmd, shell=False, stdout=None, stderr=None):
global procs_index
assert shell == True
assert stdout == subprocess.PIPE
assert stderr == subprocess.PIPE
index = procs_index
procs_index += 1
return mock_proc(cmd, index)
def set_kube_mock(mock_subproc):
mock_subproc.side_effect = mock_subproc_side_effect
def create_remote_ctr_config_json():
str_conf = '\
{\n\
"join_latency_on_boot_seconds": 2,\n\
"retry_join_interval_seconds": 0,\n\
"retry_labels_update_seconds": 0,\n\
"revert_to_local_on_wait_seconds": 5\n\
}\n'
fname = "/tmp/remote_ctr.config.json"
with open(fname, "w") as s:
s.write(str_conf)
return fname
def load_mod_from_file(modname, fpath):
spec = importlib.util.spec_from_loader(modname,
importlib.machinery.SourceFileLoader(modname, fpath))
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
sys.modules[modname] = mod
return mod

389
src/sonic-ctrmgrd/tests/container_startup_test.py Executable file
View File

@ -0,0 +1,389 @@
import sys
from unittest.mock import MagicMock, patch
import pytest
from . import common_test
sys.path.append("ctrmgr")
import container_startup
# container_startup test cases
# NOTE: Ensure state-db entry is complete in PRE as we need to
# overwrite any context left behind from last test run.
#
startup_test_data = {
0: {
common_test.DESCR: "local container starting",
common_test.ARGS: "container_startup -f snmp -o local -v 20201230.11",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "local",
"container_id": "snmp",
"container_version": "20201230.11"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_20201230.11_enabled": "false"
}
}
}
}
},
1: {
common_test.DESCR: "kube container starting with set_owner as local",
common_test.ARGS: "container_startup -f snmp -o kube -v any",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no_change",
"container_version": "NO_CHANGE",
"current_owner": "no_change",
"remote_state": "no_change",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no_change",
"container_version": "NO_CHANGE",
"current_owner": "no_change",
"remote_state": "no_change",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
}
},
2: {
common_test.DESCR: "kube container starting when system not up",
common_test.ARGS: "container_startup -f snmp -o kube -v any",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no_change",
"container_version": "NO_CHANGE",
"current_owner": "no_change",
"remote_state": "no_change",
"system_state": "down"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no_change",
"container_version": "NO_CHANGE",
"current_owner": "no_change",
"remote_state": "no_change",
"system_state": "down"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
}
},
3: {
common_test.DESCR: "kube container starting with lower version",
common_test.ARGS: "container_startup -f snmp -o kube -v 20201230.11",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no_change",
"container_version": "20201230.77",
"current_owner": "no_change",
"remote_state": "no_change",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no_change",
"container_version": "20201230.77",
"current_owner": "no_change",
"remote_state": "no_change",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
}
},
4: {
common_test.DESCR: "kube container starting with mode set to pending",
common_test.ARGS: "container_startup -f snmp -o kube -v 20201230.11",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no_change",
"container_version": "20201230.10",
"current_owner": "no_change",
"remote_state": "none",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no_change",
"container_version": "20201230.11",
"current_owner": "no_change",
"remote_state": "pending",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
}
},
5: {
common_test.DESCR: "kube container starting with mode set to ready",
common_test.ARGS: "container_startup -f snmp -o kube -v 20201230.11",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "any",
"container_version": "20201230.10",
"current_owner": "any",
"remote_state": "ready",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_version": "20201230.11",
"current_owner": "kube",
"remote_state": "running",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
}
},
6: {
common_test.DESCR: "kube container starting with no current version",
common_test.ARGS: "container_startup -f snmp -o kube -v 20201230.11",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "any",
"container_version": "",
"current_owner": "any",
"remote_state": "ready",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_version": "20201230.11",
"current_owner": "kube",
"remote_state": "running",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "no_change"
}
}
}
}
},
7: {
common_test.DESCR: "kube container starting with this version blocked",
common_test.ARGS: "container_startup -f snmp -o kube -v 20201230.11",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no change",
"container_version": "no change",
"current_owner": "no change",
"remote_state": "no change",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_20201230.11_enabled": "false"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "no change",
"container_version": "no change",
"current_owner": "no change",
"remote_state": "no change",
"system_state": "up"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_20201230.11_enabled": "false"
}
}
}
}
}
}
class TestContainerStartup(object):
@patch("container_startup.swsscommon.DBConnector")
@patch("container_startup.swsscommon.Table")
def test_start(self, mock_table, mock_conn):
container_startup.UNIT_TESTING = 1
common_test.set_mock(mock_table, mock_conn)
for (i, ct_data) in startup_test_data.items():
common_test.do_start_test("container_startup", i, ct_data)
with patch('sys.argv', ct_data[common_test.ARGS].split()):
container_startup.main()
ret = common_test.check_tables_returned()
assert ret == 0

532
src/sonic-ctrmgrd/tests/container_test.py Executable file
View File

@ -0,0 +1,532 @@
import os
from unittest.mock import MagicMock, patch
import pytest
from . import common_test
common_test.load_mod_from_file("docker",
os.path.join(os.path.dirname(os.path.realpath(__file__)), "mock_docker.py"))
container = common_test.load_mod_from_file("container",
os.path.join(os.path.dirname(os.path.realpath(__file__)), "../ctrmgr/container"))
# container_start test cases
#
start_test_data = {
0: {
common_test.DESCR: "container start for local",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "xxx"
}
}
}
},
common_test.POST: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "none",
"system_state": "up",
"current_owner": "local",
"container_id": "snmp"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "false"
}
}
}
},
common_test.ACTIONS: {
"snmp": [ "start" ]
}
},
1: {
common_test.DESCR: "container start for kube with fallback possible",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "none"
}
}
}
},
common_test.POST: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "none",
"system_state": "up",
"current_owner": "local",
"container_id": "snmp"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "true"
}
}
}
},
common_test.ACTIONS: {
"snmp": [ "start" ]
}
},
2: {
common_test.DESCR: "start for kube with fallback *not* possible",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "stopped",
"current_owner": "none",
"container_id": ""
}
}
}
},
common_test.POST: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "stopped",
"system_state": "up",
"current_owner": "none",
"container_id": ""
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "true"
}
}
}
},
common_test.ACTIONS: {
}
}
}
# container_stop test cases
# test case 0 -- container stop local
# test case 1 -- container stop kube
#
stop_test_data = {
0: {
common_test.DESCR: "container stop for local",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "none",
"system_state": "up",
"current_owner": "local",
"container_id": "snmp"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "none",
"system_state": "down",
"current_owner": "none",
"container_id": "",
"container_version": ""
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "false"
}
}
}
},
common_test.ACTIONS: {
"snmp": [ "stop" ]
}
},
1: {
common_test.DESCR: "container stop for kube",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "xxx",
"system_state": "up",
"current_owner": "kube",
"remote_state": "running"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "stopped",
"system_state": "down",
"current_owner": "none",
"container_id": "",
"container_version": ""
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "false"
}
}
}
},
common_test.ACTIONS: {
"xxx": [ "stop" ]
}
}
}
# container_kill test cases
# test case 0 -- container kill local
# -- no change in state-db
# -- no label update
# test case 1 -- container kill kube -- set label
# -- no change in state-db
# -- label update
#
kill_test_data = {
0: {
common_test.DESCR: "container kill for local",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "none",
"system_state": "up",
"current_owner": "local",
"container_id": "snmp"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "none",
"system_state": "up",
"current_owner": "local",
"container_id": "snmp"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
}
}
}
},
common_test.ACTIONS: {
"snmp": [ "kill" ]
}
},
1: {
common_test.DESCR: "container kill for kube",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "xxx",
"system_state": "up",
"current_owner": "kube",
"remote_state": "running"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "xxx",
"system_state": "up",
"current_owner": "kube",
"remote_state": "running"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "false"
}
}
}
},
common_test.ACTIONS: {
"xxx": [ "kill" ]
}
}
}
# container_wait test cases
# test case 0 -- container wait local
# -- no change in state-db
# -- no label update
# test case 1 -- container wait kube with fallback
# -- change in state-db
# -- no label update
#
wait_test_data = {
0: {
common_test.DESCR: "container wait for local",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "none",
"system_state": "up",
"current_owner": "local",
"container_id": "snmp"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"remote_state": "none",
"system_state": "up",
"current_owner": "local",
"container_id": "snmp"
}
}
}
},
common_test.ACTIONS: {
"snmp": [ "wait" ]
}
},
1: {
common_test.DESCR: "container wait for kube",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "",
"system_state": "up",
"current_owner": "kube",
"remote_state": "pending"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"container_id": "",
"system_state": "up",
"current_owner": "kube",
"remote_state": "none"
}
}
}
},
common_test.ACTIONS: {
}
}
}
class TestContainer(object):
def init(self):
container.CTR_STATE_SCR_PATH = __file__
container.SONIC_CTR_CONFIG = (
common_test.create_remote_ctr_config_json())
@patch("container.swsscommon.DBConnector")
@patch("container.swsscommon.Table")
@patch("container.docker.from_env")
def test_start(self, mock_docker, mock_table, mock_conn):
self.init()
common_test.set_mock(mock_table, mock_conn, mock_docker)
for (i, ct_data) in start_test_data.items():
common_test.do_start_test("container_test:container_start", i, ct_data)
ret = container.container_start("snmp")
assert ret == 0
ret = common_test.check_tables_returned()
assert ret == 0
ret = common_test.check_mock_containers()
assert ret == 0
@patch("container.swsscommon.DBConnector")
@patch("container.swsscommon.Table")
@patch("container.docker.from_env")
def test_stop_ct(self, mock_docker, mock_table, mock_conn):
self.init()
common_test.set_mock(mock_table, mock_conn, mock_docker)
for (i, ct_data) in stop_test_data.items():
common_test.do_start_test("container_test:container_stop", i, ct_data)
ret = container.container_stop("snmp")
ret = common_test.check_tables_returned()
assert ret == 0
ret = common_test.check_mock_containers()
assert ret == 0
@patch("container.swsscommon.DBConnector")
@patch("container.swsscommon.Table")
@patch("container.docker.from_env")
def test_kill(self, mock_docker, mock_table, mock_conn):
self.init()
common_test.set_mock(mock_table, mock_conn, mock_docker)
for (i, ct_data) in kill_test_data.items():
common_test.do_start_test("container_test:container_kill", i, ct_data)
ret = container.container_kill("snmp")
ret = common_test.check_tables_returned()
assert ret == 0
ret = common_test.check_mock_containers()
assert ret == 0
@patch("container.swsscommon.DBConnector")
@patch("container.swsscommon.Table")
@patch("container.docker.from_env")
def test_wait(self, mock_docker, mock_table, mock_conn):
self.init()
common_test.set_mock(mock_table, mock_conn, mock_docker)
for (i, ct_data) in wait_test_data.items():
common_test.do_start_test("container_test:container_wait", i, ct_data)
ret = container.container_wait("snmp")
ret = common_test.check_tables_returned()
assert ret == 0
ret = common_test.check_mock_containers()
assert ret == 0
@patch("container.swsscommon.DBConnector")
@patch("container.swsscommon.Table")
@patch("container.docker.from_env")
def test_main(self, mock_docker, mock_table, mock_conn):
self.init()
common_test.set_mock(mock_table, mock_conn, mock_docker)
for (k,v) in [ ("start", start_test_data),
("stop", stop_test_data),
("kill", kill_test_data),
("wait", wait_test_data),
("id", wait_test_data)]:
common_test.do_start_test("container_main:{}".format(k), 0, v[0])
with patch('sys.argv', ['container', k, 'snmp']):
container.main()

292
src/sonic-ctrmgrd/tests/ctrmgr_tools_test.py Executable file
View File

@ -0,0 +1,292 @@
import os
import sys
from unittest.mock import MagicMock, patch
import pytest
from . import common_test
common_test.load_mod_from_file("docker",
os.path.join(os.path.dirname(os.path.realpath(__file__)), "mock_docker.py"))
sys.path.append("ctrmgr")
import ctrmgr_tools
# ctr_image_names.json data for the test cases
#
str_ctr_image_names_json = '\
{\n\
"snmp" : "docker-snmp",\n\
"lldp" : "docker-lldp"\n\
}\n'
# ctrmgr_tools test cases
# NOTE: Ensure state-db entry is complete in PRE as we need to
# overwrite any context left behind from last test run.
#
tools_test_data = {
0: {
common_test.DESCR: "Tag all features",
common_test.ARGS: "ctrmgr_tools tag-all",
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "kube",
"container_id": "snmp_docker_id"
},
"lldp": {
"current_owner": "kube",
"container_id": "lldp_docker_id"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "kube",
"container_id": "snmp_docker_id"
},
"lldp": {
"current_owner": "kube",
"container_id": "lldp_docker_id"
}
}
}
},
common_test.ACTIONS: {
"snmp_docker_id": [
{
common_test.IMAGE_TAG: {
"image_name": "docker-snmp",
"image_tag": "latest",
"force": True
}
}
],
"snmp": ["remove"],
"lldp_docker_id": [
{
common_test.IMAGE_TAG: {
"image_name": "docker-lldp",
"image_tag": "latest",
"force": True
}
}
],
"lldp": ["remove"]
}
},
1: {
common_test.DESCR: "Tag a feature",
common_test.ARGS: "ctrmgr_tools tag -f snmp",
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "kube",
"container_id": "snmp_docker_id"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "kube",
"container_id": "snmp_docker_id"
}
}
}
},
common_test.ACTIONS: {
"snmp_docker_id": [
{
common_test.IMAGE_TAG: {
"image_name": "docker-snmp",
"image_tag": "latest",
"force": True
}
}
],
"snmp": ["remove"]
}
},
2: {
common_test.DESCR: "Skip tag local feature",
common_test.ARGS: "ctrmgr_tools tag -f snmp",
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "local",
"container_id": "any"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "local",
"container_id": "any"
}
}
}
}
},
3: {
common_test.DESCR: "Invoke missing required args",
common_test.ARGS: "ctrmgr_tools",
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "local",
"container_id": "any"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "local",
"container_id": "any"
}
}
}
}
},
4: {
common_test.DESCR: "Kill all features",
common_test.ARGS: "ctrmgr_tools kill-all",
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"system_state": "up",
"current_owner": "kube",
"container_id": "snmp_docker_id"
},
"lldp": {
"system_state": "down",
"current_owner": "kube",
"container_id": "lldp_docker_id"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"system_state": "up",
"current_owner": "kube",
"container_id": "snmp_docker_id"
},
"lldp": {
"system_state": "down",
"current_owner": "kube",
"container_id": "lldp_docker_id"
}
}
}
},
common_test.ACTIONS: {
"snmp_docker_id": ["kill"]
}
},
5: {
common_test.DESCR: "Throw exception in container get",
common_test.ARGS: "ctrmgr_tools tag-all",
common_test.RETVAL: -1,
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "kube",
"container_id": common_test.TRIGGER_GET_THROW
}
}
}
}
},
6: {
common_test.DESCR: "Throw exception in container rm",
common_test.ARGS: "ctrmgr_tools tag -f throw_on_rm -n test",
common_test.RETVAL: -1,
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
common_test.TRIGGER_RM_THROW: {
"current_owner": "kube",
"container_id": "any"
}
}
}
},
common_test.ACTIONS: {
"any": [
{
common_test.IMAGE_TAG: {
"image_name": "test",
"image_tag": "latest",
"force": True
}
}
],
"throw_on_rm": []
}
},
7: {
common_test.DESCR: "Throw exception in container get",
common_test.ARGS: "ctrmgr_tools kill-all",
common_test.RETVAL: -1,
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"system_state": "up",
"current_owner": "kube",
"container_id": common_test.TRIGGER_GET_THROW
}
}
}
}
}
}
class TestCtrmgrTools(object):
@patch("ctrmgr_tools.swsscommon.DBConnector")
@patch("ctrmgr_tools.swsscommon.Table")
@patch("ctrmgr_tools.docker.from_env")
def test_tools(self, mock_docker, mock_table, mock_conn):
fname = "/tmp/ctr_image_names.json"
with open(fname, "w") as s:
s.write(str_ctr_image_names_json)
ctrmgr_tools.CTR_NAMES_FILE = fname
common_test.set_mock(mock_table, mock_conn, mock_docker)
for (i, ct_data) in tools_test_data.items():
common_test.do_start_test("ctrmgr_tools", i, ct_data)
with patch('sys.argv', ct_data[common_test.ARGS].split()):
ret = ctrmgr_tools.main()
if common_test.RETVAL in ct_data:
assert ret == ct_data[common_test.RETVAL]
ret = common_test.check_tables_returned()
assert ret == 0
ret = common_test.check_mock_containers()
assert ret == 0

488
src/sonic-ctrmgrd/tests/ctrmgrd_test.py Executable file
View File

@ -0,0 +1,488 @@
import os
import sys
from unittest.mock import MagicMock, patch
import pytest
from . import common_test
sys.path.append("ctrmgr")
import ctrmgrd
# ctrmgrd test cases
# NOTE: Ensure state-db entry is complete in PRE as we need to
# overwrite any context left behind from last test run.
#
server_test_data = {
0: {
common_test.DESCR: "Connect using init config with join delay",
common_test.ARGS: "ctrmgrd",
common_test.KUBE_RETURN: 1,
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"ip": "10.10.10.10"
}
}
}
},
common_test.UPD: {
common_test.CONFIG_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"ip": "10.10.10.10",
"insecure": "true"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"connected": "true",
"ip": "10.10.10.10",
"port": "6443"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"sonic_version": "20201230.111",
"hwsku": "mock",
"deployment_type": "unknown"
}
}
}
},
common_test.KUBE_CMD: {
common_test.KUBE_JOIN: {
"ip": "10.10.10.10",
"port": "6443",
"insecure": "true"
}
}
},
1: {
common_test.DESCR: "Connect using init config with no join delay",
common_test.ARGS: "ctrmgrd",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"ip": "10.10.10.10"
}
}
},
common_test.STATE_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"update_time": "2020-12-03 23:18:06"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"connected": "true",
"ip": "10.10.10.10",
"port": "6443"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"sonic_version": "20201230.111",
"hwsku": "mock",
"deployment_type": "unknown"
}
}
}
},
common_test.KUBE_CMD: {
common_test.KUBE_JOIN: {
"ip": "10.10.10.10",
"port": "6443",
"insecure": "false"
}
}
},
2: {
common_test.DESCR: "Join followed by reset on next update",
common_test.ARGS: "ctrmgrd",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"ip": "10.10.10.10"
}
}
},
common_test.STATE_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"update_time": "2020-12-03 23:18:06"
}
}
}
},
common_test.UPD: {
common_test.CONFIG_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"disable": "true"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.SERVER_TABLE: {
common_test.SERVER_KEY: {
"connected": "false"
}
}
}
},
common_test.KUBE_CMD: {
common_test.KUBE_JOIN: {
"ip": "10.10.10.10",
"port": "6443",
"insecure": "false"
},
common_test.KUBE_RESET: {
"flag": "true"
}
}
}
}
feature_test_data = {
0: {
common_test.DESCR: "set_owner = local with current_owner != local",
common_test.ARGS: "ctrmgrd",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "not local"
}
}
}
},
common_test.UPD: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "local"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "not local"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"restart": "true"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "false"
}
}
}
}
},
1: {
common_test.DESCR: "set_owner = kube with pending remote state",
common_test.ARGS: "ctrmgrd",
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "kube"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "any",
"remote_state": "pending"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"restart": "true"
}
},
common_test.KUBE_LABEL_TABLE: {
"SET": {
"snmp_enabled": "true"
}
}
}
}
},
2: {
common_test.DESCR: "Run with systemstate inactive. No-op",
common_test.ARGS: "ctrmgrd",
common_test.ACTIVE: 1,
common_test.PRE: {
common_test.CONFIG_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"set_owner": "any"
}
}
},
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "any"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.FEATURE_TABLE: {
"snmp": {
"current_owner": "any"
}
}
}
}
}
}
labels_test_data = {
0: {
common_test.DESCR: "simple update",
common_test.ARGS: "ctrmgrd",
common_test.CONNECTED: True,
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.KUBE_LABEL_TABLE: {
"SET": {
"foo": "bar",
"hello": "world"
}
}
}
},
common_test.UPD: {
common_test.STATE_DB_NO: {
"xyz": {
"xxx": {
"foo": "bar",
"hello": "world"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.KUBE_LABEL_TABLE: {
"SET": {
"foo": "bar",
"hello": "world"
}
}
}
},
common_test.KUBE_CMD: {
common_test.KUBE_WR: {
"foo": "bar",
"hello": "world"
}
}
},
1: {
common_test.DESCR: "simple update - not connected",
common_test.ARGS: "ctrmgrd",
common_test.CONNECTED: False,
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.KUBE_LABEL_TABLE: {
"SET": {
"foo": "bar",
"hello": "world"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.KUBE_LABEL_TABLE: {
"SET": {
"foo": "bar",
"hello": "world"
}
}
}
}
},
2: {
common_test.DESCR: "Emulate write failure",
common_test.ARGS: "ctrmgrd",
common_test.CONNECTED: True,
common_test.KUBE_RETURN: 1,
common_test.PRE: {
common_test.STATE_DB_NO: {
common_test.KUBE_LABEL_TABLE: {
"SET": {
"foo": "bar",
"hello": "world"
}
}
}
},
common_test.POST: {
common_test.STATE_DB_NO: {
common_test.KUBE_LABEL_TABLE: {
"SET": {
"foo": "bar",
"hello": "world"
}
}
}
},
common_test.KUBE_CMD: {
common_test.KUBE_WR: {
"foo": "bar",
"hello": "world"
}
}
}
}
class TestContainerStartup(object):
def init(self):
ctrmgrd.UNIT_TESTING = 1
ctrmgrd.SONIC_CTR_CONFIG = (
common_test.create_remote_ctr_config_json())
@patch("ctrmgrd.swsscommon.DBConnector")
@patch("ctrmgrd.swsscommon.Table")
@patch("ctrmgrd.swsscommon.Select")
@patch("ctrmgrd.swsscommon.SubscriberStateTable")
@patch("ctrmgrd.kube_commands.kube_reset_master")
@patch("ctrmgrd.kube_commands.kube_join_master")
@patch("ctrmgrd.kube_commands.kube_write_labels")
def test_server(self, mock_kube_wr, mock_kube_join, mock_kube_rst, mock_subs,
mock_select, mock_table, mock_conn):
self.init()
ret = 0
common_test.set_mock(mock_table, mock_conn)
common_test.set_mock_sel(mock_select, mock_subs)
common_test.set_mock_kube(mock_kube_wr, mock_kube_join, mock_kube_rst)
common_test.mock_selector.SLEEP_SECS = 1
for (i, ct_data) in server_test_data.items():
common_test.do_start_test("ctrmgrd:server", i, ct_data)
if common_test.KUBE_RETURN in ct_data:
common_test.kube_return = ct_data[common_test.KUBE_RETURN]
else:
common_test.kube_return = 0
with patch('sys.argv', ct_data[common_test.ARGS].split()):
ret = ctrmgrd.main()
ret = common_test.check_tables_returned()
assert ret == 0
ret = common_test.check_kube_actions()
assert ret == 0
common_test.mock_selector.SLEEP_SECS = 0
@patch("ctrmgrd.swsscommon.DBConnector")
@patch("ctrmgrd.swsscommon.Table")
@patch("ctrmgrd.swsscommon.Select")
@patch("ctrmgrd.swsscommon.SubscriberStateTable")
@patch("ctrmgrd.kube_commands.kube_reset_master")
@patch("ctrmgrd.kube_commands.kube_join_master")
@patch("ctrmgrd.kube_commands.kube_write_labels")
def test_feature(self, mock_kube_wr, mock_kube_join, mock_kube_rst, mock_subs,
mock_select, mock_table, mock_conn):
self.init()
ret = 0
common_test.set_mock(mock_table, mock_conn)
common_test.set_mock_sel(mock_select, mock_subs)
common_test.set_mock_kube(mock_kube_wr, mock_kube_join, mock_kube_rst)
for (i, ct_data) in feature_test_data.items():
common_test.do_start_test("ctrmgrd:feature", i, ct_data)
if common_test.ACTIVE in ct_data:
ctrmgrd.UNIT_TESTING_ACTIVE = ct_data[common_test.ACTIVE]
print("systemctl active = {}".format(ctrmgrd.UNIT_TESTING_ACTIVE))
with patch('sys.argv', ct_data[common_test.ARGS].split()):
ret = ctrmgrd.main()
ret = common_test.check_tables_returned()
assert ret == 0
@patch("ctrmgrd.swsscommon.DBConnector")
@patch("ctrmgrd.swsscommon.Table")
@patch("ctrmgrd.swsscommon.Select")
@patch("ctrmgrd.swsscommon.SubscriberStateTable")
@patch("ctrmgrd.kube_commands.kube_reset_master")
@patch("ctrmgrd.kube_commands.kube_join_master")
@patch("ctrmgrd.kube_commands.kube_write_labels")
def test_labels(self, mock_kube_wr, mock_kube_join, mock_kube_rst, mock_subs,
mock_select, mock_table, mock_conn):
self.init()
ret = 0
common_test.set_mock(mock_table, mock_conn)
common_test.set_mock_sel(mock_select, mock_subs)
common_test.set_mock_kube(mock_kube_wr, mock_kube_join, mock_kube_rst)
for (i, ct_data) in labels_test_data.items():
common_test.do_start_test("ctrmgrd:feature", i, ct_data)
ctrmgrd.remote_connected = ct_data[common_test.CONNECTED]
print("remote_connected= {}".format(ctrmgrd.remote_connected))
if common_test.KUBE_RETURN in ct_data:
common_test.kube_return = ct_data[common_test.KUBE_RETURN]
else:
common_test.kube_return = 0
with patch('sys.argv', ct_data[common_test.ARGS].split()):
ret = ctrmgrd.main()
ret = common_test.check_tables_returned()
assert ret == 0
ret = common_test.check_kube_actions()
assert ret == 0

391
src/sonic-ctrmgrd/tests/kube_commands_test.py Executable file
View File

@ -0,0 +1,391 @@
import json
import os
import shutil
import sys
from unittest.mock import MagicMock, patch
import pytest
from . import common_test
sys.path.append("ctrmgr")
import kube_commands
KUBE_ADMIN_CONF = "/tmp/kube_admin.conf"
# kube_commands test cases
# NOTE: Ensure state-db entry is complete in PRE as we need to
# overwrite any context left behind from last test run.
#
read_labels_test_data = {
0: {
common_test.DESCR: "read labels",
common_test.RETVAL: 0,
common_test.PROC_CMD: ["\
kubectl --kubeconfig {} get nodes --show-labels |\
grep None | tr -s ' ' | cut -f6 -d' '".format(KUBE_ADMIN_CONF)],
common_test.PROC_OUT: ["foo=bar,hello=world"],
common_test.POST: {
"foo": "bar",
"hello": "world"
},
common_test.PROC_KILLED: 0
},
1: {
common_test.DESCR: "read labels timeout",
common_test.TRIGGER_THROW: True,
common_test.RETVAL: -1,
common_test.PROC_CMD: ["\
kubectl --kubeconfig {} get nodes --show-labels |\
grep None | tr -s ' ' | cut -f6 -d' '".format(KUBE_ADMIN_CONF)],
common_test.POST: {
},
common_test.PROC_KILLED: 1
},
2: {
common_test.DESCR: "read labels fail",
common_test.RETVAL: -1,
common_test.PROC_CMD: ["\
kubectl --kubeconfig {} get nodes --show-labels |\
grep None | tr -s ' ' | cut -f6 -d' '".format(KUBE_ADMIN_CONF)],
common_test.PROC_OUT: [""],
common_test.PROC_ERR: ["command failed"],
common_test.POST: {
},
common_test.PROC_KILLED: 0
}
}
write_labels_test_data = {
0: {
common_test.DESCR: "write labels: skip/overwrite/new",
common_test.RETVAL: 0,
common_test.ARGS: { "foo": "bar", "hello": "World!", "test": "ok" },
common_test.PROC_CMD: [
"kubectl --kubeconfig {} get nodes --show-labels |\
grep None | tr -s ' ' | cut -f6 -d' '".format(KUBE_ADMIN_CONF),
"kubectl --kubeconfig {} label --overwrite nodes None hello-".format(
KUBE_ADMIN_CONF),
"kubectl --kubeconfig {} label --overwrite nodes None hello=World! test=ok".format(
KUBE_ADMIN_CONF)
],
common_test.PROC_OUT: ["foo=bar,hello=world", "", ""]
},
1: {
common_test.DESCR: "write labels: skip as no change",
common_test.RETVAL: 0,
common_test.ARGS: { "foo": "bar", "hello": "world" },
common_test.PROC_CMD: [
"kubectl --kubeconfig {} get nodes --show-labels |\
grep None | tr -s ' ' | cut -f6 -d' '".format(KUBE_ADMIN_CONF)
],
common_test.PROC_OUT: ["foo=bar,hello=world"]
},
2: {
common_test.DESCR: "write labels",
common_test.RETVAL: 0,
common_test.ARGS: { "any": "thing" },
common_test.RETVAL: -1,
common_test.PROC_CMD: [
"kubectl --kubeconfig {} get nodes --show-labels |\
grep None | tr -s ' ' | cut -f6 -d' '".format(KUBE_ADMIN_CONF)
],
common_test.PROC_ERR: ["read failed"]
}
}
join_test_data = {
0: {
common_test.DESCR: "Regular insecure join",
common_test.RETVAL: 0,
common_test.ARGS: ["10.3.157.24", 6443, True, False],
common_test.PROC_CMD: [
'sed *',
'rm -f *',
"kubectl --kubeconfig {} --request-timeout 20s drain None \
--ignore-daemonsets".format(KUBE_ADMIN_CONF),
"kubectl --kubeconfig {} --request-timeout 20s delete node \
None".format(KUBE_ADMIN_CONF),
"kubeadm reset -f",
"rm -rf /etc/cni/net.d",
"systemctl stop kubelet",
"modprobe br_netfilter",
"systemctl start kubelet",
"kubeadm join --discovery-file {} --node-name None".format(
KUBE_ADMIN_CONF)
],
common_test.PROC_RUN: [True, True]
},
1: {
common_test.DESCR: "Regular secure join",
common_test.RETVAL: 0,
common_test.ARGS: ["10.3.157.24", 6443, False, False],
common_test.PROC_CMD: [
'sed *',
'rm -f *',
"kubectl --kubeconfig {} --request-timeout 20s drain None \
--ignore-daemonsets".format(KUBE_ADMIN_CONF),
"kubectl --kubeconfig {} --request-timeout 20s delete node \
None".format(KUBE_ADMIN_CONF),
"kubeadm reset -f",
"rm -rf /etc/cni/net.d",
"systemctl stop kubelet",
"modprobe br_netfilter",
"systemctl start kubelet",
"kubeadm join --discovery-file {} --node-name None".format(
KUBE_ADMIN_CONF)
],
common_test.PROC_RUN: [True, True]
},
2: {
common_test.DESCR: "Skip join as already connected",
common_test.RETVAL: 0,
common_test.ARGS: ["10.3.157.24", 6443, True, False],
common_test.NO_INIT: True,
common_test.PROC_CMD: [
"systemctl start kubelet"
]
},
3: {
common_test.DESCR: "Regular join: fail file update",
common_test.RETVAL: -1,
common_test.ARGS: ["10.3.157.24", 6443, False, False],
common_test.PROC_CMD: [
'sed *',
'rm -f *',
"kubectl --kubeconfig {} --request-timeout 20s drain None \
--ignore-daemonsets".format(KUBE_ADMIN_CONF),
"kubectl --kubeconfig {} --request-timeout 20s delete node \
None".format(KUBE_ADMIN_CONF),
"kubeadm reset -f",
"rm -rf /etc/cni/net.d",
"systemctl stop kubelet",
"modprobe br_netfilter",
"systemctl start kubelet",
"kubeadm join --discovery-file {} --node-name None".format(
KUBE_ADMIN_CONF)
],
common_test.PROC_RUN: [True, True],
common_test.PROC_FAIL: [True]
},
4: {
common_test.DESCR: "Regular join: fail file update",
common_test.RETVAL: -1,
common_test.ARGS: ["10.3.157.24", 6443, False, False],
common_test.PROC_CMD: [
'sed *',
'rm -f *',
"kubectl --kubeconfig {} --request-timeout 20s drain None \
--ignore-daemonsets".format(KUBE_ADMIN_CONF),
"kubectl --kubeconfig {} --request-timeout 20s delete node \
None".format(KUBE_ADMIN_CONF),
"kubeadm reset -f",
"rm -rf /etc/cni/net.d",
"systemctl stop kubelet",
"modprobe br_netfilter",
"systemctl start kubelet",
"kubeadm join --discovery-file {} --node-name None".format(
KUBE_ADMIN_CONF)
],
common_test.PROC_RUN: [True, True],
common_test.PROC_FAIL: [True],
common_test.PROC_THROW: [True]
},
5: {
common_test.DESCR: "Regular join: fail due to unable to lock",
common_test.RETVAL: -1,
common_test.ARGS: ["10.3.157.24", 6443, False, False],
common_test.FAIL_LOCK: True
}
}
reset_test_data = {
0: {
common_test.DESCR: "non force reset",
common_test.RETVAL: 0,
common_test.DO_JOIN: True,
common_test.ARGS: [False],
common_test.PROC_CMD: [
"kubectl --kubeconfig {} --request-timeout 20s drain None \
--ignore-daemonsets".format(KUBE_ADMIN_CONF),
"kubectl --kubeconfig {} --request-timeout 20s delete node \
None".format(KUBE_ADMIN_CONF),
"kubeadm reset -f",
"rm -rf /etc/cni/net.d",
"rm -f {}".format(KUBE_ADMIN_CONF),
"systemctl stop kubelet"
]
},
1: {
common_test.DESCR: "force reset",
common_test.RETVAL: 0,
common_test.ARGS: [False],
common_test.PROC_CMD: [
"kubectl --kubeconfig {} --request-timeout 20s drain None \
--ignore-daemonsets".format(KUBE_ADMIN_CONF),
"kubectl --kubeconfig {} --request-timeout 20s delete node \
None".format(KUBE_ADMIN_CONF),
"kubeadm reset -f",
"rm -rf /etc/cni/net.d",
"rm -f {}".format(KUBE_ADMIN_CONF),
"systemctl stop kubelet"
]
},
1: {
common_test.DESCR: "force reset",
common_test.RETVAL: 0,
common_test.ARGS: [True],
common_test.PROC_CMD: [
"kubeadm reset -f",
"rm -rf /etc/cni/net.d",
"rm -f {}".format(KUBE_ADMIN_CONF),
"systemctl stop kubelet"
]
},
2: {
common_test.DESCR: "skip reset as not connected",
common_test.RETVAL: -1,
common_test.ARGS: [False],
common_test.PROC_CMD: [
"systemctl stop kubelet"
]
}
}
class TestKubeCommands(object):
def init(self):
conf_str = "\
apiVersion: v1\n\
clusters:\n\
- cluster:\n\
server: https://10.3.157.24:6443\n\
"
self.admin_conf_file = "/tmp/kube_admin_url.info"
with open(self.admin_conf_file, "w") as s:
s.write(conf_str)
kubelet_yaml = "/tmp/kubelet_config.yaml"
with open(kubelet_yaml, "w") as s:
s.close()
kube_commands.KUBELET_YAML = kubelet_yaml
kube_commands.SERVER_ADMIN_URL = "file://{}".format(self.admin_conf_file)
kube_commands.KUBE_ADMIN_CONF = KUBE_ADMIN_CONF
@patch("kube_commands.subprocess.Popen")
def test_read_labels(self, mock_subproc):
self.init()
common_test.set_kube_mock(mock_subproc)
for (i, ct_data) in read_labels_test_data.items():
common_test.do_start_test("kube:read-labels", i, ct_data)
(ret, labels) = kube_commands.kube_read_labels()
if common_test.RETVAL in ct_data:
assert ret == ct_data[common_test.RETVAL]
if common_test.PROC_KILLED in ct_data:
assert (common_test.procs_killed ==
ct_data[common_test.PROC_KILLED])
if (common_test.POST in ct_data and
(ct_data[common_test.POST] != labels)):
print("expect={} labels={} mismatch".format(
json.dumps(ct_data[common_test.POST], indent=4),
json.dumps(labels, indent=4)))
assert False
# Exercist through main
common_test.do_start_test("kube:main:read-labels", 0,
read_labels_test_data[0])
with patch('sys.argv', "kube_commands get-labels".split()):
ret = kube_commands.main()
assert ret == 0
# Exercist through main with no args
common_test.do_start_test("kube:main:none", 0, read_labels_test_data[0])
with patch('sys.argv', "kube_commands".split()):
ret = kube_commands.main()
assert ret == -1
@patch("kube_commands.subprocess.Popen")
def test_write_labels(self, mock_subproc):
self.init()
common_test.set_kube_mock(mock_subproc)
for (i, ct_data) in write_labels_test_data.items():
common_test.do_start_test("kube:write-labels", i, ct_data)
ret = kube_commands.kube_write_labels(ct_data[common_test.ARGS])
if common_test.RETVAL in ct_data:
assert ret == ct_data[common_test.RETVAL]
if common_test.PROC_KILLED in ct_data:
assert (common_test.procs_killed ==
ct_data[common_test.PROC_KILLED])
if (common_test.POST in ct_data and
(ct_data[common_test.POST] != labels)):
print("expect={} labels={} mismatch".format(
json.dumps(ct_data[common_test.POST], indent=4),
json.dumps(labels, indent=4)))
assert False
@patch("kube_commands.subprocess.Popen")
def test_join(self, mock_subproc):
self.init()
common_test.set_kube_mock(mock_subproc)
for (i, ct_data) in join_test_data.items():
lock_file = ""
common_test.do_start_test("kube:join", i, ct_data)
if not ct_data.get(common_test.NO_INIT, False):
os.system("rm -f {}".format(KUBE_ADMIN_CONF))
if ct_data.get(common_test.FAIL_LOCK, False):
lock_file = kube_commands.LOCK_FILE
kube_commands.LOCK_FILE = "/xxx/yyy/zzz"
args = ct_data[common_test.ARGS]
(ret, _, _) = kube_commands.kube_join_master(
args[0], args[1], args[2], args[3])
if common_test.RETVAL in ct_data:
assert ret == ct_data[common_test.RETVAL]
if lock_file:
kube_commands.LOCK_FILE = lock_file
# Exercist through main is_connected
common_test.do_start_test("kube:main:is_connected", 0, join_test_data[0])
with patch('sys.argv', "kube_commands connected".split()):
ret = kube_commands.main()
assert ret == 1
# test to_str()
f = "abcd"
f == kube_commands.to_str(str.encode(f))
@patch("kube_commands.subprocess.Popen")
def test_reset(self, mock_subproc):
self.init()
common_test.set_kube_mock(mock_subproc)
for (i, ct_data) in reset_test_data.items():
common_test.do_start_test("kube:reset", i, ct_data)
if ct_data.get(common_test.DO_JOIN, False):
shutil.copyfile(self.admin_conf_file, KUBE_ADMIN_CONF)
else:
os.system("rm -f {}".format(KUBE_ADMIN_CONF))
(ret, _) = kube_commands.kube_reset_master(
ct_data[common_test.ARGS][0])
if common_test.RETVAL in ct_data:
assert ret == ct_data[common_test.RETVAL]

2
src/sonic-ctrmgrd/tests/mock_docker.py Normal file
View File

@ -0,0 +1,2 @@
def from_env():
return None