Currently, the build dockers are created as a user dockers(docker-base-stretch-<user>, etc) that are
specific to each user. But the sonic dockers (docker-database, docker-swss, etc) are
created with a fixed docker name and common to all the users.
docker-database:latest
docker-swss:latest
When multiple builds are triggered on the same build server that creates parallel building issue because
all the build jobs are trying to create the same docker with latest tag.
This happens only when sonic dockers are built using native host dockerd for sonic docker image creation.
This patch creates all sonic dockers as user sonic dockers and then, while
saving and loading the user sonic dockers, it rename the user sonic
dockers into correct sonic dockers with tag as latest.
docker-database:latest <== SAVE/LOAD ==> docker-database-<user>:tag
The user sonic docker names are derived from 'DOCKER_USERNAME and DOCKER_USERTAG' make env
variable and using Jinja template, it replaces the FROM docker name with correct user sonic docker name for
loading and saving the docker image.
Why I did it
To sign SONiC kernel image and allow secure boot based system to verify SONiC image before loading into the system.
How I did it
Pass following parameter to rules/config.user
Ex:
SONIC_ENABLE_SECUREBOOT_SIGNATURE := y
SIGNING_KEY := /path/to/key/private.key
SIGNING_CERT := /path/to/public/public.cert
How to verify it
Secure boot enabled system enrolled with right public key of the, image in the platform UEFI database will able to verify image before load.
Alternatively one can verify with offline sbsign tool as below.
export SBSIGN_KEY=/abc/bcd/xyz/
sbverify --cert $SBSIGN_KEY/public_cert.cert fsroot-platform-XYZ/boot/vmlinuz-5.10.0-8-2-amd64 mage
O/P:
Signature verification OK
Implement infrastructure that allows enabling address sanitizer
for docker containers. Enable address sanitizer for SWSS container.
- Why I did it
To add a possibility to compile SONiC applications with address sanitizer (ASAN).
ASAN is a memory error detector for C/C++. It finds:
1. Use after free (dangling pointer dereference)
2. Heap buffer overflow
3. Stack buffer overflow
4. Global buffer overflow
5. Use after return
6. Use after the scope
7. Initialization order bugs
8. Memory leaks
- How I did it
By adding new ENABLE_ASAN configuration option.
- How to verify it
By default ASAN is disabled and the SONiC image is not affected.
When ASAN is enabled it inspects all allocation, deallocation, and memory usage that the application does in run time. To verify whether the application has memory errors tests that trigger memory usage of the application should be run. Ideally, the whole regression tests should be run. Memory leaks reports will be placed in /var/log/asan/ directory of SONiC host OS.
Signed-off-by: Oleksandr Ivantsiv <oivantsiv@nvidia.com>
Support saiserver v2 with python3 and thrift 0.13.0
add variables to support the saiserverv2
build different thrift in saithrift depends on saiserver version
build differernt versions of saiserver
make the saiserver and saiserver docker with version number
test done:
build two different versions of sasiserver in local build environment
add saiserver to buster
Co-authored-by: richard.yu <richard.yu@microsoft.comwq>
* [arm64]: Fix registration of the qemu interpreters
The current code doesn't properly run the container that registers the
qemu interpreters. It checks to see if the container is "known" by
Docker, but that doesn't indicate whether it's been run or not.
Therefore, just always register the qemu interpreters in the kernel, to
make sure the binary that's in the slave images that we build is used.
* [build]: Reduce the number of python calls
Modify the BLDENV and PROJECT_ROOT variables in slave.mk to be
immediate execution instead of lazy execution. Neither of these
variables should be changing for the duration of the build in each slave
container, so just run it once instead of every time they're referenced.
When running `make configure` for broadcom arm64 (where all of the slave
images are already built) on an amd64 host, this reduces the time spent
in each slave container from 4.5-5 minutes to 2 minutes.
* [sonic-slave]: Upgrade the qemu used for Bullseye arm64 to 6.1.0
Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
- Why I did it
Fixes#8898
Dockerd for multiarch build by default use host OS config file ("/etc/docker/daemon.json").
Default configuration used on host OS may not work for containers run inside sonic-slave.
"DOCKER_CONFIG_FILE_FOR_MULTIARCH" variable allows overriding path to the
config file that will be used for multiarch dockerd.
- How I did it
Added "DOCKER_CONFIG_FILE_FOR_MULTIARCH" to Makefile.work file that allow to
override path to dockerd config file through environment variable:
DOCKER_CONFIG_FILE_FOR_MULTIARCH=${path_to_file}/daemon.json make ...
If the env variable is not set build the system preserves its default behavior.
- How to verify it
Set DOCKER_CONFIG_FILE_FOR_MULTIARCH env variable
Run build
While build is running execute ps -eo pid,cmd | grep "[0-9] dockerd.*march" command
Verify that --config-file parameter is set to the same path that was specified in DOCKER_CONFIG_FILE_FOR_MULTIARCH variable.
#### Why I did it
Changes required for feature "Event Driven TechSupport Invocation & CoreDump Mgmt". [HLD](https://github.com/Azure/SONiC/pull/818 )
Requires: https://github.com/Azure/sonic-utilities/pull/1796.
Merging in any order would be fine.
Summary of the changes:
- Added the YANG Models for the new tables introduces as a part of this feature.
- Enhanced init_cfg.json with the default config required
- Added a compile Time flag which enables/disables the config required for this feature inside the init_cfg.json
- Enhanced the supervisor-proc-exit-listener script to populate `<feature>:<critical_proc> = <comm>:<pid>` info in the STATE_DB when it observes an proc exit notification for the critical processes running inside the docker.
Linkmgrd monitors link status, mux status, and link state. Has
the link becomes unhealthy, linkmgrd will trigger mux switchover
on a standby ToR ensuring uninterrupted service to servers/blades.
This PR is initial implementation of linkmgrd.
Also, docker-mux container hold packages related to maintaining and managing
mux cable. It currently runs linkmgrd binary that monitor and switches
the mux if needed.
This PR also introduces mux-container and starts linkmgrd as startup when
build is configured with INCLUDE_MUX=y
Edit: linkmgrd PR will follow.
signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
Related work items: #2315, #3146150
Why I did it
Pre-requisite: #8269
To be able to generate multi-asic KVM image.
To provide flexibility to generate single asic or both single and multi-asic images.
How I did it
Add a new build param, BUILD_MULTIASIC_KVM, if set to "y", the multi-asic VS target KVM images will be generated. If not, only single asic VS image will be generated.
Make changes to build_image.sh to generate 4-asic and 6-asic KVM images if BUILD_MULTIASIC_KVM parameter is set to y
How to verify it
Generate single-asic VS as currently done, no change in build steps:
make configure PLATFORM=vs
make target/sonic-vs.img.gz - will generate only single asic KVM image.
make BUILD_MULTIASIC_KVM=y target/sonic-vs.img.gz - will generate single asic and multi-asic KVM images.
should generate:
sonic-vs.bin
sonic-vs.img.gz
sonic-4asic-vs.img.gz
sonic-6asic-vs.img.gz
Why I did it
Support to build armhf/arm64 platforms on arm based system without qemu simulator.
When building the armhf/arm64 on arm based system, it is not necessary to use qemu simulator.
How I did it
Build armhf on armhf system, or build arm64 on arm64 system, by default, qemu simulator will not be used.
When building armhf on arm64, and you have enabled armhf docker, then it will build images without simulator automatically. It is based how the docker service is run.
Docker base image change:
For amd64, change from debian:to amd64/debian:
For arm64, change from multiarch/debian-debootstrap:arm64- to arm64v8/debian:
For armhf, change from multiarch/debian-debootstrap:armhf- to arm32v7/debian:
See https://github.com/docker-library/official-images#architectures-other-than-amd64
The mapping relations:
arm32v6 --- armel
arm32v7 --- armhf
arm64v8 --- arm64
Docker image armhf deprecated info: https://hub.docker.com/r/armhf/debian, using arm32v7 instead.
- Why I did it
Make DHCP relay docker an extension. DHCP relay now carries dhcp relay commands CLI plugin and has a complete manifest.
It is installed as extension if INCLUDE_DHCP_REALY is set to y.
DEPENDS on #5939
- How I did it
Modify DHCP relay docker makefile and dockerfile. Make changes to sonic_debian_extension.j2 to install sonic packages.
I moved DHCP related CLI tests from sonic-utilities to DHCP relay docker.
This PR introduces a way to write a plugin as part of docker image and run the tests from cli-plugin-tests directory under docker directory.
The test result is available in target/docker-dhcp-relay.gz.log:
[ REASON ] : target/docker-dhcp-relay.gz does not exist NON-EXISTENT PREREQUISITES: docker-start target/docker-config-engine-buster.gz-load target/python-wheels/sonic_utilities-1.2-py3-none-any.whl-in
stall target/debs/buster/python3-swsscommon_1.0.0_amd64.deb-install
[ FLAGS FILE ] : []
[ FLAGS DEPENDS ] : []
[ FLAGS DIFF ] : []
============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-3.10.1, py-1.7.0, pluggy-0.8.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /sonic/dockers/docker-dhcp-relay/cli-plugin-tests, inifile:
plugins: cov-2.6.0
collecting ... collected 10 items
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_plugin_registration PASSED [ 10%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_nonexist_vlanid PASSED [ 20%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_invalid_vlanid PASSED [ 30%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_invalid_ip PASSED [ 40%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_exist_ip PASSED [ 50%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_del_dhcp_relay_dest PASSED [ 60%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_remove_nonexist_dhcp_relay_dest PASSED [ 70%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_remove_dhcp_relay_dest_with_nonexist_vlanid PASSED [ 80%]
test_show_dhcp_relay.py::TestVlanDhcpRelay::test_plugin_registration PASSED [ 90%]
test_show_dhcp_relay.py::TestVlanDhcpRelay::test_dhcp_relay_column_output PASSED [100%]
=============================== warnings summary ===============================
/usr/local/lib/python3.7/dist-packages/tabulate.py:7
/usr/local/lib/python3.7/dist-packages/tabulate.py:7: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import namedtuple, Iterable
-- Docs: https://docs.pytest.org/en/latest/warnings.html
==================== 10 passed, 1 warnings in 0.35 seconds =====================
This adds the Makefile changes to use the Bullseye slave image, but
doesn't use it by default. There should be no functional changes with
this change (Buster will still be used for now).
Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
#### Why I did it
Usecase:
export DOCKER_EXTRA_OPTS="--registry-mirror=https://some.host" - to avoid DockerHub pull rate limiting.
#### How I did it
Added DOCKER_EXTRA_OPTS
#### How to verify it
export DOCKER_EXTRA_OPTS="--registry-mirror=https://some.host"
make target/sonic-mellanox.bin
Why I did it
After PR #7344, 'make init' and/or 'make reset' will also build sonic slave dockers.
'-include rules/config.user' is supposed to be fine when the file is missing. However, when the file is missing, it generates a delayed error which later causes make init and make reset trying to build the sonic slave dockers.
How I did it
Define a do-nothing target for config.user to catch config.user build therefore preventing other builds to be triggered unexpectedly.
How to verify it
did make init and it is now only doing submodule init.
- Support compile sonic arm image on arm server. If arm image compiling is executed on arm server instead of using qemu mode on x86 server, compile time can be saved significantly.
- Add kernel argument systemd.unified_cgroup_hierarchy=0 for upgrade systemd to version 247, according to #7228
- rename multiarch docker to sonic-slave-${distro}-march-${arch}
Co-authored-by: Xianghong Gu <xgu@centecnetworks.com>
Co-authored-by: Shi Lei <shil@centecnetworks.com>
rules/config.user allows overriding default properties without
touching tracked files. This change makes sure all properties
can be set and not just the ones used in slave.mk.
Signed-off-by: Christian Svensson <blue@cmd.nu>
Lack of having the support for no_proxy in SONiC build environment limits enterprise companies to put some internal hacks to ensure the proxy's dont refer to some Intranet site for artifactory downloads etc. Today. Using no_proxy is familiar in proxy settings terminology and excludes traffic destined to certain hosts.
Most Web clients hence support connection to proxy servers via environment variables:
http_proxy / HTTP_PROXY
https_proxy / HTTPS_PROXY
no_proxy / NO_PROXY
These variables tell the client what URL should be used to access the proxy servers and which exceptions should be made.
How to verify it
Simply set up the variable in the bash shell at build time.
export no_proxy=internal.example.com, internal2.example.com
Usage is:
no_proxy is a comma- or space-separated list of machine or domain names, with optional :port part. If no :port
part is present, it applies to all ports on that domain.
- Why I did it
To move ‘sonic-host-service’ which is currently built as a separate package to ‘sonic-host-services' package.
- How I did it
- Moved 'sonic-host-server' to 'src/sonic-host-services' and included it as part of the python3 wheel.
- Other files were moved to 'src/sonic-host-services-data' and included as part of the deb package.
- Changed build option ‘INCLUDE_HOST_SERVICE’ to ‘ENABLE_HOST_SERVICE_ON_START’ for enabling sonic-hostservice at boot-up by default.
sonic-slave tag only allows all lower case. In case the user
name is mixed case, we need to change user name to all lower case.
Signed-off-by: Guohan Lu <lguohan@gmail.com>
- add config option ENABLE_DOCKER_BASE_PULL to pull sonic-slave docker from registry
- use REGISTRY_PORT, REGISTRY_SERVER to specify docker registry
Signed-off-by: Guohan Lu <lguohan@gmail.com>
for exmaple, for arm64, the sonic-slave docker image name
is sonic-slave-arm64-$(USER)
for amd64, the docker image is kept as it is
Signed-off-by: Guohan Lu <lguohan@gmail.com>
* [make reset] update submodule's remotes
This change is to address an issue where when using a long standing repo to build image, the submodule could become out-dated with their remotes so that make init is no longer to bring all submoudles to the latest. The consequence is weird build failures.
When we update/switch branch in a long standing repo, we will issue 'make reset' to get repo ready for next build. Updating all submodule's remotes during "make reset" address the issue.
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
Introduced a new build parameter 'SONIC_IMAGE_VERSION' that allows build
system users to build SONiC image with a specific version string. If
'SONIC_IMAGE_VERSION' was not passed by the user, SONIC_IMAGE_VERSION will be
set to the output of functions.sh:sonic_get_version function.
Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
Most of the devs know how they configured their build.
Instead of generating a huge header for every make operation, allow the
user to silence it via an environment variable.
SONIC_BUILD_QUIETER=y make ...
Also add SONIC_BUILDER_EXTRA_ENV variable which allows the user to
inject arbitrary options to the docker run cmdline for the builder.
SONIC_BUILDER_EXTRA_CMDLINE="-e test=var -e var='with spaces'" make ...
Add a master switch so that the sync/async mode can be configured.
Example usage of the switch:
1. Configure mode while building an image
`make ENABLE_SYNCHRONOUS_MODE=y <target>`
2. Configure when the device is running
Change CONFIG_DB with `sonic-cfggen -a '{"DEVICE_METADATA":{"localhost": {"synchronous_mode": "enable"}}}' --write-to-db`
Restart swss with `systemctl restart swss`
Removes installation of kube-proxy (117 MB) and flannel (53 MB) images from Kubernetes-enabled devices. These images are tested to be unnecessary for our use case, as we do not rely on ClusterIPs for Kubernetes Services or a CNI for pod networking.
1. remove container feature table
2. do not generate feature entry if the feature is not included
in the image
3. rename ENABLE_* to INCLUDE_* for better clarity
4. rename feature status to feature state
5. [submodule]: update sonic-utilities
* 9700e45 2020-08-03 | [show/config]: combine feature and container feature cli (#1015) (HEAD, origin/master, origin/HEAD) [lguohan]
* c9d3550 2020-08-03 | [tests]: fix drops_group_test failure on second run (#1023) [lguohan]
* dfaae69 2020-08-03 | [lldpshow]: Fix input device is not a TTY error (#1016) [Arun Saravanan Balachandran]
* 216688e 2020-08-02 | [tests]: rename sonic-utilitie-tests to tests (#1022) [lguohan]
Signed-off-by: Guohan Lu <lguohan@gmail.com>
use lsb_release -cs to get build env within the docker
for jessie: jessie
for stretch: stretch
for buster: buster
Signed-off-by: Guohan Lu <lguohan@gmail.com>
following command add additional mount in the sonic:
DOCKER_BUILDER_USER_MOUNT=/data2:/data2,/data:/data make sonic-slave-bash
Signed-off-by: Guohan Lu <lguohan@gmail.com>
--init: Run an init inside the container that forwards signals and
reaps processes
Before the change, process 1 is make, which does not do well to forward
signals and reaps process. We could see zombie process left if user
issues ctrl+c to interrupt the make process. with --init option,
a docker-init process will forwards the signals and reaps processes.
zombie process is no longer observed, and ctrl+c can reliably interrupt
the make process.
Before:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
lgh 1 0.3 0.0 12604 11908 pts/0 S+ 10:54 0:00 make ...
After:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
lgh 1 0.0 0.0 1080 4 pts/0 Ss 10:54 0:00 /sbin/docker-init -- make ...
lgh 7 0.3 0.0 12604 11908 pts/0 S+ 10:54 0:00 make ...
Signed-off-by: Guohan Lu <lguohan@gmail.com>
This PR updates kubernetes version to 1.18.6 and kubernetes-cni version to 0.8.6
signed-off by: Isabel Li isabel.li@microsoft.com
Why I did it
Previous kubernetes-cni version (0.7.5) introduced Kubernetes Man In The Middle Vulnerability. “A vulnerability was found in all versions of containernetworking/plugins before version 0.8.6, that allows malicious containers in Kubernetes clusters to perform man-in-the-middle (MitM) attacks. A malicious container can exploit this flaw by sending rogue IPv6 router advertisements to the host or other containers, to redirect traffic to the malicious container.”
How I did it
Defined kubernetes-cni version to be 0.8.6 and updated kubernetes version to be 1.18.6
How to verify it
Check versions by running dpkg -l | grep kube
Add build flag TELEMETRY_WRITABLE. When set to "y" it will add a go build flag in the telemetry build that will enable telemetry write mode to allow configuration via gNMI Set RPC as well as operations via the gNOI RPC's. The default for TELEMETRY_WRITABLE is unset in which case telemetry is read-only. In read-only mode the Set RPC and all gNOI RPC's are disabled and will return an "Unsupported" error when called.
authored-by: Eric Seifert <eric@seifert.casa>
