Why I did it
The SONiC switches get their docker images from local repo, populated during install with container images pre-built into SONiC FW. With the introduction of kubernetes, new docker images available in remote repo could be deployed. This requires dockerd to be able to pull images from remote repo.
Depending on the Switch network domain & config, it may or may not be able to reach the remote repo. In the case where remote repo is unreachable, we could potentially make Kubernetes server to also act as http-proxy.
How I did it
When admin explicitly enables, the kubernetes-server could be configured as docker-proxy. But any update to docker-proxy has to be via service-conf file environment variable, implying a "service restart docker" is required. But restart of dockerd is vey expensive, as it would restarts all dockers, including database docker.
To avoid dockerd restart, pre-configure an http_proxy using an unused IP. When k8s server is enabled to act as http-proxy, an IP table entry would be created to direct all traffic to the configured-unused-proxy-ip to the kubernetes-master IP. This way any update to Kubernetes master config would be just manipulating IPTables, which will be transparent to all modules, until dockerd needs to download from remote repo.
How to verify it
Configure a switch such that image repo is unreachable
Pre-configure dockerd with http_proxy.conf using an unused IP (e.g. 172.16.1.1)
Update ctrmgrd.service to invoke ctrmgrd.py with "-p" option.
Configure a k8s server, and deploy an image for feature with set_owner="kube"
Check if switch could successfully download the image or not.
#### Why I did it
The PR checkers do not re-run the sonic-config-engine test cases, caused by some of the config files changes not detected.
https://sonic-jenkins.westus2.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all/660/console
…
07:13:24 ======================================================================
07:13:24 ERROR: test_bgpd_quagga (tests.test_j2files.TestJ2Files)
07:13:24 ----------------------------------------------------------------------
…
07:13:24 ======================================================================
07:13:24 ERROR: test_zebra_quagga (tests.test_j2files.TestJ2Files)
07:13:24 ----------------------------------------------------------------------
…
07:13:24 error: Test failed: <unittest.runner.TextTestResult run=161 errors=2 failures=0>
07:13:24 [ FAIL LOG END ] [ target/python-wheels/sonic_config_engine-1.0-py2-none-any.whl ]
07:13:24 make: *** [slave.mk:603: target/python-wheels/sonic_config_engine-1.0-py2-none-any.whl] Error 1
07:13:24 Makefile.work:292: recipe for target 'target/sonic-mellanox.bin' failed
07:13:24 make[1]: *** [target/sonic-mellanox.bin] Error 2
07:13:24 make[1]: Leaving directory '/data2/johnar/workspace/mellanox/buildimage-mlnx-all'
07:13:24 Makefile:7: recipe for target 'target/sonic-mellanox.bin' failed
07:13:24 make: *** [target/sonic-mellanox.bin] Error 2
See PR: https://github.com/Azure/sonic-buildimage/pull/7476
#### How I did it
Add the depended files.
See src/sonic-config-engine/tests/test_j2files.py
Signed-off-by: Yong Zhao yozhao@microsoft.com
Why I did it
Currently we leveraged the Supervisor to monitor the running status of critical processes in each container and it is more reliable and flexible than doing the monitoring by Monit. So we removed the functionality of monitoring the critical processes by Monit.
How I did it
I removed the script process_checker and corresponding Monit configuration entries of critical processes.
How to verify it
I verified this on the device str-7260cx3-acs-1.
- Introduced TS common file in docker as well and moved common functions.
- TSA/B/C scripts run only in BGP instances for front end ASICs.
In addition skip enforcing it on route maps used between internal BGP sessions.
admin@str--acs-1:~$ sudo /usr/bin/TSA
System Mode: Normal -> Maintenance
and in case of Multi-ASIC
admin@str--acs-1:~$ sudo /usr/bin/TSA
BGP0 : System Mode: Normal -> Maintenance
BGP1 : System Mode: Normal -> Maintenance
BGP2 : System Mode: Normal -> Maintenance
Why I did it
Support readonly version of the command vtysh
How I did it
Check if the command starting with "show", and verify only contains single command in script.
Compiling ethtool from source is causing ethtool unit tests to fail on ARM Platforms.
These tests are failing: (By default netlink-interface is enabled while compiling ethtool)
Link: ([Test File Link](https://salsa.debian.org/kernel-team/ethtool/-/blob/debian/1%255.9-1/test-cmdline.c#L28))
```
FAIL: test-cmdline
==================
E: ethtool 16_char_devname! returns 1
E: ethtool
127_char_devname0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcde returns 1
E: ethtool --change devname xcvr external returns 0
E: ethtool --change devname speed 100 duplex half port tp autoneg on advertise 0x1 phyad 1 xcvr external wol p sopass 01:23:45:67:89:ab msglvl 1 returns 0
FAIL test-cmdline (exit status: 1)
```
Tested this on Local ARM Emulated Container:
```
(Docker Container Emulating ARM)
vkarri@3a03c70eed35:/tmp/ethtool$ ./ethtool 16_char_devname!
netlink interface initialization failed, device name longer than 15 not supported
vkarri@3a03c70eed35:/tmp/ethtool$ echo $?
1 (Expected 0)
vkarri@3a03c70eed35:~/ethtool$ ./ethtool 16_char_devnameee
netlink interface initialization failed, device name longer than 15 not supported
Checked for dependencies: (all are present)
vkarri@3a03c70eed35:~/ethtool$ apt-cache policy libmnl0
libmnl0:
Installed: 1.0.4-2
Candidate: 1.0.4-2
Version table:
*** 1.0.4-2 500
500 http://deb.debian.org/debian buster/main armhf Packages
500 http://packages.trafficmanager.net/debian/debian buster/main armhf Packages
100 /var/lib/dpkg/status
vkarri@3a03c70eed35:~/ethtool$ apt-cache policy libc6
libc6:
Installed: 2.28-10
Candidate: 2.28-10
Version table:
*** 2.28-10 500
500 http://deb.debian.org/debian buster/main armhf Packages
500 http://packages.trafficmanager.net/debian/debian buster/main armhf Packages
100 /var/lib/dpkg/status
```
#### How I did it
Disabled netlink-interface for ethtool.
Even though Netlink is not available, it doesn't seem to impact what ethtool was supposed to do. In fact the older version which was in use before this PR [#5725](https://github.com/Azure/sonic-buildimage/pull/5725) did not have netlink support and everything seemed to work well
Article on Netlink-Support for ethtool: https://lwn.net/Articles/783633/
Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
Update FRR to 7.5.1. The following is a list of new commits.
```
df7ab485b FRRouting Release 7.5.1
f4ed841b8 Merge pull request #8187 from opensourcerouting/rpmfixes-75
86d5a20e3 Merge pull request #8193 from mjstapp/fix_signals_7_5
b339cc149 lib: avoid signal-handling race with event loop poll call
0f7b432c3 lib: add debug output for signal mask
c0290c86d lib: add sigevent_check api
7a5348665 doc: Fix CentOS 7 Documentation
2a8e69f48 Merge pull request #8064 from donaldsharp/foo
cf4d1a744 redhat: Fix changelog incorrect date format
b78dcb209 Merge pull request #8181 from idryzhov/7.5-zebra-blackhole
2032e7e72 zebra: don't use kernel nexthops for blackhole routes
e52003567 bgpd: When deleting a neighbor from a peer-group the PGNAME is optional
aa86a6a6f Merge pull request #8161 from mjstapp/fix_sa_7_5_backports
13a8efb4b Merge pull request #8156 from idryzhov/7.5-backports-2021-02-26
58911c6ed lib: Free memory leak in error path in clippy
556dfd211 lib: use right type for wconv() return val
bd9caa8f1 lib: fix some misc SA warnings
683b3fe3f lib: register dependency between control plane protocol and vrf nb nodes
b45248fb6 lib: add definitions for vrf xpaths
7b9f10d04 lib: add ability to register dependencies between northbound nodes
9c240815c bgpd: Bgp peer group issue
d1b43634b bgpd: upon bgp deletion, do not systematically ask to remove main bgp
f5d1dc55e bgpd: Fix crash when we don't have a nexthop
c2e463478 frr-reload: rpki context exiting uses exit and not end
f11db1698 bgpd: Blackhole nexthops are not reachable
c628e94ff staticd: fix vrf enabling
49b079ef1 staticd: fix nexthop creation and installation
0077038e9 staticd: fix nexthop validation
be3dfbbc7 zebra: use AF_INET for protocol family
```
This package was outdated and did not properly install on buster due to dependency changes.
This version should work on all DCS-7280CR3*-32[PD]4 platforms.
Signed-off-by: Samuel Angebault <staphylo@arista.com>
Update FRR 7.5 head. The following is a list of new commits.
```
e2f17ae47ad047e66923c2ff1e84c9ba10d4ad38 Merge pull request #8096 from idryzhov/7.5-backports-2021-02-16
380341362ced8e317c18b7395acb012de1f23acd ospf6d: Don't send hellos on loopback interface
7fa78b659f8e720466e0df62689327ea4b9ff867 bgpd: send correct BMP down message when nht fails
385faf6c079a41def1e6eb882cbfd50047559644 [filter]: change return code for errors
d9a0e9a2934f2f75c64496fe4c724a18aa581fcb bfdd: fix session lookup
08afa0a75311a4e8cb2a18116384b603f7f2d751 ospf6d : fix issue in ecmp inter area route
2299afa1a9128d87d5169742b993c0ada575eb83 ospfd: Prevent duplicate packet read in certain vrf situations
ff42a28af659ee61c0efb877b10738a5812f4bc2 vrf: use wrappers to change VRF_CONFIGURED flag
2bdc59ca21da2d67b77ec70a2fadffbca60690cd vrf: mark vrf as configured when entering vrf node
b9611f65a71adc0b8fa14a5a4d1a8f44e04dcd85 ospf6d: Fix LSA formatting out-of-bounds access
610ebf56913fa56167b0a2a127b07afe020a1efe bfdd: Prevent use after free ( again )
35b0cd5d753dda9aa70ea1c06db61a8d4b8671e3 *: Fix usage of bfd_adj_event
95b8915d0f4de3eae5438632ecd0827061ef48e8 ospf6d: Fix LSA formatting inconsistent retvals
49d73d8be84dbd23d767697474019165e511786c pimd: SGRpt prune received during prune didn't override holdtime
1d0d19afa9bb7cd4bc476d00c887876bc04eee95 eigrpd: Correctly set the mtu for eigrp packets sent
bbb08db69f8eb554d23b4920c1c1e3982d8d2a91 zebra: Prevent sending of unininted data
0813d650a8120458ab7d9317061f3864dbc6f2f7 ospf6d: prevent use after free
2f2e981d967b36b240fca82fea8a961d927ef43c lib: Prevent unininted usage of data
6171becdb391ea5b88916a3a28b04b555e1fc518 bfdd: Prevent storage of ifp pointer that has been deleted
9ebb41cf4bb51e0872796530bf8c7a4d819053db bfdd: Prevent unininited data transmittal
72e16db6fea3629111537f9eb10c86f2d275adcb eigrpd: Prevent uninitialized value from being used
72b61a5bb09d59c3cc0d1d401d51de96949dff52 zebra: disallow resolution to duplicate nexthops
1083bae40b00c0ed2c9f3521ae1ab9675a87202e bgpd: Initialize bgp_notify.raw_data before passing to bgp_notify_receive()
31df7314310416f10c133dcfe9c4586edadf3fbb doc: ebgp-requires-policy requires manuall session clearing
ecc8ec678d2d8a1c3d1d50a22732f9fc4bad689c watchfrr: fix SA warning
9d9365d161979a031de817c1fbcab6508dfee013 watchfrr: fix crash on missing optional argument
907e600d63c1c5b6bda40b0a08344a72533b1787 pimd: Prevent use after free
b47374f0e95d99c93bfe2d14afe55219a9fda455 doc: Update bgp doc for more rfc-8212 talk
4fbeef60cc8dc5362ff84fc91d1a4e343e4e32c7 docker: centos 7, 8 yang bump and repo fixes
808e6d731f330df4a91fdfd6df6a3c8dce1651a6 docker: prefer alpine:latest for building
91b3c471f1c48818370a0f218add917f0d46aa47 Merge pull request #8092 from donaldsharp/7.5_track
60be43c0bf63c16ca42008fa802d0a2050f3fce2 Merge pull request #8090 from ton31337/fix/static_network_vrf_7.5
1f6785aa60cc57a5c8d5de98c9c09a344a0c9262 ospf6d: Track wait_timer and disable when needed
c89e326be91312bed066eb2447ea8944e25a225e bgpd: Check for peer->su_remote if not NULL when handling IPv6 nexthop
15e070f6448870c98c030b6b5013ad8750d8918b Merge pull request #8047 from pguibert6WIND/nhrp_shortcut_routes_75
912994efec94082ae7d8c5e014c410964bea19f4 Merge pull request #8034 from qlyoung/fix-gnu-readline-bracketed-paste-7.5.1
9f50536993f1eb900fbfbe98d21b8c072bbd9c15 nhrpd: replace nhrp route nexthop with onlink route when prefix=nh
8c185008246db31c34574d7b79358001ac411f84 nhrpd: shortcut routes installed with nexthop.
c46c87d19758040bc3f3902ab8e4a0f1bb908721 vtysh: disable bracketed paste in readline
20b35e4c3386de798f3b0cb9f2a7e6b04d995485 Merge pull request #8018 from ton31337/fix/drop_aggregate_as_attribute_if_malformed_7.5
fa25d7327fd64613cc7530aba2edfcde038da074 bgpd: Unset only aggregator flag when AGGREGATOR_AS is 0
3ee9a3726fe1a526d946c1978487a4509fe98f29 bgpd: Drop aggregator_as attribute if malformed in case of BGP_AS_ZERO
be88595c6a2011f0e882bfa663baa61c86ede14e Merge pull request #8005 from opensourcerouting/snap-libyang1-fix-75
fd840ad37f2e836b210c6e60fc6325a4c3e495ce snapcraft: Update rtrlib to 0.7.0
3d00552fa9aedb96acd7ea773bc14fd2b77e7e0f snapcraft: Fix passthrough path for Libyang 1.x
```
* Remove makefile and
As part of #6046 we have moved to python3 of dbsyncd.
Cleanup the python2 makefile and dependency.
Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
* Revert "Remove makefile and dependency for python2 of dbsyncd."
This reverts commit ceaa4f8dd9.
* Fix
Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
* Bump version number to 2.0.32-1 to include a fix for a memory-leak
found during testing. A wrong API is used to free the cJSON
data-structure, which only frees the first pointed-to structure.
The proper API should recursively free all structures.
Signed-off-by: Garrick He <garrick_he@dell.com>
Some commands used during build will prompt user interactively, but this is not expected during build. Since most output is collected into log file, user could not see the prompt and feel the build process hangs.
- How I did it
Use mv command in non interactive mode
Redirect stdin to null if command output is collected into log file.
This commit fixes a typo in the fix delivered in PR #6538
syncd fails on the armhf platform within sonic-config-engine/portconfig.py when importing the following
'from swsscommon.swsscommon import ConfigDBConnector'
- combine docker-ptf-saithrift into docker-ptf docker
- build docker-ptf under platform vs
- remove docker-ptf for other platforms
Signed-off-by: Guohan Lu <lguohan@gmail.com>
- Support for non-template based FRR configurations (BGP, route-map, OSPF, static route..etc) using config DB schema.
- Support for save & restore - Jinja template based config-DB data read and apply to FRR during startup
**- How I did it**
- add frrcfgd service
- when frr_mgmg_framework_config is set, frrcfgd starts in bgp container
- when user changed the BGP or other related table entries in config DB, frrcfgd will run corresponding VTYSH commands to program on FRR.
- add jinja template to generate FRR config file to be used by FRR daemons while bgp container restarted
**- How to verify it**
1. Add/delete data on config DB and then run VTYSH "show running-config" command to check if FRR configuration changed.
1. Restart bgp container and check if generated FRR config file is correct and run VTYSH "show running-config" command to check if FRR configuration is consistent with attributes in config DB
Co-authored-by: Zhenhong Zhao <zhenhong.zhao@dell.com>
**- Why I did it**
To enable running Pytest unit tests before building the Python 3 sonic-platform-common package
**- How I did it**
- Add Python 3 sonic-config-engine package as a dependency of Python 3 sonic-platform-common package (needed for both runtime and unit tests)
- No longer disable unit tests when building Python 3 sonic-platform-common package
The only Python code in the SwSS package, restore_neighbors.py, was recently converted to Python 3 and most dependencies were updated as part of #6207. However, the SwSS makefile still listed the Python 2 version of the swsscommon package as a dependency. This caused Python 2-related packages to be installed in containers unnecessarily.
1. Fixes the missing DPKG file for gbsyncd-vs package
2. Fixes the softlink issue on the Platform-common and ztp package
3. Fixes the PYTHNON_DEBS list is missing for DBG dockers.
Fix#119
when parallel build is enable, multiple dpkg-buildpackage
instances are running at the same time. /var/lib/dpkg is shared
by all instances and the /var/lib/dpkg/updates could be corrupted
and cause the build failure.
the fix is to use overlay fs to mount separate /var/lib/dpkg
for each dpkg-buildpackage instance so that they are not affecting
each other.
Signed-off-by: Guohan Lu <lguohan@gmail.com>
build frr dbg image force to install frr in the build process
which breaks the current build and is uneccessary.
Signed-off-by: Guohan Lu <lguohan@gmail.com>
frr build requires libyang 1.0.184 which conflicts with
libyang 1.0.73. Solution here is to compile frr and libyang 1.0.184
first, and then uninstall libyang 1.0.184 after frr build.
Then, compile libyang 1.0.73 and all packages depend on it later.
Signed-off-by: Guohan Lu <lguohan@gmail.com>
**- Why I did it**
python2 is end of life and SONiC is going to support python3. This PR is going to support:
1. Build pmon daemons with python3
2. Install and run python3 version pmon daemons
**- How I did it**
1. Change pmon daemons make files to build bothe python2 and python3 whl
2. Change docker-platform-monitor make files to install both python2 and python3 whl
3. Change pmon docker startup files to start pmon daemons according to the supported platform API version
- add config option ENABLE_DOCKER_BASE_PULL to pull sonic-slave docker from registry
- use REGISTRY_PORT, REGISTRY_SERVER to specify docker registry
Signed-off-by: Guohan Lu <lguohan@gmail.com>
**- Why I did it**
As part of migrating SONiC codebase from Python 2 to Python 3
**- How I did it**
- No longer install Python 2 in docker-base-buster or docker-config-engine-buster.
- Install Python 2 and pip2 in the following containers until we can completely eliminate it there:
- docker-platform-monitor
- docker-sonic-mgmt-framework
- docker-sonic-vs
- Pin pip2 version <21 where it is still temporarily needed, as pip version 21 will drop support for Python 2
- Also preform some other cleanup, ensuring that pip3, setuptools and wheel packages are installed in docker-base-buster, and then removing any attempts to re-install them in derived containers
* First cut image update for kubernetes support.
With this,
1) dockers dhcp_relay, lldp, pmon, radv, snmp, telemetry are enabled
for kube management
init_cfg.json configure set_owner as kube for these
2) Each docker's start.sh updated to call container_startup.py to register going up
As part of this call, it registers the current owner as local/kube and its version
The images are built with its version ingrained into image during build
3) Update all docker's bash script to call 'container start/stop/wait' instead of 'docker start/stop/wait'.
For all locally managed containers, it calls docker commands, hence no change for locally managed.
4) Introduced a new ctrmgrd service, that helps with transition between owners as kube & local and carry over any labels update from STATE-DB to API server
5) hostcfgd updated to handle owner change
6) Reboot scripts are updatd to tag kube running images as local, so upon reboot they run the same image.
7) Added kube_commands.py to handle all updates with Kubernetes API serrver -- dedicated for k8s interaction only.
