Signed-off-by: Stepan Blyschak stepanb@mellanox.com
Why I did it
To support building DHCP relay as extension and installing it during build time.
How I did it
Created infrastructure. Users need to define their packages in rules/sonic-packages.mk
How to verify it
Together with #6531
Why I did it
To Fix SONiC build with INSTALL_DEBUG_TOOLS=y
How I did it
Fixed generate_manifest function to use version prefix in file names
How to verify it
Build SONiC with INSTALL_DEBUG_TOOLS=y
Why I did it
Quagga is no longer being used. Remove quagga-related code (e.g., docker-fpm-quagga, sonic-quagga, etc.).
How I did it
Remove quagga-related code.
#### Why I did it
- To build flashrom properly with dependency tracking.
#### How I did it
- Moved flashrom code from platform/broadcom/sonic-platform-modules-dell/tools directory to src/flashrom directory.
- At the end, flashrom_0.9.7_amd64.deb package is build which will be installed in the devices.
- Currently flashrom builds only for Dell S6100 platforms.
Introduce new sonic-buildimage images for Broadcom DNX ASIC family.
sonic-broadcom-dnx.bin
sonic-aboot-broadcom-dnx.swi
How I did it
NO CHANGE to existing make commands
make init; make configure PLATFORM=broadcom; make target/sonic-aboot-broadcom.swi; make target/sonic-broadcom.bin
The difference now is that it will result in new broadcom images for DNX asic family as well.
sonic-broadcom.bin, sonic-broadcom-dnx.bin
sonic-aboot-broadcom.swi, sonic-aboot-broadcom-dnx.swi
Note: This PR also adds support for Broadcom SAI 5.0 (based on 1.8 SAI ) for DNX based platform + changes in platform x86_64-arista_7280cr3_32p4 bcm config files and platform_env.conf files
Why I did it
The SONiC switches get their docker images from local repo, populated during install with container images pre-built into SONiC FW. With the introduction of kubernetes, new docker images available in remote repo could be deployed. This requires dockerd to be able to pull images from remote repo.
Depending on the Switch network domain & config, it may or may not be able to reach the remote repo. In the case where remote repo is unreachable, we could potentially make Kubernetes server to also act as http-proxy.
How I did it
When admin explicitly enables, the kubernetes-server could be configured as docker-proxy. But any update to docker-proxy has to be via service-conf file environment variable, implying a "service restart docker" is required. But restart of dockerd is vey expensive, as it would restarts all dockers, including database docker.
To avoid dockerd restart, pre-configure an http_proxy using an unused IP. When k8s server is enabled to act as http-proxy, an IP table entry would be created to direct all traffic to the configured-unused-proxy-ip to the kubernetes-master IP. This way any update to Kubernetes master config would be just manipulating IPTables, which will be transparent to all modules, until dockerd needs to download from remote repo.
How to verify it
Configure a switch such that image repo is unreachable
Pre-configure dockerd with http_proxy.conf using an unused IP (e.g. 172.16.1.1)
Update ctrmgrd.service to invoke ctrmgrd.py with "-p" option.
Configure a k8s server, and deploy an image for feature with set_owner="kube"
Check if switch could successfully download the image or not.
#### Why I did it
The PR checkers do not re-run the sonic-config-engine test cases, caused by some of the config files changes not detected.
https://sonic-jenkins.westus2.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all/660/console
…
07:13:24 ======================================================================
07:13:24 ERROR: test_bgpd_quagga (tests.test_j2files.TestJ2Files)
07:13:24 ----------------------------------------------------------------------
…
07:13:24 ======================================================================
07:13:24 ERROR: test_zebra_quagga (tests.test_j2files.TestJ2Files)
07:13:24 ----------------------------------------------------------------------
…
07:13:24 error: Test failed: <unittest.runner.TextTestResult run=161 errors=2 failures=0>
07:13:24 [ FAIL LOG END ] [ target/python-wheels/sonic_config_engine-1.0-py2-none-any.whl ]
07:13:24 make: *** [slave.mk:603: target/python-wheels/sonic_config_engine-1.0-py2-none-any.whl] Error 1
07:13:24 Makefile.work:292: recipe for target 'target/sonic-mellanox.bin' failed
07:13:24 make[1]: *** [target/sonic-mellanox.bin] Error 2
07:13:24 make[1]: Leaving directory '/data2/johnar/workspace/mellanox/buildimage-mlnx-all'
07:13:24 Makefile:7: recipe for target 'target/sonic-mellanox.bin' failed
07:13:24 make: *** [target/sonic-mellanox.bin] Error 2
See PR: https://github.com/Azure/sonic-buildimage/pull/7476
#### How I did it
Add the depended files.
See src/sonic-config-engine/tests/test_j2files.py
Why I did it
This PR adds changes in sonic-config-engine to consume configuration data in SONiC Yang schema and generate config_db entries
How I did it
Add a new file sonic_yang_cfg_generator .
This file has the functions to
parse yang data json and convert them in config_db json format.
Validate the converted config_db entries to make sure all the dependencies and constraints are met.
Add a new option -Y to the sonic-cfggen command for this purpose
Add unit tests
This capability is support only in sonic-config-engine Python3 package only
Why I did it
Quagga is no longer being used. Remove quagga-related code (e.g., docker-fpm-quagga, sonic-quagga, etc.).
How I did it
Remove quagga-related code.
Signed-off-by: Yong Zhao yozhao@microsoft.com
Why I did it
Currently we leveraged the Supervisor to monitor the running status of critical processes in each container and it is more reliable and flexible than doing the monitoring by Monit. So we removed the functionality of monitoring the critical processes by Monit.
How I did it
I removed the script process_checker and corresponding Monit configuration entries of critical processes.
How to verify it
I verified this on the device str-7260cx3-acs-1.
- Why I did it
To give SONiC Application Extension developers an environment to run and develop their apps.
- How I did it
Created sonic-sdk and sonic-sdk-buildenv dockers and their dbg versions.
- How to verify it
Build:
$ make -f slave target/sonic-sdk.gz target/sonic-sdk-buildenv.gz
#### Why I did it
Fix https://github.com/Azure/sonic-telemetry/issues/71
#### How I did it
Added memory limit for telemetry docker.
Historical docker memory usage shows telemetry docker consuming 150-200MB memory. Adding some extra buffer.
Platform library changes
- Fix the use of /proc/modules during testing, fixes#7463
- Add `libsfp-eeprom.so` build to read/write xcvr eeproms in C
- Add some more reboot-cause information
- Write down temperature hw thresholds to the sensors
- Report software thresholds through platform api
- Writ `port_name sysfs` file of optoe`
- Tests enhancements
- Fix dependency issues for chassis provisioning
Platform configuration changes
- Add `pcie.yaml` configuration for a few platforms
- Mount `libsfp-eeprom.so` inside `pmon`
- Fix `Arista-7050SX3-48C8` and `Arista-7050SX3-48YC8' platform and hwsku
- Miscellaneous fixes
Co-authored-by: Boyang Yu <byu@arista.com>
Co-authored-by: Zhi Yuan Carl Zhao <zyzhao@arista.com>
Signed-off-by: Stepan Blyschak stepanb@nvidia.com
This PR is part of SONiC Application Extension
Depends on #5938
- Why I did it
To provide an infrastructure change in order to support SONiC Application Extension feature.
- How I did it
Label every installable SONiC Docker with a minimal required manifest and auto-generate packages.json file based on
installed SONiC images.
- How to verify it
Build an image, execute the following command:
admin@sonic:~$ docker inspect docker-snmp:1.0.0 | jq '.[0].Config.Labels["com.azure.sonic.manifest"]' -r | jq
Cat /var/lib/sonic-package-manager/packages.json file to verify all dockers are listed there.
Why I did it
Support readonly version of the command vtysh
How I did it
Check if the command starting with "show", and verify only contains single command in script.
#### Why I did it
To build flashrom properly with dependency tracking.
#### How I did it
Moved flashrom code from platform/broadcom/sonic-platform-modules-dell/tools directory to src/flashrom directory.
At the end, flashrom_0.9.7_amd64.deb package is build which will be installed in the devices.
Compiling ethtool from source is causing ethtool unit tests to fail on ARM Platforms.
These tests are failing: (By default netlink-interface is enabled while compiling ethtool)
Link: ([Test File Link](https://salsa.debian.org/kernel-team/ethtool/-/blob/debian/1%255.9-1/test-cmdline.c#L28))
```
FAIL: test-cmdline
==================
E: ethtool 16_char_devname! returns 1
E: ethtool
127_char_devname0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcde returns 1
E: ethtool --change devname xcvr external returns 0
E: ethtool --change devname speed 100 duplex half port tp autoneg on advertise 0x1 phyad 1 xcvr external wol p sopass 01:23:45:67:89:ab msglvl 1 returns 0
FAIL test-cmdline (exit status: 1)
```
Tested this on Local ARM Emulated Container:
```
(Docker Container Emulating ARM)
vkarri@3a03c70eed35:/tmp/ethtool$ ./ethtool 16_char_devname!
netlink interface initialization failed, device name longer than 15 not supported
vkarri@3a03c70eed35:/tmp/ethtool$ echo $?
1 (Expected 0)
vkarri@3a03c70eed35:~/ethtool$ ./ethtool 16_char_devnameee
netlink interface initialization failed, device name longer than 15 not supported
Checked for dependencies: (all are present)
vkarri@3a03c70eed35:~/ethtool$ apt-cache policy libmnl0
libmnl0:
Installed: 1.0.4-2
Candidate: 1.0.4-2
Version table:
*** 1.0.4-2 500
500 http://deb.debian.org/debian buster/main armhf Packages
500 http://packages.trafficmanager.net/debian/debian buster/main armhf Packages
100 /var/lib/dpkg/status
vkarri@3a03c70eed35:~/ethtool$ apt-cache policy libc6
libc6:
Installed: 2.28-10
Candidate: 2.28-10
Version table:
*** 2.28-10 500
500 http://deb.debian.org/debian buster/main armhf Packages
500 http://packages.trafficmanager.net/debian/debian buster/main armhf Packages
100 /var/lib/dpkg/status
```
#### How I did it
Disabled netlink-interface for ethtool.
Even though Netlink is not available, it doesn't seem to impact what ethtool was supposed to do. In fact the older version which was in use before this PR [#5725](https://github.com/Azure/sonic-buildimage/pull/5725) did not have netlink support and everything seemed to work well
Article on Netlink-Support for ethtool: https://lwn.net/Articles/783633/
Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
* Updating version of hsflow daemon to apply
fix, which resolves problem of switching
between IPv4 and IPv6, in case when the
IPv4 has deleted for the interface.
The new release of hsflowd contains the fix for the issue: sflow/host-sflow@2703ecb
How I did it
HSFLOWD_VERSION env variable has changed in the rules to be pointed to the latest release of hsflowd.
How to verify it
sudo config sflow enable
sudo config loopback add Loopback1
sudo config int ip add Loopback1 a84f:97ff:fea7:33a5::fe80/64
sudo config int ip add Loopback1 192.168.101.1/24
sudo config sflow agent-id add Loopback1
sudo config sflow collector add Collector1 192.168.101.1
sudo config sflow collector add Collector2 a84f:97ff:fea7:33a5::fe80
use sudo sflowtool -p 6343 -l for checking sflow data
remove and add again the ipv4 entry of Loopback1.
hsflowd should change agent ip from IPv4 to IPv6 and wise versa, depending on IPv4 entry present or not.
Switching between IPs is being performed by hsflowd, based on IP address priority ranking.
Signed-off-by: Maksym Belei <Maksym_Belei@jabil.com>
Update FRR to 7.5.1. The following is a list of new commits.
```
df7ab485b FRRouting Release 7.5.1
f4ed841b8 Merge pull request #8187 from opensourcerouting/rpmfixes-75
86d5a20e3 Merge pull request #8193 from mjstapp/fix_signals_7_5
b339cc149 lib: avoid signal-handling race with event loop poll call
0f7b432c3 lib: add debug output for signal mask
c0290c86d lib: add sigevent_check api
7a5348665 doc: Fix CentOS 7 Documentation
2a8e69f48 Merge pull request #8064 from donaldsharp/foo
cf4d1a744 redhat: Fix changelog incorrect date format
b78dcb209 Merge pull request #8181 from idryzhov/7.5-zebra-blackhole
2032e7e72 zebra: don't use kernel nexthops for blackhole routes
e52003567 bgpd: When deleting a neighbor from a peer-group the PGNAME is optional
aa86a6a6f Merge pull request #8161 from mjstapp/fix_sa_7_5_backports
13a8efb4b Merge pull request #8156 from idryzhov/7.5-backports-2021-02-26
58911c6ed lib: Free memory leak in error path in clippy
556dfd211 lib: use right type for wconv() return val
bd9caa8f1 lib: fix some misc SA warnings
683b3fe3f lib: register dependency between control plane protocol and vrf nb nodes
b45248fb6 lib: add definitions for vrf xpaths
7b9f10d04 lib: add ability to register dependencies between northbound nodes
9c240815c bgpd: Bgp peer group issue
d1b43634b bgpd: upon bgp deletion, do not systematically ask to remove main bgp
f5d1dc55e bgpd: Fix crash when we don't have a nexthop
c2e463478 frr-reload: rpki context exiting uses exit and not end
f11db1698 bgpd: Blackhole nexthops are not reachable
c628e94ff staticd: fix vrf enabling
49b079ef1 staticd: fix nexthop creation and installation
0077038e9 staticd: fix nexthop validation
be3dfbbc7 zebra: use AF_INET for protocol family
```
This package was outdated and did not properly install on buster due to dependency changes.
This version should work on all DCS-7280CR3*-32[PD]4 platforms.
Signed-off-by: Samuel Angebault <staphylo@arista.com>
- Made python2 to python3 changes
- Removed ord() func as python3 return int instead of str
- Had to change chr(..) to bytes([..]) function while using ctypes class methods
Update FRR 7.5 head. The following is a list of new commits.
```
e2f17ae47ad047e66923c2ff1e84c9ba10d4ad38 Merge pull request #8096 from idryzhov/7.5-backports-2021-02-16
380341362ced8e317c18b7395acb012de1f23acd ospf6d: Don't send hellos on loopback interface
7fa78b659f8e720466e0df62689327ea4b9ff867 bgpd: send correct BMP down message when nht fails
385faf6c079a41def1e6eb882cbfd50047559644 [filter]: change return code for errors
d9a0e9a2934f2f75c64496fe4c724a18aa581fcb bfdd: fix session lookup
08afa0a75311a4e8cb2a18116384b603f7f2d751 ospf6d : fix issue in ecmp inter area route
2299afa1a9128d87d5169742b993c0ada575eb83 ospfd: Prevent duplicate packet read in certain vrf situations
ff42a28af659ee61c0efb877b10738a5812f4bc2 vrf: use wrappers to change VRF_CONFIGURED flag
2bdc59ca21da2d67b77ec70a2fadffbca60690cd vrf: mark vrf as configured when entering vrf node
b9611f65a71adc0b8fa14a5a4d1a8f44e04dcd85 ospf6d: Fix LSA formatting out-of-bounds access
610ebf56913fa56167b0a2a127b07afe020a1efe bfdd: Prevent use after free ( again )
35b0cd5d753dda9aa70ea1c06db61a8d4b8671e3 *: Fix usage of bfd_adj_event
95b8915d0f4de3eae5438632ecd0827061ef48e8 ospf6d: Fix LSA formatting inconsistent retvals
49d73d8be84dbd23d767697474019165e511786c pimd: SGRpt prune received during prune didn't override holdtime
1d0d19afa9bb7cd4bc476d00c887876bc04eee95 eigrpd: Correctly set the mtu for eigrp packets sent
bbb08db69f8eb554d23b4920c1c1e3982d8d2a91 zebra: Prevent sending of unininted data
0813d650a8120458ab7d9317061f3864dbc6f2f7 ospf6d: prevent use after free
2f2e981d967b36b240fca82fea8a961d927ef43c lib: Prevent unininted usage of data
6171becdb391ea5b88916a3a28b04b555e1fc518 bfdd: Prevent storage of ifp pointer that has been deleted
9ebb41cf4bb51e0872796530bf8c7a4d819053db bfdd: Prevent unininited data transmittal
72e16db6fea3629111537f9eb10c86f2d275adcb eigrpd: Prevent uninitialized value from being used
72b61a5bb09d59c3cc0d1d401d51de96949dff52 zebra: disallow resolution to duplicate nexthops
1083bae40b00c0ed2c9f3521ae1ab9675a87202e bgpd: Initialize bgp_notify.raw_data before passing to bgp_notify_receive()
31df7314310416f10c133dcfe9c4586edadf3fbb doc: ebgp-requires-policy requires manuall session clearing
ecc8ec678d2d8a1c3d1d50a22732f9fc4bad689c watchfrr: fix SA warning
9d9365d161979a031de817c1fbcab6508dfee013 watchfrr: fix crash on missing optional argument
907e600d63c1c5b6bda40b0a08344a72533b1787 pimd: Prevent use after free
b47374f0e95d99c93bfe2d14afe55219a9fda455 doc: Update bgp doc for more rfc-8212 talk
4fbeef60cc8dc5362ff84fc91d1a4e343e4e32c7 docker: centos 7, 8 yang bump and repo fixes
808e6d731f330df4a91fdfd6df6a3c8dce1651a6 docker: prefer alpine:latest for building
91b3c471f1c48818370a0f218add917f0d46aa47 Merge pull request #8092 from donaldsharp/7.5_track
60be43c0bf63c16ca42008fa802d0a2050f3fce2 Merge pull request #8090 from ton31337/fix/static_network_vrf_7.5
1f6785aa60cc57a5c8d5de98c9c09a344a0c9262 ospf6d: Track wait_timer and disable when needed
c89e326be91312bed066eb2447ea8944e25a225e bgpd: Check for peer->su_remote if not NULL when handling IPv6 nexthop
15e070f6448870c98c030b6b5013ad8750d8918b Merge pull request #8047 from pguibert6WIND/nhrp_shortcut_routes_75
912994efec94082ae7d8c5e014c410964bea19f4 Merge pull request #8034 from qlyoung/fix-gnu-readline-bracketed-paste-7.5.1
9f50536993f1eb900fbfbe98d21b8c072bbd9c15 nhrpd: replace nhrp route nexthop with onlink route when prefix=nh
8c185008246db31c34574d7b79358001ac411f84 nhrpd: shortcut routes installed with nexthop.
c46c87d19758040bc3f3902ab8e4a0f1bb908721 vtysh: disable bracketed paste in readline
20b35e4c3386de798f3b0cb9f2a7e6b04d995485 Merge pull request #8018 from ton31337/fix/drop_aggregate_as_attribute_if_malformed_7.5
fa25d7327fd64613cc7530aba2edfcde038da074 bgpd: Unset only aggregator flag when AGGREGATOR_AS is 0
3ee9a3726fe1a526d946c1978487a4509fe98f29 bgpd: Drop aggregator_as attribute if malformed in case of BGP_AS_ZERO
be88595c6a2011f0e882bfa663baa61c86ede14e Merge pull request #8005 from opensourcerouting/snap-libyang1-fix-75
fd840ad37f2e836b210c6e60fc6325a4c3e495ce snapcraft: Update rtrlib to 0.7.0
3d00552fa9aedb96acd7ea773bc14fd2b77e7e0f snapcraft: Fix passthrough path for Libyang 1.x
```
- Introduced TS common file in docker as well and moved common functions.
- TSA/B/C scripts run only in BGP instances for front end ASICs.
In addition skip enforcing it on route maps used between internal BGP sessions.
admin@str--acs-1:~$ sudo /usr/bin/TSA
System Mode: Normal -> Maintenance
and in case of Multi-ASIC
admin@str--acs-1:~$ sudo /usr/bin/TSA
BGP0 : System Mode: Normal -> Maintenance
BGP1 : System Mode: Normal -> Maintenance
BGP2 : System Mode: Normal -> Maintenance
- Why I did it
To move ‘sonic-host-service’ which is currently built as a separate package to ‘sonic-host-services' package.
- How I did it
- Moved 'sonic-host-server' to 'src/sonic-host-services' and included it as part of the python3 wheel.
- Other files were moved to 'src/sonic-host-services-data' and included as part of the deb package.
- Changed build option ‘INCLUDE_HOST_SERVICE’ to ‘ENABLE_HOST_SERVICE_ON_START’ for enabling sonic-hostservice at boot-up by default.
* Remove makefile and
As part of #6046 we have moved to python3 of dbsyncd.
Cleanup the python2 makefile and dependency.
Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
* Revert "Remove makefile and dependency for python2 of dbsyncd."
This reverts commit ceaa4f8dd9.
* Fix
Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
* Bump version number to 2.0.32-1 to include a fix for a memory-leak
found during testing. A wrong API is used to free the cJSON
data-structure, which only frees the first pointed-to structure.
The proper API should recursively free all structures.
Signed-off-by: Garrick He <garrick_he@dell.com>
Some commands used during build will prompt user interactively, but this is not expected during build. Since most output is collected into log file, user could not see the prompt and feel the build process hangs.
- How I did it
Use mv command in non interactive mode
Redirect stdin to null if command output is collected into log file.
This commit fixes a typo in the fix delivered in PR #6538
syncd fails on the armhf platform within sonic-config-engine/portconfig.py when importing the following
'from swsscommon.swsscommon import ConfigDBConnector'
- combine docker-ptf-saithrift into docker-ptf docker
- build docker-ptf under platform vs
- remove docker-ptf for other platforms
Signed-off-by: Guohan Lu <lguohan@gmail.com>
- Support for non-template based FRR configurations (BGP, route-map, OSPF, static route..etc) using config DB schema.
- Support for save & restore - Jinja template based config-DB data read and apply to FRR during startup
**- How I did it**
- add frrcfgd service
- when frr_mgmg_framework_config is set, frrcfgd starts in bgp container
- when user changed the BGP or other related table entries in config DB, frrcfgd will run corresponding VTYSH commands to program on FRR.
- add jinja template to generate FRR config file to be used by FRR daemons while bgp container restarted
**- How to verify it**
1. Add/delete data on config DB and then run VTYSH "show running-config" command to check if FRR configuration changed.
1. Restart bgp container and check if generated FRR config file is correct and run VTYSH "show running-config" command to check if FRR configuration is consistent with attributes in config DB
Co-authored-by: Zhenhong Zhao <zhenhong.zhao@dell.com>
**- Why I did it**
To enable running Pytest unit tests before building the Python 3 sonic-platform-common package
**- How I did it**
- Add Python 3 sonic-config-engine package as a dependency of Python 3 sonic-platform-common package (needed for both runtime and unit tests)
- No longer disable unit tests when building Python 3 sonic-platform-common package
The only Python code in the SwSS package, restore_neighbors.py, was recently converted to Python 3 and most dependencies were updated as part of #6207. However, the SwSS makefile still listed the Python 2 version of the swsscommon package as a dependency. This caused Python 2-related packages to be installed in containers unnecessarily.
1. Fixes the missing DPKG file for gbsyncd-vs package
2. Fixes the softlink issue on the Platform-common and ztp package
3. Fixes the PYTHNON_DEBS list is missing for DBG dockers.
Fix#119
when parallel build is enable, multiple dpkg-buildpackage
instances are running at the same time. /var/lib/dpkg is shared
by all instances and the /var/lib/dpkg/updates could be corrupted
and cause the build failure.
the fix is to use overlay fs to mount separate /var/lib/dpkg
for each dpkg-buildpackage instance so that they are not affecting
each other.
Signed-off-by: Guohan Lu <lguohan@gmail.com>
The HLD about MACsec feature is at :
https://github.com/Azure/SONiC/blob/master/doc/macsec/MACsec_hld.md
- How to verify it
This PR doesn't set MACsec container automatically start, You should manually start the container by docker run docker-macsec
wpa_supplicant binary can be found at MACsec container.
This PR depends on the PR, WPA_SUPPLICANT, and The MACsec container will be set as automatically start by later PR.
Signed-off-by: zegan <zegan@microsoft.com>
build frr dbg image force to install frr in the build process
which breaks the current build and is uneccessary.
Signed-off-by: Guohan Lu <lguohan@gmail.com>
frr build requires libyang 1.0.184 which conflicts with
libyang 1.0.73. Solution here is to compile frr and libyang 1.0.184
first, and then uninstall libyang 1.0.184 after frr build.
Then, compile libyang 1.0.73 and all packages depend on it later.
Signed-off-by: Guohan Lu <lguohan@gmail.com>
**- Why I did it**
python2 is end of life and SONiC is going to support python3. This PR is going to support:
1. Build pmon daemons with python3
2. Install and run python3 version pmon daemons
**- How I did it**
1. Change pmon daemons make files to build bothe python2 and python3 whl
2. Change docker-platform-monitor make files to install both python2 and python3 whl
3. Change pmon docker startup files to start pmon daemons according to the supported platform API version
- add config option ENABLE_DOCKER_BASE_PULL to pull sonic-slave docker from registry
- use REGISTRY_PORT, REGISTRY_SERVER to specify docker registry
Signed-off-by: Guohan Lu <lguohan@gmail.com>
**- Why I did it**
As part of migrating SONiC codebase from Python 2 to Python 3
**- How I did it**
- No longer install Python 2 in docker-base-buster or docker-config-engine-buster.
- Install Python 2 and pip2 in the following containers until we can completely eliminate it there:
- docker-platform-monitor
- docker-sonic-mgmt-framework
- docker-sonic-vs
- Pin pip2 version <21 where it is still temporarily needed, as pip version 21 will drop support for Python 2
- Also preform some other cleanup, ensuring that pip3, setuptools and wheel packages are installed in docker-base-buster, and then removing any attempts to re-install them in derived containers
* First cut image update for kubernetes support.
With this,
1) dockers dhcp_relay, lldp, pmon, radv, snmp, telemetry are enabled
for kube management
init_cfg.json configure set_owner as kube for these
2) Each docker's start.sh updated to call container_startup.py to register going up
As part of this call, it registers the current owner as local/kube and its version
The images are built with its version ingrained into image during build
3) Update all docker's bash script to call 'container start/stop/wait' instead of 'docker start/stop/wait'.
For all locally managed containers, it calls docker commands, hence no change for locally managed.
4) Introduced a new ctrmgrd service, that helps with transition between owners as kube & local and carry over any labels update from STATE-DB to API server
5) hostcfgd updated to handle owner change
6) Reboot scripts are updatd to tag kube running images as local, so upon reboot they run the same image.
7) Added kube_commands.py to handle all updates with Kubernetes API serrver -- dedicated for k8s interaction only.
HLD: Azure/SONiC#646
Introducing chassisd process to monitor status of the control, line and fabric cards in a modular chassis.
- Why I did it
Modular Chassis has control-cards, line-cards and fabric-cards along with other peripherals. Chassisd will be a central entity that has visibility of the entire chassis.
- How I did it
Chassisd process will monitor cards in the main thread. Another configuation_handling_task is created to listen to CONFIG_DB for admin_status up/down events. The monitored status is persisted in REDIS-DB.
libxslt-dev and libz-dev are dependencies for lxml==4.6.1 which is required for pyangbind==0.8.1
lxml-4.6.2-cp37-cp37m-manylinux1_x86_64.whl is directly downloaded in amd64 whereas in arm this is built from lxml-4.6.2.tar.gz
Signed-off-by: Sabareesh Kumar Anandan <sanandan@marvell.com>
python2 is end of life and SONiC is going to support python3. This PR is going to support:
1. Mellanox SONiC platform API python3 support
2. Install both python2 and python3 verson of Mellanox SONiC platform API or pmon and host side
Barefoot platform vendors' sonic_platform packages import the Python 'thrift' library. Previously, our custom-built package was being installed in the PMon container and host OS. However, we are only building a Python 2 version of that package, which was only intended for use with saithrift.
Fixes#6077
The soinc-frr module has src/sonic-frr/frr submodule. The FRR sub module dependency files are not added to the DPKG file tracking. The patch includes the following.
- Included the submodule dependency files
- Removes the symbolic files.
It fixes the following error during build.
>>> fatal: not a git repository (or any of the parent directories): .git
"SONIC_DPKG_CACHE_METHOD" : "cache"
"DPKG_CACHE_PATH" : "/var/cache/sonic/artifacts"
Made changes so that Lldp docker start using py3 of sonic-db-syncd
submodule update sonic-db-syncd
5cc29a1b32d8d1f4dfbc967bfea2727c50a49c76 (HEAD -> master, origin/master, origin/HEAD) Changes to convert sonic-dbsyncd from python 2 to 3
Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
Submodule updates include the following commits:
* src/sonic-utilities 9dc58ea...f9eb739 (18):
> Remove unnecessary calls to str.encode() now that the package is Python 3; Fix deprecation warning (#1260)
> [generate_dump] Ignoring file/directory not found Errors (#1201)
> Fixed porstat rate and util issues (#1140)
> fix error: interface counters is mismatch after warm-reboot (#1099)
> Remove unnecessary calls to str.decode() now that the package is Python 3 (#1255)
> [acl-loader] Make list sorting compliant with Python 3 (#1257)
> Replace hard-coded fast-reboot with variable. And some typo corrections (#1254)
> [configlet][portconfig] Remove calls to dict.has_key() which is not available in Python 3 (#1247)
> Remove unnecessary conversions to list() and calls to dict.keys() (#1243)
> Clean up LGTM alerts (#1239)
> Add 'requests' as install dependency in setup.py (#1240)
> Convert to Python 3 (#1128)
> Fix mock SonicV2Connector in python3: use decode_responses mode so caller code will be the same as python2 (#1238)
> [tests] Do not trim from PATH if we did not append to it; Clean up/fix shebangs in scripts (#1233)
> Updates to bgp config and show commands with BGP_INTERNAL_NEIGHBOR table (#1224)
> [cli]: NAT show commands newline issue after migrated to Python3 (#1204)
> [doc]: Update Command-Reference.md (#1231)
> Added 'import sys' in feature.py file (#1232)
* src/sonic-py-swsssdk 9d9f0c6...1664be9 (2):
> Fix: no need to decode() after redis client scan, so it will work for both python2 and python3 (#96)
> FieldValueMap `contains`(`in`) will also work when migrated to libswsscommon(C++ with SWIG wrapper) (#94)
- Also fix Python 3-related issues:
- Use integer (floor) division in config_samples.py (sonic-config-engine)
- Replace print statement with print function in eeprom.py plugin for x86_64-kvm_x86_64-r0 platform
- Update all platform plugins to be compatible with both Python 2 and Python 3
- Remove shebangs from plugins files which are not intended to be executable
- Replace tabs with spaces in Python plugin files and fix alignment, because Python 3 is more strict
- Remove trailing whitespace from plugins files
**- Why I did it**
We were building a custom version of Supervisor because I had added patches to prevent hangs and crashes if the system clock ever rolled backward. Those changes were merged into the upstream Supervisor repo as of version 3.4.0 (http://supervisord.org/changes.html#id9), therefore, we should be able to simply install the vanilla package via pip. This will also allow us to easily move to Python 3, as Python 3 support was added in version 4.0.0.
**- How I did it**
- Remove Makefiles and patches for building supervisor package from source
- Install Python 3 supervisor package version 4.2.1 in Buster base container
- Also install Python 3 version of supervisord-dependent-startup in Buster base container
- Debian package installed binary in `/usr/bin/`, but pip package installs in `/usr/local/bin/`, so rather than update all absolute paths, I changed all references to simply call `supervisord` and let the system PATH find the executable to prevent future need for changes just in case we ever need to switch back to build a Debian package, then we won't need to modify these again.
- Install Python 2 supervisor package >= 3.4.0 in Stretch and Jessie base containers
sonic-bgpcfgd build fails in the absence of Python 2, as it attempts to explicitly call sonic-cfggen using `/usr/bin/python2.7`. Also, it attempts to call sonic-cfggen using a local, relative path. Since the sonic-config-engine package is not installed, neither are its dependencies.
Now, we configure the Python 3 sonic-config-engine as a dependency of sonic-bgpcfgd, which ensures the Python 3 sonic-config-engine package and its dependencies are installed before sonic-bgpcfgd is built/tested.
Submodule updates include the following commits. Also adding sonic-swsscommon build dependencies where necessary.
* src/sonic-py-swsssdk 1ea30d2...9d9f0c6 (1):
> [SonicV2Connector] make decode_responses=True as default, so python2 application need no code change when migrated to python3(#93)
* src/sonic-snmpagent 6e4a796...57e54d9 (3):
> Interact with Redis by str instead of bytes, migrate to SonicV2Connector with `decode_responses=True` (#171)
> Add a test case for LLDP_LOC_CHASSIS.lldp_loc_man_addr has only one IPv6 address (#167)
> [LLDP]: Update init_db to load global database config (#166)
* src/sonic-utilities acfa824...b693cf6 (11):
> Remove SKU create pytest output directory before execution of the script (#1226)
> [show][techsupport][multi-ASIC] Add support to collect tech support on multi ASIC platform (#1193)
> [show] Fix `show ip bgp sum` (#1194)
> [sonic_sku_create] Move tests from sonic-utilities-tests/ folder to tests/ folder (#1222)
> Replace swsssdk.SonicV2Connector with swsscommon.SonicV2Connector (SWIG wrapper of C++ implementation) in production code (#1217)
> Copy missing values from INIT_CFG to config_db as part of db_migration task (#1209)
> [connect][clear] Support --devicename option for connect/clear line commands (#1214)
> [consutil][show] Remove root need from show line command (#1218)
> [Mellanox] SKU creator Tool (#1163)
> SONiC installer - fix string formatting during image type check (#1197)
> [show/fgnhg.py] Fix whitespace issues (#1211)
This change introduces PDDF which is described here: https://github.com/Azure/SONiC/pull/536
Most of the platform bring up effort goes in developing the platform device drivers, SONiC platform APIs and validating them. Typically each platform vendor writes their own drivers and platform APIs which is very tailor made to that platform. This involves writing code, building, installing it on the target platform devices and testing. Many of the details of the platform are hard coded into these drivers, from the HW spec. They go through this cycle repetitively till everything works fine, and is validated before upstreaming the code.
PDDF aims to make this platform driver and platform APIs development process much simpler by providing a data driven development framework. This is enabled by:
JSON descriptor files for platform data
Generic data-driven drivers for various devices
Generic SONiC platform APIs
Vendor specific extensions for customisation and extensibility
Signed-off-by: Fuzail Khan <fuzail.khan@broadcom.com>
Add Python 3 sonic-yang-mgmt package as a dependency in sonic-utilities makefile.
Recently updated the sonic-utilities Jenkins build environment [here](https://github.com/Azure/sonic-build-tools/pull/185) to begin running tests on the Python 3 version of the sonic-utilities package. However, the build is failing because it attempts to copy the Python 3 sonic-yang-mgmt package from the artifacts of the latest VS build, but the package does not exist because there are no targets which specify it as a dependency. This PR will ensure the Python 3 package is built during the image build process.
* Convert bgpcfgd to python3
Convert bgpmon to python3
Fix some issues in bgpmon
* Add python3-swsscommon as depends
* Install dependencies
* reorder deps
Co-authored-by: Pavel Shirshov <pavel.contrib@gmail.com>
* Build and install openssh from source
* Copy openssh deb package to dest folder
* Update make rule
* Update sonic debian extension
* Append empty line before EOF
* Update openssh patch
* Add openssh-server to base image dependency
* Fix indent type
* Fix comments
* Use commit id instead of tag id and add comment
Signed-off-by: Jing Kan jika@microsoft.com
- Change `/run/arista` mount to pmon by `/var/run/platform_cache`
- Python3 by default for Arista platform initialisation
- Fix outstanding py2/3 compatibility issues (eeprom mostly)
- Use pytest for unit testing
- Miscellaneous modular fixes
The orchagent and syncd need to have the same default synchronous mode configuration. This PR adds a template file to translate the default value in CONFIG_DB (empty field) to an explicit mode so that the orchagent and syncd could have the same default mode.
- Enable thermalctld support for our platforms
- Fix Chassis.get_num_sfp which had an off by one
- Implement read_eeprom and write_eeprom in SfpBase
- Refactor of Psus and PsuSlots. Psus they are now detected and metadata reported
- Improvements to modular support
Co-authored-by: Zhi Yuan Carl Zhao <zyzhao@arista.com>
**- Why I did it**
Install all host services and their data files in package format rather than file-by-file
**- How I did it**
- Create sonic-host-services Python wheel package, currently including procdockerstatsd
- Also add the framework for unit tests by adding one simple procdockerstatsd test case
- Create sonic-host-services-data Debian package which is responsible for installing the related systemd unit files to control the services in the Python wheel. This package will also be responsible for installing any Jinja2 templates and other data files needed by the host services.
bring up chassisdb service on sonic switch according to the design in
Distributed Forwarding in VoQ Arch HLD
Signed-off-by: Honggang Xu <hxu@arista.com>
**- Why I did it**
To bring up new ChassisDB service in sonic as designed in ['Distributed forwarding in a VOQ architecture HLD' ](90c1289eaf/doc/chassis/architecture.md).
**- How I did it**
Implement the section 2.3.1 Global DB Organization of the VOQ architecture HLD.
**- How to verify it**
ChassisDB service won't start without chassisdb.conf file on the existing platforms.
ChassisDB service is accessible with global.conf file in the distributed arichitecture.
Signed-off-by: Honggang Xu <hxu@arista.com>
We were building our own python-click package because we needed features/bug fixes available as of version 7.0.0, but the most recent version available from Debian was in the 6.x range.
"Click" is needed for building/testing and installing sonic-utilities. Now that we are building sonic-utilities as a wheel, with Click specified as a dependency in the setup.py file, setuptools will install a more recent version of Click in the sonic-slave-buster container when building the package, and pip will install a more recent version of Click in the host OS of SONiC when installing the sonic-utilities package. Also, we don't need to worry about installing the Python 2 or 3 version of the package, as the proper one will be installed as necessary.
* system health first commit
* system health daemon first commit
* Finish healthd
* Changes due to lower layer logic change
* Get ASIC temperature from TEMPERATURE_INFO table
* Add system health make rule and service files
* fix bugs found during manual test
* Change make file to install system-health library to host
* Set system LED to blink on bootup time
* Caught exceptions in system health checker to make it more robust
* fix issue that fan/psu presence will always be true
* fix issue for external checker
* move system-health service to right after rc-local service
* Set system-health service start after database service
* Get system up time via /proc/uptime
* Provide more information in stat for CLI to use
* fix typo
* Set default category to External for external checker
* If external checker reported OK, save it to stat too
* Trim string for external checker output
* fix issue: PSU voltage check always return OK
* Add unit test cases for system health library
* Fix LGTM warnings
* fix demo comments: 1. get boot up timeout from monit configuration file; 2. set system led in library instead of daemon
* Remove boot_timeout configuration because it will get from monit config file
* Fix argument miss
* fix unit test failure
* fix issue: summary status is not correct
* Fix format issues found in code review
* rename th to threshold to make it clearer
* Fix review comment: 1. add a .dep file for system health; 2. deprecated daemon_base and uses sonic-py-common instead
* Fix unit test failure
* Fix LGTM alert
* Fix LGTM alert
* Fix review comments
* Fix review comment
* 1. Add relevant comments for system health; 2. rename external_checker to user_define_checker
* Ignore check for unknown service type
* Fix unit test issue
* Rename user define checker to user defined checker
* Rename user_define_checkers to user_defined_checkers for configuration file
* Renmae file user_define_checker.py -> user_defined_checker.py
* Fix typo
* Adjust import order for config.py
Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>
* Adjust import order for src/system-health/health_checker/hardware_checker.py
Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>
* Adjust import order for src/system-health/scripts/healthd
Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>
* Adjust import orders in src/system-health/tests/test_system_health.py
* Fix typo
* Add new line after import
* If system health configuration file not exist, healthd should exit
* Fix indent and enable pytest coverage
* Fix typo
* Fix typo
* Remove global logger and use log functions inherited from super class
* Change info level logger to notice level
Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>
print cache status when use cached file in the build process
Without DPKG cache support :
[ building ] [ target/docker-base.gz ]
[ finished ] [ target/docker-base.gz ]
With DPKG cache support :
[ building ] [ target/docker-base.gz ]
[ cached ] [ target/docker-base.gz ]
extracted from PR 4595 by Kalimuthu Velappan
Signed-off-by: Guohan Lu <lguohan@gmail.com>
implements a new feature: "BGP Allow list."
This feature allows us to control which IP prefixes are going to be advertised via ebgp from the routes received from EBGP neighbors.
implements a new feature: "BGP Allow list."
This feature allows us to control which IP prefixes are going to be advertised via ebgp from the routes received from EBGP neighbors.
We want to let Monit to unmonitor the processes in containers which are disabled in `FEATURE` table such that
Monit will not generate false alerting messages into the syslog.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
We are moving toward building all Python packages for SONiC as wheel packages rather than Debian packages. This will also allow us to more easily transition to Python 3.
Python files are now packaged in "sonic-utilities" Pyhton wheel. Data files are now packaged in "sonic-utilities-data" Debian package.
**- How I did it**
- Build and install sonic-utilities as a Python package
- Remove explicit installation of wheel dependencies, as these will now get installed implicitly by pip when installing sonic-utilities as a wheel
- Build and install new sonic-utilities-data package to install data files required by sonic-utilities applications
- Update all references to sonic-utilities scripts/entrypoints to either reference the new /usr/local/bin/ location or remove absolute path entirely where applicable
Submodule updates:
* src/sonic-utilities aa27dd9...2244d7b (5):
> Support building sonic-utilities as a Python wheel package instead of a Debian package (#1122)
> [consutil] Display remote device name in show command (#1120)
> [vrf] fix check state_db error when vrf moving (#1119)
> [consutil] Fix issue where the ConfigDBConnector's reference is missing (#1117)
> Update to make config load/reload backward compatible. (#1115)
* src/sonic-ztp dd025bc...911d622 (1):
> Update paths to reflect new sonic-utilities install location, /usr/local/bin/ (#19)
This PR limited the number of calls to sonic-cfggen to one call
per iteration instead of current 3 calls per iteration.
The PR also installs jq on host for future scripts if needed.
signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
Remove radvd Makefile and patch, change docker-router-advertiser Dockerfile template to simply install the vanilla radvd package using apt-get.
- In PR https://github.com/Azure/sonic-buildimage/pull/2795, we started building radvd from source and patching it to prevent it from erroring out when advertising an MTU of 9100 which was greater than the MTU size configured on the bridge interface (1500), which was due to a limitation in the 4.9 Linux kernel.
- Master branch is now using Linux kernel 4.19. As of 4.18, the kernel supports setting a bridge MTU to a value > 1500.
- PR https://github.com/Azure/sonic-swss/pull/1393 modified vlanmgrd to take advantage of this and now configures the MTU of bridge interfaces in SONiC to the proper size of 9100. Therefore, we no longer need to patch radvd. Since we no longer need to patch radvd, we no longer need to build it from source, so we can save build time by going back to simply installing the vanilla radvd Debian package in the router-advertiser container.
Add a master switch so that the sync/async mode can be configured.
Example usage of the switch:
1. Configure mode while building an image
`make ENABLE_SYNCHRONOUS_MODE=y <target>`
2. Configure when the device is running
Change CONFIG_DB with `sonic-cfggen -a '{"DEVICE_METADATA":{"localhost": {"synchronous_mode": "enable"}}}' --write-to-db`
Restart swss with `systemctl restart swss`
- Why I did it
When SONiC is configured with the management framework and/or telemetry services, the applications running inside those containers need to access some functionality on the host system. The following is a non-exhaustive list of such functionality:
Image management
Configuration save and load
ZTP enable/disable and status
Show tech support
- How I did it
The host service is a Python process that listens for requests via D-Bus. It will then service those requests and send a response back to the requestor.
This PR only introduces the host service infrastructure. Applications that need access to the host services must add applets that will register on D-Bus endpoints to service the appropriate functionality.
- How to verify it
- Description for the changelog
Add SONiC Host Service for container to execute select commands in host
Signed-off-by: Nirenjan Krishnan <Nirenjan.Krishnan@dell.com>
As part of migrating all Python-based package installers to wheel format rather than Debian packages. Also to allow for easily building a Python 3 version of the package in the near future. ledd and psud were converted in earlier PRs. This PR converts the remainder:
- pcied
- syseepromd
- thermalctld
- xcvrd
As part of migrating all Python-based package installers to wheel format rather than Debian packages. Also to allow for easily building a Python 3 version of the package in the near future.
As part of migrating all Python-based package installers to wheel format rather than Debian packages. Also to allow for easily building a Python 3 version of the package in the near future.
- Also remove some references to sonic-daemon-base which I previously missed and add missing sonic-py-common dependency for sonic-pcied.
python3-swsscommon is newly added build target besides originally python2 version of swsscommon library. Add this target to buildimage rules so we could make target/debs/buster/python3-swsscommon_1.0.0_amd64.deb
Removes installation of kube-proxy (117 MB) and flannel (53 MB) images from Kubernetes-enabled devices. These images are tested to be unnecessary for our use case, as we do not rely on ClusterIPs for Kubernetes Services or a CNI for pod networking.
1. remove container feature table
2. do not generate feature entry if the feature is not included
in the image
3. rename ENABLE_* to INCLUDE_* for better clarity
4. rename feature status to feature state
5. [submodule]: update sonic-utilities
* 9700e45 2020-08-03 | [show/config]: combine feature and container feature cli (#1015) (HEAD, origin/master, origin/HEAD) [lguohan]
* c9d3550 2020-08-03 | [tests]: fix drops_group_test failure on second run (#1023) [lguohan]
* dfaae69 2020-08-03 | [lldpshow]: Fix input device is not a TTY error (#1016) [Arun Saravanan Balachandran]
* 216688e 2020-08-02 | [tests]: rename sonic-utilitie-tests to tests (#1022) [lguohan]
Signed-off-by: Guohan Lu <lguohan@gmail.com>
As part of consolidating all common Python-based functionality into the new sonic-py-common package, this pull request:
1. Redirects all Python applications/scripts in sonic-buildimage repo which previously imported sonic_device_util or sonic_daemon_base to instead import sonic-py-common, which was added in https://github.com/Azure/sonic-buildimage/pull/5003
2. Replaces all calls to `sonic_device_util.get_platform_info()` to instead call `sonic_py_common.get_platform()` and removes any calls to `sonic_device_util.get_machine_info()` which are no longer necessary (i.e., those which were only used to pass the results to `sonic_device_util.get_platform_info()`.
3. Removes unused imports to the now-deprecated sonic-daemon-base package and sonic_device_util.py module
This is the next step toward resolving https://github.com/Azure/sonic-buildimage/issues/4999
Also reverted my previous change in which device_info.get_platform() would first try obtaining the platform ID string from Config DB and fall back to gathering it from machine.conf upon failure because this function is called by sonic-cfggen before the data is in the DB, in which case, the db_connect() call will hang indefinitely, which was not the behavior I expected. As of now, the function will always reference machine.conf.
virtual-chassis test uses multiple vs instances to simulate a
modular switch and a redis-chassis service is required to run on
the vs instance that represents a supervisor card.
This change allows vs docker start redis-chassis service according
to external config file.
**- Why I did it**
To support virtual-chassis setup, so that we can test distributed forwarding feature in virtual sonic environment, see `Distributed forwarding in a VOQ architecture HLD` pull request at https://github.com/Azure/SONiC/pull/622
**- How I did it**
The sonic-vs start.sh is enhanced to start new redis_chassis service if external chassis config file found. The config file doesn't exist in current vs environment, start.sh will behave like before.
**- How to verify it**
The swss/test still pass. The chassis_db service is verified in virtual-chassis topology and tests which are in following PRs.
Signed-off-by: Honggang Xu <hxu@arista.com>
(cherry picked from commit c1d45cf81ce3238be2dcbccae98c0780944981ce)
Co-authored-by: Honggang Xu <hxu@arista.com>
Verify that /etc/apt/sources.list points to buster using docker exec bgp cat /etc/apt/sources.list
BGP neighborship is established.
root@sonic:~# show ip bgp summary
IPv4 Unicast Summary:
BGP router identifier 10.1.0.1, local AS number 65100 vrf-id 0
BGP table version 1
RIB entries 1, using 184 bytes of memory
Peers 1, using 20 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
6.1.1.1 4 100 96 96 0 0 0 01:32:04 0
Total number of neighbors 1
root@sonic:~#
Signed-off-by: Joyas Joseph <joyas_joseph@dell.com>
when parallel build is enabled, both docker-fpm-frr and docker-syncd-brcm
is built at the same time, docker-fpm-frr requires swss which requires to
install libsaivs-dev. docker-syncd-brcm requires syncd package which requires
to install libsaibcm-dev.
since libsaivs-dev and libsaibcm-dev install the sai header in the same
location, these two packages cannot be installed at the same time. Therefore,
we need to serialize the build between these two packages. Simply uninstall
the conflict package is not enough to solve this issue. The correct solution
is to have one package wait for another package to be uninstalled.
For example, if syncd is built first, then it will install libsaibcm-dev.
Meanwhile, if the swss build job starts and tries to install libsaivs-dev,
it will first try to query if libsaibcm-dev is installed or not. if it is
installed, then it will wait until libsaibcm-dev is uninstalled. After syncd
job is finished, it will uninstall libsaibcm-dev and swss build job will be
unblocked.
To solve this issue, _UNINSTALLS is introduced to uninstall a package that
is no longer needed and to allow blocked job to continue.
Signed-off-by: Guohan Lu <lguohan@gmail.com>
Consolidate common SONiC Python-language functionality into one shared package (sonic-py-common) and eliminate duplicate code.
The package currently includes three modules:
- daemon_base
- device_info
- logger
This PR updates kubernetes version to 1.18.6 and kubernetes-cni version to 0.8.6
signed-off by: Isabel Li isabel.li@microsoft.com
Why I did it
Previous kubernetes-cni version (0.7.5) introduced Kubernetes Man In The Middle Vulnerability. “A vulnerability was found in all versions of containernetworking/plugins before version 0.8.6, that allows malicious containers in Kubernetes clusters to perform man-in-the-middle (MitM) attacks. A malicious container can exploit this flaw by sending rogue IPv6 router advertisements to the host or other containers, to redirect traffic to the malicious container.”
How I did it
Defined kubernetes-cni version to be 0.8.6 and updated kubernetes version to be 1.18.6
How to verify it
Check versions by running dpkg -l | grep kube
also update submodule
* 01f810f 2020-07-02 | fix compiling issue for gcc8.3 (#1339) [lguohan]
* 9b13120 2020-07-03 | Fix in script to avoid orchagent crash when port down followed by fdb delete (#1340) [rupesh-k]
* 9b01844 2020-07-01 | [qosorch] Update QoS scheduler params for shaping features (#1296) [Michael Li]
* 86b5e99 2020-07-02 | [mirrororch] Port Mirroring implementation (#1314) [rupesh-k]
* c05601c 2020-06-24 | [portsyncd]: add debug message if a port cannot be found in port able (#1328) [lguohan]
* a0b6412 2020-06-23 | COPP_DEL_fix: DEL for one trap group from SONIC is resetting all the trap IDs (#1273) [SinghMinu]
Signed-off-by: Guohan Lu <lguohan@gmail.com>
make swss build depends only on libsairedis instead of syncd. This allows to build swss without depending
on vendor sai library.
Currently, libsairedis build also buils syncd which requires vendor SAI lib. This makes difficult to build
swss docker in buster while still keeping syncd docker in stretch, as swss requires libsairedis which also
build syncd and requires vendor to provide SAI for buster. As swss docker does not really contain syncd
binary, so it is not necessary to build syncd for swss docker.
* [submodule]: update sonic-sairedis
* ccbb3bc 2020-06-28 | add option to build without syncd (HEAD, origin/master, origin/HEAD) [Guohan Lu]
* 4247481 2020-06-28 | install saidiscovery into syncd package [Guohan Lu]
* 61b8e8e 2020-06-26 | Revert "sonic-sairedis: Add support to sonic-sairedis for gearbox phys (#624)" (#630) [Danny Allen]
* 85e543c 2020-06-26 | add a README to tests directory to describe how to run 'make check' (#629) [Syd Logan]
* 2772f15 2020-06-26 | sonic-sairedis: Add support to sonic-sairedis for gearbox phys (#624) [Syd Logan]
Signed-off-by: Guohan Lu <lguohan@gmail.com>
* [sonic-buildimage] Changes to make network specific sysctl
common for both host and docker namespace (in multi-npu).
This change is triggered with issue found in multi-npu platforms
where in docker namespace
net.ipv6.conf.all.forwarding was 0 (should be 1) because of
which RS/RA message were triggered and link-local router were learnt.
Beside this there were some other sysctl.net.ipv6* params whose value
in docker namespace is not same as host namespace.
So to make we are always in sync in host and docker namespace
created common file that list all sysctl.net.* params and used
both by host and docker namespace. Any change will get applied
to both namespace.
Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
* Address Review Comments and made sure to invoke augtool
only one and do string concatenation of all set commands
* Address Review Comments.
Since we can not refer a dir in sonic-buildimage while jenkins testing of sonic-utilities.
We need to create build dependency on sonic_yang_models PKG too.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
sonic-utils has sonic-yang-mgmt as build time deps, which inturn installs libyang.
libyang is needed to run newly added test.
If sonic-yang-mgmt is already built then libyang will not be installed in slave docker
without this PR and test will not run.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
Add build flag TELEMETRY_WRITABLE. When set to "y" it will add a go build flag in the telemetry build that will enable telemetry write mode to allow configuration via gNMI Set RPC as well as operations via the gNOI RPC's. The default for TELEMETRY_WRITABLE is unset in which case telemetry is read-only. In read-only mode the Set RPC and all gNOI RPC's are disabled and will return an "Unsupported" error when called.
authored-by: Eric Seifert <eric@seifert.casa>
- Why I did it
New repo sonic-mgmt-common is introduced for the common translib related code. This commit adds build rules for this new repo.
- How I did it
Added sonic-mgmt-common submodule
Added build rules for the new sonic-mgmt-common repo. It creates two deb packages -- sonic-mgmt-common_1.0.0_{arch}.deb and sonic-mgmt-common-codegen_1.0.0_{arch}.deb. Package cache is enabled.
Added dependency on sonic-mgmt-common for mgmt-framework and telemetry debs and dockers.
- How to verify it
Full build and incremental builds
Basic ACL and interface opreations through REST, KLISH CLI and gNMI
- Description for the changelog
Git submodule and build rules for the new sonic-mgmt-common repo.
To run the unit test for sonic-utilities/config/config_mgmt.py, we need sonic-yang-mgmt.
It is decided to not mock sonic-yang-mgmt because config_mgmt is highly dependent on sonic-yang-mgmt.
So It is good to catch any compatibility issues during build time rather than run time.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
The `-sv2` suffix was used to differentiate SNMP Dockers when we transitioned from "SONiCv1" to "SONiCv2", about four years ago. The old Docker materials were removed long ago; there is no need to keep this suffix. Removing it aligns the name with all the other Dockers.
Also edit Monit configuration to detect proper snmp-subagent command line in Buster, and make snmpd command line matching more robust.
The -sv2 suffix was used to differentiate SNMP Dockers when we transitioned from "SONiCv1" to "SONiCv2", about four years ago. The old Docker materials were removed long ago; there is no need to keep this suffix. Removing it aligns the name with all the other Dockers.
This changes is needed to support parallel build Jobs.
Made LIBYANG_PY2 and LIBYANG_PY3 depend on LIBYANG and LIBYANG_CPP. Also LIBYANG_CPP depends on LIBYANG.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
Changes:
-- Removing the part where build dependencies are installed in setup.py.
-- Adding build dependencies in corresponsing rules\..*.mk file.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
**- What I did**
#### wheel package Makefiles
- wheel package Makefiles for sonic-yang-mgmt package.
#### libyang Python APIs:
- python APIs based on libyang
- functions to load/merge yang models and Yang data files
- function to validate data trees based on Yang models
- functions to merge yang data files/trees
- add/set/delete node in schema and data trees
- find data/schema nodes from xpath from the Yang data/schema tree in memory
- find dependencies
- dump the data tree in json/xml
#### Extension of libyang Python APIs:
-- Cropping input config based on Yang Model.
-- Translate input config based on Yang Model.
-- rev Translate input config based on Yang Model.
-- Find xpath of port, portleaf and a yang list.
-- Find if node is key of a list while deletion if yes, then delete the parent.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
Signed-off-by: Ping Mao pmao@linkedin.com
This change adds support to build dockers using buster as base.
- Define docker-base-buster using docker-base-stretch as starting point
- Define docker-config-engine-buster using docker-config-engine-stretch as starting point.
- sonic-mgmt-framework docker is updated to build using buster as base
Signed-off-by: Joyas Joseph <joyas_joseph@dell.com>
[baseimage]: upgrade base image to debian buster
bring up the base image to debian buster 4.19 kernel.
using the merge commits to preserve the individual commits to better track the history.
- build SONIC_STRETCH_DOCKERS in sonic-slave-stretch docker
- build image related module in sonic-slave-buster docker.
This includes all kernels modules and some packages
Signed-off-by: Guohan Lu <lguohan@gmail.com>
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL.
YANG models as per Guidelines.
Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md
[sonic-yang-models/tests]: YANG model test code and JSON input for testing.
[sonic-yang-models/setup.py]: Build infra for yang models.
**- What I did**
Created Yang model for Sonic.
Tables: PORT, VLAN, VLAN_INTERFACE, VLAN_MEMBER, ACL_RULE, ACL_TABLE, INTERFACE.
Created build infra files using which a new package (sonic-yang-models) can be build and can be deployed on sonic switches. Yang models will be part of this new package.
**- How I did it**
Wrote yang models based on Guideline doc:
https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md
and
https://github.com/Azure/SONiC/wiki/Configuration.
Wrote python wheel Package infra which runs test for these Yang models using a json files which consists configuration as per yang models. These configs are for negative tests, which means we want to test that most must condition, pattern and when condition works as expected.
**- How to verify it**
Build Logs and testing:
———————————————————————————————————
```
/sonic/src/sonic-yang-models /sonic
running test
running egg_info
writing top-level names to sonic_yang_models.egg-info/top_level.txt
writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt
writing sonic_yang_models.egg-info/PKG-INFO
reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
running build_ext
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
running bdist_wheel
running build
running build_py
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../libyang_1.0.73_amd64.deb ...
Unpacking libyang (1.0.73) over (1.0.73) ...
Setting up libyang (1.0.73) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Processing triggers for man-db (2.7.6.1-2) ...
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../libyang-cpp_1.0.73_amd64.deb ...
Unpacking libyang-cpp (1.0.73) over (1.0.73) ...
Setting up libyang-cpp (1.0.73) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../python3-yang_1.0.73_amd64.deb ...
Unpacking python3-yang (1.0.73) over (1.0.73) ...
Setting up python3-yang (1.0.73) ...
INFO:YANG-TEST:module: sonic-vlan is loaded successfully
ERROR:YANG-TEST:Could not get module: sonic-head
INFO:YANG-TEST:module: sonic-portchannel is loaded successfully
INFO:YANG-TEST:module: sonic-acl is loaded successfully
INFO:YANG-TEST:module: sonic-loopback-interface is loaded successfully
ERROR:YANG-TEST:Could not get module: sonic-port
INFO:YANG-TEST:module: sonic-interface is loaded successfully
INFO:YANG-TEST:
------------------- Test 1: Configure a member port in VLAN_MEMBER table which does not exist.---------------------
libyang[0]: Leafref "/sonic-port:sonic-port/sonic-port:PORT/sonic-port:PORT_LIST/sonic-port:port_name" of value "Ethernet156" points to a non
-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlan_name='Vlan100'][port='Ethernet156']/port)
INFO:YANG-TEST:Configure a member port in VLAN_MEMBER table which does not exist. Passed
INFO:YANG-TEST:
------------------- Test 2: Configure non-existing ACL_TABLE in ACL_RULE.---------------------
libyang[0]: Leafref "/sonic-acl:sonic-acl/sonic-acl:ACL_TABLE/sonic-acl:ACL_TABLE_LIST/sonic-acl:ACL_TABLE_NAME" of value "NOT-EXIST" points
to a non-existing leaf. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NOT-EXIST'][RULE_NAME='Rule_20']/ACL_TABLE_NAME)
INFO:YANG-TEST:Configure non-existing ACL_TABLE in ACL_RULE. Passed
INFO:YANG-TEST:
------------------- Test 3: Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU
LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/ICMPV6_CODE)
INFO:YANG-TEST:Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE. Passed
INFO:YANG-TEST:
INFO:YANG-TEST:
------------------- Test 4: Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU
LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_20']/SRC_IPV6)
INFO:YANG-TEST:Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE. Passed
------------------- Test 5: Configure l4_src_port_range as 99999-99999 in ACL_RULE---------------------
libyang[0]: Value "99999-99999" does not satisfy the constraint "([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])-([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6'][RULE_NAME='Rule_20']/L4_SRC_PORT_RANGE)
INFO:YANG-TEST:Configure l4_src_port_range as 99999-99999 in ACL_RULE Passed
INFO:YANG-TEST:
------------------- Test 6: Configure empty string as ip-prefix in INTERFACE table.---------------------
libyang[0]: Invalid value "" in "ip-prefix" element. (path: /sonic-interface:sonic-interface/INTERFACE/INTERFACE_LIST[interface='Ethernet8'][ip-prefix='']/ip-prefix)
INFO:YANG-TEST:Configure empty string as ip-prefix in INTERFACE table. Passed
INFO:YANG-TEST:
------------------- Test 7: Configure Wrong family with ip-prefix for VLAN_Interface Table---------------------
libyang[0]: Must condition "(contains(../ip-prefix, ':') and current()='IPv6') or (contains(../ip-prefix, '.') and current()='IPv4')" not satisfied. (path: /sonic-vlan:sonic-vlan/VLAN_INTERFACE/VLAN_INTERFACE_LIST[vlanid='100'][ip-prefix='2a04:5555:66:7777::1/64']/family)
INFO:YANG-TEST:Configure Wrong family with ip-prefix for VLAN_Interface Table Passed
INFO:YANG-TEST:
------------------- Test 8: Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPV6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NS
W-PACL-V6'][RULE_NAME='Rule_20']/DST_IPV6)
INFO:YANG-TEST:Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE. Passed
INFO:YANG-TEST:
------------------- Test 9: Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE.---------------------
libyang[0]: Value "0x080C" does not satisfy the constraint "(0x88CC|0x8100|0x8915|0x0806|0x0800|0x86DD|0x8847)" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/INNER_ETHER_TYPE)
INFO:YANG-TEST:Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE. Passed
INFO:YANG-TEST:
------------------- Test 10: Add dhcp_server which is not in correct ip-prefix format.---------------------
libyang[0]: Invalid value "10.186.72.566" in "dhcp_servers" element. (path: /sonic-vlan:sonic-vlan/VLAN/VLAN_LIST/dhcp_servers[.='10.186.72.566'])
INFO:YANG-TEST:Add dhcp_server which is not in correct ip-prefix format. Passed
INFO:YANG-TEST:
------------------- Test 11: Configure undefined acl_table_type in ACL_TABLE table.---------------------
libyang[0]: Invalid value "LAYER3V4" in "type" element. (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6']/type)
INFO:YANG-TEST:Configure undefined acl_table_type in ACL_TABLE table. Passed
INFO:YANG-TEST:
------------------- Test 12: Configure undefined packet_action in ACL_RULE table.---------------------
libyang[0]: Invalid value "SEND" in "PACKET_ACTION" element. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST/PACKET_ACTION)
INFO:YANG-TEST:Configure undefined packet_action in ACL_RULE table. Passed
INFO:YANG-TEST:
------------------- Test 13: Configure wrong value for tagging_mode.---------------------
libyang[0]: Invalid value "non-tagged" in "tagging_mode" element. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST/tagging_mode)
INFO:YANG-TEST:Configure wrong value for tagging_mode. Passed
INFO:YANG-TEST:
------------------- Test 14: Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table.---------------------
libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid)
libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid)
INFO:YANG-TEST:Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table. Passed
INFO:YANG-TEST:All Test Passed
../../target/debs/stretch/libyang0.16_0.16.105-1_amd64.deb installtion failed
../../target/debs/stretch/libyang-cpp0.16_0.16.105-1_amd64.deb installtion failed
../../target/debs/stretch/python2-yang_0.16.105-1_amd64.deb installtion failed
YANG Tests passed
Passed: pyang -f tree ./yang-models/*.yang > ./yang-models/sonic_yang_tree
copying tests/yangModelTesting.py -> build/lib/tests
copying tests/test_sonic_yang_models.py -> build/lib/tests
copying tests/__init__.py -> build/lib/tests
running egg_info
writing top-level names to sonic_yang_models.egg-info/top_level.txt
writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt
writing sonic_yang_models.egg-info/PKG-INFO
reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/yangModelTesting.py -> build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/test_sonic_yang_models.py -> build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/wheel/tests
running install_data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-head.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-acl.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-loopback-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-port.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-portchannel.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-vlan.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
```
* Install kubernetes worker node packages, if enabled.
* Minor updates
* Added some comments
* Updates per review comments.
Built a private image to test to work fine.
* Remove the removed file.
* Update per comments
Make a fix, as kubeadm no demands a higher version of kubelet & kubectl.
As kubeadm auto install kubectl & kubelet, removing explicit install is an easier/robust fix.
* Changes per review comments.
* Updates per comments.
1) Dropped helper & pod scripts
2) Made install verbose
* Drop creation of pods subdir, as this PR does not use them.
* From comments to 'n' per review comments.
* 1) kubeadm.conf is created as part of kubeadm package install. Hence dropped explicit copy.
As per SONiC CLI requirements, we wanted the collector port to default to 6343 if not explicitly configured. In the Click based CLI, if the collector port is omitted, the port will be set to 6343 by default (https://github.com/Azure/sonic-utilities/blob/master/config/main.py#L2543).
However, if sonic-mgmt-framework based CLI is used, hsflowd was using port 0 as default. host-sflow v2.0.26-4 addresses this issue (i.e. use 6343 in case “collector_port” is not set).
Upgrade hsflowd to v2.0.26-4 to pick up https://github.com/sflow/host-sflow/releases/tag/v2.0.26-4.
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.
The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.
This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.
- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=
where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
none: no caching
cache: cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.
For example: dependency files for Linux Kernel - src/sonic-linux-kernel,
SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST)
SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))
DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
$(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)
$(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA
$(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS)
$(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format
FORMAT:
<module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
Eg:
linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz
< 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
< 24-byte MOD SHA value > - the SHA value is derived from either of the following.
GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule
GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.
$(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
$(call dpkg_depend,$(DEBS_PATH)/%.dep )
$(HEADER)
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
.....
# Rules for Generating the target DEB file.
.....
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
$(FOOTER)
The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.
Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
It also updates the target-specific variable to indicate the cache is loaded or not.
The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it
The caching functionality is verified by enabling it in Linux kernel submodule.
It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)
DOCUMENT PR:
https://github.com/Azure/SONiC/pull/559
This patch upgrade the kernel from version
4.9.0-9-2 (4.9.168-1+deb9u3) to 4.9.0-11-2 (4.9.189-3+deb9u2)
Co-authored-by: rajendra-dendukuri <47423477+rajendra-dendukuri@users.noreply.github.com>
* [database] Implement the auto-restart feature for database container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [database] Remove the duplicate dependency in service files. Since we
already have updategraph ---> config_setup ---> database, we do not need
explicitly add database.service in all other container service files.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [event listener] Reorganize the line 73 in event listener script.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [database] update the file sflow.service.j2 to remove the duplicate
dependency.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [event listener] Add comments in event listener.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [event listener] Update the comments in line 56.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [event listener] Add parentheses for if statement in line 76 in event listener.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
Changed logic for building frr. Previously we used latest commit in the FRR_BRANCH. Now the buildsystem will use a tag to identify a commit for building. New approach will let us to update sonic-frr without corrupting building sonic-buildimage.
* Changes in sonic-buildimage for the NAT feature
- Docker for NAT
- installing the required tools iptables and conntrack for nat
Signed-off-by: kiran.kella@broadcom.com
* Add redis-tools dependencies in the docker nat compilation
* Addressed review comments
* add natsyncd to warm-boot finalizer list
* addressed review comments
* using swsscommon.DBConnector instead of swsssdk.SonicV2Connector
* Enable NAT application in docker-sonic-vs
- move single instance services into their own folder
- generate Systemd templates for any multi-instance service files in slave.mk
- detect single or multi-instance platform in systemd-sonic-generator based on asic.conf platform specific file.
- update container hostname after creation instead of during creation (docker_image_ctl)
- run Docker containers in a network namespace if specified
- add a service to create a simulated multi-ASIC topology on the virtual switch platform
Signed-off-by: Lawrence Lee <t-lale@microsoft.com>
Signed-off-by: Suvarna Meenakshi <Suvarna.Meenaksh@microsoft.com>
* Add a monit config file for teamd container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* Add a copy mechanism to put the monit config file in teamd container
into base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* Add a monit config file for snmp container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* Add a copy mechanism to put the monit config file of snmp container into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* Add a monit config file for dhcp_relay container in the dir
base_image_files.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* Add a copy mechanism to put the monit config file of dhcp_relay
container into base image under /etc/monit/conf.d.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* Add a monit config file for router advertiser container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* Add a copy mechanism to put the monit config file of router advertiser
contianer into base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-Pmon] Add a monit config file for pmon container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-Pmon] Add a copy mechanism to put the monit config file into the
base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-lldp] Add a monit config file for lldp container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-lldp] Add a copy mechanism to put the monit config file into the
base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-bgp] Add a monit config file for BGP container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-bgp] Add a copy mechanism to put monit config file into the base
image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-swss] Add a monit config file for the swss container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-swss] Add a copy mechanism to put monit config file into the
base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a monit config file for syncd container on barefoot
platform.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image on barefoot.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a monit config file for syncd container on broadcom.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image on broadcom.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a monit config file for syncd container on cavium.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-centec] Add a monit config file for syncd container on centen
platform.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a monit config file for syncd container on centen
platform.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a monit config file for syncd container on marvell.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit conifg file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a monit config file for syncd container on
marvell-arm64.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image on marvell-arm64.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a monit config file for syncd container on
marvell-armhf.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a monit config file for syncd container on mellanox.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a monit config file for syncd container on nephos.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-sflow] Add a monit config file for sflow container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-sflow] Add a copy mechanism to put the monit conifg file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-telemetry] Add a monit config file for telemetry container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-telemetry] Add a copy mechanism to put the monit config file
into the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-database] Add a monit config file for database container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-database] Add a copy mechanism to put the monit config file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-Dhcprelay] Change a typo.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-Dhcprelay] Change the process name in monit config file to
dhcrelay.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] There is no desserve process in syncd container on
barefoot.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] There is no process desserve in syncd container on
cavium.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] There is no process named desserve in syncd on centec.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] There is no process named desserve in syncd on marvell.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Should not delete the process desserve in syncd container
on marvell.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Delete the process dsserve in syncd on marvell.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Delete the process dsserve in syncd container on
marvell-arm64.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Delete the process dsserve in syncd container on
marvell-armhf.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Delete the process dsserve in syncd container on
mellanox.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-Radv] Change the process name to radvd.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-telemetry] Correct a typo in monit_telemetry.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-teamd] Delete the monit config file for teamd.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-teamd] Delete the mechanism to copy the monit config file into
base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-dhcprelay] Delete the monit config file for dhcp_relay
container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-dhcprelay] Delete the mechanism to copy the monit config file
into the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-radv] Delete the monit config file foe radv container.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-radv] Delete the mechanism to copy the monit config file into
the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-bgp] change the monit config file for BGP container such that
monit only generates alert if the process is not running for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-snmp] Change the monit config file for snmp container such that
monit only generates alret if the process is not running for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-pmon] Change the monit config file for pmon container such that
monit only generates alert if the processes are not running for 5
minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-lldp] Change the monit config file for lldp container such that
monit only generates alerts if some processes are not running for 5
minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-pmon] Delete the monit config file for pmon container since some
of processes are not running depended on the type of box.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-pmon] Delete the copy mechanism to copy the monit config file
into the base image.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-lldp] Change the matching name for the process lldpd.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-swss] Change the monit config file for swss container such that
monit only generates alerts if the processes are not running for 5
minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Change the monit config file for syncd container on
barefoot such that monit only generates alerts if the process is not
running for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Correct a typo in monit config file.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Change the monit config file for syncd container on
broadcom such that monit only generates alerts if the processes are not
running for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Change the monit config file for syncd container on
cavium such that monit only generates alerts if the process is not
running for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Change the monit config file for syncd container such
that monit only generates alerts if the process is not running for 5
minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Change the monit config file for syncd container on
marvell such that monit only generates alerts if the process is not
running for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Change the monit config file for syncd container on
marvell-arm64 such that monit only generates alerts if the process is
not running for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Change the monit config file for syncd container on
marvell-armhf such that monit will generate alert if the process is not
running for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Change the monit config file for syncd container on
mellanox such that monit only generates alerts if the process is not
running for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-sycnd] Change the monit config file for syncd container such
that monit only generates alerts if the processes are not running for 5
minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-sflow] Change the monit config file for sflow container such
that monit only generates alerts if the process is not running for 5
minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-telemetry] Change the monit config file for telemetry container
such that monit only generates alerts if the processes are not running
for 5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-database] Change the monit config file for database container
such that monit only generates alerts if the process is not running for
5 minutes.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-database] Use 4 spaces to replace 2 spaces in monit config file.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-bgp] Use 4 spcess to replace 2 spaces in monit config file.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-lldp] Use 4 spaces to replace 2 spaces in monit config file.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-swss] Use 4 spaces to replace 2 space in monit config file.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-sflow] Use 4 spaces to replace 2 spaces in monit config file.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-snmp] Use 4 spaces to replace 2 spaces in monit config file.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-telemetry] Use 4 spaces to replace 2 spaces in monit config
file.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on barefoot.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on broadcom.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on cavium.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on centec.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on marvell.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on mellanox.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-syncd] Use 4 spaces to repalce 2 spaces in the monit config file
on nephos.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
* [Docker-bgp] Remove the trailing extra spaces in monit config file.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
DHCP relay MONitor (dhcpmon) keeps track of DORA messages. If DHCP Relay
is detected to be not forwarding DORA message, dhcpmon will log such event
to syslog. Under the hood dhcpmon keeps counts of clients DR messages,
forwarded DR messages, DHCP server OA messages, and forwarded OA messages.
dhcpmon will check every 12 sec (configurable) if counts are monotonically
increasing and record snapshot of those counters. dhcpmon will report
discrepancies when detected between current counters and snapshot counters.
pull-request: https://github.com/Azure/sonic-buildimage/pull/3886
signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
Packaging libnl 3.5.0 based off libnl 3.2.27 packaging. libnl contains various bug fixes that are nice to have.
pull-request: https://github.com/Azure/sonic-buildimage/pull/3967
signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
* Added sonic-mgmt-framework as submodule / docker
* fix build issues
* update sonic-mgmt-framework submodule branch to master
* Merged changes 70007e6d2ba3a4c0b371cd693ccc63e0a8906e77..00d4fcfed6a759e40d7b92120ea0ee1f08300fc6
00d4fcfed6a759e40d7b92120ea0ee1f08300fc6 Modified environemnt variables
* Changes to build sonic-mgmt-framework docker
* bumped up sonic-mgmt-framework commit-id
* version bump for sonic-mgmt-framework commit-it
* bumped up sonic-mgmt-framework commit-id
* Add python packages to docker
* Build fix for docker with python packages
* added libyang as dependent package
* Allow building images on NFS-mounted clones
Prior to this change, `build_debian.sh` would generate a Debian
filesystem in `./fsroot`. This needs root permissions, and one of the
tests that is performed is whether the user can create a character
special file in the filesystem (using mknod).
On most NFS deployments, `root` is the least privileged user, and cannot
run mknod. Also, attempting to run commands like rm or mv as root would
fail due to permission errors, since the root user gets mapped to an
unprivileged user like `nobody`.
This commit changes the location of the Debian filesystem to `/fsroot`,
which is a tmpfs mount within the slave Docker. The default squashfs,
docker tarball and zip files are also created within /tmp, before being
copied back to /sonic as the regular user.
The side effect of this change is that the contents of `/fsroot` are no
longer available once the slave container exits, however they are
available within the squashfs image.
Signed-off-by: Nirenjan Krishnan <Nirenjan.Krishnan@dell.com>
* bumped up sonc-mgmt-framework commit to include PR #18
* REST Server startup script is enahnced to read the settings from
ConfigDB. Below table provides mapping of db field to command line
argument name.
============================================================
ConfigDB entry key Field name REST Server argument
============================================================
REST_SERVER|default port -port
REST_SERVER|default client_auth -client_auth
REST_SERVER|default log_level -v
DEVICE_METADATA|x509 server_crt -cert
DEVICE_METADATA|x509 server_key -key
DEVICE_METADATA|x509 ca_crt -cacert
============================================================
* Replace src/telemetry as submodule to sonic-telemetry
* Update telemetry commit HEAD
* Update sonic-telemetry commit HEAD
* libyang env path update
* Add libyang dependency to telemetry
* Add scripts to create JSON files for CLI backend
Scripts to create /var/platform/syseeprom and /var/platform/system, which are back-end
files for CLI, for system EEPROM and system information.
Signed-off-by: Howard Persh <Howard_Persh@dell.com>
* In startup script, create directory where CLI back-end files live
Signed-off-by: Howard Persh <Howard_Persh@dell.com>
* build dependency pkgs added to docker for build failure fix
* Changes to fix build issue for mgmt framework
* Fix exec path issue with telemetry
* s5232[device] PSU detecttion and default led state support
* Processing of first boot in rc.local should not have premature exit
Signed-off-by: Howard Persh <Howard_Persh@dell.com>
* docker mount options added for platform, system features
* bumped up sonic-mgmt-framework commit id to pick 23rd July 2019 changes
* Added mount options for telemetry docker to get access for system and platform info.
* Update commit for sonic-utilities
* [dell]: Corrected dport map and renamed config files for S5232F
* Fix telemetry submodule commit
* added support for sonic-cli console
* [Dell S5232F, Z9264F] Harden FPGA driver kernel module
For Dell S5232F and Z9264F platforms, be more strict when checking state
in ISR of FPGA driver, to harden against spurious interrupts.
Signed-off-by: Howard Persh <Howard_Persh@dell.com>
* update mgmt-framework submodule to 27th Aug commit.
* remove changes not related to mgmt-framework and sonic-telemetry
* Revert "Replace src/telemetry as submodule to sonic-telemetry"
This reverts commit 11c3192975.
* Revert "Replace src/telemetry as submodule to sonic-telemetry"
This reverts commit 11c3192975.
* make submodule changes and remove a change not related to PR
* more changes
* Update .gitmodules
* Update Dockerfile.j2
* Update .gitmodules
* Update .gitmodules
* Update .gitmodules
reverting experimental change
* Removed syspoll for release_1.0
Signed-off-by: Jeff Yin <29264773+jeff-yin@users.noreply.github.com>
* Update docker-sonic-mgmt-framework.mk
* Update sonic-mgmt-framework.mk
* Update sonic-mgmt-framework.mk
* Update docker-sonic-mgmt-framework.mk
* Update docker-sonic-mgmt-framework.mk
* Revert "Processing of first boot in rc.local should not have premature exit"
This reverts commit e99a91ffc2.
* Remove old telemetry directory
* Update docker-sonic-mgmt-framework.mk
* Resolving merge conflict with Azure
* Reverting the wrong merge
* Use CVL_SCHEMA_PATH instead of changing directory for telemetry startup
* Add missing export
* Add python mmh3 to slave dockerfile
* Remove sonic-mgmt-framework build dep for telemetry, fix dialout startup issues
* Provided flag to disable compiling mgmt-framework
* Update sonic-utilites point latest commit id
* Point sonic-utilities to Azure accepted SHA
* Updating mgmt framework to right sha
* Add sonic-telemetry submodule
* Update the mgmt-framework commit id
Co-authored-by: jghalam <joe.ghalam@gmail.com>
Co-authored-by: Partha Dutta <51353699+dutta-partha@users.noreply.github.com>
Co-authored-by: srideepDell <srideep_devireddy@dell.com>
Co-authored-by: nirenjan <nirenjan@users.noreply.github.com>
Co-authored-by: Sachin Holla <51310506+sachinholla@users.noreply.github.com>
Co-authored-by: Eric Seifert <seiferteric@gmail.com>
Co-authored-by: Howard Persh <hpersh@yahoo.com>
Co-authored-by: Jeff Yin <29264773+jeff-yin@users.noreply.github.com>
Co-authored-by: Arunsundar Kannan <31632515+arunsundark@users.noreply.github.com>
Co-authored-by: rvasanthm <51932293+rvasanthm@users.noreply.github.com>
Co-authored-by: Ashok Daparthi-Dell <Ashok_Daparthi@Dell.com>
Co-authored-by: anand-kumar-subramanian <51383315+anand-kumar-subramanian@users.noreply.github.com>
Put a flag for fast-reboot to the db using EXPIRE feature. Using this flag in other part of SONiC to start in Fast-reboot mode. If we reload a config, the state in the db will be removed.
* Build sonic-ztp package
- Add changes in make rules to conditionally include sonic-ztp package
Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
* In the event of a kernel crash, we need to gather as much information
as possible to understand and identify the root cause of the crash.
Currently, the kernel does not provide much information, which make
kernel crash investigation difficult and time consuming.
Fortunately, there is a way in the kernel to provide more information
in the case of a kernel crash. kdump is a feature of the Linux kernel
that creates crash dumps in the event of a kernel crash. This PR
will add kermel kdump support.
An extension to the CLI utilities config and show is provided to
configure and manage kdump:
- enable / disable kdump functionality
- configure kdump (how many kernel crash logs can be saved, memory
allocated for capture kernel)
- view kernel crash logs
* Makefile: ARM kernel support from sonic-linux-kernel
* Fix for multiarch build docker spawn
Platform: Install the DTB deb for the platform
Signed-off-by: Antony Rheneus <arheneus@marvell.com>
slave.mk: add SONIC_PLATFORM_API_PY2 as dependency of host
sonic_debian_extension.j2: install sonic_daemon_base and Mellanox-specific sonic_platform on host
mlnx-platform-api.mk: export mlnx_platform_api_py2_wheel_path for sonic_debian_extension.j2
sonic-daemon-base.mk: export daemon_base_py2_wheel_path for sonic_debian_extension.j2
daemon_base.py: hind unnecessary dependency of swss_common on host
Introduce a new "sflow" container (if ENABLE_SFLOW is set). The new docker will include:
hsflowd : host-sflow based daemon is the sFlow agent
psample : Built from libpsample repository. Useful in debugging sampled packets/groups.
sflowtool : Locally dump sflow samples (e.g. with a in-unit collector)
In case of SONiC-VS, enable psample & act_sample kernel modules.
VS' syncd needs iproute2=4.20.0-2~bpo9+1 & libcap2-bin=1:2.25-1 to support tc-sample
tc-syncd is provided as a convenience tool for debugging (e.g. tc-syncd filter show ...)
Both python2 and python3 wheels being built out of the same
source directory can interfere with each other when
SONIC_BUILD_JOBS > 1.
Signed-off-by: Greg Paussa <greg.paussa@broadcom.com>
* [kernel] Upgrade kernel to version 4.9.168-1+deb9u5
- Picking up security patches.
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
* advance linux-kernel sub module head
- What I did
Move the enabling of Systemd services from sonic_debian_extension to a new systemd generator
- How I did it
Create a new systemd generator to manually create symlinks to enable systemd services
Add rules/Makefile to build generator
Add services to be enabled to /etc/sonic/generated_services.conf to be read by the generator at boot time
Signed-off-by: Lawrence Lee <t-lale@microsoft.com>
ARM Architecture support in SONIC
make configure platform=[ASIC_VENDOR_ARCH] PLATFORM_ARCH=[ARM_ARCH]
SONIC_ARCH: default amd64
armhf - arm32bit
arm64 - arm64bit
Signed-off-by: Antony Rheneus <arheneus@marvell.com>
* Upgrade ifupdown2 to version 1.2.8
Required by ZTP to support ZTP over IPv6 transport
Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
* Added debug symbols to many debug dockers.
* For debug images *only*:
1) Archive source files into debug image
2) Archived source is copied into /src
3) Created an empty dir /debug
4) Mount both /src as ro & /debug as rw into every docker
5) Login banner will give some details on /src & /debug
6) Devs can copy core file into /debug and view it from inside a container.
7) Dev may create all gdb logs and other data directly into /debug.
* Dropped redundant REDIS_TOOLS per review comments.
* Added debug symbols to frr package and hence FRR based BGP docker.
* 1) Moved dbg_files.sh to scripts/
2) Src directories to archive are now collected from individual Makefiles.
3) Added few more debug symbols
4) Added few more debug dockers.
Here after no more changes except per review comments.
To debug:
Install required version of debug image in Switch or VM.
Copy core file into /debug of host
Get into Docker
gdb /usr/bin/<daemon> -c /debug/<your core file>
set directory /src/... <-- inside gdb to get the source
For non-in-depth debugging:
Download corresponding debug Docker image (docker-...-dbg.gz) to your VM
Load the image
Run image with entrypoint as 'bash' with dir containing core mapped in.
Run gdb on the core.
This is a follow-up of sonic-snmpagent PR 92
Now that licensing issues have been solved FRR is distributed with SNMP
support compiled-in. This PR adds the last bits of configuration to get
the frr-snmp debian packages added to the docker container and the
config bits to enable the snmp module in FRR
This PR brings the functionality of being able to poll bgpd for routes
and peer status.
Signed-off-by: Michel Moriniaux <m.moriniaux@criteo.com>
* [submodule] update sonic-linux-kernel
* update linux kernel version
* Fix many version strings
* update mellanox components (built with new kernel)
* [mlnx] add make files for SDK WJH libs
* Update arista driver submodule (#8)
Make the debian packaging point to a newer kernel version.
* Update frr to frr-7.0.1
* Fix a typo
* Set right permissions on /etc/frr
* Convert external file links from debian to Azure
* Revert python3 fix
* Build frr using more than 1 job
* Add SWIG as dependency for libswss-common
* [frr]: change frr as default sonic routing stack
* fix quagga configuration
* [vstest]: fix bgp test for frr
* [vstest]: skip bgp/test_invalid_nexthop.py for frr
Signed-off-by: Guohan Lu <gulv@microsoft.com>
* [service] Restart SwSS Docker container if orchagent exits unexpectedly
* Configure systemd to stop restarting swss if it attempts to restart more than 3 times in 20 minutes
* Move supervisor-proc-exit-listener script
* [docker-dhcp-relay] Enhance wait_for_intf.sh.j2 to utilize STATEDB
* Ensure dependent services stop/start/restart with SwSS
* Change 'StartLimitInterval' to 'StartLimitIntervalSec', as Stretch installs systemd 232 (>= v230)
* Also update journald.conf options
* Remove 'PartOf' option from unit files
* Add '$(SUPERVISOR_PROC_EXIT_LISTENER_SCRIPT)' to new shared docker-orchagent makefile
* Make supervisor-proc-exit-listener script read from 'critical_processes' file inside container
* Update critical_processes file for swss container
* Base DHCP relay Docker image on Strech base Docker
* Change URL for isc-dhcp source repository
* Upgrade isc-dhcp source branch to 4.3.5-3.1
* Update patch #0001 to apply to isc-dhcp 4.3.5-3.1
* Update patch #0002 to apply to isc-dhcp 4.3.5-3.1
* Update patch #0003 to apply to isc-dhcp 4.3.5-3.1
* Update patch #0004 to apply to isc-dhcp 4.3.5-3.1
* Remove security patches, as they are now applied as part of 4.3.5-3.1 source
* Reorder patches to apply bug fix first, then features
* Extend makefile to build debug Docker image
* Update commit that series file applies against
Overall goal: Build debug images for every stretch docker.
An earlier PR (#2789) made the first cut, by transforming broadcom/orchagent to build target/docker-orhagent-dbg.gz.
Changes in this PR:
Made docker-orchagent build to be platform independent.
1.1) Created rules/docker_orchagent.mk
1.2) Removed platform//docker-orchagent-*.mk
1.3) Removed the corresponding entry from platform//rules.mk
Extended the debug docker image build to stretch based syncd dockers.
2.1) For now, only mellanox & barefoot are stretch based.
2.2) All the common variable definitions are put in one place platform/template/docker-syncd-base.mk
2.3) platform/[mellanox, bfn]/docker-syncd-[mlnx, bfn].mk are updated as detailed below.
2.3.1) Set platform code and include template base file
2.3.2) Add the dependencies & debug dependencies and any update over what base template offers.
Extended all stretch based non-platform dockers to build debug dockers too.
3.1) Affected are:
docker-database.mk,
docker-platform-monitor.mk,
docker-router-advertiser.mk,
docker-teamd.mk,
docker-telemetry.mk
Next: Build debug flavor of final images with regular dockers replaced with debug dockers where available.
* [dockers]: Upgrade SNMP docker to stretch build
* Removed patch-> 0003-CHANGES-BUG-2743-snmpd-crashes-when-receiving-a-GetN.patch
* update platform-common submodule
* adding PyYAML package to stretch dockerfile
* Installing redis package via pip in stretch dockerfile
* again updating platform-common submodule
* revert back the snmpd version to 5.7.3+dfsg-1.5
* upgrading the snmpd version to 5.7.3+dfsg-1.7+deb9u1 with openssl backport support
* update sonic-snmpagent submodule
* purge libdpkg-perl package in Dockerfile
* revert back the snmpd version to 5.7.3+dfsg-1.5
* minor change in series file
* Updated Makefile infrastructure to build debug images.
As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target.
Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image.
This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside.
"docker run -it --entrypoint=/bin/bash e47a8fb8ed38"
You may map the core file path to this docker run.
* Dropped the regular binary using DBG_PACKAGES and a small name change to help readability.
* Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y.
When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag.
Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior.
* [vm build] force Linux to drop cache before calling kvm
KVM need to allocate 2G memory for this build. The system memory might
be occupied by cache at the moment and doesn't have 2G chunk to give
out. Forcing Kernel to drop cache to boost the chance of getting 2G
memory.
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
* [make] add option to enable/disable VS build memory preparation
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
DaemonBase is to wrap the common function of daemons, such as logging,
signal handling, db connector, load specific platform plugins.
Signed-off-by: Kevin Wang <kevinw@mellanox.com>
* Upgrade database and telemetry docker to stretch build
Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>
* Remove SONIC_STRETCH_DEBS list add for redis and telemetry
Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>
* [docker-radvd]: upgrade docker radvd to stretch based
* install jinja>=2.10
Signed-off-by: Guohan Lu <gulv@microsoft.com>
* install pip packages for testing sonic-utilities
Signed-off-by: Guohan Lu <gulv@microsoft.com>
* set storage driver to vfs
Signed-off-by: Guohan Lu <gulv@microsoft.com>
