Why I did it
Enable the reproducible build for PR build for master branch
Fix the reproducible build variable display error in the slave container.
The below config is none, although the config is set and takes effect.
"SONIC_VERSION_CONTROL_COMPONENTS": "none"
How I did it
Passing the variable through the slave container command line.
The variable has been passed to the slave container and the other docker container by a config file, it is only used to display the value during the build.
How to verify it
See https://dev.azure.com/mssonic/build/_build/results?buildId=247960&view=logs&j=88ce9a53-729c-5fa9-7b6e-3d98f2488e3f&t=88f376cf-c35d-5783-0a48-9ad83a873284
"SONIC_VERSION_CONTROL_COMPONENTS": "deb,py2,py3,web,git,docker"
Why I did it
We don't use snapshots of debian mirrors for sonic-slave containers even if MIRROR_SNAPSHOT is enabled.
How I did it
Export MIRROR_SNAPSHOT in Makefile.work to generate sources.list for sonic-slave containers using debian snapshot mirror
How to verify it
- Why I did it
To solve an issue with upgrade with fast-reboot including FW upgrade which has been introduced since moving to fast-reboot over warm-reboot infrastructure.
As well, this introduces fast-reboot finalizing logic to determine fast-reboot is done.
- How I did it
Added logic to finalize-warmboot script to handle fast-reboot as well, this makes sense as using fast-reboot over warm-reboot this script will be invoked. The script will clear fast-reboot entry from state-db instead of previous implementation that relied on timer. The timer could expire in some scenarios between fast-reboot finished causing fallback to cold-reboot and possible crashes.
As well this PR updates all services/scripts reading fast-reboot state-db entry to look for the updated value representing fast-reboot is active.
- How to verify it
Run fast-reboot and check that fast-reboot entry exists in state-db right after startup and being cleared as warm-reboot is finalized and not due to a timer.
#### Why I did it
Enhance the error message output mechanism during swss docker creating
#### How I did it
Capture the output to stderr of `sonic-cfggen` and output it using `echo` to make sure the error message will be logged in syslog.
#### How to verify it
Manually test
#### Why I did it
When removing port from LAG while traffic is running thorough LAG there is traffic disruption of 60 seconds.
Fix issue https://github.com/sonic-net/sonic-buildimage/issues/14381
#### How I did it
The patch I added introduces "port_removing" op and call it right before Kernel is asked to remove the port.
Implement the op in LACP runner to disable the port which leads to proper LACPDU send.
#### How to verify it
Set LAG between 2 switches.
Set LAGs to be router port and set ip address.
In switch A send ping to ip address of LAG in switch B.
In switch B, while ping is running remove port from LAG.
Verify ping is not stopping.
#### Why I did it
Fix issue: wrong teamd link watch state after warm reboot due to TEAM_ATTR_PORT_CHANGED lost
The flag TEAM_ATTR_PORT_CHANGED is maintained by kernel team driver:
- a flag "changed" is maintained in struct team_port struct
- the flag is set by __team_port_change_send once relevant information is updated, including port linkup (together with speed, duplex), adding or removing
- the flag is cleared by team_nl_fill_one_port_get once the updated information has been notified to user space via RTNL
In the userspace, the change flag is maintained by libteam in struct team_port.
The team daemon calls port_priv_change_handler_func on receiving port change event.
The logic in port_priv_change_handler_func
1. creates the port if it did not exist, which triggers port add event and eventually calls lacp_port_added callback.
2. triggers port change event if team_port->changed is true, which eventually calls lw_ethtool_event_watch_port_changed to update port state for link watch ethtool.
3. removes the port if team_port->removed is removed
In lacp_port_added, it calls team_refresh to refresh ifinfo, port info, and option info from the kernel via RTNL.
In this step, port_priv_change_handler_func is called recursively.
- In the inner call, it won't get TEAM_ATTR_PORT_CHANGED flag because kernel has already notified that.
- As a result, team_port->changed flag is cleared in the libteam.
- The port change event won't be triggered from either inner or outer call of port_priv_change_handler_func.
If the port has been up when the port is being added to the team device, the "port up" information is carried in the outer call but will be lost.
In case the flag TEAM_ATTR_PORT_CHANGED is set only in the inner call, function port_priv_change_handler_func can be called in the inner call.
However, it will fail to fetch "enable" options because option_list_init has not be called.
Signed-off-by: Stephen Sun <stephens@nvidia.com>
#### How I did it
Fix:
Do not call check_call_change_handlers when parsing RTNL function is called from another check_call_change_handlers recursively.
#### How to verify it
- Manually test
- Regression test
- warm reboot
- warm reboot sad lag
- warm reboot sad lag member
- warm reboot sad (partial)
Why I did it
Orchagent sometimes take additional time to execute Tunnel tasks. This cause write_standby script to error out and mux state machines are not initialized. It results in show mux status missing some ports in output.
Mar 13 20:36:52.337051 m64-tor-0-yy41 INFO systemd[1]: Starting MUX Cable Container...
Mar 13 20:37:52.480322 m64-tor-0-yy41 ERR write_standby: Timed out waiting for tunnel MuxTunnel0, mux state will not be written
Mar 13 20:37:58.983412 m64-tor-0-yy41 NOTICE swss#orchagent: :- doTask: Tunnel(s) added to ASIC_DB.
How I did it
Increase timeout from 60s to 90s
How to verify it
Verified that mux state machine is initialized and show mux status has all needed ports in it.
Why I did it
[Build] Support to use the snapshot mirror for debian base image
How I did it
If the MIRROR_SNAPSHOT=n, then use the default mirror http://deb.debian.org/debian
If the MIRROR_SNAPSHOT=y, then use the snapshot mirror, for instance http://packages.trafficmanager.net/snapshot/debian/20230330T000330Z/.
How to verify it
+ scripts/build_debian_base_system.sh amd64 bullseye ./fsroot-vs
I: Target architecture can be executed
I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id A4285295FC7B1A81600062A9605C66F00D6C9793)
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://packages.trafficmanager.net/snapshot/debian/20230331T000125Z...
I: Retrieving libacl1 2.2.53-10
Why I did it
At service start up time, there are chances that the networking service is being restarted by interface-config service. When that happens, write_standby could fail to make DB connections due to loopback interface is being reconfigured.
How I did it
Force the db connector to use unix socket to avoid loopback reconfig timing window.
How to verify it
Run config reload test 20+ times and no issue encountered.
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
* use unix socket instead
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
#### Why I did it
When CPU is busy, the sonic_ax_impl may not have sufficient speed to handle the notification message sent from REDIS.
Thus, the message will keep stacking in the memory space of sonic_ax_impl.
If the condition continues, the memory usage will keep increasing.
#### How I did it
Add a monit file to check if the SNMP container where sonic_ax_impl resides in use more than 4GB memory.
If yes, restart the sonic_ax_impl process.
#### How to verify it
Run a lot of this command: `while true; do ret=$(redis-cli -n 0 set LLDP_ENTRY_TABLE:test1 test1); sleep 0.1; done;`
And check the memory used by sonic_ax_impl keeps increasing.
After a period, make sure the sonic_ax_impl is restarted when the memory usage reaches the 4GB threshold.
And verify the memory usage of sonic_ax_impl drops down from 4GB.
#### Why I did it
Fixes#14277.
Fixes the inconsistent fallback behaviour for RADIUS authentication when AAA authentication is configured as "radius, local".
#### How I did it
Modified common-auth-sonic.j2 template to make sure that when no RADIUS servers are configured (with AAA authentication login method set to radius, local), the system falls back to local authentication successfully.
#### How to verify it
1. Configure authentication based on RADIUS and local.
config aaa authentication login radius local
2. Configure an unreachable RADIUS server.
config radius add 6.6.6.6
3. Try to login to switch with existing admin user credentials. This is successful.
4. Remove RADIUS server configuration.
config radius delete 6.6.6.6
5. Try to login to switch with admin user credentials. This is successful.
Based on the port breakout HLD, we are now using subport instead of channel in the CONFIG_DB PORT table to handle port breakout. The yang schema needs to be modified accordingly to handle the corresponding change.
The corresponding code changes have been merged through sonic-net/sonic-platform-daemons/pull/342 merged
Signed-off-by: Mihir Patel <patelmi@microsoft.com>
Why I did it
Support Egress Mirroring on supported Arista platforms
How I did it
Add necessary soc properties for egress mirroring recycle ports to be created
Signed-off-by: Nathan Wolfe <nwolfe@arista.com>
Updated asic_port_names for all Arista LC SKUs to follow latest naming
conventions to remove redundant ASICx suffix. For
Arista-7800R3-48CQ2-C48, added the asic_port_name mapping.
Why I did it
It is to fix the issue #13773
It only has impact on the build triggered manually inside of the slave container. Developers can go to the slave container do a build, it will print a skippable error message complaining the variable not found.
How I did it
Add the default value for variable SLAVE_DRI.
How to verify it
[S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert.
#### Why I did it
On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it.
However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully.
To avoid the false alert, improve the monitor to wait and re-check.
Steps to reproduce this issue:
1. User login to device via console, and keep the connection.
2. User login to device via SSH, check the serial-getty@ttyS1.service service, it's running.
3. Run 'monit reload' from SSH connection.
4. Check syslog 1 minutes later, there will be false alert: ' 'serial-getty' process is not running'
#### How I did it
Add check-getty.sh script to recheck again later when getty service not running.
And update monit unit to check serial-getty service status with this script to avoid false alert.
#### How to verify it
Pass all UT.
Manually check fixed code work correctly:
```
admin@***:~$ sudo systemctl stop serial-getty@ttyS1.service
admin@***:~$ sudo /usr/local/bin/check-getty.sh
admin@***:~$ echo $?
1
admin@***:~$ sudo systemctl status serial-getty@ttyS1.service
● serial-getty@ttyS1.service - Serial Getty on ttyS1
Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled)
Active: inactive (dead) since Tue 2023-03-28 07:15:21 UTC; 1min 13s ago
admin@***:~$ sudo /usr/local/bin/check-getty.sh
admin@***:~$ echo $?
0
admin@***:~$ sudo systemctl status serial-getty@ttyS1.service
● serial-getty@ttyS1.service - Serial Getty on ttyS1
Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled)
```
syslog:
```
Mar 28 07:10:37.597458 *** INFO systemd[1]: serial-getty@ttyS1.service: Succeeded.
Mar 28 07:12:43.010550 *** ERR monit[593]: 'serial-getty' status failed (1) -- no output
Mar 28 07:12:43.010744 *** INFO monit[593]: 'serial-getty' trying to restart
Mar 28 07:12:43.010846 *** INFO monit[593]: 'serial-getty' stop: '/bin/systemctl stop serial-getty@ttyS1.service'
Mar 28 07:12:43.132172 *** INFO monit[593]: 'serial-getty' start: '/bin/systemctl start serial-getty@ttyS1.service'
Mar 28 07:13:43.286276 *** INFO monit[593]: 'serial-getty' status succeeded (0) -- no output
```
#### Description for the changelog
[S6100] Improve S6100 serial-getty monitor.
#### Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
Why I did it
All these 3 services started after swss service, which used to start after interface-config service. But #13084 remove the time constraints for swss.
After that, these 3 services has the chance of start earlier when the inteface-config service is restarting the networking service, which could cause db connect request to fail.
How I did it
Delay mux/sflow/snmp timer after the interface-config service.
How to verify it
PR test.
Config reload can repro the issue in 1-3 retries. With this change. config reload run 30+ iterations without hitting the issue.
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
Change references to use bullseye instead of buster
Why I did it
Almost all daemons in 202211 and master uses bullseye, and sflow was easy to migrate.
How I did it
Replaced the references, built and tested in 202211.
How to verify it
Build with the changes, enable sflow:
admin@sonic:~$ sudo config sflow collector add test 1.2.3.4
admin@sonic:~$ sudo config sflow collector enable
tcpdump on 1.2.3.4 and see that UDP sFlow are being sent.
Signed-off-by: Christian Svensson <blue@cmd.nu>
Change references to use bullseye instead of buster
Why I did it
Almost all daemons in 202211 and master uses bullseye, and NAT seems easy to migrate.
How I did it
Replaced the references, built with 202211 branch.
How to verify it
Not sure, it builds and tests pass as far as I can tell but I don't use the feature myself.
Signed-off-by: Christian Svensson <blue@cmd.nu>
Why I did it
After sonic-install install a new image, print_menu is set echo without any data. No image info between Hit any key to stop autoboot: 0 and Start USB
Board configuration detected:
Net:
| port | Interface | PHY address |
|--------|-----------|--------------|
No ethernet found.
Hit any key to stop autoboot: 0
(Re)start USB...
USB0: Port (usbActive) : 0 Interface (usbType = 2) : USB EHCI 1.00
scanning bus 0 for devices... 3 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
How I did it
The fw_setenv print_menu is missing the double quotes. That causes the value is truncated. Using double quotes to in the environment setting.
How to verify it
Install new image with this fix. And reboot the system. The following section should be shown:
Signed-off-by: mlok <marty.lok@nokia.com>
Why I did it
We found a bug when pilot, the tag function doesn't remove the ACR domain when do tag, it makes the latest tag not work. And in the original tag function, it calls os.system and os.popen which are not recommend, need to refactor.
How I did it
Do a split("/") when get image_rep to fix the acr domain bug
Refactor the tag function code and add test cases
How to verify it
Check whether container images are tagged as latest when in kube mode.
Why I did it
Add platform files for critical processes and default qos config for Innovium platforms
How I did it
Added default files for critical processes and qos config
How to verify it
Tested with autorestart/test_container_autorestart.py::test_containers_autorestart
Signed-off-by: rck-innovium rck@innovium.com
Why I did it
mmh3's new version 3.1.0 breaks pipeline build.
bullseye/buster/jessie pined the version to 2.5.1
How I did it
Pin mmh3's version as other dists.
How to verify it
Why I did it
832ef9c4 - Fix bug in GCU vlanintf_validator ([Bcm SAI] ugprade Broadcom SAI to version 3.3.5.4m-1 #2765) (5 minutes ago) [jingwenxie]
53f611b7 - Revert "Convert IPv6 addresses to lowercase in apply-patch (Add Pegatron project to branch 201807 #2299)" (Add note for running out of disk space in /var/lib/docker to README.md #2758) (20 hours ago) [jingwenxie]
79a21cef - Revert frr route check ([mlnx] fix url inconsistency in fw.mk #2761) (8 minutes ago) [StormLiangMS]
824680ed - Resolved rc!=0 problem by replacing fgrep with awk. Added ipv4 filtering to get only v4 peers in case of show ip bgp neighbors (Improve eeprom access reliability #2756) (30 hours ago) [saurabh17g]
10f31ea6 - Revert "Replace pickle by json (Add autoneg to 7170-Q59S20 #2636)" ([hostcfgd] Default value of fallthrough for authentication set to be False. #2746) (7 days ago) [Mai Bui]
05fa7513 - Fix the show interface counters throwing exception on device with no external interfaces ([docker-platform-monitor]: Add smartmontools 6.6-1 #2703) (11 days ago) [abdosi]
f27dea0c - [route_check] remove check-frr_patch mock ([minigraph]: Mark both ERSPAN and ERSPANv6 as mirror ACL tables #2732) (11 days ago) [Stepan Blyshchak]
2d95529d - Revert "Update load minigraph to load backend acl (mlnx msn2010: default config_db.json generation with sonic-cfggen is not working #2236)" (swss stretch update broke restore_neighbors.py for neigh service #2735) (12 days ago) [Neetha John]
c869c970 - (master) Update the ref guide to reflect the vlan brief output ([teamd] update teamd docker to stretch and fix teamd_init failure #2731) (2 weeks ago) [Vivek]
76457141 - Fix fast-reboot DB migration ([teamd]: update teamd docker to stretch #2734) (2 weeks ago) [Aryeh Feigin]
f7f783bc - Enhance the logic to wait for all buffer tables to be removed in _clear_qos ([sfputil] Not able to read out values of voltage/temp/power on some cables #2720) (2 weeks ago) [Stephen Sun]
e6179afa - Remove timer from FAST_REBOOT STATE_DB entry and use finalizer (Rollback kernel submodule update. #2621) (3 weeks ago) [Aryeh Feigin]
ff688323 - [route_check] fix IPv6 address handling ([docker pmon] install fancontrol & sensord #2722) (3 weeks ago) [Stepan Blyshchak]
7a604c51 - update fast-reboot ([201811][sairedis][swss] advance sub module head of sairedis and swss #2728) (3 weeks ago) [jhli-cisco]
9f83ace9 - [GCU] Add vlanintf-validator (Revert "[device/celestica] blacklist gpio_ich kernel module on haliburton" #2697) (3 weeks ago) [jingwenxie]
338d1c05 - Check SONiC dependencies before installation. ([sonic-slave]: Add iproute2 dependencies in stretch docker #2716) (3 weeks ago) [Liu Shilong]
64d2efd2 - Improve show acl commands ([sonic-utilities] update submodule #2667) (3 weeks ago) [bingwang-ms]
2ef5b31e - [GCU] Add PFC_WD RDMA validator ([sub module] advance sonic-utilities sub module for 201811 branch #2619) (3 weeks ago) [isabelmsft]
c7aa8416 - [show][muxcable] increase timeout for displaying HW_STATUS (Fixing get_transceiver_change_event #2712) (3 weeks ago) [vdahiya12]
2fc2b826 - YANG validation for ConfigDB Updates: MIRROR_SESSION use case ([mellanox] Update SDK to 4.3.0132 #2430) (3 weeks ago) [isabelmsft]
e16bdaae - Fix non-zero status exit on non secure boot system ([service] add warmboot finializer service #2715) (3 weeks ago) [kellyyeh]
90d70152 - [route_check] implement a check for FRR routes not marked offloaded (Feature to run an option platform specific script on the first boot #2531) (3 weeks ago) [Stepan Blyshchak]
c2bc150a - [warm/fast-reboot] Backup logs from tmpfs to disk during fast/warm shutdown ([swss]: update swss docker to stretch #2714) (3 weeks ago) [Vaibhav Hemant Dixit]
a015834d - [db_migrator] Add missing attribute 'weight' to route entries in APPL DB ([device/celestica] blacklist gpio_ich kernel module on seastone #2691) (4 weeks ago) [Vaibhav Hemant Dixit]
cd519aac - [ci] Fix pipeline issue caused by sonic-slave-* change. ([201803] Modify Debian apt repos to reflect changes made by maintainers #2709) (4 weeks ago) [Liu Shilong]
2680e6f3 - [dhcp_relay] Fix dhcp_relay restart error while add/del vlan ([thrift] add a patch to revert THRIFT-3650 #2688) (4 weeks ago) [Yaqiang Zhu]
How I did it
How to verify it
Why I did it
This PR is to update the check of IP_TYPE from sonic-acl.yang.
It's because if the ACL rule is added by loading a json file with acl-loader, there is no IP_TYPE for ACL rule. If such rule exists in ACL_RULE table, the GCU (generic config updater) refuses to update any ACL rules because the existing one is invalid.
This PR updates the yang model for ACL. If the IP_TYPE leaf doesn't exist, then we don't check the field.
How I did it
Accept the rule if IP_TYPE is absent.
How to verify it
The change is verified by UT.
arp_update should resolve the missing arp/ndp static route
entries. Added code to check for missing entries and try ping to
resolve the missing entry.
Why I did it
Fixes#14179
chassis-packet: missing arp entries for static routes causing high orchagent cpu usage
It is observed that some sonic-mgmt test case calls sonic-clear arp, which clears the static arp entries as well. Orchagent or arp_update process does not try to resolve the missing arp entries after clear.
How I did it
arp_update should resolve the missing arp/ndp static route
entries. Added code to check for missing entries and try ping if any
found to resolve it.
How to verify it
After boot or config reload, check ipv4 and ipv4 neigh entries to make sure all static route entries are present
manual validation:
Use sonic-clear arp and sonic-clear ndp to clear all neighbor entries
run arp_update
Check for neigh entries. All entries should be present.
Signed-off-by: anamehra <anamehra@cisco.com>
Description for the changelog
Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
Why I did it
We don't need to install rsync in every docker container if vcache is disabled.
How I did it
Install rsync in pre_run_buildinfo script only if vcache is enabled.
How to verify it
Why I did it
Fix the installation candidate not found issue when building docker-sonic-vs
How I did it
Need to run the command "apt-get update" to update the mirror indexes before installing the package gnupg
How to verify it
Improve sudo cat command for RO user.
#### Why I did it
RO user can use sudo command show none syslog files.
#### How I did it
Improve sudo cat command for RO user.
#### How to verify it
Pass all UT.
Manually check fixed code work correctly.
#### Description for the changelog
Improve sudo cat command for RO user.
New docker versions use stderr instead of stdout to print info when build image.
As a resullt we got empty log files.
the fix is to redirect stderr to stdout when build sonic-slave images.
#### Why I did it
Fixes#9216
#### How I did it
Add support for python2 redis to the sonic-slave-stretch and sonic-slave-buster Dockerfiles
#### How to verify it
Run build steps documented in Issue #9216
Why I did it
After warm reboot, show environment prints the following error:
failed to import plugin show.plugins.macsec: [Errno 13] Permission denied: '/tmp/cache/macsec'
How I did it
Set owner back to admin after restoring counters folder.
How to verify it
sudo warm-reboot, then ensure show environement does not print errors.
Signed-off-by: Oleksandr Kolomeiets <oleksandrx.kolomeiets@intel.com>
