Found another syncd timing issue related to clock going backwards.
To be safe disable the ntp long jump.
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
- Broadcom SAI 3.5 GA code drop on 20200608.
Changes:
- CS9533198
- CS10283709
- CS00009716645
- CS00010389861
- CS00010406122
- CS00010503275
- Addressed a few memory leak issues.
- Addressed an array memory allocation issue.
- Addressed assert during SER handling.
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
I found that with IPv4Network types, calling list(ip_ntwrk.hosts()) is reliable. However, when doing the same with an IPv6Network, I found that the conversion to a list can hang indefinitely. This appears to me to be a bug in the ipaddress.IPv6Network implementation. However, I could not find any other reports on the web.
This patch changes the behavior to call next() on the ip_ntwrk.hosts() generator instead, which returns the IP address of the first host.
**- Why I did it**
When I tested auto-restart feature of swss container by manually killing one of critical processes in it, swss will be stopped. Then syncd container as the peer container should also be
stopped as expected. However, I found sometimes syncd container can be stopped, sometimes
it can not be stopped. The reason why syncd container can not be stopped is the process
(/usr/local/bin/syncd.sh stop) to execute the stop() function will be stuck between the lines 164 –167. Systemd will wait for 90 seconds and then kill this process.
164 # wait until syncd quit gracefully
165 while docker top syncd$DEV | grep -q /usr/bin/syncd; do
166 sleep 0.1
167 done
The first thing I did is to profile how long this while loop will spin if syncd container can be
normally stopped after swss container is stopped. The result is 5 seconds or 6 seconds. If syncd
container can be normally stopped, two messages will be written into syslog:
str-a7050-acs-3 NOTICE syncd#dsserve: child /usr/bin/syncd exited status: 134
str-a7050-acs-3 INFO syncd#supervisord: syncd [5] child /usr/bin/syncd exited status: 134
The second thing I did was to add a timer in the condition of while loop to ensure this while loop will be forced to exit after 20 seconds:
After that, the testing result is that syncd container can be normally stopped if swss is stopped
first. One more thing I want to mention is that if syncd container is stopped during 5 seconds or 6 seconds, then the two log messages can be still seen in syslog. However, if the execution
time of while loop is longer than 20 seconds and is forced to exit, although syncd container can be stopped, I did not see these two messages in syslog. Further, although I observed the auto-restart feature of swss container can work correctly right now, I can not make sure the issue which syncd container can not stopped will occur in future.
**- How I did it**
I added a timer around the while loop in stop() function. This while loop will exit after spinning
20 seconds.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
Since the introduction of VRF, interface-related tables in ConfigDB will have multiple entries, one of which only contains the interface name and no IP prefix. Thus, when iterating over the keys in the tables, we need to ignore the entries which do not contain IP prefixes.
Modified caclmgrd behavior to enhance control plane security as follows:
Upon starting or receiving notification of ACL table/rule changes in Config DB:
1. Add iptables/ip6tables commands to allow all incoming packets from established TCP sessions or new TCP sessions which are related to established TCP sessions
2. Add iptables/ip6tables commands to allow bidirectional ICMPv4 ping and traceroute
3. Add iptables/ip6tables commands to allow bidirectional ICMPv6 ping and traceroute
4. Add iptables/ip6tables commands to allow all incoming Neighbor Discovery Protocol (NDP) NS/NA/RS/RA messages
5. Add iptables/ip6tables commands to allow all incoming IPv4 DHCP packets
6. Add iptables/ip6tables commands to allow all incoming IPv6 DHCP packets
7. Add iptables/ip6tables commands to allow all incoming BGP traffic
8. Add iptables/ip6tables commands for all ACL rules for recognized services (currently SSH, SNMP, NTP)
9. For all services which we did not find configured ACL rules, add iptables/ip6tables commands to allow all incoming packets for those services (allows the device to accept SSH connections before the device is configured)
10. Add iptables rules to drop all packets destined for loopback interface IP addresses
11. Add iptables rules to drop all packets destined for management interface IP addresses
12. Add iptables rules to drop all packets destined for point-to-point interface IP addresses
13. Add iptables rules to drop all packets destined for our VLAN interface gateway IP addresses
14. Add iptables/ip6tables commands to allow all incoming packets with TTL of 0 or 1 (This allows the device to respond to tools like tcptraceroute)
15. If we found control plane ACLs in the configuration and applied them, we lastly add iptables/ip6tables commands to drop all other incoming packets
Dynamic threshold setting changed to 0 and WRED profile green min threshold set to 250000 for Tomahawk devices
Changed the dynamic threshold settings in pg_profile_lookup.ini
Added a macro for WRED profiles in qos.json.j2 for Tomahawk devices
Necessary changes made in qos.config.j2 to use the macro if present
Signed-off-by: Neetha John <nejo@microsoft.com>
* [dhcpmon] Filter DHCP O/A Messages of Neighboring Vlans
This code fixes a bug where two or more vlans exist. Cross contamination
happens for DHCP packets Offer/Ack when received on shared northbound links.
The code filters out those packet based on dst IP equal Vlan loopback IP.
signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
* Include platform info in name.
Get SONiC Version as parameter and use
Make additional tag as optional.
Avoid repetitions by using function.
* Per review comments, make SONIC_VERSION optional and added some comments.
* 1) Added additional params are optional
2) Handle DOCKER_IMAGE_TAG only if given
3) Use BUILD_NUMBER only if SONIC_VERSION not given
4) Tag with SONIC_VERSION if given.
Current behavior is not changed, unless SONIC_VERSION is given.
* Update per review comments
1) Added new args with options
2) Handle PORT possible being empty
3) Exhibit new behavior only if both version & platform are given.
* Drop redundant quotes
* fixes an issue when /host/warmboot/issu_bank.txt is empty/corrupted
switch is not able to over come this and enters continuos reload/reboot
failure.
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
Submodule src/sonic-utilities f431510ae..d7e8f84cf:
Fix issue of fields overwritten before display (#863)
Signed-off-by: Guohan Lu <lguohan@gmail.com>
Co-authored-by: Guohan Lu <lguohan@gmail.com>
- What I did
Add configuration to avoid ntpd from panic and exit if the drift between new time and current system time is large.
- How I did it
Added "tinker panic 0" in ntp.conf file.
- How to verify it
[this assumes that there is a valid NTP server IP in config_db/ntp.conf]
Change the current system time to a bad time with a large drift from time in ntp server; drift should be greater than 1000s.
Reboot the device.
Before the fix:
3. upon reboot, ntp-config service comes up fine, ntp service goes to active(exited) state without any error message. This is because the offset between new time (from ntp server) and the current system time is very large, ntpd goes to panic mode and exits. The system continues to show the bad time.
After the fix:
3. Upon reboot, ntp-config comes up fine, ntp services comes up from and stays in active (running) state. The system clock gets synced with the ntp server time.
admin@sonic:~$ sudo hw-management-wd.sh
Usage: hw-management-wd.sh start [timeout] | stop | tleft | check_reset | help
start - start watchdog
timeout is optional. Default value will be used in case if it's omitted
timeout provided in seconds
stop - stop watchdog
tleft - check watchdog timeout left
check_reset - check if previous reset was caused by watchdog
Prints only in case of watchdog reset
help -this help
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
Submodule src/sonic-utilities e9747899a..f431510ae:
> [201811][intfutil] set speed to 0 when interface speed is not available (#840)
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
We believe that the supervisord issue in face of clock rolling backwards
has been addressed. Therefore reverting change 2598 to allow ntp sync
to right clock at the start up time.
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
* [bgpcfgd]: Generate set src configuration dynamically
Sometimes zebra starts faster then swss configures Loopback0
In this case "set src" inside of the route map will not be
inserted to the configuration because zebra doesn't see Loopback
ips in the list of available ip addresses.
I've added extra logic to push the "set src" configuration only
when Loopback has been configured by swss.
* [interfaces-config.sh] Flush the loopback interface before configure it
Without this, you may end up with more and more ip addresses
on loopback interface after you change the loopback ip and do config reload
Signed-off-by: Zhenggen Xu <zxu@linkedin.com>
Increased netlink socket receive buffer size for zebra. Otherwise we receive following messages sometimes:
zebra[60]: netlink-listen recvmsg overrun: No buffer space available
