Commit Graph

1123 Commits

Author SHA1 Message Date
Shashanka Balakuntala
8b192a1151
[dhcp-relay]: Modify dhcp relay to pick primary address (#17012)
This is change taken as part of the HLD: sonic-net/SONiC#1470 and this is a follow up on the PR #16827 where in the docker-dhcp we pick the value of primary gateway of the interface from the VLAN_Interface table which has "secondary" flag set in the config_db

Microsoft ADO (number only): 16784946

How did I do it
-  Changes in the j2 file to add a new "-pg" parameter in the dhcpv4-relay.agents.j2, the ip would be retrieved from the config db's vlan_interface table such that the interface which are picked will have secondary field set.

- Changes in isc-dhcp to re-order the addresses of the discovered interface and which has the ip which has the passed parameter.
2023-11-22 15:05:32 -08:00
Saikrishna Arcot
730faa152a Fix docker-sonic-mgmt-framework for armhf
Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2023-11-21 18:53:15 -08:00
Ze Gan
a87cddc6c9
[docker-database-init.sh]: Fix wrong creating of database_global.json in multi asic platform (#17221)
Fix bug: #17161 (comment)
multi-asic platforms it will never go to the else part as DATABASE_TYPE is always ""


Microsoft ADO (number only): 25072889

Move the checker NAMESPACE_ID == "" back

Signed-off-by: Ze Gan <ganze718@gmail.com>
2023-11-21 09:41:20 -08:00
Xichen96
ee38e2447d
[dhcp_server] Add show dhcp_server ipv4 lease (#17125)
* Add show dhcp_server ipv4 lease
* add ut for show dhcp_server ipv4 lease
2023-11-21 08:42:07 -08:00
abdosi
4a7aa2634f
[chassis] Support advertisement of Loopback0 of all LC's across all e-BGP peers in TSA mode (#16714)
What I did:
In Chassis TSA mode Loopback0 Ip's of each LC's should be advertise through e-BGP peers of each remote LC's

How I did:

- Route-map policy to Advertise own/self Loopback IP to other internal iBGP peers with a community internal_community as define in constants.yml
- Route-map policy to match on above internal_community when route is received from internal iBGP peers and set a internal tag as define in constants.yml and also delete the internal_community so we don't send to any of e-BGP peers
- In TSA new route-map match on above internal tag and permit the route (Loopback0 IP's of remote LC's) and set the community to traffic_shift_community.
- In TSB delete the above new route-map.

How I verify:

Manual Verification

UT updated.
sonic-mgmt PR: sonic-net/sonic-mgmt#10239


Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
2023-11-20 09:42:02 -08:00
abdosi
e37b4f3cfa
Revert iBGP GTSM feature for VOQ Chassis (#17037)
What I did:

Revert the GTSM feature for VOQ iBGP session done as part of #16777.

Why I did:
On VOQ chassis BGP packets go over Recycle Port and then for Ingress Pipeline Routing making ttl as 254 and failing single hop check.

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
2023-11-17 17:03:37 -08:00
Ze Gan
9f08f88a0d
[dpu]: Add DPU database service (#17161)
Sub PRs:

sonic-net/sonic-host-services#84
#17191

Why I did it
According to the design, the database instances of DPU will be kept in the NPU host.

Microsoft ADO (number only): 25072889

How I did it
To follow the multiple ASIC design, I assume a new platform environment variable NUM_DPU will be defined in the /usr/share/sonic/device/$PLATFORM/platform_env.conf. Based on this number, NPU host will launch a corresponding number of instances for the DPU database.

Signed-off-by: Ze Gan <ganze718@gmail.com>
2023-11-17 09:10:03 -08:00
Yaqiang Zhu
3223ca0156
[dhcp_server] Add config_db monitor and customize options for dhcpservd (#17051)
Why I did it
Add config_db monitor and customize options for dhcpservd. HLD: sonic-net/SONiC#1282

Work item tracking
Microsoft ADO (number only): 25600859
How I did it
Add support to customize unassigned DHCP options. Current support type: binary, boolean, ipv4-address, string, uint8, uint16, uint32
Add db config change monitor for dhcpservd
How to verify it
Unit tests in sonic-dhcp-server all passed
2023-11-16 08:56:50 -08:00
Lawrence Lee
04b30fc378
[tph]: Detect LAG flaps from APPL_DB (#16879)
Why I did it
A race condition exists while the TPH is processing a netlink message - if a second netlink message arrives during processing it will be missed since TPH is not listening for other messages.
Another bug was found where TPH was unnecessarily restarting since it was checking admin status instead of operational status of portchannels.

How I did it
Subscribe to APPL_DB for updates on LAG operational state
Track currently sniffed interfaces

How to verify it
Send tunnel packets with destination IP of an unresolved neighbor, verify that ping commands are run
Shut down a portchannel interface, verify that sniffer does not restart
Send tunnel packets, verify ping commands are still run
Bring up portchannel interface, verify that sniffer restarts

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
2023-11-09 16:01:59 -08:00
JunhongMao
4da5099919
[VOQ][saidump] Install rdbtools into the docker base related containers. (#16466)
Fix #13561

The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access.

This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table.

Related PRs:
sonic-net/sonic-utilities#2972
sonic-net/sonic-sairedis#1288
sonic-net/sonic-sairedis#1298

How did I do it?
To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it.

1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers.

2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format.

3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps:

  For each ASIC, such as ASIC0,

  3.1. Config Redis consistency directory. 
  redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null

  3.2. Save the Redis data.
  redis-cli -h $hostname -p $port SAVE > /dev/null

  3.3. Run rdb command to convert the dump files into JSON files
    rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null

  3.4.  Run saidump -r to update the JSON files' format as same as the saidump before. 
       Then we can get the saidump's result in standard output."
       saidump -r $redis_dir/dump.json -m 100

  3.5. Clear the temporary files.
   rm -f $redis_dir/dump.rdb
   rm -f $redis_dir/dump.json

4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB.

How to verify it
On T2 setup with more than 96K routes, execute CLI command -- generate_dump
No error should be shown
Download the generate_dump result and verify the saidump file after unpacking it.
2023-11-08 11:57:25 -08:00
ganglv
c71fb3a30f
Share image for gnmi and telemetry (#16863)
Why I did it
Share docker image to support gnmi container and telemetry container

Work item tracking
Microsoft ADO 25423918:
How I did it
Create telemetry image from gnmi docker image.
Enable gnmi container and disable telemetry container by default.

How to verify it
Run end to end test.
2023-11-08 08:54:36 +08:00
ShiyanWangMS
7013b05899
Add new docker-sonic-mgmt makefile flag: LEGACY_SONIC_MGMT_DOCKER (#17070)
Why I did it
This is part of Python3 migration project. This PR will add a new makefile flag: LEGACY_SONIC_MGMT_DOCKER
Now by default: LEGACY_SONIC_MGMT_DOCKER = y will build sonic-mgmt-docker with Python2 and Python3
If LEGACY_SONIC_MGMT_DOCKER = n will will sonic-mgmt-docker with Python3 only

Work item tracking
Microsoft ADO (number only): 25254349

How I did it
Add makefile flag: LEGACY_SONIC_MGMT_DOCKER

How to verify it
By default will build sonic-mgmt-docker with Python2 and Python3. No change compared to before.
Set LEGACY_SONIC_MGMT_DOCKER=n will build sonic-mgmt-docker with Python3 only
2023-11-03 09:04:01 +08:00
Yaqiang Zhu
274d320443
[dhcp_server] Add dhcprelayd for dhcp_server feature (#16947)
Add support in dhcp_relay container for dhcp_server_ipv4 feature. HLD: sonic-net/SONiC#1282
2023-11-02 08:09:01 -07:00
ShiyanWangMS
fe735e35c6
Upgrade Ansible to 6.7.0 and make Python3 as the default interpreter in sonic-mgmt-docker (#17021)
Why I did it
This PR is part of sonic-mgmt-docker Python3 migration project.

Work item tracking
Microsoft ADO (number only): 24397943

How I did it
Upgrade Ansible to 6.7.0
Make Python3 as the default interpreter. python is a soft link to python3. If you want to use python2, use the command python2 explicitly.
Upgrade some pip packages to higher version in order to meet security requirement.

How to verify it
Build a private sonic-mgmt-docker successfully.
Verify python is python3.
Verify python2 is working with 202012 and 202205 branch.
Verify python3 is working with master branch.
Verify with github PR test.
2023-10-31 09:44:55 +08:00
Saikrishna Arcot
87f137be25
Upgrade paramiko in docker-ptf to 2.9.5 (#16897)
With Debian Bookworm, Paramiko 2.9 or newer will need to be used to be
able to connect to devices running that version of Debian
(specifically, to those running OpenSSH 9.2).

Paramiko is currently on 3.3.1. For now, upgrade to 2.9.5.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2023-10-24 22:51:16 -07:00
Yaqiang Zhu
73dd38a5ce
[dhcp_server] Add dhcpservd to dhcp_server container (#16560)
Why I did it
Part implementation of dhcp_server. HLD: sonic-net/SONiC#1282.
Add dhcpservd to dhcp_server container.

How I did it
Add installing required pkg (psutil) in Dockerfile.
Add copying required file to container in Dockerfile (kea-dhcp related and dhcpservd related)
Add critical_process and supervisor config.
Add support for generating kea config (only in dhcpservd.py) and updating lease table (in dhcpservd.py and lease_update.sh)

How to verify it
Build image with setting INCLUDE_DHCP_SERVER to y and enabled dhcp_server feature after installed image, container start as expected.
Enter container and found that all processes defined in supervisor configuration running as expected.
Kill processes defined in critical_processes, container exist.
2023-10-20 09:52:05 -07:00
Stepan Blyshchak
7ab27c1b90
[frr] fix default zebra config not inserted into empty zebra.conf (#16747)
Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
2023-10-19 08:47:24 +08:00
abdosi
7059f42385
[chassis/multi-asic] Make sure iBGP session established as directly connected (#16777)
What I did:
Make Sure for internal iBGP we are one-hop away (directly connected) by using Generic TTL security mechanism.

Why I did:
Without this change it's possible on packet chassis i-BGP can be established even if there no direct connection. Below is the example

- Let's say we have 3 LC's LC1/LC2/LC3 each having i-BGP session session with each other over Loopback4096
- Each LC's have static route towards other LC's Loopback4096 to establish i-BGP session
- LC1 learn default route 0.0.0.0/0 from it's e-BGP peers and send it over to LC2 and LC3 over i-BGP
- Now for some reason on LC2 static route towards LC3 is removed/not-present/some-issue we expect i-BGP session should go down between LC2 and LC3
- However i-BGP between LC2 and LC3 does not go down because of feature ip nht-resolve-via-default  where LC2 will use default route to reach Loopback4096 of LC3. As it's using default route BGP packets from LC2 towards LC3 will first route to LC1 and then go to LC3 from there.

Above scenario can result in packet mis-forwarding on data plane

How I fixed it:-

To make sure BGP packets between i-BGP peers are not going with extra routing hop enable using GTSM feature

neighbor PEER ttl-security hops NUMBER

This command enforces Generalized TTL Security Mechanism (GTSM), as specified in RFC 5082. With this command, only neighbors that are the specified number of hops away will be allowed to become neighbors. This command is mutually exclusive with ebgp-multihop.

We set hop count as 1 which makes FRR to reject BGP connection if we receive BGP packets if it's TTL < 255. Also setting this attribute make sure i-BGP frames are originated with IP TTL of 255.

How I verify:

Manual Verification of above scenario. See blow BGP packets receive with IP TTL 254 (additional routing hop) we are seeing FIN TCP flags as BGP is rejecting the connection

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
2023-10-10 11:51:40 -07:00
Hua Liu
6e3260098f
Enable ZMQ between GNMI and Orchanget (#16661)
Enable ZMQ on gnmi and orchagent

#### Why I did it
Improve GNMI API performance for Dash resources

#### How I did it
Modify gnmi and orchagent service start script, add ZMQ parameter.

#### How to verify it
Pass all UT & E2E test
Manually verify with create Dash resources via gnmi API.
2023-10-09 14:22:50 -07:00
vikumarks
b45ee0980b
[sonic-mgmt]: Added required python packages to run MSFT hero Test cases (#15883)
Added required python packages to run MSFT hero Test cases

dpugen==0.1.1
ctypesgen
pandas
PyYAML
ixload

Co-authored-by: Guohan Lu <lguohan@gmail.com>
2023-09-27 18:13:44 -07:00
Bob Chu
7e6790ab6b
[Telemetry] enable default service config if no config from DB (#16683)
#### Why I did it
Fix issue #16533 , telemetry service exit in master and 202305 branches due to no telemetry configs in redis DB.

#### How I did it
Enable default config if no TELEMETRY configs from redis DB.

#### How to verify it
After the fix, telemetry service would work with the following two scenarios:
1. With TELEMETRY config in redis DB, load service configs from DB.
2. No TELEMETRY config in redis DB, use default service configs.
2023-09-27 17:20:18 -07:00
Zain Budhwani
d89dde3b6d
Fix regex and process name (#16647)
### Why I did it

### How I did it

Fix regex such that dhcp bind failure event is detected as well as process name since dhcp relay processes that need to be detected are dhcprelay6 and dhcrelay.

#### How to verify it

Manual testing and nightly test event
2023-09-26 16:15:27 -07:00
abdosi
8b7b2a7f7c
[chassis/multi-asic] Enable Sending BGP Community over internal neighbors over iBGP Session (#16705)
What I did:
Enable Sending BGP Community over internal neighbors over iBGP Session

Microsoft ADO: 25268695

Why I did:
Without this change BGP community send by e-BGP Peers are not carry-forward to other e-BGP peers.


str2-xxxx-lc1-2# show bgp ipv6  20c0:a801::/64
BGP routing table entry for 20c0:a801::/64, version 52141
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  65000 65500
    2603:10e2:400::6 from 2603:10e2:400::6 (3.3.3.6)
      Origin IGP, localpref 100, valid, internal, best (First path received)
      Last update: Tue Sep 26 16:08:26 2023
str2-xxxx-lc1-2# show ip bgp 192.168.35.128/25
BGP routing table entry for 192.168.35.128/25, version 52688
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  65000 65502
    3.3.3.6 from 3.3.3.6 (3.3.3.6)
      Origin IGP, localpref 100, valid, internal, best (First path received)
      Last update: Tue Sep 26 15:45:51 2023

After the change

str2-xxxx-lc2-2(config)# router bgp 65100
str2-xxxx-lc2-2(config-router)# address-family ipv4
str2-xxxx-lc2-2(config-router-af)# neighbor INTERNAL_PEER_V4 send-community
str2-xxxx-lc2-2(config-router-af)# exit
str2-xxxx-lc2-2(config-router)# address-family ipv6
str2-xxxx-lc2-2(config-router-af)# neighbor INTERNAL_PEER_V6 send-community
str2-xxxx-lc1-2# show bgp ipv6  20c0:a801::/64
BGP routing table entry for 20c0:a801::/64, version 52400
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  65000 65500
    2603:10e2:400::6 from 2603:10e2:400::6 (3.3.3.6)
      Origin IGP, localpref 100, valid, internal, best (First path received)
      **Community: 1111:1111**
      Last update: Tue Sep 26 16:10:19 2023
str2-xxxx-lc1-2# show ip bgp 192.168.35.128/25
BGP routing table entry for 192.168.35.128/25, version 52947
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  65000 65502
    3.3.3.6 from 3.3.3.6 (3.3.3.6)
      Origin IGP, localpref 100, valid, internal, best (First path received)
      **Community: 1111:1111**
      Last update: Tue Sep 26 16:10:09 2023

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
2023-09-26 13:34:38 -07:00
Yevhen Fastiuk
52f6dd65a3
Improve remote fetch (#12795)
### Why I did it
To fix those errors:
One:
```
Connecting to urm.nvidia.com (urm.nvidia.com)|*.*.*.*|:443... connected.
GnuTLS: Error in the pull function.
Unable to establish SSL connection.
Error 4
make[1]: Leaving directory '/sonic/src/smartmontools'
[ target/debs/bullseye/smartmontools_6.6-1_amd64.deb ]
```
Second:
```
Get:90 https://debian-mirror-url buster/main amd64 librrd-dev amd64 1.7.1-2 [284 kB]
Get:91 https://debian-mirror-url buster/main amd64 psmisc amd64 23.2-1+deb10u1 [126 kB]
Get:92 https://debian-mirror-url buster/main amd64 python-smbus amd64 4.1-1 [12.2 kB]
Get:93 https://debian-mirror-url buster/main amd64 python3.7-dev amd64 3.7.3-2+deb10u3 [510 kB]
Get:94 https://debian-mirror-url buster/main amd64 python3-dev amd64 3.7.3-1 [1264 B]
Get:95 https://debian-mirror-url buster/main amd64 python3-smbus amd64 4.1-1 [12.5 kB]
Get:96 https://debian-mirror-url buster/main amd64 rrdtool amd64 1.7.1-2 [485 kB]
Fetched 122 MB in 12s (9976 kB/s)
E: Failed to fetch https://debian-mirror-url/pool/main/p/python-defaults/python2-minimal_2.7.16-1_amd64.deb  500  Internal Server Error [IP: *.*.*.* 443]
E: Failed to fetch https://debian-mirror-url/pool/main/f/fontconfig/fontconfig-config_2.13.1-2_all.deb  500  Internal Server Error [IP: *.*.*.* 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get update &&       apt-get install -y          build-essential         python3-dev             ipmitool                librrd8                 librrd-dev              rrdtool                 python-smbus            python3-smbus           dmidecode               i2c-tools               psmisc                  libpci3' returned a non-zero code: 100
[ target/docker-platform-monitor.gz ]
Error 1
```

#### How I did it
Add retry mechanism to apt, wget, and curl hooks
2023-09-23 18:07:04 -07:00
Ze Gan
83d67d4c8a
[build]: Polish protobuf build (#16119)
- Use dget to download the protobuf source code
- Add official link in sonic-mgmt Dockerfile for protobuf

Signed-off-by: Ze Gan <ganze718@gmail.com>
2023-09-23 00:25:43 -07:00
prasannam2302
c3cf42124b
[bgp]: exclude bgpmon for frrcfgd. (#16582)
Boot up a switch, if frrcfgd is enabled with frr_mgmt_framework_config being "true", 
then "bgpmon" process should not be running after this change. bgpmon should be 
running when bgpcfgd is enable with frr_mgmt_framework_config being "false"
2023-09-23 00:17:36 -07:00
Liu Shilong
bfa05c8349
[build] Fix build issue in docker-ptf-sai caused by setuptools_scm new release (#16636)
docker-ptf-sai build fails on setuptools_scm's new release on 09/20/2023.
Use old version instead.
2023-09-21 10:38:08 -07:00
Zain Budhwani
2dfdeb94d6
Load generic omprog in all dockers for rsyslog plugin support (#16601)
### Why I did it

##### Work item tracking
- Microsoft ADO **(number only)**:13366345

#### How I did it

Add generic omprog file in all dockers for rsyslog plugin support. Add file to docker-config-engine-bullseye so no need to add individually.

#### How to verify it

UT/Pipeline
2023-09-20 16:27:42 -07:00
vdahiya12
45a852233b
[pmon] update gRPC version to 1.57.0 (#16257)
Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>
2023-09-15 16:41:51 -07:00
anamehra
78981d93b8
Chassis: fix pmon docker failure when DEVICE_METADATA is not available (#16527)
Signed-off-by: anamehra anamehra@cisco.com

Added a check for DEVICE_METADATA before accessing the data. This prevents the j2 failure when var is not available.
2023-09-13 14:10:56 -07:00
ShiyanWangMS
42126ccf7d
Revert "Upgrade Ansible to 6.7.0 and make Python3 as the default interpreter in sonic-mgmt-docker (#15836)" (#16537)
This reverts commit 51fb6d7d9f.

The new sonic-mgmt docker image has ansible upgraded. Encountered some issues that are hard to debug to have a quick fix. Let's revert the change for now. The new sonic-mgmt docker image was kept for further debugging and fixing. After all the issues are fixed, we'll need to apply this change again.
2023-09-13 16:20:17 +08:00
Zain Budhwani
337a9dbcf4
Add rsyslog plugin support for frr log (#16192)
### Why I did it

Currently there is only rsyslog plugin support for /var/log/syslog, meaning we do not detect events that occur in frr logs such as BGP Hold Timer Expiry that appears in frr/bgpd.log. 

##### Work item tracking
- Microsoft ADO **(number only)**: 13366345

#### How I did it

Add omprog action to frr/bgpd.log and frr/zebra.log. Add appropriate regex for both events.

#### How to verify it

sonic-mgmt test case
2023-09-12 16:53:45 -07:00
ShiyanWangMS
51fb6d7d9f
Upgrade Ansible to 6.7.0 and make Python3 as the default interpreter in sonic-mgmt-docker (#15836)
Why I did it
This PR is part of sonic-mgmt-docker Python3 migration project.

Work item tracking
Microsoft ADO (number only): 24397943

How I did it
Upgrade Ansible to 6.7.0
Make Python3 as the default interpreter. python is a soft link to python3. If you want to use python2, use the command python2 explicitly.
Upgrade some pip packages to higher version in order to meet security requirement.

How to verify it
Build a private sonic-mgmt-docker successfully.
Verify python is python3.
Verify python2 is working with 202012 and 202205 branch.
Verify python3 is working with master branch.
2023-09-12 17:34:57 +08:00
Yaqiang Zhu
76b7cb8b64
[dhcp_server] Add dhcp_server container (#14031)
Why I did it
Add dhcp_server ipv4 feature to SONiC.
HLD: sonic-net/SONiC#1282

How I did it
To be clarify: This container is disabled by INCLUDE_DHCP_SERVER = n for now, which would cause container not build.

Add INCLUDE_DHCP_SERVER to indicate whether to build dhcp_server container
Add docker file for dhcp_server, build and install kea-dhcp4 inside container
Add template file for dhcp_server container services.
Add entry for dhcp_server to FEATURE table in config_db.
How to verify it
Build image with INCLUDE_DHCP_SERVER = y to verify:

Image can be install successfully without crush.
By config feature state dhcp_server enabled to enable dhcp_server.
2023-09-11 09:15:56 -07:00
jcaiMR
9c1c82e9ff
add show dhcp_relay ipv4 counter entry, fix interface name offset issue (#16507)
Why I did it
Add another cli entry: show dhcp_relay ipv4 counter
Fix get all interface offset issue

Work item tracking
Microsoft ADO (17271822):
How I did it
show dhcp_relay ipv4 counter -i [ifname]
show dhcp4relay_counters counts -i [ifname]

How to verify it
show dhcp4relay_counters counts | more 10
Message Type Ethernet144(RX)
2023-09-11 21:08:06 +08:00
ganglv
666879b867
Upgrade gnxi to support dash (#16498)
### Why I did it
Need new gnmi client for dash test.

### How I did it
I have updated gnxi repo, and this PR is used to get latest change.

#### How to verify it
Run end2end test for DASH.
2023-09-08 08:56:51 -07:00
judyjoseph
7d2e3cb011
Update macsec CAK keys in profile for tests to change to type7 encoded format (#16388)
* Change the CAK key length check in config plugin, macsec test profile changes

* Fix the format in add_profile api

The changes needed in various macsec unit tests and config plugin when we move to accept the type 7 encoded key format for macsec. This goes along with PR : sonic-net/sonic-swss#2892 raised earlier.
2023-09-06 21:11:02 -07:00
Konstantin Vasin
1e7db2ab01
[build]: Don't build ethtool from source (#15856)
Why I did it
There is no reason to build deb package ethtool from source code.
We can install the same version from Debian bullseye mirror.

How I did it
Remove ethtool Makefiles from sonic-buildimage.
Install ethtool via apt-get in pmon container.
2023-09-05 23:42:34 -07:00
jcaiMR
a522a63e25
[dhcp-relay]: dhcp/dhcpv6 per interface counter support (#16377)
Why I did it
Support DHCP/DHCPv6 per-interface counter, code change in sonic-build image.

Work item tracking
Microsoft ADO (17271822):

How I did it
- Introduce libjsoncpp-dev in dhcpmon and dhcprelay repo
- Show CLI changes after counter format change

How to verify it
- Manually run show command
- dhcpmon, dhcprelay integration tests
2023-09-05 10:16:39 -07:00
Mai Bui
ff5f46955c
[database] make Redis process runs as non-root user (#16326)
Why I did it
Running the Redis server as the "root" user is not recommended. It is suggested that the server should be operated by a non-privileged user.

Work item tracking
Microsoft ADO (number only): 15895240

How I did it
Ensure the Redis process is operating under the 'redis' user in supervisord and make redis user own REDIS_DIR inside db container.

How to verify it
Built new image, verify redis process is running as 'redis' user and all containers are up.

Signed-off-by: Mai Bui <maibui@microsoft.com>
2023-09-01 23:03:15 -07:00
Riff
7c1d720a65
[sonic-mgmt]: Adding sshconf 0.2.5 into sonic-mgmt container. (#16344)
Why I did it
This change is to help us running SSH config generation for our testbed in mgmt container.

Original PR in sonic-mgmt repo can be found here: sonic-net/sonic-mgmt#9773.

Work item tracking
Microsoft ADO (number only): 25007799

How I did it
Updating sonic-mgmt docker file to add sshconf 0.2.5 into pip install under venv.
2023-09-01 22:58:27 -07:00
guangyao6
80ce957d20
Add no-export to sentinel community-list (#16285)
Why I did it
Add no-export to bgp sentinel community-list. So that bgp updates from bgp sentinel service must match sentinel community and no-export, otherwise, the bgp update will be dropped.

Work item tracking
Microsoft ADO (24946274):
How I did it
Add no-export to bgp sentinel community-list.

How to verify it
Run UT, case would pass. Build the image and start the device. Add bgp sentinel and check that no-export community exist in bgp sentinel community list.
2023-08-29 09:12:19 +08:00
Zhaohui Sun
286ec3edbf
Change orchagent pop batch size from 8192 to 1024 (#16125)
### Why I did it
Background running lua script may cause redis-server quite busy if batch size is 8192.
If handling time exceeded default 5s, the redis-server will not response to other process and will cause syncd crash.

```
Aug  9 07:46:29.512326 str-s6100-acs-5 INFO database#supervisord: redis 68:M 09 Aug 2023 07:46:29.511 # Lua slow script detected: still in execution after 5186 milliseconds. You can try killing the script using the SCRIPT KILL command. Script SHA1 is: 88270a7c5c90583e56425aca8af8a4b8c39fe757
Aug  9 07:46:29.523716 str-s6100-acs-5 ERR syncd#syncd: :- checkReplyType: Expected to get redis type 5 got type 6, err: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
Aug  9 07:46:29.524818 str-s6100-acs-5 INFO syncd#supervisord: syncd terminate called after throwing an instance of '
Aug  9 07:46:29.525268 str-s6100-acs-5 ERR pmon#CCmisApi: :- checkReplyType: Expected to get redis type 5 got type 6, err: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
Aug  9 07:46:29.526148 str-s6100-acs-5 INFO syncd#supervisord: syncd std::system_error'
Aug  9 07:46:29.528308 str-s6100-acs-5 ERR pmon#psud[32]: :- checkReplyType: Expected to get redis type 5 got type 6, err: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
Aug  9 07:46:29.529048 str-s6100-acs-5 ERR lldp#python3: :- guard: RedisReply catches system_error: command: *2#015#012$3#015#012DEL#015#012$27#015#012LLDP_ENTRY_TABLE:Ethernet37#015#012, reason: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.: Input/output error
Aug  9 07:46:29.529720 str-s6100-acs-5 ERR snmp#python3: :- guard: RedisReply catches system_error: command: *2#015#012$7#015#012HGETALL#015#012$28#015#012COUNTERS:oid:0x100000000000a#015#012, reason: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.: Input/output error
```

88270a7c5c90583e56425aca8af8a4b8c39fe757 is /usr/share/swss/consumer_state_table_pops.lua
##### Work item tracking
- Microsoft ADO **24741990**:

#### How I did it
Change batch size from 8192 to1024.
#### How to verify it
Run all test cases in sonic-mgmt to verify the system stability.

### Tested branch (Please provide the tested image version)

- [x] 20220531.36
2023-08-14 17:49:49 -07:00
Ze Gan
96757a335c
Remove temporary files and import dash_api to python3 env (#16033)
1. Remove useless temporary protobuf deb packages
2. Import dash_api to python3 env

### Why I did it
1. There are some temporary Debian packages,protobuf packages, needs to be deleted
2. The dash-api was installed in the system folder that cannot be imported by the virtual python3 environment. But the testcases of DASH in sonic-mgmt are executed in virtual python3 environment.

##### Work item tracking
- Microsoft ADO **(number only)**: 17417902

#### How I did it
1. Add missed `&&` so that all protobuf debian packaged can be downloaded to the /tmp folder
2. Add ` --system-site-packages ` to env-python so that the system library can be accessed by virtual environment

#### How to verify it
Check the dash_api can be imported in env-python3
```
AzDevOps@46a900cf8477:~$ source env-python3/bin/activate
(env-python3) zegan@46a900cf8477:~$ ls
bin  env-python3
(env-python3) zegan@46a900cf8477:~$ python3
Python 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dash_api
>>>

```
2023-08-08 21:04:54 -07:00
SuvarnaMeenakshi
803c71c86a
[SNMP][IPv6]: Fix to use link local IPv6 address as snmp agentAddress (#16013)
<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it
fixes: https://github.com/sonic-net/sonic-buildimage/issues/16001
Caused by: https://github.com/sonic-net/sonic-buildimage/pull/15487

The above PR introduced change to use Management and Loopback Ipv4 and ipv6 addresses as snmpagent address in snmpd.conf file.
With this change, if Link local IP address is configured as management or Loopback IPv6 address, then snmpd tries to open socket on that ipv6 address and fails with the below error:
```
Error opening specified endpoint "udp6:[fe80::5054:ff:fe6f:16f0]:161"
Server Exiting with code 1
```
From RFC4007, if we need to specify non-global ipv6 address without ambiguity, we need to use zone id along with the ipv6 address: <address>%<zone_id>
Reference: https://datatracker.ietf.org/doc/html/rfc4007

##### Work item tracking
- Microsoft ADO **(number only)**:

#### How I did it
Modify snmpd.conf file to use the %zone_id representation for ipv6 address.
#### How to verify it
In VS testbed, modify config_db to use link local ipv6 address as management address:
    "MGMT_INTERFACE": {
        "eth0|10.250.0.101/24": {
            "forced_mgmt_routes": [
                "172.17.0.1/24"
            ],
            "gwaddr": "10.250.0.1"
        },
        "eth0|fe80::5054:ff:fe6f:16f0/64": {
            "gwaddr": "fe80::1"
        }
    },

Execute config_reload after the above change.
snmpd comes up and check if snmpd is listening on ipv4 and ipv6 addresses:
```
admin@vlab-01:~$ sudo netstat -tulnp | grep 161
tcp        0      0 127.0.0.1:3161          0.0.0.0:*               LISTEN      274060/snmpd        
udp        0      0 10.1.0.32:161           0.0.0.0:*                           274060/snmpd        
udp        0      0 10.250.0.101:161        0.0.0.0:*                           274060/snmpd        
udp6       0      0 fc00:1::32:161          :::*                                274060/snmpd        
udp6       0      0 fe80::5054:ff:fe6f::161 :::*                                274060/snmpd      -- Link local 
 
admin@vlab-01:~$ sudo ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.250.0.101  netmask 255.255.255.0  broadcast 10.250.0.255
        inet6 fe80::5054:ff:fe6f:16f0  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:6f:16:f0  txqueuelen 1000  (Ethernet)
        RX packets 36384  bytes 22878123 (21.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 261265  bytes 46585948 (44.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

admin@vlab-01:~$ docker exec -it snmp snmpget -v2c -c public fe80::5054:ff:fe6f:16f0 1.3.6.1.2.1.1.1.0
iso.3.6.1.2.1.1.1.0 = STRING: "SONiC Software Version: SONiC.master.327516-04a6031b2 - HwSku: Force10-S6000 - Distribution: Debian 11.7 - Kernel: 5.10.0-18-2-amd64"
```
Logs from snmpd:
```
Turning on AgentX master support.
NET-SNMP version 5.9
Connection from UDP/IPv6: [fe80::5054:ff:fe6f:16f0%eth0]:44308
```
Ran test_snmp_loopback test to check if loopback ipv4 and ipv6 works:
```
./run_tests.sh -n vms-kvm-t0 -d vlab-01 -c snmp/test_snmp_loopback.py  -f vtestbed.yaml -i ../ansible/veos_vtb -e "--skip_sanity --disable_loganalyzer" -u
=== Running tests in groups ===
Running: pytest snmp/test_snmp_loopback.py --inventory ../ansible/veos_vtb --host-pattern vlab-01 --testbed vms-kvm-t0 --testbed_file vtestbed.yaml --log-cli-level warning --log-file-level debug --kube_master unset --showlocals --assert plain --show-capture no -rav --allow_recover --ignore=ptftests --ignore=acstests --ignore=saitests --ignore=scripts --ignore=k8s --ignore=sai_qualify --junit-xml=logs/tr.xml --log-file=logs/test.log --skip_sanity --disable_loganalyzer
..                                                                        

snmp/test_snmp_loopback.py::test_snmp_loopback[vlab-01] PASSED 
```
<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [x] 202012
- [x] 202106
- [x] 202111
- [x] 202205
- [x] 202211
- [x] 202305

#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [ ] <!-- image version 1 -->
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
2023-08-08 14:47:33 -07:00
abdosi
c6d1dae741
Fix the Loopback0 IPv6 address of LC's in chassis not reachable from (#16026)
What I did:
Fix the Loopback0 IPv6 address of LC's in chassis not reachable from peer devices.

Why I did:
For Ipv6 Loopback0 address we only advertise /64 subnet to the peer devices. However, in case of chassis each LC will have it own /128 address of that /64 subnet . Since this /128 address does not get advertised peer devices can-not ping/reach the LC's loopback0.

How I fix:
Advertise /128 Loopback0 Ipv6 address only between i-BGP peers. This way even though /64 is advertised to e-BGP peer devices when packet reaches any of LC's it can reach the appropriate LC's.

How I verify:
Manual verification
UT added for same.

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
2023-08-06 22:36:33 -07:00
guangyao6
9567c06570
Add BGP configuration for BGPSentinel peer (#15714)
Why I did it
For route registry service, in order to block hijacked routes, IBGP session needs to be set up from BGP sentinel service to SONiC, and BGP sentinel service advertise the same route with higher local-preference and no export community. So that SONiC takes the route from BGP sentinel as the best path and does not advertise the route to EBGP peers.
In order to do that, new route-maps are needed. So this change adds a new set of templates, keeping BGPSentinel peers out of the other templates.

Work item tracking
Microsoft ADO (number only): 24451346
How I did it
Add sentinel_community in constants.yml, route from BGPSentinel do not match this community will be denied.
Add support to convert BGPSentinel related configuration in the BGPPeerPassive element of the minigraph to a new BGP_SENTINELS table in CONFIG_DB
Add a new set of "sentinels" templates to docker-fpm-frr
Add a new BGP peer manager to bgpcfgd, to add neighbors from the BGP_SENTINELS table using the "sentinels" templates
Add a test case for minigraph.py, making sure the BGPSentinel and BGPSentinelV6 elements create BGP_SENTINELS DB entry.
Add a set of test cases for the new sentinels templates in sonic-bgpcfgd tests.
Add sonic-bgp-sentinel.yang and a set of testcases for the yang file.

How to verify it
Testcases and UT newly added would pass.
Setup IPv4 and IPv6 BGPSentinel services in minigraph, and load minigraph, show CONFIG_DB and "show runningconfig bgp", configuration would be loaded successfully.
Using t1-lag topo and setup IBGP session from BGPSentinel to SONiC loopback address, IBGP session would up.
Advertise route from BGPSentinel to T1 with sentinel_community, higher local-preference and no-export communiyt. In T1, show bgp route, the result is "Not advertise to any EBGP peer".
Withdraw the route in BGPSentinel, in T1, route would advertise to EBGP peers.
Advertise route from T1 that does not match sentinel_community, in T1, would not see the route in show bgp route.
2023-07-21 09:32:29 +08:00
Ye Jianquan
7533c8ccf6
[sonic-mgmt docker image] Upgrade celery in the python3 to 5.2.7, upgrade ipython to 8.12.2 (#15911)
Upgrade celery in the python3 to 5.2.7,
Upgrade ipython to 8.12.2 since 5.4.1 requires prompt-toolkit<2.0.0,>=1.0.4,
But celery 5.2.7 relies click-repl>=0.2.0 , click-repl>=0.2.0 relies prompt-toolkit>=3.0.36.
So upgrade ipython to resolve the prompt-toolkit version incompatible issue.
2023-07-20 14:28:08 +08:00
xumia
bdef73ea96
[Build] Fix the PyYang python package installation issue (#15890)
Why I did it
Fix the armhf build failure.
How to reproduce the issue:

docker run -it debain:bullseye bash
apt-get update && apt-get install -y python3-pip
pip3 install PyYAML==5.4.1
Error message:

Collecting PyYAML==5.4.1
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /tmp/tmp6xabslgb_in_process.py get_requires_for_build_wheel /tmp/tmp_er01ztl
....
      raise AttributeError(attr)
  AttributeError: cython_sources
  ----------------------------------------
WARNING: Discarding d63f2d7597/PyYAML-5.4.1.tar.gz (sha256)=607774cbba28732bfa802b54baa7484215f530991055bb562efbed5b2f20a45e (from https://pypi.org/simple/pyyaml/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*). Command errored out with exit status 1: /usr/bin/python3 /tmp/tmp6xabslgb_in_process.py get_requires_for_build_wheel /tmp/tmp_er01ztl Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement PyYAML==5.4.1
ERROR: No matching distribution found for PyYAML==5.4.1
root@fa2fa92edcfd:/# 
But if adding the option --no-build-isolation, then it is good, see fix.

install "PyYAML==5.4.1" --no-build-isolation
The same error can be found in the multiple builds.

Work item tracking
Microsoft ADO (number only): 24567457

How I did it
Add a build option --no-build-isolation.
2023-07-19 06:33:49 +08:00
jcaiMR
bd413d20d2
advance dhcprelay to 6a6ce24, add default dhcpv6 dualtor source interface (#15864)
sonic-build image side change to fix source interface selection in dual tor scenario.
dhcprelay related PR:
[master]fix dhcpv6 relay dual tor source interface selection issue sonic-dhcp-relay#42

Announce dhcprelay submodule to 6a6ce24([to invoke #40 PR]([master]fix dhcpv6 relay dual tor source interface selection issue sonic-dhcp-relay#42))
2023-07-17 15:28:10 -07:00