Commit Graph

731 Commits

Author SHA1 Message Date
shlomibitton
6762f526d9
[NVMe] Add NVMe SSD disc type support to installer.sh script (#6142)
In order to install a SONiC image on top of a NVMe SSD disc properly with ONIE we must configure it properly on the installer.sh script.

Signed-off-by: Shlomi Bitton <shlomibi@nvidia.com>
2020-12-09 19:03:27 -08:00
Samuel Angebault
44f4c2ed66
[Arista] Update driver submodules (#6151)
- Enhance eeprom parsing robustness on corrupted fields
 - Add chassis provisioning service
 - Disable CPU sleep state on some systems
 - Complete refactor for FanSlots
 - Fix module unload while still in use
2020-12-08 11:17:28 -08:00
abdosi
59c1e3a78a
[multi-asic] Enhancing monit process checker for multi-asic. (#6100)
Added Support of process checker for work on multi-asic platforms.
2020-12-04 10:39:43 -08:00
Samuel Angebault
468aac92b7
[Arista] Update platform configurations for 7060DX4 and 7060PX4 (#6084)
Current support for the 7060PX4-32 and 7060DX4 was broken.
With this change, ports are now linking fine.

Co-authored-by: Zhi Yuan Carl Zhao <zyzhao@arista.com>
2020-12-04 10:11:06 -08:00
Samuel Angebault
8576911a57
[database-chassis]: Fix the way database-chassis start (#6099)
The service crash when the platform boots due to missing waits.
/usr/bin/database.sh tries to operate on a missing socket and fails.
We now wait for the chassis database to be ready the same way we do database.
2020-12-04 10:09:35 -08:00
Prabhu Sreenivasan
2895b79482
[ntp]: NTP service ordering (#6115)
Make sure ntp-config service is executed before ntpd

Updated ntp-config service files to force dependency with ntp service. Also resolved circular dependency with --no-block flag. (needed as ntp-config service internally invokes systemd to restart ntp which in turn waits for ntp-config to complete)

Signed-off-by: Prabhu Sreenivasan <prabhu.sreenivasan@broadcom.com>
2020-12-04 08:49:20 -08:00
Joe LeVeque
83f0d8240e
[pmon]: Install vanilla 'thrift' Python 2 and 3 packages for Barefoot in host and PMon (#6080)
Barefoot platform vendors' sonic_platform packages import the Python 'thrift' library. Previously, our custom-built package was being installed in the PMon container and host OS. However, we are only building a Python 2 version of that package, which was only intended for use with saithrift.

Fixes #6077
2020-12-04 08:41:17 -08:00
Joe LeVeque
905a5127bb
[Python] Align files in root dir, dockers/ and files/ with PEP8 standards (#6109)
**- Why I did it**

Align style with slightly modified PEP8 standards (extend maximum line length to 120 chars). This will also help in the transition to Python 3, where it is more strict about whitespace, plus it helps unify style among the SONiC codebase. Will tackle other directories in separate PRs.

**- How I did it**

Using `autopep8 --in-place --max-line-length 120` and some manual tweaks.
2020-12-03 15:57:50 -08:00
Sabareesh-Kumar-Anandan
fe524c37e7
[platform][marvell] Arm 32-bit Arch support changes (#5749)
- Added Arm 32-bit arch build fixes
- Added marvell armhf platform specific changes

Signed-off-by: Sabareesh Kumar Anandan <sanandan@marvell.com>
2020-12-03 12:38:50 -08:00
Garrick He
fc0e6af337
[sflow] Fix race-condition seen with mVRF configured (#6102)
Under certain conditions, the sFlow service can start before
interface configurations are sucessfully applied. This will
cause hsflowd to get a socket error.

This fix ensures all interface configurations are successfully
applied before the sFlow service (hsflowd) starts.

During testing we saw this error from hsflowd if interface configs were not successfully applied before hsflowd started.

    ERR sflow#hsflowd: socket sendto error: Network is unreachable

no FLOW samples can be seen. This can be consistently reproducible if you force sFlow service to start before interface-config.service.

Signed-off-by: Garrick He <garrick_he@dell.com>
2020-12-03 01:33:10 -08:00
lguohan
4812953468
[ntp]: build ntp with various fixes (#6037)
- NTP Bug 1970 (UNLINK_EXPR_SLIST empty list) Fix
- ENOBUFS log message level set to WARN
- Fix audit message seen on console apparmor
- add force-confold option when install ntp

Signed-off-by: Guohan Lu <lguohan@gmail.com>
Co-authored-by: Prabhu Sreenivasan <prabhu.sreenivasan@broadcom>
2020-12-02 15:02:50 -08:00
Joe LeVeque
7f4ab8fbd8
[sonic-utilities] Update submodule; Build and install as a Python 3 wheel (#5926)
Submodule updates include the following commits:

* src/sonic-utilities 9dc58ea...f9eb739 (18):
  > Remove unnecessary calls to str.encode() now that the package is Python 3; Fix deprecation warning (#1260)
  > [generate_dump] Ignoring file/directory not found Errors (#1201)
  > Fixed porstat rate and util issues (#1140)
  > fix error: interface counters is mismatch after warm-reboot (#1099)
  > Remove unnecessary calls to str.decode() now that the package is Python 3 (#1255)
  > [acl-loader] Make list sorting compliant with Python 3 (#1257)
  > Replace hard-coded fast-reboot with variable. And some typo corrections (#1254)
  > [configlet][portconfig] Remove calls to dict.has_key() which is not available in Python 3 (#1247)
  > Remove unnecessary conversions to list() and calls to dict.keys() (#1243)
  > Clean up LGTM alerts (#1239)
  > Add 'requests' as install dependency in setup.py (#1240)
  > Convert to Python 3 (#1128)
  > Fix mock SonicV2Connector in python3: use decode_responses mode so caller code will be the same as python2 (#1238)
  > [tests] Do not trim from PATH if we did not append to it; Clean up/fix shebangs in scripts (#1233)
  > Updates to bgp config and show commands with BGP_INTERNAL_NEIGHBOR table (#1224)
  > [cli]: NAT show commands newline issue after migrated to Python3 (#1204)
  > [doc]: Update Command-Reference.md (#1231)
  > Added 'import sys' in feature.py file (#1232)

* src/sonic-py-swsssdk 9d9f0c6...1664be9 (2):
  > Fix: no need to decode() after redis client scan, so it will work for both python2 and python3 (#96)
  > FieldValueMap `contains`(`in`)  will also work when migrated to libswsscommon(C++ with SWIG wrapper) (#94)

- Also fix Python 3-related issues:
    - Use integer (floor) division in config_samples.py (sonic-config-engine)
    - Replace print statement with print function in eeprom.py plugin for x86_64-kvm_x86_64-r0 platform
    - Update all platform plugins to be compatible with both Python 2 and Python 3
    - Remove shebangs from plugins files which are not intended to be executable
    - Replace tabs with spaces in Python plugin files and fix alignment, because Python 3 is more strict
    - Remove trailing whitespace from plugins files
2020-11-25 10:28:36 -08:00
abdosi
fad481edc1
Enhanced Feature table to support 'always_enabled' value for state and auto-restart fields. (#6000)
Added new flag value 'always_enabled' for the state and auto-restart field of feature table

init_cfg.json is updated to initialize state field of database/swss/syncd/teamd feature and auto-restart field of database feature
as always_enabled

Once the state/auto-restart value is initialized as "always_enabled" it is immutable and cannot be change via feature config commands. (config feature..) PR#Azure/sonic-utilities#1271

hostcfgd will not take any action if state field value is 'always_enabled'

Since we have always_enabled field for auto-restart updated supervisor-proc-exit-listener
not to have special check for database and always rely on value from Feature table.
2020-11-25 08:41:11 -08:00
Blueve
6a6e583b06
[bash.bashrc] Add reverse SSH script to bash.bashrc (#5438)
* [bash.bashrc] Add reverse SSH script to bash.bashrc
* Fix command issue and add emptt line before EOF
* Add checks for SSH_TARGET_CONSOLE_LINE

Signed-off-by: Jing Kan jika@microsoft.com
2020-11-24 14:11:53 +08:00
Sudharsan Dhamal Gopalarathnam
98a434e8c1
Copp Manager Changes (#4861)
*Introduce CoPP Manager infrastructure
Copp service to generate initial copp config template file

Co-authored-by: dgsudharsan <sudharsan_gopalarat@dell.com>
2020-11-23 09:31:42 -08:00
Sujin Kang
5b31996f7b
[reboot-history] Add reboot history to state db (#5933)
- Why I did it
Add reboot history to State db so that can be used telemetry service
- How I did it
Split the process-reboot-cause service to determine-reboot-cause and process-reboot-cause
determine-reboot-cause to determine the reboot cause
process-reboot-cause to parse the reboot cause files and put the reboot history to state db
Moved to sonic-host-service* packages
- How to verify it
Performed unit test and tested on DUT
2020-11-20 20:08:18 -08:00
Joe LeVeque
23247514f9
Fix a number of LGTM alerts (#5952)
Fix 259 alerts reported by the LGTM tool:

- 245 for Unused import
- 7 for Testing equality to None
- 5 for Duplicate key in dict literal
- 1 for Module is imported more than once
- 1 for Unused local variable
2020-11-20 10:58:48 -08:00
JiangboHe
461e43649b
fix error: interface counters is mismatch after warm-reboot (#5346)
- Why I did it
There is a issue for counters after warm-reboot:
If I clear counters by command "sonic-clear counters", then execute 'warm-reboot' and whenSONiC is restart, the counters showed with command "show interface counters" is still old counters before "sonic-clear". It is not the right counters because the counters file in '/tmp' is lost in warm-reboot process.

- How I did it
I fixed it by saving '/tmp/portstat-0' folders in '/host/' before executing 'warm-reboot' (in pull request Azure/sonic-utilities#1099 ), and restore the counters folders back to '/tmp/' after warm-reboot process is finished.

- How to verify it

Clear counters by command 'sonic-clear'
sonic-clear counters
sonic-clear dropcounters
sonic-clear pfccounters
sonic-clear queuecounters
sonic-clear rifcounters
Execute 'warm-reboot'
Use command ‘show interface counters’ to see if the counters is right.
2020-11-20 10:37:45 -08:00
Joe LeVeque
7bf05f7f4f
[supervisor] Install vanilla package once again, install Python 3 version in Buster container (#5546)
**- Why I did it**

We were building a custom version of Supervisor because I had added patches to prevent hangs and crashes if the system clock ever rolled backward. Those changes were merged into the upstream Supervisor repo as of version 3.4.0 (http://supervisord.org/changes.html#id9), therefore, we should be able to simply install the vanilla package via pip. This will also allow us to easily move to Python 3, as Python 3 support was added in version 4.0.0.

**- How I did it**

- Remove Makefiles and patches for building supervisor package from source
- Install Python 3 supervisor package version 4.2.1 in Buster base container
    - Also install Python 3 version of supervisord-dependent-startup in Buster base container
- Debian package installed binary in `/usr/bin/`, but pip package installs in `/usr/local/bin/`, so rather than update all absolute paths, I changed all references to simply call `supervisord` and let the system PATH find the executable to prevent future need for changes just in case we ever need to switch back to build a Debian package, then we won't need to modify these again.
- Install Python 2 supervisor package >= 3.4.0 in Stretch and Jessie base containers
2020-11-19 23:41:32 -08:00
pavel-shirshov
a92732fe5d
[bgpcfgd]: Fixes for BBR (#5956)
* Add explicit default state into the constants.yml
* Enable/disable only peer-groups, available in the config
* Retrieve updates from frr before using configuration

Co-authored-by: Pavel Shirshov <pavel.contrib@gmail.com>
2020-11-19 00:07:58 -08:00
heidinet2007
7c17c58b83
Move teamd warm reboot code to service script (#5163)
Summary: Move teamd functions to a new service script

Motivation: To segregate teamd functions in one common place. fast-reboot script calls teamd functions that should ideally be replaced by a simple call to a service script.
 
Changes: New teamd service script and path modification from /usr/bin/teamd.sh to /usr/local/bin/teamd.sh
fast-reboot script (in sonic-utilities) modification (to use new teamd.sh to stop teamd) should follow soon after this change.

Verification: VS image tests.

Signed-off-by: Vaibhav Hemant Dixit <vaibhav.dixit@microsoft.com>

Co-authored-by: heidi.ou@alibaba-inc.com <heidi.ou@alibaba-inc.com>
Co-authored-by: Ying Xie <ying.xie@microsoft.com>
2020-11-13 13:34:18 -08:00
fk410167
a3dd3f55f9
Platform Driver Developement Framework (PDDF) (#4756)
This change introduces PDDF which is described here: https://github.com/Azure/SONiC/pull/536

Most of the platform bring up effort goes in developing the platform device drivers, SONiC platform APIs and validating them. Typically each platform vendor writes their own drivers and platform APIs which is very tailor made to that platform. This involves writing code, building, installing it on the target platform devices and testing. Many of the details of the platform are hard coded into these drivers, from the HW spec. They go through this cycle repetitively till everything works fine, and is validated before upstreaming the code.
PDDF aims to make this platform driver and platform APIs development process much simpler by providing a data driven development framework. This is enabled by:

JSON descriptor files for platform data
Generic data-driven drivers for various devices
Generic SONiC platform APIs
Vendor specific extensions for customisation and extensibility

Signed-off-by: Fuzail Khan <fuzail.khan@broadcom.com>
2020-11-12 10:22:38 -08:00
Lawrence Lee
ae69fdf312
[buffers_config.j2]: Use correct cable lengths for backend devices (#5905)
* Remove 'backend' from device type strings so that backend devices ('BackEndToRRouter' and 'BackEndLeafRouter') are given the same cable lengths as regular device types.

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
2020-11-12 09:03:59 -08:00
Lawrence Lee
d0f16c0d79
Make backend device checking more robust (#5730)
Treat devices that are ToRRouters (ToRRouters and BackEndToRRouters) the same when rendering templates
 Except for BackEndToRRouters belonging to a storage cluster, since these devices have extra sub-interfaces created
Treat devices that are LeafRouters (LeafRouters and BackEndLeafRouters) the same when rendering templates

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
2020-11-10 15:06:35 -08:00
Prince Sunny
1eaaf64ed2
Set preference for forced mgmt routes (#5844)
When forced mgmt routes are present, the issue fixed as part of #5754 is not complete. 
Added a preference(priority) field to forced mgmt route ip rules
2020-11-10 14:20:13 -08:00
arlakshm
2b41f6bd5c
Add the vtysh command with newly added "-n" option for multi asic to the read_only_cmds (#5845)
In multi asic platforms the "show ip bgp summary" commands is not available for user with read only privileges, so to fix this the vtysh command with the new "-n" option, added for multi asic platforms, needs to be added to the READ_ONLY_COMMANDS list in the sudoers files. Added the command vtysh -n [0-9] -c show * to list of READ_ONLY_COMMANDS in the sudoers files in this commit.

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>
2020-11-10 12:18:49 -08:00
abdosi
4f82463670
[multi-asic] Fixed the docker mount point check for multi-asic (#5848)
API getMount() API was not updated to handle multi-asic platforms
Updated API getMount() to return abspath() for Docker Mount Point
and use that one for mount point comparison

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
2020-11-09 13:03:00 -08:00
Joe LeVeque
e0fdf45ad0
[update_chassisdb_config] Convert to Python 3 (#5838)
- Convert update_chassisdb_config script to Python 3
- Reorganize imports per PEP8 standard
- Two blank lines precede functions per PEP8 standard
2020-11-09 08:35:36 -08:00
Guohan Lu
ad2e18e856 [baseimage]: install psutil for python3
psutil is needed by process_checker which is using python3

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-11-09 00:29:10 -08:00
Praveen Chaudhary
6156cb2805
[sonic-yang-mgmt] Build PY3 & PY2 packages (#5559)
Moving sonic-yang-mgmt to PY3 to support move of sonic-utilities to PY3.

Signed-off-by: Praveen Chaudhary<pchaudhary@linkedin.com>
2020-11-07 13:03:41 -08:00
Joe LeVeque
04d0e8ab00
[hostcfgd] Convert to Python 3; Add to sonic-host-services package (#5713)
To consolidate host services and install via packages instead of file-by-file, also as part of migrating all of SONiC to Python 3, as Python 2 is no longer supported.
2020-11-07 12:48:19 -08:00
Joe LeVeque
9e7e092610
[Monit process_checker] Convert to Python 3 (#5836)
Convert process_checker script to Python 3
2020-11-07 12:46:23 -08:00
lguohan
e6796da141
[init_cfg.json.j2]: only enable gbsyncd feature for vs platform (#5815)
currently only vs platform has gdbsyncd feature built

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-11-07 00:46:18 -08:00
Stepan Blyshchak
9bc693ce6e
[hostcfgd] If feature state entry not in the cache, add a default state (#5777)
Our use case is to register new features in runtime. The previous change which introduced the cache broke this capability and caused hostcfgd crash.

Signed-off-by: Stepan Blyshchak <stepanb@nvidia.com>
2020-11-06 10:24:31 -08:00
Joe LeVeque
13ff7b38d5
[docker-wait-any] Convert to Python 3, install dependency in host OS (#5784)
- Convert docker-wait-any script to Python 3
- Install Python 3 Docker Engine API in host OS
2020-11-05 11:23:00 -08:00
Joe LeVeque
d8045987a6
[core_uploader.py] Convert to Python 3; Use logger from sonic-py-common for uniform logging (#5790)
- Convert core_uploader.py script to Python 3
- Use logger from sonic-py-common for uniform logging
- Reorganize imports alphabetically per PEP8 standard
- Two blank lines precede functions per PEP8 standard
- Remove unnecessary global variable declarations
2020-11-05 11:19:26 -08:00
Joe LeVeque
522a071ffb
[core_cleanup.py] Convert to Python 3; Fix bug; Improve code reuse (#5781)
- Convert to Python 3
- Fix bug: `CORE_FILE_DIR` previously was set to `os.path.basename(__file__)`, which would resolve to the script name. Fix this by hardcoding to `/var/core/` instead
- Remove locally-define logging functions; use Logger class from sonic-py-common instead
2020-11-05 10:01:12 -08:00
Joe LeVeque
d3262d10f7
[generate_asic_config_checksum.py] Convert to Python 3 (#5783)
- Convert script to Python 3
    - Need to open file in binary mode before hashing due to new string data type in Python 3 being unicode by default. This should probably have been done regardless.
- Reorganize imports alphabetically
- When running the script, don't explicitly call `python`. Instead let the program loader use the interpreter specified in the shebang (which is now `python3`).
2020-11-04 15:06:44 -08:00
Lawrence Lee
10ab46f7a0
Revert "[docker-base]: Rate limit priority INFO and lower in syslog" (#5763)
* This was a temporary fix for orchagent spamming log messages and causing rate limiting, leading to critical messages being dropped for the syslog. No longer needed since Azure/sonic-sairedis#680 was merged.
2020-11-02 08:49:40 -08:00
Blueve
698b5544c9
[openssh] Introduce custom openssh-server package for supporting reverse console SSH (#5717)
* Build and install openssh from source
* Copy openssh deb package to dest folder
* Update make rule
* Update sonic debian extension
* Append empty line before EOF
* Update openssh patch
* Add openssh-server to base image dependency
* Fix indent type
* Fix comments
* Use commit id instead of tag id and add comment

Signed-off-by: Jing Kan jika@microsoft.com
2020-11-02 10:31:15 +08:00
lguohan
c8a00eda95
[mgmt ip]: mvrf ip rule priority change to 32765 (#5754)
Fix Azure/SONiC#551

When eth0 IP address is configured, an ip rule is getting added for eth0 IP address through the interfaces.j2 template. 

This eth0 ip rule creates an issue when VRF (data VRF or management VRF) is also created in the system.
When any VRF (data VRF or management VRF) is created, a new rule is getting added automatically by kernel as "1000: from all lookup [l3mdev-table]".
This l3mdev IP rule is never getting deleted even if VRF is deleted.

Once if this l3mdev IP rule is added, if user configures IP address for the eth0 interface, interfaces.j2 adds an eth0 IP rule as "1000:from 100.104.47.74 lookup default ". Priority 1000 is automatically chosen by kernel and hence this rule gets higher priority than the already existing rule "1001:from all lookup local ".

This results in an issue "ping from console to eth0 IP does not work once if VRF is created" as explained in Issue 551.
More details and possible solutions are explained as comments in the Issue551.

This PR is to resolve the issue by always fixing the low priority 32765 for the IP rule that is created for the eth0 IP address.
Tested with various combinations of VRF creation, deletion and IP address configuration along with ping from console to eth0 IP address.

Co-authored-by: Kannan KVS <kannan_kvs@dell.com>
2020-10-31 20:45:59 -07:00
abdosi
dddf96933c
[monit] Adding patch to enhance syslog error message generation for monit alert action when status is failed. (#5720)
Why/How I did:

Make sure first error syslog is triggered based on FAULT TOLERANCE condition.

Added support of repeat clause with alert action. This is used as trigger
for generation of periodic syslog error messages if error is persistent

Updated the monit conf files with repeat every x cycles for the alert action
2020-10-31 17:29:49 -07:00
Renuka Manavalan
8d8aadb615
Load config after subscribe (#5740)
- Why I did it
The update_all_feature_states can run in the range of 20+ seconds to one minute. With load of AAA & Tacacs preceding it, any DB updates in AAA/TACACS during the long running feature updates would get missed. To avoid, switch the order.

- How I did it
Do a load after after updating all feature states.

- How to verify it
Not a easy one
Have a script that
restart hostcfgd
sleep 2s
run redis-cli/config command to update AAA/TACACS table

Run the script above and watch the file /etc/pam.d/common-auth-sonic for a minute.

- When it repro:
The updates will not reflect in /etc/pam.d/common-auth-sonic
2020-10-31 16:38:32 -07:00
Joe LeVeque
6333bb73b0
Explicitly call pip2 rather than pip in locations where both pip2 and pip3 are installed (#5747)
As part of the transition from Python 2 to Python 3, we are installing both pip2 and pip3 in the slave and config-engine containers. This PR replaces calls to `pip` in these containers with an explicit call to `pip2` to ensure the proper version of pip is executed, no matter which version of pip is aliased to `pip`, as we no longer rely on that alias.

Also some other pip-related cleanup
2020-10-30 09:43:14 -07:00
Joe LeVeque
e111204206
[caclmgrd] Convert to Python 3; Add to sonic-host-services package (#5739)
To consolidate host services and install via packages instead of file-by-file, also as part of migrating all of SONiC to Python 3, as Python 2 is no longer supported, convert caclmgrd to Python 3 and add to sonic-host-services package
2020-10-29 16:29:12 -07:00
Shi Su
5ee5c13f32
Enable synchronous mode by default and add in minigraph parser (#5735) 2020-10-29 09:15:12 -07:00
judyjoseph
6088bd59de
[multi-ASIC] BGP internal neighbor table support (#5520)
* Initial commit for BGP internal neighbor table support.
  > Add new template named "internal" for the internal BGP sessions
  > Add a new table in database "BGP_INTERNAL_NEIGHBOR"
  > The internal BGP sessions will be stored in this new table "BGP_INTERNAL_NEIGHBOR"

* Changes in template generation tests with the introduction of internal neighbor template files.
2020-10-28 16:41:27 -07:00
lguohan
07748a939f
[gbsyncd]: add gbsyncd to FEATURE table (#5683)
remove syncd from critical process list because
gbsyncd process will exit for platform without
gearbox.

closes #5623

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-10-27 11:40:23 -07:00
bingwang-ms
36c52cca2b
Fix 'NoSuchProcess' exception in process_checker (#5716)
The psutil library used in process_checker create a cache for each
process when calling process_iter. So, there is some possibility that
one process exists when calling process_iter, but not exists when
calling cmdline, which will raise a NoSuchProcess exception. This commit
fix the issue.

Signed-off-by: bingwang <bingwang@microsoft.com>
2020-10-27 09:25:35 +08:00
Joe LeVeque
9e34003136
[sonic-config-engine] Clean up dependencies, pin versions; install Python 3 package in Buster container (#5656)
To clean up the image build procedure, and let setuptools/pip[3] implicitly install Python dependencies. Also use ipaddress package instead of ipaddr.
2020-10-26 13:48:50 -07:00