Fixes#11873.
#### Why I did it
When loading from minigraph, for port channels, don't create the members@ array in config_db in the PORTCHANNEL table. This is no longer needed or used.
In addition, when adding a port channel member from the CLI, that member doesn't get added into the members@ array, resulting in a bit of inconsistency. This gets rid of that inconsistency.
Signed-off-by: maipbui <maibui@microsoft.com>
#### Why I did it
`subprocess` is used with `shell=True`, which is very dangerous for shell injection.
#### How I did it
remove `shell=True`, use `shell=False`
#### How to verify it
Pass UT
changes:
-- yang model for mpls_tc_to_tc_map table.
-- tests.
#### Why I did it
yang model for mpls_tc_to_tc_map table.
#### How I did it
-- yang model for mpls_tc_to_tc_map table.
-- yang model tests.
#### How to verify it
-- yang model build time tests.
#### Why I did it
Added a tool to generate configuration based on yang model
#### How I did it
Parse the yang model and create documentation based on description field.
#### How to verify it
Added UT to verify it.
#### Why I did it
Fix issue: Non compliant leaf list in config_db schema: https://github.com/Azure/sonic-buildimage/issues/9801
#### How I did it
The basic flow of DPB is like:
1. Transfer config db json value to YANG json value, name it “yangIn”
2. Validate “yangIn” by libyang
3. Generate a YANG json value to represent the target configuration, name it “yangTarget”
4. Do diff between “yangIn” and “yangTarget”
5. Apply the diff to CONFIG DB json and save it back to DB
The fix:
• For step #1, If value of a leaf-list field string type, transfer it to a list by splitting it with “,” the purpose here is to make step#2 happy. We also need to save <table_name>.<key>.<field_name> to a set named “leaf_list_with_string_value_set”.
• For step#5, loop “leaf_list_with_string_value_set” and change those fields back to a string.
#### How to verify it
1. Manual test
2. Changed sample config DB and unit test passed
#### Why I did it
Need to pass LY_CTX_DISABLE_SEARCHDIR_CWD to Context in order to disable automatically searching for schemas in current working directory (which is by default searched automatically)
#### How I did it
add additional attribute into YANG context
#### How to verify it
Create some invalid link on switch :
1) **ln -s /usr/abc xxx**
2) run **spm list**
--> There should not be these messages:
```
libyang[1]: Unable to get information about "xxx" file in "/tmp" when searching for (sub)modules (No such file or directory)
libyang[1]: Unable to get information about "xxx" file in "/tmp" when searching for (sub)modules (No such file or directory)
libyang[1]: Unable to get information about "xxx" file in "/tmp" when searching for (sub)modules (No such file or directory)
libyang[1]: Unable to get information about "xxx" file in "/tmp" when searching for (sub)modules (No such file or directory)
```
#### Why I did it
Adding exceptlionList to validation exception
#### How I did it
Check code.
#### How to verify it
Ran manually.
- Run full config validation from a KVM
- Print the thrown exception
**Before**
```
Error: Data Loading Failed
All Keys are not parsed in FEATURE
dict_keys(['telemetry'])
```
**After**
```
Error: Data Loading Failed
All Keys are not parsed in FEATURE
dict_keys(['telemetry'])
exceptionList:["'status'"]
```
#### Which release branch to backport (provide reason below if selected)
<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->
- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/SONiC/wiki/Configuration.
-->
#### A picture of a cute animal (not mandatory but encouraged)
Why I did it
Fix#9746
How I did it
Split the check condition based on non-exist and zero length.
How to verify it
Run verification script when table contains empty value
#### Why I did it
If the grouping is in the same file in the module, fetching the grouping fails
#### How I did it
fixing the `uses_module_name` when the grouping is under the same file
#### How to verify it
Enabled a grouping under the same file which is `lldp`, there is a test in sonic-yang-mgmt that translates `sample-config-db` into yang format. This test passes with grouping in `lldp` used.
#### Which release branch to backport (provide reason below if selected)
<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->
- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
#### A picture of a cute animal (not mandatory but encouraged)
#### Why I did it
Fixes https://github.com/Azure/sonic-utilities/issues/1871
From [generic-config-updater](https://github.com/Azure/sonic-utilities/tree/master/generic_config_updater) we call `sonic-yang-mgmt` multiple times in order to check a certain change to ConfigDb is valid or not. It is expected for some changes to be invalid, so always printing errors from `sonic-yang-mgmt` makes the output hard to read.
In this PR, we are adding a way to control if logs should be printed or not.
#### How I did it
- Added `print_log_enabled` flag to sonic_yang ctor
- Converted all `print` statements to `sysLog(..., doPrint=True)`
#### How to verify it
unit-test passing means the change did not break logs.
#### Info about libyang logging
libyang provides an extensive logging logic which can support a lot of scenarios:
- ly_log_level: setting logging level
- LY_LLERR
- LY_LLWRN
- ...
- ly_set_log_clb: setting log callback to customize the default behavior which is printing the msgs
- ly_log_options: setting logging options
- LY_LOLOG: If callback is set use it, otherwise just print. If flag is not set, do nothing.
- ...
For more info refer to:
- https://netopeer.liberouter.org/doc/libyang/devel/html/group__logopts.html#gaff80501597ed76344a679be2b90a1d0a
- https://netopeer.liberouter.org/doc/libyang/devel/html/group__log.html#gac88b78694dfe9efe0450a69603f7eceb
#### What's next?
Consume the new flag `print_log_enabled` in [generic-config-updater](https://github.com/Azure/sonic-utilities/tree/master/generic_config_updater) to reduce the logging clutter.
#### Which release branch to backport (provide reason below if selected)
<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->
- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
#### A picture of a cute animal (not mandatory but encouraged)
Why I did it
Add YANG model support for table CABLE_LENGTH
How I did it
Add the YANG model file
Add the test description file and config file
add list CABLE_LENGTH_LIST to the qos_maps_model list in sonic-yang-ext, as it has an inner list.
How to verify it
Build sonic-yang-model and sonic-yang-mgmt
Previously the Yang model unit tests pulled directly from the Yang model source code for their models. This worked up to a point where we had jinja templates which needed to be rendered before they could be tested on. Now we need to reference the install location of the rendered templates for the unit tests.
Additionally, PBH yang was never added to the list of models to install onto a switch so I have resolved that (another example why pulling directly from the source was not correct)
My best explanation for why we only saw this failing on mellanox was that these tests could not have possibly been running on the broadcom build (I have not investigated this specifically). There was no way for these tests to pass recently given they were referencing Yang models which did not exist in the source (ACL).
Changes:
-- pre Process Grouping section from all yang models, so it can be used
from any yang model.
-- add jsondiff in setup.py, it is useful for test debugging in case of
failures.
-- use 'stypes' instead of head.
-- pass config DB table name in _createLeafDict().
-- added test config for grouping.
-- white spaces changes.
Note: Changes are done in the way that we can add support for other
Generic YANG statement easily for translation.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
Why I did it
SONiC YANG model support for BGP & route-map features.
How I did it
Defined various BGP and route-map YANG containers and lists based on config-DB schema.
How to verify it
Built the following successfully with various BGP & route-map unit test cases.
make target/python-wheels/sonic_yang_models-1.0-py3-none-any.whl
make target/python-wheels/sonic_yang_mgmt-1.0-py3-none-any.whl
Why I did it
This PR adds changes in sonic-config-engine to consume configuration data in SONiC Yang schema and generate config_db entries
How I did it
Add a new file sonic_yang_cfg_generator .
This file has the functions to
parse yang data json and convert them in config_db json format.
Validate the converted config_db entries to make sure all the dependencies and constraints are met.
Add a new option -Y to the sonic-cfggen command for this purpose
Add unit tests
This capability is support only in sonic-config-engine Python3 package only
* [yang-models]: Remove PLY Extensions and change translation code.
With assumption that TABLE_SEPARATOR and ENTRY_SEPARATOR for configDB is always "|",
translation from configDB.json to sonicYang.json can be done based on keys specified
in YANG Lists inside YANG models. So removing extensions is good idea.
Changes:
-- Remove use of regex in Translation code.
-- Remove regex Extensions from YANG models.
-- Improved debugging i.e. log on stdout in case of any Exception from sonic-yang-mgmt,
so that failed tests can be debugged faster. Also this is good to debug Dynamic
port breakout issues.
-- Minor Test changes.
Co-authored-by: lguohan <lguohan@gmail.com>
Changes:
— Remove unnecessary file libyang-python-tests/sample_config_db.json
— Keep only config in format of sonic_yang.json in yang_model_tests/yangTest.json.
— Keep config in format of config_db.json in file tests/files/sample_config_db.json
— Add a test case to validate yang models.
**- Why I did it**
As per discussion in sonic Yang workgroup, arranging files better so that Yang model can be in force for new config, also adding a test case to validate young models upto some extent.
**- How I did it**
Changes:
— Remove unnecessary file libyang-python-tests/sample_config_db.json
— Keep only config in format of sonic_yang.json in yang_model_tests/yangTest.json.
— Keep config in format of config_db.json in file tests/files/sample_config_db.json
— Add a test case to validate yang models.
**- How to verify it**
Build PKGs----Building-----
Changes:
1.) yang models for device_metadata, device_neighbor, flex_counters, crm, versions.
2.) Test cases for above yang models.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com:
- Add .gitignore files in each subdirectory of src/, so as to reduce the size of the .gitignore file in the project root, and also make it easier to maintain (i.e., if a directory in src/ is removed, there will not be outdated entries in the root .gitignore file.
- Also add missing .gitignore entries and remove outdated entries and duplicates.
Changes:
-- Removing the part where build dependencies are installed in setup.py.
-- Adding build dependencies in corresponsing rules\..*.mk file.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
**- What I did**
#### wheel package Makefiles
- wheel package Makefiles for sonic-yang-mgmt package.
#### libyang Python APIs:
- python APIs based on libyang
- functions to load/merge yang models and Yang data files
- function to validate data trees based on Yang models
- functions to merge yang data files/trees
- add/set/delete node in schema and data trees
- find data/schema nodes from xpath from the Yang data/schema tree in memory
- find dependencies
- dump the data tree in json/xml
#### Extension of libyang Python APIs:
-- Cropping input config based on Yang Model.
-- Translate input config based on Yang Model.
-- rev Translate input config based on Yang Model.
-- Find xpath of port, portleaf and a yang list.
-- Find if node is key of a list while deletion if yes, then delete the parent.
Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
Signed-off-by: Ping Mao pmao@linkedin.com
