Commit Graph

277 Commits

Author SHA1 Message Date
Tyler Li
2398992d52
[iccpd] build iccpd deb by auto tools (#4540)
* [iccpd] build iccpd deb by auto tools
2020-05-21 09:12:51 -07:00
rajendra-dendukuri
9c7105b5f3
Install swsssdk-py3 in the base Debian image for python3 based apps (#4542)
Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
2020-05-19 11:15:05 -07:00
joyas-joseph
9084ac50fb
[docker-telemetry]: upgrade telemetry docker to buster (#4515)
Signed-off-by: Joyas Joseph <joyas_joseph@dell.com>
2020-05-19 03:12:50 -07:00
joyas-joseph
9814da1e21
[docker-lldp-sv2]: upgrade docker-lldp-sv2 to buster (#4598)
Signed-off-by: Joyas Joseph <joyas_joseph@dell.com>
2020-05-15 14:51:53 -07:00
joyas-joseph
9dea816532
Convert docker-snmp-sv2 to buster (#4529)
* Fix libsnmp-base compilation failure
* Convert docker-snmp-sv2 to buster
* Define install_python3_wheels
* Address review comments
* Address review comments
* Advance snmpagent submodule
* Bump net-snmp to the Buster version
* Revert "Fix libsnmp-base compilation failure"
* use azure storage url
2020-05-14 10:23:37 -07:00
Kalimuthu-Velappan
fd953a437e
Fix for missing dependency in the sonic device package and added .dep file for yang module (#4471) 2020-05-06 13:17:08 +08:00
Sabareesh-Kumar-Anandan
81f4c81f0c
[build]: changing debian names based on Architecture (#4508)
Replacing "amd64" with $PLATFORM_ARCH variable
Fix for compiling marvell-armhf arch

Signed-off-by: Sabareesh Kumar Anandan <sanandan@marvell.com>
2020-04-30 15:47:03 -07:00
Sumukha Tumkur Vani
e1ba5b0f5f
[docker-restapi]: add restapi config to config Db (#4495) 2020-04-30 00:25:49 -07:00
lguohan
00383051f4
[docker-iccpd]: disable iccpd build by default (#4502)
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-29 22:45:46 -07:00
joyas-joseph
fc55329b22
[mgmt-framework]: convert mgmt-framework to use buster docker (#4480)
This change adds support to build dockers using buster as base.
- Define docker-base-buster using docker-base-stretch as starting point
- Define docker-config-engine-buster using docker-config-engine-stretch as starting point.
- sonic-mgmt-framework docker is updated to build using buster as base

Signed-off-by: Joyas Joseph <joyas_joseph@dell.com>
2020-04-28 15:57:13 -07:00
Prince Sunny
4adf243260
Fix docker restapi start issue, undefined error (#4475) 2020-04-24 12:43:56 -07:00
lguohan
bddd0d1e53
[docker-sonic-mgmt]: set docker-sonic-mgmt as stretch docker (#4450)
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-21 09:57:57 -07:00
lguohan
46fad0e0c6
Merge pull request #4405 from lguohan/buster
[baseimage]: upgrade base image to debian buster

bring up the base image to debian buster 4.19 kernel.

using the merge commits to preserve the individual commits to better track the history.
2020-04-20 15:32:17 -07:00
Kalimuthu-Velappan
7b53ae3b07
[build]: DPKG caching fix for evaluation bugs (#4425)
Added DPKG support for ICCPD package
Added warning support for missing dependency files that causes the incorrect SHA generation.
2020-04-20 11:14:25 -07:00
Renuka Manavalan
583bfde439
Kubeadm v1.18.0 requires Calico images of version v3.12.1 (#4445) 2020-04-19 11:34:30 -07:00
Guohan Lu
124ce22b8c [initramfs]: porting loopback file system support patch to v0.133
explicitly install dependency linux-base

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-17 04:51:51 +00:00
Guohan Lu
4e07c7880a [baseimage]: update python-click to 7.0-1 to support python3
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-17 04:51:51 +00:00
Guohan Lu
9d6760c336 [kernel]: build linux kernel 4.19.0-6
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-16 10:26:18 +00:00
Guohan Lu
01cb7934b0 [build]: add buster docker as the last step of the build proces
- build SONIC_STRETCH_DOCKERS in sonic-slave-stretch docker
- build image related module in sonic-slave-buster docker.
  This includes all kernels modules and some packages

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-16 10:26:18 +00:00
Sumukha Tumkur Vani
12400a447c
[docker-restapi]: Fix authentication in restapi (#4383)
- enabling http/https endpoint and mounting the certificates directory
- updating the sonic-restapi submodule
2020-04-15 02:08:45 -07:00
Praveen Chaudhary
a02255e2f4
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. (#3730)
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL.

YANG models as per Guidelines.

Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md

[sonic-yang-models/tests]: YANG model test code and JSON input for testing.

[sonic-yang-models/setup.py]: Build infra for yang models.

**- What I did**
Created Yang model for Sonic.
Tables:  PORT, VLAN, VLAN_INTERFACE, VLAN_MEMBER, ACL_RULE, ACL_TABLE, INTERFACE.

Created build infra files using which a new package (sonic-yang-models) can be build and can be deployed on sonic switches. Yang models will be part of this new package.

**- How I did it**
Wrote yang models based on Guideline doc: 
https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md
and 
https://github.com/Azure/SONiC/wiki/Configuration.

Wrote python wheel Package infra which runs test for these Yang models using a json files which consists configuration as per yang models. These configs are for negative tests, which means we want to test that most must condition, pattern and when condition works as expected.

**- How to verify it**
Build Logs and testing:
———————————————————————————————————

```
/sonic/src/sonic-yang-models /sonic
running test
running egg_info
writing top-level names to sonic_yang_models.egg-info/top_level.txt
writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt
writing sonic_yang_models.egg-info/PKG-INFO
reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
running bdist_wheel
running build
running build_py
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../libyang_1.0.73_amd64.deb ...
Unpacking libyang (1.0.73) over (1.0.73) ...
Setting up libyang (1.0.73) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Processing triggers for man-db (2.7.6.1-2) ...
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../libyang-cpp_1.0.73_amd64.deb ...
Unpacking libyang-cpp (1.0.73) over (1.0.73) ...
Setting up libyang-cpp (1.0.73) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../python3-yang_1.0.73_amd64.deb ...
Unpacking python3-yang (1.0.73) over (1.0.73) ...
Setting up python3-yang (1.0.73) ...
INFO:YANG-TEST:module: sonic-vlan is loaded successfully
ERROR:YANG-TEST:Could not get module: sonic-head
INFO:YANG-TEST:module: sonic-portchannel is loaded successfully
INFO:YANG-TEST:module: sonic-acl is loaded successfully
INFO:YANG-TEST:module: sonic-loopback-interface is loaded successfully
ERROR:YANG-TEST:Could not get module: sonic-port
INFO:YANG-TEST:module: sonic-interface is loaded successfully
INFO:YANG-TEST:
------------------- Test 1: Configure a member port in VLAN_MEMBER table which does not exist.---------------------
libyang[0]: Leafref "/sonic-port:sonic-port/sonic-port:PORT/sonic-port:PORT_LIST/sonic-port:port_name" of value "Ethernet156" points to a non
-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlan_name='Vlan100'][port='Ethernet156']/port)
INFO:YANG-TEST:Configure a member port in VLAN_MEMBER table which does not exist. Passed

INFO:YANG-TEST:
------------------- Test 2: Configure non-existing ACL_TABLE in ACL_RULE.---------------------
libyang[0]: Leafref "/sonic-acl:sonic-acl/sonic-acl:ACL_TABLE/sonic-acl:ACL_TABLE_LIST/sonic-acl:ACL_TABLE_NAME" of value "NOT-EXIST" points
to a non-existing leaf. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NOT-EXIST'][RULE_NAME='Rule_20']/ACL_TABLE_NAME)
INFO:YANG-TEST:Configure non-existing ACL_TABLE in ACL_RULE. Passed

INFO:YANG-TEST:
------------------- Test 3: Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU
LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/ICMPV6_CODE)
INFO:YANG-TEST:Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE. Passed
INFO:YANG-TEST:

INFO:YANG-TEST:
------------------- Test 4: Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU
LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_20']/SRC_IPV6)
INFO:YANG-TEST:Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE. Passed

------------------- Test 5: Configure l4_src_port_range as 99999-99999 in ACL_RULE---------------------
libyang[0]: Value "99999-99999" does not satisfy the constraint "([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])-([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6'][RULE_NAME='Rule_20']/L4_SRC_PORT_RANGE)
INFO:YANG-TEST:Configure l4_src_port_range as 99999-99999 in ACL_RULE Passed

INFO:YANG-TEST:
------------------- Test 6: Configure empty string as ip-prefix in INTERFACE table.---------------------
libyang[0]: Invalid value "" in "ip-prefix" element. (path: /sonic-interface:sonic-interface/INTERFACE/INTERFACE_LIST[interface='Ethernet8'][ip-prefix='']/ip-prefix)
INFO:YANG-TEST:Configure empty string as ip-prefix in INTERFACE table. Passed

INFO:YANG-TEST:
------------------- Test 7: Configure Wrong family with ip-prefix for VLAN_Interface Table---------------------
libyang[0]: Must condition "(contains(../ip-prefix, ':') and current()='IPv6') or                               (contains(../ip-prefix, '.') and current()='IPv4')" not satisfied. (path: /sonic-vlan:sonic-vlan/VLAN_INTERFACE/VLAN_INTERFACE_LIST[vlanid='100'][ip-prefix='2a04:5555:66:7777::1/64']/family)
INFO:YANG-TEST:Configure Wrong family with ip-prefix for VLAN_Interface Table Passed

INFO:YANG-TEST:
------------------- Test 8: Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPV6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NS
W-PACL-V6'][RULE_NAME='Rule_20']/DST_IPV6)
INFO:YANG-TEST:Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE. Passed

INFO:YANG-TEST:
------------------- Test 9: Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE.---------------------
libyang[0]: Value "0x080C" does not satisfy the constraint "(0x88CC|0x8100|0x8915|0x0806|0x0800|0x86DD|0x8847)" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/INNER_ETHER_TYPE)
INFO:YANG-TEST:Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE. Passed

INFO:YANG-TEST:
------------------- Test 10: Add dhcp_server which is not in correct ip-prefix format.---------------------
libyang[0]: Invalid value "10.186.72.566" in "dhcp_servers" element. (path: /sonic-vlan:sonic-vlan/VLAN/VLAN_LIST/dhcp_servers[.='10.186.72.566'])
INFO:YANG-TEST:Add dhcp_server which is not in correct ip-prefix format. Passed

INFO:YANG-TEST:
------------------- Test 11: Configure undefined acl_table_type in ACL_TABLE table.---------------------
libyang[0]: Invalid value "LAYER3V4" in "type" element. (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6']/type)
INFO:YANG-TEST:Configure undefined acl_table_type in ACL_TABLE table. Passed

INFO:YANG-TEST:
------------------- Test 12: Configure undefined packet_action in ACL_RULE table.---------------------
libyang[0]: Invalid value "SEND" in "PACKET_ACTION" element. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST/PACKET_ACTION)
INFO:YANG-TEST:Configure undefined packet_action in ACL_RULE table. Passed

INFO:YANG-TEST:
------------------- Test 13: Configure wrong value for tagging_mode.---------------------
libyang[0]: Invalid value "non-tagged" in "tagging_mode" element. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST/tagging_mode)
INFO:YANG-TEST:Configure wrong value for tagging_mode. Passed

INFO:YANG-TEST:
------------------- Test 14: Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN  table.---------------------
libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid)
libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid)
INFO:YANG-TEST:Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN  table. Passed

INFO:YANG-TEST:All Test Passed
../../target/debs/stretch/libyang0.16_0.16.105-1_amd64.deb installtion failed
../../target/debs/stretch/libyang-cpp0.16_0.16.105-1_amd64.deb installtion failed
../../target/debs/stretch/python2-yang_0.16.105-1_amd64.deb installtion failed
YANG Tests passed
Passed: pyang -f tree ./yang-models/*.yang > ./yang-models/sonic_yang_tree
copying tests/yangModelTesting.py -> build/lib/tests
copying tests/test_sonic_yang_models.py -> build/lib/tests
copying tests/__init__.py -> build/lib/tests
running egg_info
writing top-level names to sonic_yang_models.egg-info/top_level.txt
writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt
writing sonic_yang_models.egg-info/PKG-INFO
reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/yangModelTesting.py -> build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/test_sonic_yang_models.py -> build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/wheel/tests
running install_data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-head.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-acl.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-loopback-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-port.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-portchannel.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-vlan.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
```
2020-04-14 15:36:02 -07:00
Renuka Manavalan
f128153706
[baseimage]: Install Kubernetes packages if enabled in image (#4374)
* Install kubernetes worker node packages, if enabled.

* Minor updates

* Added some comments

* Updates per review comments.
Built a private image to test to work fine.

* Remove the removed file.

* Update per comments
Make a fix, as kubeadm no demands a higher version of kubelet & kubectl.
As kubeadm auto install kubectl & kubelet, removing explicit install is an easier/robust fix.

* Changes per review comments.

* Updates per comments.
1) Dropped helper & pod scripts
2) Made install verbose

* Drop creation of pods subdir, as this PR does not use them.

* From comments to 'n' per review comments.

* 1) kubeadm.conf is created as part of kubeadm package install. Hence dropped explicit copy.
2020-04-13 08:41:18 -07:00
Danny Allen
4f8f1d45e4
[docker-sonic-mgmt] Merge spytest dependencies into mgmt docker (#4411)
Signed-off-by: Danny Allen <daall@microsoft.com>
2020-04-12 13:35:28 -07:00
shine4chen
524cf9e56a
MCLAG feature for SONIC (#2514)
* MCLAG feature for sonic

* MCLAG feature for sonic

* remove binary file

* remove unused dockerfile

update docker-iccpd to stretch-based container

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* minor fix for isolation port setting

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* iccpd docker would start on demand

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* Add x attribute on mclagdctl file

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* add warm-reboot support for MCLAG

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* merge to master branch and reformat iccpd file

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* fix some bugs and make peer-link configuration optional

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* refactor code per Brcm review

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* correct a typo

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* * optimize iccpd arp/mac sync process
* refine code according to brcm opinoin
* unify function return value

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* * optimize warm-reboot process
* estabish iccpd connection with configurated src-ip

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* fix a typo

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* optimize some code
* add some debug info
* optimize bridge mac setting
* fix vlan mac sync issue on standby node

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* optimize some code

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* fix some bugs for warm-reboot

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* refine log level

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* refine iccpd syslog & skip arp packet whose src ip is local ip

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* remove iccpd dependency with teamd

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* print log level when dump mclag status

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* revise per community review

Signed-off-by: shine.chen <shine.chen@mediatek.com>

Co-authored-by: shine.chen <shine.chen@nephosinc.com>
Co-authored-by: shine.chen <shine.chen@mediatek.com>
2020-04-04 15:24:06 -07:00
pavel-shirshov
8585005a36
[FRR]: Update frr to latest 7.2.1-s3 (#4294)
- Updated to latest frr 7.2.1 from the master.
- Updated patches accordingly
2020-04-01 12:57:19 -07:00
SuvarnaMeenakshi
4b8067e913
Multi-ASIC implementation (#3888)
Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.
2020-03-31 10:06:19 -07:00
padmanarayana
eede9a9db9
sFlow : Upgrade hsflowd to v2.0.26-4 (#4301)
As per SONiC CLI requirements, we wanted the collector port to default to 6343 if not explicitly configured. In the Click based CLI, if the collector port is omitted, the port will be set to 6343 by default (https://github.com/Azure/sonic-utilities/blob/master/config/main.py#L2543).
However, if sonic-mgmt-framework based CLI is used, hsflowd was using port 0 as default. host-sflow v2.0.26-4 addresses this issue (i.e. use 6343 in case “collector_port” is not set).

Upgrade hsflowd to v2.0.26-4 to pick up https://github.com/sflow/host-sflow/releases/tag/v2.0.26-4.
2020-03-28 23:49:09 -07:00
xumia
46df9ad576
[build]: Fix DPKG derived package issue (#4333) 2020-03-27 12:57:14 -07:00
xumia
6c8ed042da
Fix docker-sonic-mgmt-framework-dbg wrong dependency issue (#4314) 2020-03-24 19:32:51 +08:00
lguohan
760e763935
[build]: allow to specify timestamp format in the build log (#4311)
only simple/none are supported currently

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-03-23 09:45:43 -07:00
Sangita Maity
0dc047c327
[build]: add dependency of sonic-config-engine on 'sonic-platform-common' package (#3912)
Signed-off-by: Sangita Maity <sangitamaity0211@gmail.com>
2020-03-22 12:30:55 -07:00
lguohan
20260ceb1d
[build]: add SONIC_CONFIG_BUILD_LOG_TIMESTAMP to add timestamp in build log (#4269)
add timestamp in each job build log

example:

   [01:39:21] dh clean  --with autotools-dev
   [01:39:22]    dh_auto_clean
   [01:39:27]      make -j16 distclean

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-03-21 14:21:26 -07:00
Kalimuthu-Velappan
2d23495ef7
[build]: Fix for missing derived package in libream.mk durinig DPKG caching (#4287) 2020-03-19 12:37:47 -07:00
Kalimuthu-Velappan
7d2ebf8116
[build]: support for DPKG local caching (#4117)
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.

The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.

This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.

- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=

    where  SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
                            none:     no caching
                            cache:    cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.

    For example: dependency files for Linux Kernel - src/sonic-linux-kernel,

            SPATH       := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
            DEP_FILES   := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
            DEP_FILES   += $(SONIC_COMMON_BASE_FILES_LIST)
            SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))

            DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
                         $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)

            $(LINUX_HEADERS_COMMON)_CACHE_MODE  := GIT_CONTENT_SHA
            $(LINUX_HEADERS_COMMON)_DEP_FLAGS   := $(DEP_FLAGS)
            $(LINUX_HEADERS_COMMON)_DEP_FILES   := $(DEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format

    FORMAT:
            <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
            Eg:
              linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz

            < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
            < 24-byte MOD SHA value > - the SHA value is derived from either of the following.
                    GIT_COMMIT_SHA  - SHA value of the last git commit ID if it is a submodule
                    GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.

            $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
                    $(call dpkg_depend,$(DEBS_PATH)/%.dep )
            $(HEADER)


            # Load the target deb from DPKG cache
            $(call LOAD_CACHE,$*,$@)


            # Skip building the target if it is already loaded from cache
            if [ -z '$($*_CACHE_LOADED)' ] ; then

                  .....
                 # Rules for Generating the target DEB file.
                  .....

                  # Save the target deb into DPKG cache
                  $(call SAVE_CACHE,$*,$@)
            fi


            $(FOOTER)


    The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.

    Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
    The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
    It also updates the target-specific variable to indicate the cache is loaded or not.
    The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it

    The caching functionality is verified by enabling it in Linux kernel submodule.
    It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

DOCUMENT PR:

           https://github.com/Azure/SONiC/pull/559
2020-03-11 20:04:52 -07:00
pavel-shirshov
fc91f81dab
[libteam]: Update libteam to the latest master (#4216) 2020-03-09 16:13:49 -07:00
Junchao-Mellanox
be549db395
Add thermal control support for SONiC (#3949) 2020-03-09 10:41:10 -07:00
Joe LeVeque
64a6989d02
[Services] Restart NAT service upon unexpected critical process exit. (#4208) 2020-03-05 15:27:21 -08:00
Danny Allen
f906a758cd
[spytest] Create new sonic-mgmt docker for running spytests (#4212)
Signed-off-by: Danny Allen <daall@microsoft.com>
2020-03-04 11:47:31 -08:00
yozhao101
23ff55a709
[Services] Restart BGP service upon unexpected critical process exit. (#4207) 2020-03-03 16:50:32 -08:00
abdosi
f5ab24d8f1
Made Change to have Configurable option to enable/disable docker image (#4171)
* Made Change to have Configurable option to enable/disable docker image

* [Submodule-Update] sonic-utilities to latest master
2020-02-21 10:20:07 -08:00
Danny Allen
6005f4c976
Revert "Update frr to latest 7.2.1 (#4145)" (#4170)
This reverts commit 4b42a48f45.
2020-02-20 13:47:21 -08:00
Danny Allen
236013142c
[frr] Update FRR build to include version tags (#4172)
Signed-off-by: Danny Allen <daall@microsoft.com>
2020-02-20 11:20:50 -08:00
Prince Sunny
31fb631cd3
Fix service and container name to be same (#4151) 2020-02-14 11:08:57 -08:00
pavel-shirshov
4b42a48f45
Update frr to latest 7.2.1 (#4145) 2020-02-13 18:45:37 -08:00
Olivier Singla
6a0dcb1b16
[kernel]: security kernel update to 4.9.189 (#3913)
This patch upgrade the kernel from version
4.9.0-9-2 (4.9.168-1+deb9u3) to 4.9.0-11-2 (4.9.189-3+deb9u2)

Co-authored-by: rajendra-dendukuri <47423477+rajendra-dendukuri@users.noreply.github.com>
2020-02-12 17:41:58 -08:00
yozhao101
729f343f77
[Services] Restart database service upon unexpected critical process exit. (#4138)
* [database] Implement the auto-restart feature for database container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [database] Remove the duplicate dependency in service files. Since we
already have updategraph ---> config_setup ---> database, we do not need
explicitly add database.service in all other container service files.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [event listener] Reorganize the line 73 in event listener script.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [database] update the file sflow.service.j2 to remove the duplicate
dependency.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [event listener] Add comments in event listener.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [event listener] Update the comments in line 56.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [event listener] Add parentheses for if statement in line 76 in event listener.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2020-02-11 14:03:02 -08:00
pavel-shirshov
c63b5fc719
[frr]: Use tag for building frr (#4082)
Changed logic for building frr. Previously we used latest commit in the FRR_BRANCH. Now the buildsystem will use a tag to identify a commit for building. New approach will let us to update sonic-frr without corrupting building sonic-buildimage.
2020-02-07 07:51:01 -08:00
Mykola F
f8e530577d
[docker] remove leftover --net=host docker run options (#4119)
Fixes #4118

Signed-off-by: Mykola Faryma <mykolaf@mellanox.com>
2020-02-06 16:27:20 -08:00
lguohan
2e0f9ca0f4
[hsflowd]: remove linux kernel header dependency from hsflowd package (#4111)
hsflowd compilation does not linux kernel header

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-02-05 19:04:19 -08:00
Kiran Kumar Kella
97165a0d69
Changes in sonic-buildimage to support the NAT feature (#3494)
* Changes in sonic-buildimage for the NAT feature
- Docker for NAT
- installing the required tools iptables and conntrack for nat

Signed-off-by: kiran.kella@broadcom.com

* Add redis-tools dependencies in the docker nat compilation

* Addressed review comments

* add natsyncd to warm-boot finalizer list

* addressed review comments

* using swsscommon.DBConnector instead of swsssdk.SonicV2Connector

* Enable NAT application in docker-sonic-vs
2020-01-29 17:40:43 -08:00