Commit Graph

323 Commits

Author SHA1 Message Date
joyas-joseph
9dea816532
Convert docker-snmp-sv2 to buster (#4529)
* Fix libsnmp-base compilation failure
* Convert docker-snmp-sv2 to buster
* Define install_python3_wheels
* Address review comments
* Address review comments
* Advance snmpagent submodule
* Bump net-snmp to the Buster version
* Revert "Fix libsnmp-base compilation failure"
* use azure storage url
2020-05-14 10:23:37 -07:00
Kalimuthu-Velappan
fd953a437e
Fix for missing dependency in the sonic device package and added .dep file for yang module (#4471) 2020-05-06 13:17:08 +08:00
Sabareesh-Kumar-Anandan
81f4c81f0c
[build]: changing debian names based on Architecture (#4508)
Replacing "amd64" with $PLATFORM_ARCH variable
Fix for compiling marvell-armhf arch

Signed-off-by: Sabareesh Kumar Anandan <sanandan@marvell.com>
2020-04-30 15:47:03 -07:00
Sumukha Tumkur Vani
e1ba5b0f5f
[docker-restapi]: add restapi config to config Db (#4495) 2020-04-30 00:25:49 -07:00
lguohan
00383051f4
[docker-iccpd]: disable iccpd build by default (#4502)
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-29 22:45:46 -07:00
joyas-joseph
fc55329b22
[mgmt-framework]: convert mgmt-framework to use buster docker (#4480)
This change adds support to build dockers using buster as base.
- Define docker-base-buster using docker-base-stretch as starting point
- Define docker-config-engine-buster using docker-config-engine-stretch as starting point.
- sonic-mgmt-framework docker is updated to build using buster as base

Signed-off-by: Joyas Joseph <joyas_joseph@dell.com>
2020-04-28 15:57:13 -07:00
Prince Sunny
4adf243260
Fix docker restapi start issue, undefined error (#4475) 2020-04-24 12:43:56 -07:00
lguohan
bddd0d1e53
[docker-sonic-mgmt]: set docker-sonic-mgmt as stretch docker (#4450)
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-21 09:57:57 -07:00
lguohan
46fad0e0c6
Merge pull request #4405 from lguohan/buster
[baseimage]: upgrade base image to debian buster

bring up the base image to debian buster 4.19 kernel.

using the merge commits to preserve the individual commits to better track the history.
2020-04-20 15:32:17 -07:00
Kalimuthu-Velappan
7b53ae3b07
[build]: DPKG caching fix for evaluation bugs (#4425)
Added DPKG support for ICCPD package
Added warning support for missing dependency files that causes the incorrect SHA generation.
2020-04-20 11:14:25 -07:00
Renuka Manavalan
583bfde439
Kubeadm v1.18.0 requires Calico images of version v3.12.1 (#4445) 2020-04-19 11:34:30 -07:00
Guohan Lu
124ce22b8c [initramfs]: porting loopback file system support patch to v0.133
explicitly install dependency linux-base

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-17 04:51:51 +00:00
Guohan Lu
4e07c7880a [baseimage]: update python-click to 7.0-1 to support python3
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-17 04:51:51 +00:00
Guohan Lu
9d6760c336 [kernel]: build linux kernel 4.19.0-6
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-16 10:26:18 +00:00
Guohan Lu
01cb7934b0 [build]: add buster docker as the last step of the build proces
- build SONIC_STRETCH_DOCKERS in sonic-slave-stretch docker
- build image related module in sonic-slave-buster docker.
  This includes all kernels modules and some packages

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-16 10:26:18 +00:00
Sumukha Tumkur Vani
12400a447c
[docker-restapi]: Fix authentication in restapi (#4383)
- enabling http/https endpoint and mounting the certificates directory
- updating the sonic-restapi submodule
2020-04-15 02:08:45 -07:00
Praveen Chaudhary
a02255e2f4
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. (#3730)
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL.

YANG models as per Guidelines.

Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md

[sonic-yang-models/tests]: YANG model test code and JSON input for testing.

[sonic-yang-models/setup.py]: Build infra for yang models.

**- What I did**
Created Yang model for Sonic.
Tables:  PORT, VLAN, VLAN_INTERFACE, VLAN_MEMBER, ACL_RULE, ACL_TABLE, INTERFACE.

Created build infra files using which a new package (sonic-yang-models) can be build and can be deployed on sonic switches. Yang models will be part of this new package.

**- How I did it**
Wrote yang models based on Guideline doc: 
https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md
and 
https://github.com/Azure/SONiC/wiki/Configuration.

Wrote python wheel Package infra which runs test for these Yang models using a json files which consists configuration as per yang models. These configs are for negative tests, which means we want to test that most must condition, pattern and when condition works as expected.

**- How to verify it**
Build Logs and testing:
———————————————————————————————————

```
/sonic/src/sonic-yang-models /sonic
running test
running egg_info
writing top-level names to sonic_yang_models.egg-info/top_level.txt
writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt
writing sonic_yang_models.egg-info/PKG-INFO
reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
running bdist_wheel
running build
running build_py
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../libyang_1.0.73_amd64.deb ...
Unpacking libyang (1.0.73) over (1.0.73) ...
Setting up libyang (1.0.73) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Processing triggers for man-db (2.7.6.1-2) ...
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../libyang-cpp_1.0.73_amd64.deb ...
Unpacking libyang-cpp (1.0.73) over (1.0.73) ...
Setting up libyang-cpp (1.0.73) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../python3-yang_1.0.73_amd64.deb ...
Unpacking python3-yang (1.0.73) over (1.0.73) ...
Setting up python3-yang (1.0.73) ...
INFO:YANG-TEST:module: sonic-vlan is loaded successfully
ERROR:YANG-TEST:Could not get module: sonic-head
INFO:YANG-TEST:module: sonic-portchannel is loaded successfully
INFO:YANG-TEST:module: sonic-acl is loaded successfully
INFO:YANG-TEST:module: sonic-loopback-interface is loaded successfully
ERROR:YANG-TEST:Could not get module: sonic-port
INFO:YANG-TEST:module: sonic-interface is loaded successfully
INFO:YANG-TEST:
------------------- Test 1: Configure a member port in VLAN_MEMBER table which does not exist.---------------------
libyang[0]: Leafref "/sonic-port:sonic-port/sonic-port:PORT/sonic-port:PORT_LIST/sonic-port:port_name" of value "Ethernet156" points to a non
-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlan_name='Vlan100'][port='Ethernet156']/port)
INFO:YANG-TEST:Configure a member port in VLAN_MEMBER table which does not exist. Passed

INFO:YANG-TEST:
------------------- Test 2: Configure non-existing ACL_TABLE in ACL_RULE.---------------------
libyang[0]: Leafref "/sonic-acl:sonic-acl/sonic-acl:ACL_TABLE/sonic-acl:ACL_TABLE_LIST/sonic-acl:ACL_TABLE_NAME" of value "NOT-EXIST" points
to a non-existing leaf. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NOT-EXIST'][RULE_NAME='Rule_20']/ACL_TABLE_NAME)
INFO:YANG-TEST:Configure non-existing ACL_TABLE in ACL_RULE. Passed

INFO:YANG-TEST:
------------------- Test 3: Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU
LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/ICMPV6_CODE)
INFO:YANG-TEST:Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE. Passed
INFO:YANG-TEST:

INFO:YANG-TEST:
------------------- Test 4: Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU
LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_20']/SRC_IPV6)
INFO:YANG-TEST:Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE. Passed

------------------- Test 5: Configure l4_src_port_range as 99999-99999 in ACL_RULE---------------------
libyang[0]: Value "99999-99999" does not satisfy the constraint "([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])-([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6'][RULE_NAME='Rule_20']/L4_SRC_PORT_RANGE)
INFO:YANG-TEST:Configure l4_src_port_range as 99999-99999 in ACL_RULE Passed

INFO:YANG-TEST:
------------------- Test 6: Configure empty string as ip-prefix in INTERFACE table.---------------------
libyang[0]: Invalid value "" in "ip-prefix" element. (path: /sonic-interface:sonic-interface/INTERFACE/INTERFACE_LIST[interface='Ethernet8'][ip-prefix='']/ip-prefix)
INFO:YANG-TEST:Configure empty string as ip-prefix in INTERFACE table. Passed

INFO:YANG-TEST:
------------------- Test 7: Configure Wrong family with ip-prefix for VLAN_Interface Table---------------------
libyang[0]: Must condition "(contains(../ip-prefix, ':') and current()='IPv6') or                               (contains(../ip-prefix, '.') and current()='IPv4')" not satisfied. (path: /sonic-vlan:sonic-vlan/VLAN_INTERFACE/VLAN_INTERFACE_LIST[vlanid='100'][ip-prefix='2a04:5555:66:7777::1/64']/family)
INFO:YANG-TEST:Configure Wrong family with ip-prefix for VLAN_Interface Table Passed

INFO:YANG-TEST:
------------------- Test 8: Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPV6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NS
W-PACL-V6'][RULE_NAME='Rule_20']/DST_IPV6)
INFO:YANG-TEST:Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE. Passed

INFO:YANG-TEST:
------------------- Test 9: Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE.---------------------
libyang[0]: Value "0x080C" does not satisfy the constraint "(0x88CC|0x8100|0x8915|0x0806|0x0800|0x86DD|0x8847)" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/INNER_ETHER_TYPE)
INFO:YANG-TEST:Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE. Passed

INFO:YANG-TEST:
------------------- Test 10: Add dhcp_server which is not in correct ip-prefix format.---------------------
libyang[0]: Invalid value "10.186.72.566" in "dhcp_servers" element. (path: /sonic-vlan:sonic-vlan/VLAN/VLAN_LIST/dhcp_servers[.='10.186.72.566'])
INFO:YANG-TEST:Add dhcp_server which is not in correct ip-prefix format. Passed

INFO:YANG-TEST:
------------------- Test 11: Configure undefined acl_table_type in ACL_TABLE table.---------------------
libyang[0]: Invalid value "LAYER3V4" in "type" element. (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6']/type)
INFO:YANG-TEST:Configure undefined acl_table_type in ACL_TABLE table. Passed

INFO:YANG-TEST:
------------------- Test 12: Configure undefined packet_action in ACL_RULE table.---------------------
libyang[0]: Invalid value "SEND" in "PACKET_ACTION" element. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST/PACKET_ACTION)
INFO:YANG-TEST:Configure undefined packet_action in ACL_RULE table. Passed

INFO:YANG-TEST:
------------------- Test 13: Configure wrong value for tagging_mode.---------------------
libyang[0]: Invalid value "non-tagged" in "tagging_mode" element. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST/tagging_mode)
INFO:YANG-TEST:Configure wrong value for tagging_mode. Passed

INFO:YANG-TEST:
------------------- Test 14: Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN  table.---------------------
libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid)
libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid)
INFO:YANG-TEST:Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN  table. Passed

INFO:YANG-TEST:All Test Passed
../../target/debs/stretch/libyang0.16_0.16.105-1_amd64.deb installtion failed
../../target/debs/stretch/libyang-cpp0.16_0.16.105-1_amd64.deb installtion failed
../../target/debs/stretch/python2-yang_0.16.105-1_amd64.deb installtion failed
YANG Tests passed
Passed: pyang -f tree ./yang-models/*.yang > ./yang-models/sonic_yang_tree
copying tests/yangModelTesting.py -> build/lib/tests
copying tests/test_sonic_yang_models.py -> build/lib/tests
copying tests/__init__.py -> build/lib/tests
running egg_info
writing top-level names to sonic_yang_models.egg-info/top_level.txt
writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt
writing sonic_yang_models.egg-info/PKG-INFO
reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/yangModelTesting.py -> build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/test_sonic_yang_models.py -> build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/wheel/tests
running install_data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-head.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-acl.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-loopback-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-port.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-portchannel.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-vlan.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
```
2020-04-14 15:36:02 -07:00
Renuka Manavalan
f128153706
[baseimage]: Install Kubernetes packages if enabled in image (#4374)
* Install kubernetes worker node packages, if enabled.

* Minor updates

* Added some comments

* Updates per review comments.
Built a private image to test to work fine.

* Remove the removed file.

* Update per comments
Make a fix, as kubeadm no demands a higher version of kubelet & kubectl.
As kubeadm auto install kubectl & kubelet, removing explicit install is an easier/robust fix.

* Changes per review comments.

* Updates per comments.
1) Dropped helper & pod scripts
2) Made install verbose

* Drop creation of pods subdir, as this PR does not use them.

* From comments to 'n' per review comments.

* 1) kubeadm.conf is created as part of kubeadm package install. Hence dropped explicit copy.
2020-04-13 08:41:18 -07:00
Danny Allen
4f8f1d45e4
[docker-sonic-mgmt] Merge spytest dependencies into mgmt docker (#4411)
Signed-off-by: Danny Allen <daall@microsoft.com>
2020-04-12 13:35:28 -07:00
shine4chen
524cf9e56a
MCLAG feature for SONIC (#2514)
* MCLAG feature for sonic

* MCLAG feature for sonic

* remove binary file

* remove unused dockerfile

update docker-iccpd to stretch-based container

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* minor fix for isolation port setting

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* iccpd docker would start on demand

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* Add x attribute on mclagdctl file

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* add warm-reboot support for MCLAG

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* merge to master branch and reformat iccpd file

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* fix some bugs and make peer-link configuration optional

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* refactor code per Brcm review

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* correct a typo

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* * optimize iccpd arp/mac sync process
* refine code according to brcm opinoin
* unify function return value

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* * optimize warm-reboot process
* estabish iccpd connection with configurated src-ip

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* fix a typo

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* optimize some code
* add some debug info
* optimize bridge mac setting
* fix vlan mac sync issue on standby node

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* optimize some code

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* fix some bugs for warm-reboot

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* refine log level

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* refine iccpd syslog & skip arp packet whose src ip is local ip

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* remove iccpd dependency with teamd

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* print log level when dump mclag status

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* revise per community review

Signed-off-by: shine.chen <shine.chen@mediatek.com>

Co-authored-by: shine.chen <shine.chen@nephosinc.com>
Co-authored-by: shine.chen <shine.chen@mediatek.com>
2020-04-04 15:24:06 -07:00
pavel-shirshov
8585005a36
[FRR]: Update frr to latest 7.2.1-s3 (#4294)
- Updated to latest frr 7.2.1 from the master.
- Updated patches accordingly
2020-04-01 12:57:19 -07:00
SuvarnaMeenakshi
4b8067e913
Multi-ASIC implementation (#3888)
Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.
2020-03-31 10:06:19 -07:00
padmanarayana
eede9a9db9
sFlow : Upgrade hsflowd to v2.0.26-4 (#4301)
As per SONiC CLI requirements, we wanted the collector port to default to 6343 if not explicitly configured. In the Click based CLI, if the collector port is omitted, the port will be set to 6343 by default (https://github.com/Azure/sonic-utilities/blob/master/config/main.py#L2543).
However, if sonic-mgmt-framework based CLI is used, hsflowd was using port 0 as default. host-sflow v2.0.26-4 addresses this issue (i.e. use 6343 in case “collector_port” is not set).

Upgrade hsflowd to v2.0.26-4 to pick up https://github.com/sflow/host-sflow/releases/tag/v2.0.26-4.
2020-03-28 23:49:09 -07:00
xumia
46df9ad576
[build]: Fix DPKG derived package issue (#4333) 2020-03-27 12:57:14 -07:00
xumia
6c8ed042da
Fix docker-sonic-mgmt-framework-dbg wrong dependency issue (#4314) 2020-03-24 19:32:51 +08:00
lguohan
760e763935
[build]: allow to specify timestamp format in the build log (#4311)
only simple/none are supported currently

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-03-23 09:45:43 -07:00
Sangita Maity
0dc047c327
[build]: add dependency of sonic-config-engine on 'sonic-platform-common' package (#3912)
Signed-off-by: Sangita Maity <sangitamaity0211@gmail.com>
2020-03-22 12:30:55 -07:00
lguohan
20260ceb1d
[build]: add SONIC_CONFIG_BUILD_LOG_TIMESTAMP to add timestamp in build log (#4269)
add timestamp in each job build log

example:

   [01:39:21] dh clean  --with autotools-dev
   [01:39:22]    dh_auto_clean
   [01:39:27]      make -j16 distclean

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-03-21 14:21:26 -07:00
Kalimuthu-Velappan
2d23495ef7
[build]: Fix for missing derived package in libream.mk durinig DPKG caching (#4287) 2020-03-19 12:37:47 -07:00
Kalimuthu-Velappan
7d2ebf8116
[build]: support for DPKG local caching (#4117)
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.

The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.

This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.

- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=

    where  SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
                            none:     no caching
                            cache:    cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.

    For example: dependency files for Linux Kernel - src/sonic-linux-kernel,

            SPATH       := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
            DEP_FILES   := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
            DEP_FILES   += $(SONIC_COMMON_BASE_FILES_LIST)
            SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))

            DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
                         $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)

            $(LINUX_HEADERS_COMMON)_CACHE_MODE  := GIT_CONTENT_SHA
            $(LINUX_HEADERS_COMMON)_DEP_FLAGS   := $(DEP_FLAGS)
            $(LINUX_HEADERS_COMMON)_DEP_FILES   := $(DEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format

    FORMAT:
            <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
            Eg:
              linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz

            < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
            < 24-byte MOD SHA value > - the SHA value is derived from either of the following.
                    GIT_COMMIT_SHA  - SHA value of the last git commit ID if it is a submodule
                    GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.

            $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
                    $(call dpkg_depend,$(DEBS_PATH)/%.dep )
            $(HEADER)


            # Load the target deb from DPKG cache
            $(call LOAD_CACHE,$*,$@)


            # Skip building the target if it is already loaded from cache
            if [ -z '$($*_CACHE_LOADED)' ] ; then

                  .....
                 # Rules for Generating the target DEB file.
                  .....

                  # Save the target deb into DPKG cache
                  $(call SAVE_CACHE,$*,$@)
            fi


            $(FOOTER)


    The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.

    Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
    The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
    It also updates the target-specific variable to indicate the cache is loaded or not.
    The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it

    The caching functionality is verified by enabling it in Linux kernel submodule.
    It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

DOCUMENT PR:

           https://github.com/Azure/SONiC/pull/559
2020-03-11 20:04:52 -07:00
pavel-shirshov
fc91f81dab
[libteam]: Update libteam to the latest master (#4216) 2020-03-09 16:13:49 -07:00
Junchao-Mellanox
be549db395
Add thermal control support for SONiC (#3949) 2020-03-09 10:41:10 -07:00
Joe LeVeque
64a6989d02
[Services] Restart NAT service upon unexpected critical process exit. (#4208) 2020-03-05 15:27:21 -08:00
Danny Allen
f906a758cd
[spytest] Create new sonic-mgmt docker for running spytests (#4212)
Signed-off-by: Danny Allen <daall@microsoft.com>
2020-03-04 11:47:31 -08:00
yozhao101
23ff55a709
[Services] Restart BGP service upon unexpected critical process exit. (#4207) 2020-03-03 16:50:32 -08:00
abdosi
f5ab24d8f1
Made Change to have Configurable option to enable/disable docker image (#4171)
* Made Change to have Configurable option to enable/disable docker image

* [Submodule-Update] sonic-utilities to latest master
2020-02-21 10:20:07 -08:00
Danny Allen
6005f4c976
Revert "Update frr to latest 7.2.1 (#4145)" (#4170)
This reverts commit 4b42a48f45.
2020-02-20 13:47:21 -08:00
Danny Allen
236013142c
[frr] Update FRR build to include version tags (#4172)
Signed-off-by: Danny Allen <daall@microsoft.com>
2020-02-20 11:20:50 -08:00
Prince Sunny
31fb631cd3
Fix service and container name to be same (#4151) 2020-02-14 11:08:57 -08:00
pavel-shirshov
4b42a48f45
Update frr to latest 7.2.1 (#4145) 2020-02-13 18:45:37 -08:00
Olivier Singla
6a0dcb1b16
[kernel]: security kernel update to 4.9.189 (#3913)
This patch upgrade the kernel from version
4.9.0-9-2 (4.9.168-1+deb9u3) to 4.9.0-11-2 (4.9.189-3+deb9u2)

Co-authored-by: rajendra-dendukuri <47423477+rajendra-dendukuri@users.noreply.github.com>
2020-02-12 17:41:58 -08:00
yozhao101
729f343f77
[Services] Restart database service upon unexpected critical process exit. (#4138)
* [database] Implement the auto-restart feature for database container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [database] Remove the duplicate dependency in service files. Since we
already have updategraph ---> config_setup ---> database, we do not need
explicitly add database.service in all other container service files.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [event listener] Reorganize the line 73 in event listener script.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [database] update the file sflow.service.j2 to remove the duplicate
dependency.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [event listener] Add comments in event listener.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [event listener] Update the comments in line 56.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [event listener] Add parentheses for if statement in line 76 in event listener.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2020-02-11 14:03:02 -08:00
pavel-shirshov
c63b5fc719
[frr]: Use tag for building frr (#4082)
Changed logic for building frr. Previously we used latest commit in the FRR_BRANCH. Now the buildsystem will use a tag to identify a commit for building. New approach will let us to update sonic-frr without corrupting building sonic-buildimage.
2020-02-07 07:51:01 -08:00
Mykola F
f8e530577d
[docker] remove leftover --net=host docker run options (#4119)
Fixes #4118

Signed-off-by: Mykola Faryma <mykolaf@mellanox.com>
2020-02-06 16:27:20 -08:00
lguohan
2e0f9ca0f4
[hsflowd]: remove linux kernel header dependency from hsflowd package (#4111)
hsflowd compilation does not linux kernel header

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-02-05 19:04:19 -08:00
Kiran Kumar Kella
97165a0d69
Changes in sonic-buildimage to support the NAT feature (#3494)
* Changes in sonic-buildimage for the NAT feature
- Docker for NAT
- installing the required tools iptables and conntrack for nat

Signed-off-by: kiran.kella@broadcom.com

* Add redis-tools dependencies in the docker nat compilation

* Addressed review comments

* add natsyncd to warm-boot finalizer list

* addressed review comments

* using swsscommon.DBConnector instead of swsssdk.SonicV2Connector

* Enable NAT application in docker-sonic-vs
2020-01-29 17:40:43 -08:00
SuvarnaMeenakshi
c9483796dc [baseimage]: support building multi-asic component (#3856)
- move single instance services into their own folder
- generate Systemd templates for any multi-instance service files in slave.mk
- detect single or multi-instance platform in systemd-sonic-generator based on asic.conf platform specific file.
- update container hostname after creation instead of during creation (docker_image_ctl)
- run Docker containers in a network namespace if specified
- add a service to create a simulated multi-ASIC topology on the virtual switch platform

Signed-off-by: Lawrence Lee <t-lale@microsoft.com>
Signed-off-by: Suvarna Meenakshi <Suvarna.Meenaksh@microsoft.com>
2020-01-26 13:56:42 -08:00
pavel-shirshov
61858807ce [frr]: Update FRR to 7.2.1 (#4066)
Update rules for frr package.
Update frr submodule
2020-01-25 19:43:22 -08:00
pavel-shirshov
08f60f652a
Update lldpd to latest stable version (#3905)
* Change lldpd version

* Remove patches not-required patches
2020-01-24 11:52:52 -08:00
byu343
b45d2b4130 [arista]: Add support for bald eagle phy (#3922)
* Add support for bald edgle phy
2020-01-17 10:36:12 -08:00
Sumukha Tumkur Vani
ebb07d9942 [docker-restapi]: Avoid building RestAPI docker by default (#4018)
Provide build option for RESTAPI docker
2020-01-14 17:13:45 -08:00
padmanarayana
918fdc6c44 [sflow]: Upgrade hsflowd to v2.0.26-3 (#4020) 2020-01-14 14:16:33 -08:00
yozhao101
b7e48b422f [Services] Allow monit system tool to monitor the critical processes status running in various SONiC containers. (#3940)
* Add a monit config file for teamd container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* Add a copy mechanism to put the monit config file in teamd container
into base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* Add a monit config file for snmp container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* Add a copy mechanism to put the monit config file of snmp container into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* Add a monit config file for dhcp_relay container in the dir
base_image_files.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* Add a copy mechanism to put the monit config file of dhcp_relay
container into base image under /etc/monit/conf.d.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* Add a monit config file for router advertiser container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* Add a copy mechanism to put the monit config file of router advertiser
contianer into base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-Pmon] Add a monit config file for pmon container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-Pmon] Add a copy mechanism to put the monit config file into the
base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-lldp] Add a monit config file for lldp container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-lldp] Add a copy mechanism to put the monit config file into the
base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-bgp] Add a monit config file for BGP container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-bgp] Add a copy mechanism to put monit config file into the base
image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-swss] Add a monit config file for the swss container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-swss] Add a copy mechanism to put monit config file into the
base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a monit config file for syncd container on barefoot
platform.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image on barefoot.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a monit config file for syncd container on broadcom.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image on broadcom.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a monit config file for syncd container on cavium.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-centec] Add a monit config file for syncd container on centen
platform.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a monit config file for syncd container on centen
platform.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a monit config file for syncd container on marvell.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit conifg file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a monit config file for syncd container on
marvell-arm64.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image on marvell-arm64.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a monit config file for syncd container on
marvell-armhf.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a monit config file for syncd container on mellanox.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a monit config file for syncd container on nephos.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Add a copy mechanism to put the monit config file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-sflow] Add a monit config file for sflow container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-sflow] Add a copy mechanism to put the monit conifg file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-telemetry] Add a monit config file for telemetry container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-telemetry] Add a copy mechanism to put the monit config file
into the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-database] Add a monit config file for database container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-database] Add a copy mechanism to put the monit config file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-Dhcprelay] Change a typo.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-Dhcprelay] Change the process name in monit config file to
dhcrelay.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] There is no desserve process in syncd container on
barefoot.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] There is no process desserve in syncd container on
cavium.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] There is no process named desserve in syncd on centec.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] There is no process named desserve in syncd on marvell.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Should not delete the process desserve in syncd container
on marvell.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Delete the process dsserve in syncd on marvell.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Delete the process dsserve in syncd container on
marvell-arm64.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Delete the process dsserve in syncd container on
marvell-armhf.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Delete the process dsserve in syncd container on
mellanox.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-Radv] Change the process name to radvd.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-telemetry] Correct a typo in monit_telemetry.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-teamd] Delete the monit config file for teamd.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-teamd] Delete the mechanism to copy the monit config file into
base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-dhcprelay] Delete the monit config file for dhcp_relay
container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-dhcprelay] Delete the mechanism to copy the monit config file
into the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-radv] Delete the monit config file foe radv container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-radv] Delete the mechanism to copy the monit config file into
the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-bgp] change the monit config file for BGP container such that
monit only generates alert if the process is not running for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-snmp] Change the monit config file for snmp container such that
monit only generates alret if the process is not running for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-pmon] Change the monit config file for pmon container such that
monit only generates alert if the processes are not running for 5
minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-lldp] Change the monit config file for lldp container such that
monit only generates alerts if some processes are not running for 5
minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-pmon] Delete the monit config file for pmon container since some
of processes are not running depended on the type of box.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-pmon] Delete the copy mechanism to copy the monit config file
into the base image.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-lldp] Change the matching name for the process lldpd.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-swss] Change the monit config file for swss container such that
monit only generates alerts if the processes are not running for 5
minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Change the monit config file for syncd container on
barefoot such that monit only generates alerts if the process is not
running for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Correct a typo in monit config file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Change the monit config file for syncd container on
broadcom such that monit only generates alerts if the processes are not
running for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Change the monit config file for syncd container on
cavium such that monit only generates alerts if the process is not
running for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Change the monit config file for syncd container such
that monit only generates alerts if the process is not running for 5
minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Change the monit config file for syncd container on
marvell such that monit only generates alerts if the process is not
running for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Change the monit config file for syncd container on
marvell-arm64 such that monit only generates alerts if the process is
not running for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Change the monit config file for syncd container on
marvell-armhf such that monit will generate alert if the process is not
running for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Change the monit config file for syncd container on
mellanox such that monit only generates alerts if the process is not
running for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-sycnd] Change the monit config file for syncd container such
that monit only generates alerts if the processes are not running for 5
minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-sflow] Change the monit config file for sflow container such
that monit only generates alerts if the process is not running for 5
minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-telemetry] Change the monit config file for telemetry container
such that monit only generates alerts if the processes are not running
for 5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-database] Change the monit config file for database container
such that monit only generates alerts if the process is not running for
5 minutes.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-database] Use 4 spaces to replace 2 spaces in monit config file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-bgp] Use 4 spcess to replace 2 spaces in monit config file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-lldp] Use 4 spaces to replace 2 spaces in monit config file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-swss] Use 4 spaces to replace 2 space in monit config file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-sflow] Use 4 spaces to replace 2 spaces in monit config file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-snmp] Use 4 spaces to replace 2 spaces in monit config file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-telemetry] Use 4 spaces to replace 2 spaces in monit config
file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on barefoot.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on broadcom.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on cavium.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on centec.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on marvell.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Use 4 spaces to replace 2 spaces in the monit config file
on mellanox.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-syncd] Use 4 spaces to repalce 2 spaces in the monit config file
on nephos.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

* [Docker-bgp] Remove the trailing extra spaces in monit config file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2020-01-10 16:21:02 -08:00
Sumukha Tumkur Vani
3b1ee5dc2b
Changes to build restapi docker (#3993) 2020-01-10 13:46:32 -08:00
Tamer Ahmed
2658ab8add
[dhcp-relay]: Add DHCP Relay Monitor (#3886)
DHCP relay MONitor (dhcpmon) keeps track of DORA messages. If DHCP Relay
is detected to be not forwarding DORA message, dhcpmon will log such event
to syslog. Under the hood dhcpmon keeps counts of clients DR messages,
forwarded DR messages, DHCP server OA messages, and forwarded OA messages.
dhcpmon will check every 12 sec (configurable) if counts are monotonically
increasing and record snapshot of those counters. dhcpmon will report
discrepancies when detected between current counters and snapshot counters.

pull-request: https://github.com/Azure/sonic-buildimage/pull/3886
signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
2020-01-07 17:48:03 -08:00
tahmed-dev
df04809cb8
[libnl]: Debian Packaging libnl version 3.5.0 (#3967)
Packaging libnl 3.5.0 based off libnl 3.2.27 packaging. libnl contains various bug fixes that are nice to have.

pull-request: https://github.com/Azure/sonic-buildimage/pull/3967
signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
2020-01-06 17:01:28 -08:00
Joe LeVeque
24a0c46464
[monit] Build from source and patch to use MemAvailable value if available on system (#3875) 2019-12-30 18:25:57 -08:00
Prabhu Sreenivasan
87f70108cb SONiC Management Framework Release 1.0 (#3488)
* Added sonic-mgmt-framework as submodule / docker

* fix build issues

* update sonic-mgmt-framework submodule branch to master

* Merged changes 70007e6d2ba3a4c0b371cd693ccc63e0a8906e77..00d4fcfed6a759e40d7b92120ea0ee1f08300fc6

00d4fcfed6a759e40d7b92120ea0ee1f08300fc6 Modified environemnt variables

* Changes to build sonic-mgmt-framework docker

* bumped up sonic-mgmt-framework commit-id

* version bump for sonic-mgmt-framework commit-it

* bumped up sonic-mgmt-framework commit-id

* Add python packages to docker

* Build fix for docker with python packages

* added libyang as dependent package

* Allow building images on NFS-mounted clones

Prior to this change, `build_debian.sh` would generate a Debian
filesystem in `./fsroot`. This needs root permissions, and one of the
tests that is performed is whether the user can create a character
special file in the filesystem (using mknod).

On most NFS deployments, `root` is the least privileged user, and cannot
run mknod. Also, attempting to run commands like rm or mv as root would
fail due to permission errors, since the root user gets mapped to an
unprivileged user like `nobody`.

This commit changes the location of the Debian filesystem to `/fsroot`,
which is a tmpfs mount within the slave Docker. The default squashfs,
docker tarball and zip files are also created within /tmp, before being
copied back to /sonic as the regular user.

The side effect of this change is that the contents of `/fsroot` are no
longer available once the slave container exits, however they are
available within the squashfs image.

Signed-off-by: Nirenjan Krishnan <Nirenjan.Krishnan@dell.com>

* bumped up sonc-mgmt-framework commit to include PR #18

*     REST Server startup script is enahnced to read the settings from
    ConfigDB. Below table provides mapping of db field to command line
    argument name.

    ============================================================
    ConfigDB entry key      Field name      REST Server argument
    ============================================================
    REST_SERVER|default     port            -port
    REST_SERVER|default     client_auth     -client_auth
    REST_SERVER|default     log_level       -v
    DEVICE_METADATA|x509    server_crt      -cert
    DEVICE_METADATA|x509    server_key      -key
    DEVICE_METADATA|x509    ca_crt          -cacert
    ============================================================

* Replace src/telemetry as submodule to sonic-telemetry

* Update telemetry commit HEAD

* Update sonic-telemetry commit HEAD

* libyang env path update

* Add libyang dependency to telemetry

* Add scripts to create JSON files for CLI backend

Scripts to create /var/platform/syseeprom and /var/platform/system, which are back-end
files for CLI, for system EEPROM and system information.

Signed-off-by: Howard Persh <Howard_Persh@dell.com>

* In startup script, create directory where CLI back-end files live

Signed-off-by: Howard Persh <Howard_Persh@dell.com>

* build dependency pkgs added to docker for build failure fix

* Changes to fix build issue for mgmt framework

* Fix exec path issue with telemetry

* s5232[device] PSU detecttion and default led state support

* Processing of first boot in rc.local should not have premature exit

Signed-off-by: Howard Persh <Howard_Persh@dell.com>

*  docker mount options added for platform, system features

* bumped up sonic-mgmt-framework commit id to pick 23rd July 2019 changes

* Added mount options for telemetry docker to get access for system and platform info.

* Update commit for sonic-utilities

* [dell]: Corrected dport map and renamed config files for S5232F

* Fix telemetry submodule commit

* added support for sonic-cli console

* [Dell S5232F, Z9264F] Harden FPGA driver kernel module

For Dell S5232F and Z9264F platforms, be more strict when checking state
in ISR of FPGA driver, to harden against spurious interrupts.

Signed-off-by: Howard Persh <Howard_Persh@dell.com>

* update mgmt-framework submodule to 27th Aug commit.

* remove changes not related to mgmt-framework and sonic-telemetry

* Revert "Replace src/telemetry as submodule to sonic-telemetry"

This reverts commit 11c3192975.

* Revert "Replace src/telemetry as submodule to sonic-telemetry"

This reverts commit 11c3192975.

* make submodule changes and remove a change not related to PR

* more changes

* Update .gitmodules

* Update Dockerfile.j2

* Update .gitmodules

* Update .gitmodules

* Update .gitmodules

reverting experimental change

* Removed syspoll for release_1.0

Signed-off-by: Jeff Yin <29264773+jeff-yin@users.noreply.github.com>

* Update docker-sonic-mgmt-framework.mk

* Update sonic-mgmt-framework.mk

* Update sonic-mgmt-framework.mk

* Update docker-sonic-mgmt-framework.mk

* Update docker-sonic-mgmt-framework.mk

* Revert "Processing of first boot in rc.local should not have premature exit"

This reverts commit e99a91ffc2.

* Remove old telemetry directory

* Update docker-sonic-mgmt-framework.mk

* Resolving merge conflict with Azure

* Reverting the wrong merge

* Use CVL_SCHEMA_PATH instead of changing directory for telemetry startup

* Add missing export

* Add python mmh3 to slave dockerfile

* Remove sonic-mgmt-framework build dep for telemetry, fix dialout startup issues

* Provided flag to disable compiling mgmt-framework

* Update sonic-utilites point latest commit id

* Point sonic-utilities to Azure accepted SHA

* Updating mgmt framework to right sha

* Add sonic-telemetry submodule

* Update the mgmt-framework commit id

Co-authored-by: jghalam <joe.ghalam@gmail.com>
Co-authored-by: Partha Dutta <51353699+dutta-partha@users.noreply.github.com>
Co-authored-by: srideepDell <srideep_devireddy@dell.com>
Co-authored-by: nirenjan <nirenjan@users.noreply.github.com>
Co-authored-by: Sachin Holla <51310506+sachinholla@users.noreply.github.com>
Co-authored-by: Eric Seifert <seiferteric@gmail.com>
Co-authored-by: Howard Persh <hpersh@yahoo.com>
Co-authored-by: Jeff Yin <29264773+jeff-yin@users.noreply.github.com>
Co-authored-by: Arunsundar Kannan <31632515+arunsundark@users.noreply.github.com>
Co-authored-by: rvasanthm <51932293+rvasanthm@users.noreply.github.com>
Co-authored-by: Ashok Daparthi-Dell <Ashok_Daparthi@Dell.com>
Co-authored-by: anand-kumar-subramanian <51383315+anand-kumar-subramanian@users.noreply.github.com>
2019-12-23 21:47:16 -08:00
pavel-shirshov
1848fb262b [fast-reboot]: Save fast-reboot state into the db (#3741)
Put a flag for fast-reboot to the db using EXPIRE feature. Using this flag in other part of SONiC to start in Fast-reboot mode. If we reload a config, the state in the db will be removed.
2019-12-04 14:10:19 -08:00
rajendra-dendukuri
eec594adf2 [sonic-ztp]: Build sonic-ztp package (#3299)
* Build sonic-ztp package

- Add changes in make rules to conditionally include sonic-ztp package

Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
2019-12-04 04:50:56 -08:00
yozhao101
67fc68513e [Services] Restart Sflow service upon unexpected critical process exit. (#3751)
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2019-11-25 13:02:00 -08:00
padmanarayana
62a7846960 [sFlow]: Upgrade hsflowd to 2.0.26-1 (#3812) 2019-11-24 15:21:43 -08:00
Andriy Kokhan
f943440fcb Moved telemetry exit listener from process package into Docker (#3805)
Signed-off-by: Andriy Kokhan <akokhan@barefootnetworks.com>
2019-11-22 14:28:40 -08:00
yozhao101
df11b2b9f1 [Services] Restart Telemetry service upon unexpected critical process exit. (#3768)
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2019-11-18 16:56:44 -08:00
Olivier Singla
c70d8bca9f [baseimage]: kdump support (#3722)
* In the event of a kernel crash, we need to gather as much information
as possible to understand and identify the root cause of the crash.
Currently, the kernel does not provide much information, which make
kernel crash investigation difficult and time consuming.

Fortunately, there is a way in the kernel to provide more information
in the case of a kernel crash. kdump is a feature of the Linux kernel
that creates crash dumps in the event of a kernel crash. This PR
will add kermel kdump support.

An extension to the CLI utilities config and show is provided to
configure and manage kdump:
 - enable / disable kdump functionality
 - configure kdump (how many kernel crash logs can be saved, memory
   allocated for capture kernel)
 - view kernel crash logs
2019-11-08 23:08:42 -08:00
Ping Mao
bdf1b7c607 libyang 1.0.73 (#3710)
- build libyang1.0.73 debian pacakge from libyang github source
    - build libyang python2 and python3 debian packages
2019-11-07 13:45:38 -08:00
pavel-shirshov
aa1a13677d
[frr]: Move to version 7.2 (#3704)
* Use 7.2 tree to generate frr packages

* Adapt patches for frr/7.2

* Use vrf_id
2019-11-06 11:57:23 -08:00
yozhao101
a117b25446 [Services] Restart LLDP service upon unexpected critical process exit. (#3713)
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2019-11-06 11:02:57 -08:00
yozhao101
ed79f54569 [Services] Restart DHCP-Relay service upon unexpected critical process exit. (#3667)
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2019-11-05 18:32:14 -08:00
yozhao101
4c31ef3cd2 [Services] Restart Teamd service upon unexpected critical process exit. (#3703)
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2019-11-04 17:45:41 -08:00
yozhao101
4fa3a1e27e [Services] Restart Platform-monitor service upon unexpected critical process exit. (#3689)
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2019-11-04 17:44:01 -08:00
padmanarayana
841949f099 [hsflowd]: Update hsflowd to 2.0.25-4 to take care of admin_state changes (#3683) 2019-10-31 09:15:30 -07:00
yozhao101
cff30c59d0 [Services] Restart Router-advertiser service upon unexpected critical process exit (#3681)
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2019-10-30 16:41:55 -07:00
yozhao101
a0fbeeaca5 [Services] Restart SNMP service upon unexpected critical process exit. (#3650)
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
2019-10-22 14:41:12 -07:00
arheneus@marvell.com
2694e66074 [build]: build ARM kernel support from sonic-linux-kernel (#3556)
* Makefile: ARM kernel support from sonic-linux-kernel
        * Fix for multiarch  build docker spawn
Platform: Install the DTB deb for the platform

Signed-off-by: Antony Rheneus <arheneus@marvell.com>
2019-10-10 15:11:26 -07:00
pavel-shirshov
ce2ecf2680
[frr]: Update frr version to 7.1 (#3575)
* Build 7.1 without patches

* Port patches
2019-10-08 07:30:27 -07:00
Andriy Moroz
976850fc00 [submodule update] Add SSD Health tools (#3218)
Signed-off-by: Andriy Moroz <c_andriym@mellanox.com>
2019-10-04 10:52:58 -07:00
Stephen Sun
c34a4783e0 [build] install new platform api on host (#3282)
slave.mk: add SONIC_PLATFORM_API_PY2 as dependency of host
sonic_debian_extension.j2: install sonic_daemon_base and Mellanox-specific sonic_platform on host
mlnx-platform-api.mk: export mlnx_platform_api_py2_wheel_path for sonic_debian_extension.j2
sonic-daemon-base.mk: export daemon_base_py2_wheel_path for sonic_debian_extension.j2
daemon_base.py: hind unnecessary dependency of swss_common on host
2019-09-25 11:00:24 -07:00
yangshiping
503d87b901 [lldp]: expose lldpcli command of lldp docker to host (#3473)
Lldp can not be configured in host environment, so we expose lldpcli command of lldp docker to host for configuring more convenient.
2019-09-18 11:39:31 -07:00
padmanarayana
75104bb35d [sflow]: Build infrastructure changes to support sflow docker and utilities (#3251)
Introduce a new "sflow" container (if ENABLE_SFLOW is set). The new docker will include:
hsflowd : host-sflow based daemon is the sFlow agent
psample : Built from libpsample repository. Useful in debugging sampled packets/groups.
sflowtool : Locally dump sflow samples (e.g. with a in-unit collector)

In case of SONiC-VS, enable psample & act_sample kernel modules.

VS' syncd needs iproute2=4.20.0-2~bpo9+1 & libcap2-bin=1:2.25-1 to support tc-sample

tc-syncd is provided as a convenience tool for debugging (e.g. tc-syncd filter show ...)
2019-09-14 20:27:09 -07:00
Greg Paussa
a2cd26a2e1 [build] Avoid race condition in sonic-platform-common (#3457)
Both python2 and python3 wheels being built out of the same
source directory can interfere with each other when
SONIC_BUILD_JOBS > 1.

Signed-off-by: Greg Paussa <greg.paussa@broadcom.com>
2019-09-13 22:09:34 -07:00
zzhiyuan
2095d6ec03 [devices]: Update arista submodule (#3431)
- Add support for fans in new platform API
- Mount platform API in pmon for arista platforms
2019-09-10 14:37:05 -07:00
Ying Xie
8b674c8da4
[kernel] Upgrade kernel to version 4.9.168-1+deb9u5 (#3369)
* [kernel] Upgrade kernel to version 4.9.168-1+deb9u5

- Picking up security patches.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* advance linux-kernel sub module head
2019-08-21 09:16:55 -07:00
Jipan Yang
0ec5de4fb3 [build]: Fix INSTALL_DEBUG_TOOLS build config option (#2564)
* Fix INSTALL_DEBUG_TOOLS build config option

Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>
2019-08-15 02:20:18 -07:00
arheneus@marvell.com
1ca75d320f [Makefile] multi arch variable fix (#3263)
Signed-off-by: Antony Rheneus <arheneus@marvell.com>
2019-08-01 16:25:09 -07:00
Stepan Blyshchak
67463f18b2 [docker-sonic-mgmt] install ptf framework in mgmt container and upgrade ptf (#3239)
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2019-07-30 10:19:48 -07:00
Lawrence Lee
7271fe598f [build]: Move Systemd service start to systemd generator (#3172)
- What I did

 Move the enabling of Systemd services from sonic_debian_extension to a new systemd generator

- How I did it

  Create a new systemd generator to manually create symlinks to enable systemd services
  Add rules/Makefile to build generator
  Add services to be enabled to /etc/sonic/generated_services.conf to be read by the generator at boot time

Signed-off-by: Lawrence Lee <t-lale@microsoft.com>
2019-07-29 15:52:15 -07:00
pavel-shirshov
b4517b9591
[bgp]: Implement Universal Traffic Shift for SONiC (#3209)
* [bgp]: Implement Universal Traffic Shift for SONiC

* Fix issue with ipv6 loopback match

* Add tests
2019-07-26 14:31:56 -07:00
arheneus@marvell.com
50fe458592 [build]: SONiC buildimage ARM arch support (#2980)
ARM Architecture support in SONIC

make configure platform=[ASIC_VENDOR_ARCH] PLATFORM_ARCH=[ARM_ARCH]
SONIC_ARCH: default amd64
armhf - arm32bit
arm64 - arm64bit

Signed-off-by: Antony Rheneus <arheneus@marvell.com>
2019-07-25 22:06:41 -07:00
Andriy Kokhan
4eb356874e [platform]: fixed syseeprom access for BFN platform (#3216)
Signed-off-by: Andriy Kokhan <akokhan@barefootnetworks.com>
2019-07-25 08:43:30 -07:00
rajendra-dendukuri
40c8bc14cd [baseimage]: Upgrade ifupdown2 to version 1.2.8 (#3180)
* Upgrade ifupdown2 to version 1.2.8

Required by ZTP to support ZTP over IPv6 transport

Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
2019-07-19 23:09:14 -07:00
Renuka Manavalan
6c4ec396ba [debug]: Fix src path mistake for sonic-swss. (#3186) 2019-07-19 06:55:39 -07:00
Stepan Blyshchak
e4300b8f76 [sairedis.mk] do not build rpc sairedis when not needed (#3109)
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2019-07-06 14:55:31 -07:00
lguohan
cf063bd3af
[redis]: build redis-tools-dbgsym instead of redis-server-dbgsym (#3125)
redis-server-dbgsym is not available since redis server 4
2019-07-05 11:00:01 -07:00
Jipan Yang
9a8202a39d [database]: Update redis to 5.0.3 (#3066)
Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>
2019-07-03 22:16:09 -07:00
Renuka Manavalan
76bf5a0bc4 [build]: Added debug symbols to many debug dockers. (#3098)
* Added debug symbols to many debug dockers.

* For debug images *only*:
1) Archive source files into debug image
2) Archived source is copied into /src
3) Created an empty dir /debug
4) Mount both /src as ro & /debug as rw into every docker
5) Login banner will give some details on /src & /debug
6) Devs can copy core file into /debug and view it from inside a container.
7) Dev may create all gdb logs and other data directly into /debug.

* Dropped redundant REDIS_TOOLS per review comments.

* Added debug symbols to frr package and hence FRR based BGP docker.

* 1) Moved dbg_files.sh to scripts/
2) Src directories to archive are now collected from individual Makefiles.
3) Added few more debug symbols
4) Added few more debug dockers.

Here after no more changes except per review comments.

To debug:
Install required version of debug image in Switch or VM.
Copy core file into /debug of host
Get into Docker
gdb /usr/bin/<daemon> -c /debug/<your core file>
set directory /src/... <-- inside gdb to get the source

For non-in-depth debugging:

Download corresponding debug Docker image (docker-...-dbg.gz) to your VM
Load the image
Run image with entrypoint as 'bash' with dir containing core mapped in.
Run gdb on the core.
2019-07-03 22:13:55 -07:00
Stepan Blyshchak
1bcd54d38c [build]: Upgrade docker to 18.09 in stretch slave (#3049)
Also add an option to use docker buildkit for image build

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2019-06-20 09:19:33 -07:00
Qi Luo
69b8ad7be0 [submodule] update sonic-linux-kernel to 4.9.168-1+deb9u3 (#3034) 2019-06-19 14:20:38 -07:00
Michel Moriniaux
18544530d3 [FRR] Enable SNMP support (#2981)
This is a follow-up of sonic-snmpagent PR 92
Now that licensing issues have been solved FRR is distributed with SNMP
support compiled-in. This PR adds the last bits of configuration to get
the frr-snmp debian packages added to the docker container and the
config bits to enable the snmp module in FRR

This PR brings the functionality of being able to poll bgpd for routes
and peer status.

Signed-off-by: Michel Moriniaux <m.moriniaux@criteo.com>
2019-06-19 01:24:42 -07:00
Kebo Liu
8a08595006 [Pmon] Add new daemon "syseepromd" to pmon docker (#2866) 2019-06-18 11:02:24 -07:00