Commit Graph

70 Commits

Author SHA1 Message Date
Tamer Ahmed
c9c2826520 Merged PR 3845699: [linkmgrd]: Introduce MUX cable linkmgrd
Linkmgrd monitors link status, mux status, and link state. Has
the link becomes unhealthy, linkmgrd will trigger mux switchover
on a standby ToR ensuring uninterrupted service to servers/blades.
This PR is initial implementation of linkmgrd.

Also, docker-mux container hold packages related to maintaining and managing
mux cable. It currently runs linkmgrd binary that monitor and switches
the mux if needed.
This PR also introduces mux-container and starts linkmgrd as startup when
build is configured with INCLUDE_MUX=y

Edit: linkmgrd PR will follow.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

Related work items: #2315, #3146150
2021-10-15 09:59:59 -07:00
SuvarnaMeenakshi
1646678fb8
[vs][multi-asic]: Add support to build multi-asic vs images (#8270)
Why I did it
Pre-requisite: #8269
To be able to generate multi-asic KVM image.
To provide flexibility to generate single asic or both single and multi-asic images.
How I did it
Add a new build param, BUILD_MULTIASIC_KVM, if set to "y", the multi-asic VS target KVM images will be generated. If not, only single asic VS image will be generated.
Make changes to build_image.sh to generate 4-asic and 6-asic KVM images if BUILD_MULTIASIC_KVM parameter is set to y
How to verify it
Generate single-asic VS as currently done, no change in build steps:
make configure PLATFORM=vs
make target/sonic-vs.img.gz - will generate only single asic KVM image.

make BUILD_MULTIASIC_KVM=y target/sonic-vs.img.gz - will generate single asic and multi-asic KVM images.
should generate:
sonic-vs.bin
sonic-vs.img.gz
sonic-4asic-vs.img.gz
sonic-6asic-vs.img.gz
2021-09-01 17:28:13 -07:00
賓少鈺
aa59bfeab7
[PDE]: introduce the SONiC Platform Development Env (#7510)
The PDE silicon test harness and platform test harness can be found in
src/sonic-platform-pdk-pde
2021-07-24 16:24:43 -07:00
Renuka Manavalan
3a96eb933e
Get Docker proxy info from config (#8205)
This helps not to hard code the docker proxy IP, but take it from config file during build time.
2021-07-19 21:17:47 -07:00
Renuka Manavalan
c5dff0c640
Revert "Revert "[Kubernetes]: The kube server could be used as http-proxy for docker (#7469)" (#8023)" (#8158)
This reverts commit 7236fa98e8.

Restore original PR #7469
2021-07-15 19:48:55 -07:00
Stepan Blyshchak
b3b6938fda
[dhcp-relay] make DHCP relay an extension (#6531)
- Why I did it
Make DHCP relay docker an extension. DHCP relay now carries dhcp relay commands CLI plugin and has a complete manifest.
It is installed as extension if INCLUDE_DHCP_REALY is set to y.

DEPENDS on #5939

- How I did it
Modify DHCP relay docker makefile and dockerfile. Make changes to sonic_debian_extension.j2 to install sonic packages.
I moved DHCP related CLI tests from sonic-utilities to DHCP relay docker.
This PR introduces a way to write a plugin as part of docker image and run the tests from cli-plugin-tests directory under docker directory.
The test result is available in target/docker-dhcp-relay.gz.log:

[ REASON ] :      target/docker-dhcp-relay.gz does not exist   NON-EXISTENT PREREQUISITES: docker-start target/docker-config-engine-buster.gz-load target/python-wheels/sonic_utilities-1.2-py3-none-any.whl-in
stall target/debs/buster/python3-swsscommon_1.0.0_amd64.deb-install
[ FLAGS  FILE    ] : []
[ FLAGS  DEPENDS ] : []
[ FLAGS  DIFF    ] : []
============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-3.10.1, py-1.7.0, pluggy-0.8.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /sonic/dockers/docker-dhcp-relay/cli-plugin-tests, inifile:
plugins: cov-2.6.0
collecting ... collected 10 items

test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_plugin_registration PASSED [ 10%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_nonexist_vlanid PASSED [ 20%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_invalid_vlanid PASSED [ 30%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_invalid_ip PASSED [ 40%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_exist_ip PASSED [ 50%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_del_dhcp_relay_dest PASSED [ 60%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_remove_nonexist_dhcp_relay_dest PASSED [ 70%]
test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_remove_dhcp_relay_dest_with_nonexist_vlanid PASSED [ 80%]
test_show_dhcp_relay.py::TestVlanDhcpRelay::test_plugin_registration PASSED [ 90%]
test_show_dhcp_relay.py::TestVlanDhcpRelay::test_dhcp_relay_column_output PASSED [100%]

=============================== warnings summary ===============================
/usr/local/lib/python3.7/dist-packages/tabulate.py:7
  /usr/local/lib/python3.7/dist-packages/tabulate.py:7: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import namedtuple, Iterable

-- Docs: https://docs.pytest.org/en/latest/warnings.html
==================== 10 passed, 1 warnings in 0.35 seconds =====================
2021-07-15 10:35:56 -07:00
jusherma
ad3f861a77
[build]: Allow build-time specification of alternative docker registries (#8134)
Why I did it
Allows users to host their own local docker registries and utilize them via the REGISTRY_SERVER and REGISTRY_PORT environmental variables

How I did it
Only set REGISTRY_SERVER and REGISTRY_PORT in rules/config if they are unset.

How to verify it
Export environmental variables REGISTRY_SERVER and REGISTRY_PORT to an alternative docker registry. Export the environmental variable ENABLE_DOCKER_BASE_PULL to y.
Ensure the required sonic-slave docker images are not present locally, but are available in the docker registry
Execute make init and make configure
Confirm that the appropriate docker images were pulled from the appropriate docker registry, and not built locally
2021-07-14 01:27:25 -07:00
Ying Xie
7236fa98e8
Revert "[Kubernetes]: The kube server could be used as http-proxy for docker (#7469)" (#8023)
This change causes nightly test to fail due to the fake proxy IP is not reachable.

Reverts #7469

This reverts commit f7ed82f44a.
2021-06-29 18:43:53 -07:00
Shi Su
f52ba3b496
Remove quagga-related code (#7898)
Why I did it
Quagga is no longer being used. Remove quagga-related code (e.g., docker-fpm-quagga, sonic-quagga, etc.).

How I did it
Remove quagga-related code.
2021-06-23 09:15:56 -07:00
Renuka Manavalan
f7ed82f44a
[Kubernetes]: The kube server could be used as http-proxy for docker (#7469)
Why I did it
The SONiC switches get their docker images from local repo, populated during install with container images pre-built into SONiC FW. With the introduction of kubernetes, new docker images available in remote repo could be deployed. This requires dockerd to be able to pull images from remote repo.

Depending on the Switch network domain & config, it may or may not be able to reach the remote repo. In the case where remote repo is unreachable, we could potentially make Kubernetes server to also act as http-proxy.

How I did it
When admin explicitly enables, the kubernetes-server could be configured as docker-proxy. But any update to docker-proxy has to be via service-conf file environment variable, implying a "service restart docker" is required. But restart of dockerd is vey expensive, as it would restarts all dockers, including database docker.

To avoid dockerd restart, pre-configure an http_proxy using an unused IP. When k8s server is enabled to act as http-proxy, an IP table entry would be created to direct all traffic to the configured-unused-proxy-ip to the kubernetes-master IP. This way any update to Kubernetes master config would be just manipulating IPTables, which will be transparent to all modules, until dockerd needs to download from remote repo.

How to verify it
Configure a switch such that image repo is unreachable
Pre-configure dockerd with http_proxy.conf using an unused IP (e.g. 172.16.1.1)
Update ctrmgrd.service to invoke ctrmgrd.py with "-p" option.
Configure a k8s server, and deploy an image for feature with set_owner="kube"
Check if switch could successfully download the image or not.
2021-06-16 07:46:01 -07:00
Qi Luo
658ed4fd37
Revert "Remove quagga related code (#7476)" (#7831)
Reverts Azure/sonic-buildimage#7476
It remove bgpd.conf.j2 and zebra.conf.j2, which is still used by sonic-config-engine unit test.
2021-06-09 18:52:45 -07:00
Shi Su
62a4603eef
Remove quagga related code (#7476)
Why I did it
Quagga is no longer being used. Remove quagga-related code (e.g., docker-fpm-quagga, sonic-quagga, etc.).

How I did it
Remove quagga-related code.
2021-06-07 16:44:54 -07:00
Ze Gan
4068944202
[MACsec]: Set MACsec feature to be auto-start (#6678)
1. Add supervisord as the entrypoint of docker-macsec
2. Add wpa_supplicant conf into docker-macsec
3. Set the macsecmgrd as the critical_process
4. Configure supervisor to monitor macsecmgrd
5. Set macsec in the features list
6. Add config variable `INCLUDE_MACSEC`
7. Add macsec.service

**- How to verify it**

Change the `/etc/sonic/config_db.json` as follow
```
{
    "PORT": {
        "Ethernet0": {
            ...
            "macsec": "test"
         }
    }
    ...
    "MACSEC_PROFILE": {
        "test": {
            "priority": 64,
            "cipher_suite": "GCM-AES-128",
            "primary_cak": "0123456789ABCDEF0123456789ABCDEF",
            "primary_ckn": "6162636465666768696A6B6C6D6E6F707172737475767778797A303132333435",
            "policy": "security"
        }
    }
}
```
To execute `sudo config reload -y`, We should find the following new items were inserted in app_db of redis
```
127.0.0.1:6379> keys *MAC*
1) "MACSEC_EGRESS_SC_TABLE:Ethernet0:72152375678227538"
2) "MACSEC_PORT_TABLE:Ethernet0"
127.0.0.1:6379> hgetall "MACSEC_EGRESS_SC_TABLE:Ethernet0:72152375678227538"
1) "ssci"
2) ""
3) "encoding_an"
4) "0"
127.0.0.1:6379> hgetall "MACSEC_PORT_TABLE:Ethernet0"
 1) "enable"
 2) "false"
 3) "cipher_suite"
 4) "GCM-AES-128"
 5) "enable_protect"
 6) "true"
 7) "enable_encrypt"
 8) "true"
 9) "enable_replay_protect"
10) "false"
11) "replay_window"
12) "0"
```

Signed-off-by: Ze Gan <ganze718@gmail.com>
2021-02-23 13:22:45 -08:00
Arun Saravanan Balachandran
3015de1dd0
[sonic-host-service] Move to sonic-host-services package (#6273)
- Why I did it

To move ‘sonic-host-service’ which is currently built as a separate package to ‘sonic-host-services' package. 

- How I did it

- Moved 'sonic-host-server' to 'src/sonic-host-services' and included it as part of the python3 wheel.
- Other files were moved to 'src/sonic-host-services-data' and included as part of the deb package.
- Changed build option ‘INCLUDE_HOST_SERVICE’ to ‘ENABLE_HOST_SERVICE_ON_START’ for enabling sonic-hostservice at boot-up by default.
2021-02-08 19:35:08 -08:00
lguohan
a79fcb49d4
[build]: add option to pull sonic-slave docker from registry (#6300)
- add config option ENABLE_DOCKER_BASE_PULL to pull sonic-slave docker from registry
- use REGISTRY_PORT, REGISTRY_SERVER to specify docker registry

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-12-26 10:30:51 -08:00
xumia
0a36de3a89
Recover "Support SONiC Reproduceable Build-debian/pip/web packages (#6255)
* Revert "Revert "Support SONiC Reproduceable Build-debian/pip/web packages (#5718)""

This reverts commit 17497a65e3.

* Revert "Revert "Remove unnecessary sudo authority in build Makefile (#6237)""

This reverts commit 163b7111b5.
2020-12-21 15:31:10 +08:00
Guohan Lu
17497a65e3 Revert "Support SONiC Reproduceable Build-debian/pip/web packages (#5718)"
This reverts commit 55a707586b.
2020-12-18 23:37:27 -08:00
xumia
55a707586b
Support SONiC Reproduceable Build-debian/pip/web packages (#5718)
* Support SONiC reproduceable build for deb/py2/py3/web

* Remove j2 files

* Fix bug

* Fix some issues

1. Change some code format issues
2. Fix curl calling wget command, pip2 calling pip3 issue
3. Fix wget/curl downloading multiple urls issue

* Fix some code format issue

* Fix bug

* Fix bug

* Fix command path hard code in build info scripts issue

* Add debian package sonic-build-tools

* Fix auto debian package removed issue

* Change build debian package name, and change the folder

* Collect the pre-versions and post-versions

* Change to use debian:buster

* Remove apt-mark and improve code

* Remove set_build_hooks

* Change docker trusted gpg files

* Fix docker build COPY directory name issue

* Move the trusted gpg files into the sonic-build-hooks package
2020-12-17 13:06:53 +08:00
fk410167
a3dd3f55f9
Platform Driver Developement Framework (PDDF) (#4756)
This change introduces PDDF which is described here: https://github.com/Azure/SONiC/pull/536

Most of the platform bring up effort goes in developing the platform device drivers, SONiC platform APIs and validating them. Typically each platform vendor writes their own drivers and platform APIs which is very tailor made to that platform. This involves writing code, building, installing it on the target platform devices and testing. Many of the details of the platform are hard coded into these drivers, from the HW spec. They go through this cycle repetitively till everything works fine, and is validated before upstreaming the code.
PDDF aims to make this platform driver and platform APIs development process much simpler by providing a data driven development framework. This is enabled by:

JSON descriptor files for platform data
Generic data-driven drivers for various devices
Generic SONiC platform APIs
Vendor specific extensions for customisation and extensibility

Signed-off-by: Fuzail Khan <fuzail.khan@broadcom.com>
2020-11-12 10:22:38 -08:00
shi-su
339cfbf9af
Remove the configuration of synchronous mode from init_cfg.json (#5308)
Remove the configuration of synchronous mode from init_cfg.json
2020-09-10 01:26:10 -07:00
shi-su
f3feb56c8a
Add switch for synchronous mode (#5237)
Add a master switch so that the sync/async mode can be configured.
Example usage of the switch:
1.  Configure mode while building an image
    `make ENABLE_SYNCHRONOUS_MODE=y <target>`
2. Configure when the device is running 
    Change CONFIG_DB with `sonic-cfggen -a '{"DEVICE_METADATA":{"localhost": {"synchronous_mode": "enable"}}}' --write-to-db`
    Restart swss with `systemctl restart swss`
2020-08-24 14:04:10 -07:00
nirenjan
bb57ccecd4
[sonic-host-service]: Add SONiC Host Services infrastructure (#4840)
- Why I did it

When SONiC is configured with the management framework and/or telemetry services, the applications running inside those containers need to access some functionality on the host system. The following is a non-exhaustive list of such functionality:

Image management
Configuration save and load
ZTP enable/disable and status
Show tech support
- How I did it

The host service is a Python process that listens for requests via D-Bus. It will then service those requests and send a response back to the requestor.

This PR only introduces the host service infrastructure. Applications that need access to the host services must add applets that will register on D-Bus endpoints to service the appropriate functionality.

- How to verify it

- Description for the changelog

Add SONiC Host Service for container to execute select commands in host

Signed-off-by: Nirenjan Krishnan <Nirenjan.Krishnan@dell.com>
2020-08-21 15:34:14 -07:00
isabelmsft
19a3452ddc
[Kubernetes Setup] Remove flannel, kube-proxy images (#5098)
Removes installation of kube-proxy (117 MB) and flannel (53 MB) images from Kubernetes-enabled devices. These images are tested to be unnecessary for our use case, as we do not rely on ClusterIPs for Kubernetes Services or a CNI for pod networking.
2020-08-06 18:23:27 -05:00
lguohan
082c26a27d
[build]: combine feature and container feature table (#5081)
1. remove container feature table
2. do not generate feature entry if the feature is not included
   in the image
3. rename ENABLE_* to INCLUDE_* for better clarity
4. rename feature status to feature state
5. [submodule]: update sonic-utilities

* 9700e45 2020-08-03 | [show/config]: combine feature and container feature cli (#1015) (HEAD, origin/master, origin/HEAD) [lguohan]
* c9d3550 2020-08-03 | [tests]: fix drops_group_test failure on second run (#1023) [lguohan]
* dfaae69 2020-08-03 | [lldpshow]: Fix input device is not a TTY error (#1016) [Arun Saravanan Balachandran]
* 216688e 2020-08-02 | [tests]: rename sonic-utilitie-tests to tests (#1022) [lguohan]

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-08-05 13:23:12 -07:00
isabelmsft
55d3ecd267
Update Kubernetes and kubernetes-cni versions (#5024)
This PR updates kubernetes version to 1.18.6 and kubernetes-cni version to 0.8.6

signed-off by: Isabel Li isabel.li@microsoft.com

Why I did it
Previous kubernetes-cni version (0.7.5) introduced Kubernetes Man In The Middle Vulnerability. “A vulnerability was found in all versions of containernetworking/plugins before version 0.8.6, that allows malicious containers in Kubernetes clusters to perform man-in-the-middle (MitM) attacks. A malicious container can exploit this flaw by sending rogue IPv6 router advertisements to the host or other containers, to redirect traffic to the malicious container.”

How I did it
Defined kubernetes-cni version to be 0.8.6 and updated kubernetes version to be 1.18.6

How to verify it
Check versions by running dpkg -l | grep kube
2020-07-24 10:14:24 -07:00
Eric Seifert
de07029352
Add TELEMETRY_WRITABLE build option to enable telemetry write mode (default read-only) (#4209)
Add build flag TELEMETRY_WRITABLE. When set to "y" it will add a go build flag in the telemetry build that will enable telemetry write mode to allow configuration via gNMI Set RPC as well as operations via the gNOI RPC's. The default for TELEMETRY_WRITABLE is unset in which case telemetry is read-only. In read-only mode the Set RPC and all gNOI RPC's are disabled and will return an "Unsupported" error when called.

authored-by: Eric Seifert <eric@seifert.casa>
2020-06-25 08:04:43 -07:00
Renuka Manavalan
edeb40ffcf
[k8s]: switching to Flannel from Calico. (#4768)
Switching to Flannel from Calico which brings down the image size by around 500+MB.
2020-06-12 18:06:08 -07:00
xumia
e28cb8f58e
[Secure Boot] Support to sign swi image (#4627)
* [secure boot] Support to sign swi image

* Fix build issue

* fix tab format issue

* Fix typing issue

* Change the sign_image.sh command line

* Remove SONIC_CETIFICATE_PATH

* Fix bugs
2020-06-09 15:25:17 +08:00
lguohan
00383051f4
[docker-iccpd]: disable iccpd build by default (#4502)
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-29 22:45:46 -07:00
Renuka Manavalan
583bfde439
Kubeadm v1.18.0 requires Calico images of version v3.12.1 (#4445) 2020-04-19 11:34:30 -07:00
Renuka Manavalan
f128153706
[baseimage]: Install Kubernetes packages if enabled in image (#4374)
* Install kubernetes worker node packages, if enabled.

* Minor updates

* Added some comments

* Updates per review comments.
Built a private image to test to work fine.

* Remove the removed file.

* Update per comments
Make a fix, as kubeadm no demands a higher version of kubelet & kubectl.
As kubeadm auto install kubectl & kubelet, removing explicit install is an easier/robust fix.

* Changes per review comments.

* Updates per comments.
1) Dropped helper & pod scripts
2) Made install verbose

* Drop creation of pods subdir, as this PR does not use them.

* From comments to 'n' per review comments.

* 1) kubeadm.conf is created as part of kubeadm package install. Hence dropped explicit copy.
2020-04-13 08:41:18 -07:00
shine4chen
524cf9e56a
MCLAG feature for SONIC (#2514)
* MCLAG feature for sonic

* MCLAG feature for sonic

* remove binary file

* remove unused dockerfile

update docker-iccpd to stretch-based container

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* minor fix for isolation port setting

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* iccpd docker would start on demand

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* Add x attribute on mclagdctl file

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* add warm-reboot support for MCLAG

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* merge to master branch and reformat iccpd file

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* fix some bugs and make peer-link configuration optional

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* refactor code per Brcm review

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* correct a typo

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* * optimize iccpd arp/mac sync process
* refine code according to brcm opinoin
* unify function return value

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* * optimize warm-reboot process
* estabish iccpd connection with configurated src-ip

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* fix a typo

Signed-off-by: shine.chen <shine.chen@nephosinc.com>

* optimize some code
* add some debug info
* optimize bridge mac setting
* fix vlan mac sync issue on standby node

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* optimize some code

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* fix some bugs for warm-reboot

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* refine log level

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* refine iccpd syslog & skip arp packet whose src ip is local ip

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* remove iccpd dependency with teamd

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* print log level when dump mclag status

Signed-off-by: shine.chen <shine.chen@mediatek.com>

* revise per community review

Signed-off-by: shine.chen <shine.chen@mediatek.com>

Co-authored-by: shine.chen <shine.chen@nephosinc.com>
Co-authored-by: shine.chen <shine.chen@mediatek.com>
2020-04-04 15:24:06 -07:00
lguohan
760e763935
[build]: allow to specify timestamp format in the build log (#4311)
only simple/none are supported currently

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-03-23 09:45:43 -07:00
lguohan
20260ceb1d
[build]: add SONIC_CONFIG_BUILD_LOG_TIMESTAMP to add timestamp in build log (#4269)
add timestamp in each job build log

example:

   [01:39:21] dh clean  --with autotools-dev
   [01:39:22]    dh_auto_clean
   [01:39:27]      make -j16 distclean

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-03-21 14:21:26 -07:00
Kalimuthu-Velappan
7d2ebf8116
[build]: support for DPKG local caching (#4117)
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.

The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.

This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.

- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=

    where  SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
                            none:     no caching
                            cache:    cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.

    For example: dependency files for Linux Kernel - src/sonic-linux-kernel,

            SPATH       := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
            DEP_FILES   := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
            DEP_FILES   += $(SONIC_COMMON_BASE_FILES_LIST)
            SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))

            DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
                         $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)

            $(LINUX_HEADERS_COMMON)_CACHE_MODE  := GIT_CONTENT_SHA
            $(LINUX_HEADERS_COMMON)_DEP_FLAGS   := $(DEP_FLAGS)
            $(LINUX_HEADERS_COMMON)_DEP_FILES   := $(DEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format

    FORMAT:
            <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
            Eg:
              linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz

            < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
            < 24-byte MOD SHA value > - the SHA value is derived from either of the following.
                    GIT_COMMIT_SHA  - SHA value of the last git commit ID if it is a submodule
                    GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.

            $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
                    $(call dpkg_depend,$(DEBS_PATH)/%.dep )
            $(HEADER)


            # Load the target deb from DPKG cache
            $(call LOAD_CACHE,$*,$@)


            # Skip building the target if it is already loaded from cache
            if [ -z '$($*_CACHE_LOADED)' ] ; then

                  .....
                 # Rules for Generating the target DEB file.
                  .....

                  # Save the target deb into DPKG cache
                  $(call SAVE_CACHE,$*,$@)
            fi


            $(FOOTER)


    The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.

    Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
    The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
    It also updates the target-specific variable to indicate the cache is loaded or not.
    The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it

    The caching functionality is verified by enabling it in Linux kernel submodule.
    It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

DOCUMENT PR:

           https://github.com/Azure/SONiC/pull/559
2020-03-11 20:04:52 -07:00
abdosi
f5ab24d8f1
Made Change to have Configurable option to enable/disable docker image (#4171)
* Made Change to have Configurable option to enable/disable docker image

* [Submodule-Update] sonic-utilities to latest master
2020-02-21 10:20:07 -08:00
Sumukha Tumkur Vani
ebb07d9942 [docker-restapi]: Avoid building RestAPI docker by default (#4018)
Provide build option for RESTAPI docker
2020-01-14 17:13:45 -08:00
Prabhu Sreenivasan
87f70108cb SONiC Management Framework Release 1.0 (#3488)
* Added sonic-mgmt-framework as submodule / docker

* fix build issues

* update sonic-mgmt-framework submodule branch to master

* Merged changes 70007e6d2ba3a4c0b371cd693ccc63e0a8906e77..00d4fcfed6a759e40d7b92120ea0ee1f08300fc6

00d4fcfed6a759e40d7b92120ea0ee1f08300fc6 Modified environemnt variables

* Changes to build sonic-mgmt-framework docker

* bumped up sonic-mgmt-framework commit-id

* version bump for sonic-mgmt-framework commit-it

* bumped up sonic-mgmt-framework commit-id

* Add python packages to docker

* Build fix for docker with python packages

* added libyang as dependent package

* Allow building images on NFS-mounted clones

Prior to this change, `build_debian.sh` would generate a Debian
filesystem in `./fsroot`. This needs root permissions, and one of the
tests that is performed is whether the user can create a character
special file in the filesystem (using mknod).

On most NFS deployments, `root` is the least privileged user, and cannot
run mknod. Also, attempting to run commands like rm or mv as root would
fail due to permission errors, since the root user gets mapped to an
unprivileged user like `nobody`.

This commit changes the location of the Debian filesystem to `/fsroot`,
which is a tmpfs mount within the slave Docker. The default squashfs,
docker tarball and zip files are also created within /tmp, before being
copied back to /sonic as the regular user.

The side effect of this change is that the contents of `/fsroot` are no
longer available once the slave container exits, however they are
available within the squashfs image.

Signed-off-by: Nirenjan Krishnan <Nirenjan.Krishnan@dell.com>

* bumped up sonc-mgmt-framework commit to include PR #18

*     REST Server startup script is enahnced to read the settings from
    ConfigDB. Below table provides mapping of db field to command line
    argument name.

    ============================================================
    ConfigDB entry key      Field name      REST Server argument
    ============================================================
    REST_SERVER|default     port            -port
    REST_SERVER|default     client_auth     -client_auth
    REST_SERVER|default     log_level       -v
    DEVICE_METADATA|x509    server_crt      -cert
    DEVICE_METADATA|x509    server_key      -key
    DEVICE_METADATA|x509    ca_crt          -cacert
    ============================================================

* Replace src/telemetry as submodule to sonic-telemetry

* Update telemetry commit HEAD

* Update sonic-telemetry commit HEAD

* libyang env path update

* Add libyang dependency to telemetry

* Add scripts to create JSON files for CLI backend

Scripts to create /var/platform/syseeprom and /var/platform/system, which are back-end
files for CLI, for system EEPROM and system information.

Signed-off-by: Howard Persh <Howard_Persh@dell.com>

* In startup script, create directory where CLI back-end files live

Signed-off-by: Howard Persh <Howard_Persh@dell.com>

* build dependency pkgs added to docker for build failure fix

* Changes to fix build issue for mgmt framework

* Fix exec path issue with telemetry

* s5232[device] PSU detecttion and default led state support

* Processing of first boot in rc.local should not have premature exit

Signed-off-by: Howard Persh <Howard_Persh@dell.com>

*  docker mount options added for platform, system features

* bumped up sonic-mgmt-framework commit id to pick 23rd July 2019 changes

* Added mount options for telemetry docker to get access for system and platform info.

* Update commit for sonic-utilities

* [dell]: Corrected dport map and renamed config files for S5232F

* Fix telemetry submodule commit

* added support for sonic-cli console

* [Dell S5232F, Z9264F] Harden FPGA driver kernel module

For Dell S5232F and Z9264F platforms, be more strict when checking state
in ISR of FPGA driver, to harden against spurious interrupts.

Signed-off-by: Howard Persh <Howard_Persh@dell.com>

* update mgmt-framework submodule to 27th Aug commit.

* remove changes not related to mgmt-framework and sonic-telemetry

* Revert "Replace src/telemetry as submodule to sonic-telemetry"

This reverts commit 11c3192975.

* Revert "Replace src/telemetry as submodule to sonic-telemetry"

This reverts commit 11c3192975.

* make submodule changes and remove a change not related to PR

* more changes

* Update .gitmodules

* Update Dockerfile.j2

* Update .gitmodules

* Update .gitmodules

* Update .gitmodules

reverting experimental change

* Removed syspoll for release_1.0

Signed-off-by: Jeff Yin <29264773+jeff-yin@users.noreply.github.com>

* Update docker-sonic-mgmt-framework.mk

* Update sonic-mgmt-framework.mk

* Update sonic-mgmt-framework.mk

* Update docker-sonic-mgmt-framework.mk

* Update docker-sonic-mgmt-framework.mk

* Revert "Processing of first boot in rc.local should not have premature exit"

This reverts commit e99a91ffc2.

* Remove old telemetry directory

* Update docker-sonic-mgmt-framework.mk

* Resolving merge conflict with Azure

* Reverting the wrong merge

* Use CVL_SCHEMA_PATH instead of changing directory for telemetry startup

* Add missing export

* Add python mmh3 to slave dockerfile

* Remove sonic-mgmt-framework build dep for telemetry, fix dialout startup issues

* Provided flag to disable compiling mgmt-framework

* Update sonic-utilites point latest commit id

* Point sonic-utilities to Azure accepted SHA

* Updating mgmt framework to right sha

* Add sonic-telemetry submodule

* Update the mgmt-framework commit id

Co-authored-by: jghalam <joe.ghalam@gmail.com>
Co-authored-by: Partha Dutta <51353699+dutta-partha@users.noreply.github.com>
Co-authored-by: srideepDell <srideep_devireddy@dell.com>
Co-authored-by: nirenjan <nirenjan@users.noreply.github.com>
Co-authored-by: Sachin Holla <51310506+sachinholla@users.noreply.github.com>
Co-authored-by: Eric Seifert <seiferteric@gmail.com>
Co-authored-by: Howard Persh <hpersh@yahoo.com>
Co-authored-by: Jeff Yin <29264773+jeff-yin@users.noreply.github.com>
Co-authored-by: Arunsundar Kannan <31632515+arunsundark@users.noreply.github.com>
Co-authored-by: rvasanthm <51932293+rvasanthm@users.noreply.github.com>
Co-authored-by: Ashok Daparthi-Dell <Ashok_Daparthi@Dell.com>
Co-authored-by: anand-kumar-subramanian <51383315+anand-kumar-subramanian@users.noreply.github.com>
2019-12-23 21:47:16 -08:00
rajendra-dendukuri
eec594adf2 [sonic-ztp]: Build sonic-ztp package (#3299)
* Build sonic-ztp package

- Add changes in make rules to conditionally include sonic-ztp package

Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
2019-12-04 04:50:56 -08:00
padmanarayana
75104bb35d [sflow]: Build infrastructure changes to support sflow docker and utilities (#3251)
Introduce a new "sflow" container (if ENABLE_SFLOW is set). The new docker will include:
hsflowd : host-sflow based daemon is the sFlow agent
psample : Built from libpsample repository. Useful in debugging sampled packets/groups.
sflowtool : Locally dump sflow samples (e.g. with a in-unit collector)

In case of SONiC-VS, enable psample & act_sample kernel modules.

VS' syncd needs iproute2=4.20.0-2~bpo9+1 & libcap2-bin=1:2.25-1 to support tc-sample

tc-syncd is provided as a convenience tool for debugging (e.g. tc-syncd filter show ...)
2019-09-14 20:27:09 -07:00
Jipan Yang
0ec5de4fb3 [build]: Fix INSTALL_DEBUG_TOOLS build config option (#2564)
* Fix INSTALL_DEBUG_TOOLS build config option

Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>
2019-08-15 02:20:18 -07:00
Stepan Blyshchak
1bcd54d38c [build]: Upgrade docker to 18.09 in stretch slave (#3049)
Also add an option to use docker buildkit for image build

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2019-06-20 09:19:33 -07:00
lguohan
f35daa7694
[frr]: change frr as default sonic routing stack (#2863)
* [frr]: change frr as default sonic routing stack

* fix quagga configuration

* [vstest]: fix bgp test for frr

* [vstest]: skip bgp/test_invalid_nexthop.py for frr

Signed-off-by: Guohan Lu <gulv@microsoft.com>
2019-05-07 23:40:40 -07:00
Ying Xie
9d6bac567b [build] force Linux to drop cache before calling kvm (#2717)
* [vm build] force Linux to drop cache before calling kvm

KVM need to allocate 2G memory for this build. The system memory might
be occupied by cache at the moment and doesn't have 2G chunk to give
out. Forcing Kernel to drop cache to boost the chance of getting 2G
memory.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* [make] add option to enable/disable VS build memory preparation

Signed-off-by: Ying Xie <ying.xie@microsoft.com>
2019-03-29 15:25:17 -07:00
Joe LeVeque
0fd4f1806b
[build] Enable streaming telemetry Docker container by default (#2354) 2019-02-28 16:27:29 -08:00
Nikos
4ed5cb4ef1 [docker-frr]: Move FRR from 4.0 to 6.0.2 and make the new frr version and debian package compile (#2454)
Signed-off-by: nikos <ntriantafillis@gmail.com>
2019-01-16 18:34:41 -08:00
Nikos
7056b49af7 Routing application split config support (#2286)
* Routing application split config support

Signed-off-by: nikos <ntriantafillis@gmail.com>

* Routing application split config support
Routing application split config support

Signed-off-by: nikos <Nikos Triantafillis>
2018-11-26 18:19:12 -08:00
Wataru Ishida
4db98b46e4 [build]: add build option to use native docker instead of dind for build (#2215)
using native docker is faster than dind dockerd with vfs storage driver

https://github.com/Azure/sonic-buildimage/pull/2016

https://github.com/Azure/draft/issues/181

Signed-off-by: Wataru Ishida <ishida@nel-america.com>
2018-11-01 18:00:00 -07:00
zhenggen-xu
673bb6580e [sonic-frr]: FRR 4.0 integration with SONiC (#2099)
* FRR 4.0 integration with SONiC

-- Uses SONiC FRR repo frr/4.0 (which has SONiC support) to build image
-- Makefile changes to make frr4.0 builtable.
-- Updated/Added FRR configuration files
-- bgpd jinja template fixes

To build SONiC images with FRR4.0, simply edit rules/config file and change
routing stack to following:

SONIC_ROUTING_STACK = frr

and then build images as usual.

* Used integrated-vtysh-config in FRR
Changed to single template: frr.conf.j2 for configuration and added tests
2018-10-02 10:24:59 -07:00
Ying Xie
3c33737d58
[make] introducing new build option KERNEL_PROCURE_METHOD (#1837)
* [make] introducing new build option KERNEL_BUILD_METHOD

- Kernel could be built from source files with method 'build'
- Kernel could be downloaded from Azure storage with method 'download'

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* Replace BUILD with PROCURE

* fix typo
2018-07-25 08:14:18 -07:00