sonic-buildimage/slave.mk

1186 lines
52 KiB
Makefile
Raw Normal View History

Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Presettings
###############################################################################
# Select bash for commands
.ONESHELL:
SHELL = /bin/bash
.SHELLFLAGS += -e
USER = $(shell id -un)
UID = $(shell id -u)
GUID = $(shell id -g)
ifeq ($(SONIC_IMAGE_VERSION),)
override SONIC_IMAGE_VERSION := $(shell export BUILD_TIMESTAMP=$(BUILD_TIMESTAMP) && export BUILD_NUMBER=$(BUILD_NUMBER) && . functions.sh && sonic_get_version)
endif
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
.SECONDEXPANSION:
NULL :=
SPACE := $(NULL) $(NULL)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## General definitions
###############################################################################
SRC_PATH = src
RULES_PATH = rules
TARGET_PATH = target
DOCKERS_PATH = dockers
BLDENV = $(shell lsb_release -cs)
DEBS_PATH = $(TARGET_PATH)/debs/$(BLDENV)
FILES_PATH = $(TARGET_PATH)/files/$(BLDENV)
PYTHON_DEBS_PATH = $(TARGET_PATH)/python-debs
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
PYTHON_WHEELS_PATH = $(TARGET_PATH)/python-wheels
PROJECT_ROOT = $(shell pwd)
JESSIE_DEBS_PATH = $(TARGET_PATH)/debs/jessie
JESSIE_FILES_PATH = $(TARGET_PATH)/files/jessie
STRETCH_DEBS_PATH = $(TARGET_PATH)/debs/stretch
STRETCH_FILES_PATH = $(TARGET_PATH)/files/stretch
BUSTER_DEBS_PATH = $(TARGET_PATH)/debs/buster
BUSTER_FILES_PATH = $(TARGET_PATH)/files/buster
DBG_IMAGE_MARK = dbg
DBG_SRC_ARCHIVE_FILE = $(TARGET_PATH)/sonic_src.tar.gz
DPKG_ADMINDIR_PATH = /sonic/dpkg
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
CONFIGURED_PLATFORM := $(shell [ -f .platform ] && cat .platform || echo generic)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
PLATFORM_PATH = platform/$(CONFIGURED_PLATFORM)
CONFIGURED_ARCH := $(shell [ -f .arch ] && cat .arch || echo amd64)
ifeq ($(PLATFORM_ARCH),)
override PLATFORM_ARCH = $(CONFIGURED_ARCH)
endif
DOCKER_BASE_ARCH := $(CONFIGURED_ARCH)
ifeq ($(CONFIGURED_ARCH),armhf)
override DOCKER_BASE_ARCH = arm32v7
else
ifeq ($(CONFIGURED_ARCH),arm64)
override DOCKER_BASE_ARCH = arm64v8
endif
endif
IMAGE_DISTRO := buster
IMAGE_DISTRO_DEBS_PATH = $(TARGET_PATH)/debs/$(IMAGE_DISTRO)
IMAGE_DISTRO_FILES_PATH = $(TARGET_PATH)/files/$(IMAGE_DISTRO)
export BUILD_NUMBER
export BUILD_TIMESTAMP
export SONIC_IMAGE_VERSION
[barefoot]: Support for platforms based on Barefoot Networks' device (#1796) * Initial commit * Add Ingrasys S9180-32X platform dirver. Signed-off-by: Wade He <chihen.he@gmail.com> * Add bfn.service for init barefoot. Signed-off-by: Wade He <chihen.he@gmail.com> * [Barefoot Beta] Add some functions and fixed some bugs. 1. Update sensors.conf. 2. Fixed IO expander init. 3. Fixed PSU EEPROM. 4. Fixed MB EEPROM. 5. Add fancontrol and fan init. 6. Add SYS LED control (sys, fan, fan tray). 7. 2.5V compute and setup max and min. 8. Fixed typo MB eeprom delete address. 9. Remove coretemp to BMC. 10. Add active CPLD. 11. Modify SFP+ GPIO slave address. 12. Modify tmp75 Near Port 32 slave address. Signed-off-by: Wade He <chihen.he@gmail.com> * Add bfn script in /etc/init.d/ Signed-off-by: Wade He <chihen.he@gmail.com> * Add bfn service in debian Signed-off-by: Wade He <chihen.he@gmail.com> * Fixed CPLD switch LED behavior. Signed-off-by: Wade He <chihen.he@gmail.com> * [Barefoot Beta] Fixed sensors and hwmon order. 1. Fixed ignore sensors Vbat. 2. Reorg hwmon order. Signed-off-by: Wade He <chihen.he@gmail.com> * Fixed PSU1 and PSU2 EEPROM order. Signed-off-by: Wade He <chihen.he@gmail.com> * initial barefoot checkin october 2017 * update refpoint * update refpoints * update refpoints to bf-master * update refpoint * update refpoint to tested version * change to platform from asic * update refpoint for swss * revert core creation setting * update refpoints * add telnet for debug shell * update refpoints 11/17/17 * missed change in file on previous merge * [CPLD] Fixed blink LED issue. * Fixed blink LED mask set error. Signed-off-by: Wade He <chihen.he@gmail.com> * Update bf_kdrv.c for 6.0.2.39 * Update bf kernel driver * Add bf_fun kernel module. * Update bf_tun for fixed build error * merge with Azure master (12/12/17) * update swss refpoint * update refpoint of swss * library dependency for stack unroll * update refpoint to bf-master * [DHCP relay]: Fix circuit ID and remote ID bugs (#1248) * [DHCP relay]: Fix circuit ID and remote ID bugs * Set circuit_id_len after setting circuit_id_len to ip->name * [Platform] Add Psuutil and update sensors.conf for S9100-32X, S8810-32Q and S9200-64X (#1272) * Add I2C CPLD kernel module for psuutil. * Support psuutil script. * Add voltage min and max threshold. * Update sensors.conf for tmp75. Signed-off-by: Wade He <chihen.he@gmail.com> * Allow multi platform support - infra (more changes to follow) * update relative path to include platform for clarity * [Platform] Add Ingrasys S9130-32X and S9230-64X with Nephos Switch ASIC for "branch 201712" (#1274) - What I did Add switch ASIC vendor: Nephos Add Nephos platforms: Ingrasys S9130-32X, Ingrasys S9230-64X - How I did it Add platform/nephos files Add platform/nephos/sonic-platform-modules-ingrasys submodule Add device/ingrasys/x86_64-ingrasys_s9130_32x-r0 files Add device/ingrasys/x86_64-ingrasys_s9230_64x-r0 files Add SONiC to support Nephos platform Update Head of submodule src/sonic-sairedis to "3b817bb" - How to verify it To build SONiC installer image and docker images, run the following commands: make configure PLATFORM=nephos make target/sonic-nephos.bin Check system and network feature is worked as well - Description for the changelog Add switch ASIC vendor and platforms for Nephos - A picture of a cute animal (not mandatory but encouraged) Signed-off-by: Sam Yang <yang.kaiyu@gmail.com> * change source of files to github (from dropbox), update sairedis refpoint * update refpoint of sairedis * [centec] support CENTEC SAI 1.0 on 201712 branch and update e582-48x6q board (#1269) * [marvel]: Marvell's updates for SONiC.201712 & SAI v1.0 (#1287) * update sairedis (fast-boot refpoint) * fix syncd rpc make files * update refpoint to handle Makefile change (no functional change) * [Marvell]: Add support for SLM5401-54x device (#1307) * Marvell's updates for SONiC.201712 & SAI v1.0 * [Platform] Add Marvell's SLM5401-54x for branch 201712 * [Broadcom]: Update Boradcom SAI package to 3.0.3.3-3 (#1312) (#1321) - update Arista 7050-QX32S config.bcm file - update Accton th-as771*-32x100G.config.bcm files * update refpoint for Makefile chnage in sairedis * update refpoint - sairedis * update sairedis to older refpoint till we debug clean build * export asic platform for build * update refpoint for makefiles * [PLATFORM] Centec update E582 driver fan/epprom/sensor (#1332) * Upload wnc-osw1800 * Modify for Barefoot suggest * Revert bfn-platform.mk * Update bfn-platform-wnc.mk Update parameter name * Update parameter name * initial support for WNC platform * change switch name to "switch" * Delete bf modules for rel_7_0 * Add Ingrasys S9180 platform Signed-off-by: Wade He <chihen.he@gmail.com> * Modify bfnsdk for Ingrasys S9180 platform Signed-off-by: Wade He <chihen.he@gmail.com> * Resolved the conflict. * Resolved the conflict. * Update submodule path and url. * Delete unused file. * Update PSU GPIO and EEPROM for psuutil. * Add psuutil in S9180-32X Signed-off-by: Wade He <chihen.he@gmail.com> * update refpoint * update refpoint * change contact email, update refpoint * cleanup and update kernel modules * updates based on review * update refpoint * update refpoint * fix typo in config script to check for platforms * remove stale file * resolve conflicts * cleanup diffs with Azure repo and update SDK debs * update refpoints to Azure * address review comments * revert refpoint of swss-common * porting the build fix from master * porting build fix from master * Minor Fix * Minor fix * Temp to sde deb packages url * Update sonic - sairedis,swss & swss-common refpoints * Update git modules url path to bfn repo * updated paths for swss, swss-common & sairedis * Update refpoint for sonic-swss to local bfn repo * Update URL for downloading sde debian packages * porting fix links of debian git server from master * porting fix links of debian git server from master * [Ingrasys] Add platform support for S9280-64X with Barefoot ASIC * Update ref points for swss, swss-common and sairedis repos * Add sonic platform scripts for bfn montara/maverick * Call sh scripts instead of calling py scripts * Address upstream PR Comments (#10) * Update bf-master with azure/master * Undo changes to some files * Revert "Address upstream PR Comments (#10)" This reverts commit a7fddb83ca1073f90fbe46955ba57a9b43742c73. * Address upstream comments (#11) * Remove all non bfn specific changes from upstream PR * Revert "Address upstream comments (#11)" This reverts commit 559132103e5c73e43f4282d1559ede03f16abfea. * Undo non bfn changes * Little more cleanup * Add back code removed in merge * export CONFIGURED_PLATFORM * Update sairedis and swss refpoints * Address Upstream PR comment * change deb pkg dependency from 3.16.0-4-amd64 to 3.16.0-5-amd64 * Set default tx queue len for usb0 interface to 64 * Update sairedis refpoint * Update swss ref point * Add bfn buffer cfg files for montara/maverick as per new design * Update buffer cfg templates for bfn montara * add non zero size to buffer profile * add macro to generate port lists * Update buffer cfg templates for bfn mavericks * add non zero size for buffer profiles * add port generation macro * Add missing psmisc package * BGP docker seems to be missing killall utility being used by fast-reboot script. This is causing non graceful termination of BGP sessions. Adding psmisc to resolve this issue. * Update swss ref point * Update swss ref point * Update sairedis refpoint * Update sairedis refpoint * Update sairedis refpoint * Update sairedis refpoint * Update refpoint for sairedis and swss * sairedis to azure master * swss to latest bfn bf-master * Update gitmodules Update url for sairedis to azure master * Correct typo in bfn platform script * Update swss and sairedis ref points * Update swss ref point * Address Review comments * Update swws path in gitmodules to azure master * update swss refpoint * update base docker j2 file -remove psmisc package (could be a concern, would cause fast reboot to not work correctly will fix in another PR) * Fix sairedis refpoint broken in by previous merge * Remove psmisc from docker base image * This will break fast reboot as killall is required for killing bgp process and initiating graceful termination of BGP session. Will fix this in a seperate PR. Need this for SONIC upstreaming * Address upstream comments * Remove bmc interface from interface jinja template and sample output interfaces file * Add bmc interface at boot time to network interfaces for bfn bmc based platforms * Remove autogen ingrasys debian files * Revert "Remove autogen ingrasys debian files" * Buffer and qos config template fix for bfn platforms (#21) SWI-1509 Buffer and qos config template fix for bfn platforms * Fix qos config files for montara & mavericks (#22) * Reference only ppg 3,4 in qos files as no profiles are attached to 0,1 in buffer configs * Fix vs test (#23)
2018-07-24 12:23:12 -05:00
export CONFIGURED_PLATFORM
export CONFIGURED_ARCH
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. (#3730) [sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. YANG models as per Guidelines. Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md [sonic-yang-models/tests]: YANG model test code and JSON input for testing. [sonic-yang-models/setup.py]: Build infra for yang models. **- What I did** Created Yang model for Sonic. Tables: PORT, VLAN, VLAN_INTERFACE, VLAN_MEMBER, ACL_RULE, ACL_TABLE, INTERFACE. Created build infra files using which a new package (sonic-yang-models) can be build and can be deployed on sonic switches. Yang models will be part of this new package. **- How I did it** Wrote yang models based on Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md and https://github.com/Azure/SONiC/wiki/Configuration. Wrote python wheel Package infra which runs test for these Yang models using a json files which consists configuration as per yang models. These configs are for negative tests, which means we want to test that most must condition, pattern and when condition works as expected. **- How to verify it** Build Logs and testing: ——————————————————————————————————— ``` /sonic/src/sonic-yang-models /sonic running test running egg_info writing top-level names to sonic_yang_models.egg-info/top_level.txt writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt writing sonic_yang_models.egg-info/PKG-INFO reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt' writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt' running build_ext ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK running bdist_wheel running build running build_py (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../libyang_1.0.73_amd64.deb ... Unpacking libyang (1.0.73) over (1.0.73) ... Setting up libyang (1.0.73) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... Processing triggers for man-db (2.7.6.1-2) ... (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../libyang-cpp_1.0.73_amd64.deb ... Unpacking libyang-cpp (1.0.73) over (1.0.73) ... Setting up libyang-cpp (1.0.73) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../python3-yang_1.0.73_amd64.deb ... Unpacking python3-yang (1.0.73) over (1.0.73) ... Setting up python3-yang (1.0.73) ... INFO:YANG-TEST:module: sonic-vlan is loaded successfully ERROR:YANG-TEST:Could not get module: sonic-head INFO:YANG-TEST:module: sonic-portchannel is loaded successfully INFO:YANG-TEST:module: sonic-acl is loaded successfully INFO:YANG-TEST:module: sonic-loopback-interface is loaded successfully ERROR:YANG-TEST:Could not get module: sonic-port INFO:YANG-TEST:module: sonic-interface is loaded successfully INFO:YANG-TEST: ------------------- Test 1: Configure a member port in VLAN_MEMBER table which does not exist.--------------------- libyang[0]: Leafref "/sonic-port:sonic-port/sonic-port:PORT/sonic-port:PORT_LIST/sonic-port:port_name" of value "Ethernet156" points to a non -existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlan_name='Vlan100'][port='Ethernet156']/port) INFO:YANG-TEST:Configure a member port in VLAN_MEMBER table which does not exist. Passed INFO:YANG-TEST: ------------------- Test 2: Configure non-existing ACL_TABLE in ACL_RULE.--------------------- libyang[0]: Leafref "/sonic-acl:sonic-acl/sonic-acl:ACL_TABLE/sonic-acl:ACL_TABLE_LIST/sonic-acl:ACL_TABLE_NAME" of value "NOT-EXIST" points to a non-existing leaf. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NOT-EXIST'][RULE_NAME='Rule_20']/ACL_TABLE_NAME) INFO:YANG-TEST:Configure non-existing ACL_TABLE in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 3: Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/ICMPV6_CODE) INFO:YANG-TEST:Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE. Passed INFO:YANG-TEST: INFO:YANG-TEST: ------------------- Test 4: Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_20']/SRC_IPV6) INFO:YANG-TEST:Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE. Passed ------------------- Test 5: Configure l4_src_port_range as 99999-99999 in ACL_RULE--------------------- libyang[0]: Value "99999-99999" does not satisfy the constraint "([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])-([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6'][RULE_NAME='Rule_20']/L4_SRC_PORT_RANGE) INFO:YANG-TEST:Configure l4_src_port_range as 99999-99999 in ACL_RULE Passed INFO:YANG-TEST: ------------------- Test 6: Configure empty string as ip-prefix in INTERFACE table.--------------------- libyang[0]: Invalid value "" in "ip-prefix" element. (path: /sonic-interface:sonic-interface/INTERFACE/INTERFACE_LIST[interface='Ethernet8'][ip-prefix='']/ip-prefix) INFO:YANG-TEST:Configure empty string as ip-prefix in INTERFACE table. Passed INFO:YANG-TEST: ------------------- Test 7: Configure Wrong family with ip-prefix for VLAN_Interface Table--------------------- libyang[0]: Must condition "(contains(../ip-prefix, ':') and current()='IPv6') or (contains(../ip-prefix, '.') and current()='IPv4')" not satisfied. (path: /sonic-vlan:sonic-vlan/VLAN_INTERFACE/VLAN_INTERFACE_LIST[vlanid='100'][ip-prefix='2a04:5555:66:7777::1/64']/family) INFO:YANG-TEST:Configure Wrong family with ip-prefix for VLAN_Interface Table Passed INFO:YANG-TEST: ------------------- Test 8: Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPV6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NS W-PACL-V6'][RULE_NAME='Rule_20']/DST_IPV6) INFO:YANG-TEST:Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 9: Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE.--------------------- libyang[0]: Value "0x080C" does not satisfy the constraint "(0x88CC|0x8100|0x8915|0x0806|0x0800|0x86DD|0x8847)" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/INNER_ETHER_TYPE) INFO:YANG-TEST:Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 10: Add dhcp_server which is not in correct ip-prefix format.--------------------- libyang[0]: Invalid value "10.186.72.566" in "dhcp_servers" element. (path: /sonic-vlan:sonic-vlan/VLAN/VLAN_LIST/dhcp_servers[.='10.186.72.566']) INFO:YANG-TEST:Add dhcp_server which is not in correct ip-prefix format. Passed INFO:YANG-TEST: ------------------- Test 11: Configure undefined acl_table_type in ACL_TABLE table.--------------------- libyang[0]: Invalid value "LAYER3V4" in "type" element. (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6']/type) INFO:YANG-TEST:Configure undefined acl_table_type in ACL_TABLE table. Passed INFO:YANG-TEST: ------------------- Test 12: Configure undefined packet_action in ACL_RULE table.--------------------- libyang[0]: Invalid value "SEND" in "PACKET_ACTION" element. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST/PACKET_ACTION) INFO:YANG-TEST:Configure undefined packet_action in ACL_RULE table. Passed INFO:YANG-TEST: ------------------- Test 13: Configure wrong value for tagging_mode.--------------------- libyang[0]: Invalid value "non-tagged" in "tagging_mode" element. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST/tagging_mode) INFO:YANG-TEST:Configure wrong value for tagging_mode. Passed INFO:YANG-TEST: ------------------- Test 14: Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table.--------------------- libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid) libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid) INFO:YANG-TEST:Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table. Passed INFO:YANG-TEST:All Test Passed ../../target/debs/stretch/libyang0.16_0.16.105-1_amd64.deb installtion failed ../../target/debs/stretch/libyang-cpp0.16_0.16.105-1_amd64.deb installtion failed ../../target/debs/stretch/python2-yang_0.16.105-1_amd64.deb installtion failed YANG Tests passed Passed: pyang -f tree ./yang-models/*.yang > ./yang-models/sonic_yang_tree copying tests/yangModelTesting.py -> build/lib/tests copying tests/test_sonic_yang_models.py -> build/lib/tests copying tests/__init__.py -> build/lib/tests running egg_info writing top-level names to sonic_yang_models.egg-info/top_level.txt writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt writing sonic_yang_models.egg-info/PKG-INFO reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt' writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt' installing to build/bdist.linux-x86_64/wheel running install running install_lib creating build/bdist.linux-x86_64/wheel creating build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/yangModelTesting.py -> build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/test_sonic_yang_models.py -> build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/wheel/tests running install_data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-head.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-acl.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-loopback-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-port.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-portchannel.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-vlan.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models ```
2020-04-14 17:36:02 -05:00
export PYTHON_WHEELS_PATH
export IMAGE_DISTRO
export IMAGE_DISTRO_DEBS_PATH
export MULTIARCH_QEMU_ENVIRON
export DOCKER_BASE_ARCH
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Utility rules
## Define configuration, help etc.
###############################################################################
.platform :
ifneq ($(CONFIGURED_PLATFORM),generic)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
@echo Build system is not configured, please run make configure
@exit 1
endif
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
configure :
@mkdir -p $(DEBS_PATH)
@mkdir -p $(JESSIE_DEBS_PATH)
@mkdir -p $(STRETCH_DEBS_PATH)
@mkdir -p $(BUSTER_DEBS_PATH)
@mkdir -p $(FILES_PATH)
@mkdir -p $(JESSIE_FILES_PATH)
@mkdir -p $(STRETCH_FILES_PATH)
@mkdir -p $(BUSTER_FILES_PATH)
@mkdir -p $(PYTHON_DEBS_PATH)
@mkdir -p $(PYTHON_WHEELS_PATH)
@mkdir -p $(DPKG_ADMINDIR_PATH)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
@echo $(PLATFORM) > .platform
@echo $(PLATFORM_ARCH) > .arch
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
2017-03-02 06:08:25 -06:00
distclean : .platform clean
@rm -f .platform
@rm -f .arch
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
list :
@$(foreach target,$(SONIC_TARGET_LIST),echo $(target);)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Include other rules
###############################################################################
include $(RULES_PATH)/config
-include $(RULES_PATH)/config.user
###############################################################################
## Version control related exports
###############################################################################
export PACKAGE_URL_PREFIX
export TRUSTED_GPG_URLS
export SONIC_VERSION_CONTROL_COMPONENTS
DEFAULT_CONTAINER_REGISTRY := $(SONIC_DEFAULT_CONTAINER_REGISTRY)
export DEFAULT_CONTAINER_REGISTRY
ifeq ($(SONIC_ENABLE_PFCWD_ON_START),y)
ENABLE_PFCWD_ON_START = y
endif
ifeq ($(SONIC_INCLUDE_SYSTEM_TELEMETRY),y)
INCLUDE_SYSTEM_TELEMETRY = y
endif
ifeq ($(SONIC_INCLUDE_RESTAPI),y)
INCLUDE_RESTAPI = y
endif
ifeq ($(SONIC_ENABLE_SYNCD_RPC),y)
ENABLE_SYNCD_RPC = y
endif
ifeq ($(SONIC_INSTALL_DEBUG_TOOLS),y)
INSTALL_DEBUG_TOOLS = y
endif
ifeq ($(SONIC_INCLUDE_SFLOW),y)
INCLUDE_SFLOW = y
endif
ifeq ($(SONIC_INCLUDE_NAT),y)
INCLUDE_NAT = y
endif
ifeq ($(SONIC_INCLUDE_MACSEC),y)
INCLUDE_MACSEC = y
endif
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
include $(RULES_PATH)/functions
ifeq ($(SONIC_USE_PDDF_FRAMEWORK),y)
PDDF_SUPPORT = y
else
PDDF_SUPPORT = n
endif
export PDDF_SUPPORT
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
include $(RULES_PATH)/*.mk
ifneq ($(CONFIGURED_PLATFORM), undefined)
ifeq ($(PDDF_SUPPORT), y)
PDDF_DIR = pddf
PLATFORM_PDDF_PATH = platform/$(PDDF_DIR)
include $(PLATFORM_PDDF_PATH)/rules.mk
endif
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
include $(PLATFORM_PATH)/rules.mk
endif
ifeq ($(USERNAME),)
override USERNAME := $(DEFAULT_USERNAME)
[baseimage]: Improve password hashing for default user account (#1748) * [slave.mk]: Fix displaying username and password in build summary We display contents of DEFAULT_USERNAME and DEFAULT_PASSWORD, while image can be build with USERNAME and/or PASSWORD given on make(1) command line. For example: $ make USERNAME=adm PASSWORD=mypass target/sonic-broadcom.bin Fix by displaying USERNAME and PASSWORD variables in build summary. Signed-off-by: Sergey Popovich <sergey.popovich@ordnance.co> * [baseimage]: Improve default user account handling There are couple of issues with current implementation of default user account management in baseimage: 1) It uses DES to encrypt accounts password. Furthermore this effectively limits password length to 8 symbols, even if more provided with PASSWORD or DEFAULT_PASSWORD from rules/config. 2) Salt value for password is same on all builds even with different password increasing attack surface. 3) During the build process password passed as command line parameter either as plain text (if given to make(1) as "make PASSWORD=...") or DES encrypted (if given to build_debian.sh) can be seen by non-build users using /proc/<pid>/cmdline file that has group and world readable permissions. Both 1) and 2) come from: perl -e 'print crypt("$(PASSWORD)", "salt"),"\n"')" that by defalt uses DES if salt does not have format $<id>$<salt>$, where <id> is hashing function id. See crypt(3) for more details on valid <id> values. To address issues above we propose following changes: 1) Do not create password by hands (e.g. using perl snippet above): put this job to chpasswd(8) which is aware about system wide password hashing policy specified in /etc/login.defs with ENCRYPT_METHOD (by default it is SHA512 for Debian 8). 2) Now chpasswd(8) will take care about proper salt value. 3) This has two steps: 3.1) For compatibility reasons accept USERNAME and PASSWORD as make(1) parameters, but warn user that this is unsafe. 3.2) Use process environment to pass USERNAME and PASSWORD variables from Makefile to build_debian.sh as more secure alternative to passing via command line parameters: /proc/<pid>/environ readable only by user running process or privileged users like root. Before change: -------------- hash1 ----- # u='admin' # p="$(LANG=C perl -e 'print crypt("YourPaSs", "salt"),"\n"')" ^^^^^^^^ 8 symbols # echo "$u:$p" | chpasswd -e # getent shadow admin admin:sazQDkwgZPfSk:17680:0:99999:7::: ^^^^^^^^^^^^^ Note the hash (DES encrypted password) hash2 ----- # u='admin' # p="$(LANG=C perl -e 'print crypt("YourPaSsWoRd", "salt"),"\n"')" ^^^^^^^^^^^^ 12 symbols # echo "$u:$p" | chpasswd -e # getent shadow admin admin:sazQDkwgZPfSk:17680:0:99999:7::: ^^^^^^^^^^^^^ Hash is the same as for "YourPaSs" After change: ------------- hash1 ----- # echo "admin:YourPaSs" | chpasswd # getent shadow admin admin:$6$1Nho1jHC$T8YwK58FYToXMFuetQta7/XouAAN2q1IzWC3bdIg86woAs6WuTg\ ^^^^^^^^ Note salt here ksLO3oyQInax/wNVq.N4de6dyWZDsCAvsZ1:17681:0:99999:7::: hash2 ----- # echo "admin:YourPaSs" | chpasswd # getent shadow admin admin:$6$yKU5g7BO$kdT02Z1wHXhr1VCniKkZbLaMPZXK0WSSVGhSLGrNhsrsVxCJ.D9\ ^^^^^^^^ Here salt completely different from case above plFpd8ksGNpw/Vb92hvgYyCL2i5cfI8QEY/:17681:0:99999:7::: Since salt is different hashes for same password different too. hash1 ----- # LANG=C perl -e 'print crypt("YourPaSs", "\$6\$salt\$"),"\n"' ^^^^^ We want SHA512 hash $6$salt$qkwPvXqUeGpexO1vatnIQFAreOTXs6rnDX.OI.Sz2rcy51JrO8dFc9aGv82bB\ yd2ELrIMJ.FQLNjgSD0nNha7/ hash2 ----- # LANG=C perl -e 'print crypt("YourPaSsWoRd", "\$6\$salt\$"),"\n"' $6$salt$1JVndGzyy/dj7PaXo6hNcttlQoZe23ob8GWYWxVGEiGOlh6sofbaIvwl6Ho7N\ kYDI8zwRumRwga/A29nHm4mZ1 Now with same "salt" and $<id>$, and same 8 symbol prefix in password, but different password length we have different hashes. Signed-off-by: Sergey Popovich <sergey.popovich@ordnance.co>
2018-06-09 13:29:16 -05:00
else
$(warning USERNAME given on command line: could be visible to other users)
endif
ifeq ($(PASSWORD),)
override PASSWORD := $(DEFAULT_PASSWORD)
[baseimage]: Improve password hashing for default user account (#1748) * [slave.mk]: Fix displaying username and password in build summary We display contents of DEFAULT_USERNAME and DEFAULT_PASSWORD, while image can be build with USERNAME and/or PASSWORD given on make(1) command line. For example: $ make USERNAME=adm PASSWORD=mypass target/sonic-broadcom.bin Fix by displaying USERNAME and PASSWORD variables in build summary. Signed-off-by: Sergey Popovich <sergey.popovich@ordnance.co> * [baseimage]: Improve default user account handling There are couple of issues with current implementation of default user account management in baseimage: 1) It uses DES to encrypt accounts password. Furthermore this effectively limits password length to 8 symbols, even if more provided with PASSWORD or DEFAULT_PASSWORD from rules/config. 2) Salt value for password is same on all builds even with different password increasing attack surface. 3) During the build process password passed as command line parameter either as plain text (if given to make(1) as "make PASSWORD=...") or DES encrypted (if given to build_debian.sh) can be seen by non-build users using /proc/<pid>/cmdline file that has group and world readable permissions. Both 1) and 2) come from: perl -e 'print crypt("$(PASSWORD)", "salt"),"\n"')" that by defalt uses DES if salt does not have format $<id>$<salt>$, where <id> is hashing function id. See crypt(3) for more details on valid <id> values. To address issues above we propose following changes: 1) Do not create password by hands (e.g. using perl snippet above): put this job to chpasswd(8) which is aware about system wide password hashing policy specified in /etc/login.defs with ENCRYPT_METHOD (by default it is SHA512 for Debian 8). 2) Now chpasswd(8) will take care about proper salt value. 3) This has two steps: 3.1) For compatibility reasons accept USERNAME and PASSWORD as make(1) parameters, but warn user that this is unsafe. 3.2) Use process environment to pass USERNAME and PASSWORD variables from Makefile to build_debian.sh as more secure alternative to passing via command line parameters: /proc/<pid>/environ readable only by user running process or privileged users like root. Before change: -------------- hash1 ----- # u='admin' # p="$(LANG=C perl -e 'print crypt("YourPaSs", "salt"),"\n"')" ^^^^^^^^ 8 symbols # echo "$u:$p" | chpasswd -e # getent shadow admin admin:sazQDkwgZPfSk:17680:0:99999:7::: ^^^^^^^^^^^^^ Note the hash (DES encrypted password) hash2 ----- # u='admin' # p="$(LANG=C perl -e 'print crypt("YourPaSsWoRd", "salt"),"\n"')" ^^^^^^^^^^^^ 12 symbols # echo "$u:$p" | chpasswd -e # getent shadow admin admin:sazQDkwgZPfSk:17680:0:99999:7::: ^^^^^^^^^^^^^ Hash is the same as for "YourPaSs" After change: ------------- hash1 ----- # echo "admin:YourPaSs" | chpasswd # getent shadow admin admin:$6$1Nho1jHC$T8YwK58FYToXMFuetQta7/XouAAN2q1IzWC3bdIg86woAs6WuTg\ ^^^^^^^^ Note salt here ksLO3oyQInax/wNVq.N4de6dyWZDsCAvsZ1:17681:0:99999:7::: hash2 ----- # echo "admin:YourPaSs" | chpasswd # getent shadow admin admin:$6$yKU5g7BO$kdT02Z1wHXhr1VCniKkZbLaMPZXK0WSSVGhSLGrNhsrsVxCJ.D9\ ^^^^^^^^ Here salt completely different from case above plFpd8ksGNpw/Vb92hvgYyCL2i5cfI8QEY/:17681:0:99999:7::: Since salt is different hashes for same password different too. hash1 ----- # LANG=C perl -e 'print crypt("YourPaSs", "\$6\$salt\$"),"\n"' ^^^^^ We want SHA512 hash $6$salt$qkwPvXqUeGpexO1vatnIQFAreOTXs6rnDX.OI.Sz2rcy51JrO8dFc9aGv82bB\ yd2ELrIMJ.FQLNjgSD0nNha7/ hash2 ----- # LANG=C perl -e 'print crypt("YourPaSsWoRd", "\$6\$salt\$"),"\n"' $6$salt$1JVndGzyy/dj7PaXo6hNcttlQoZe23ob8GWYWxVGEiGOlh6sofbaIvwl6Ho7N\ kYDI8zwRumRwga/A29nHm4mZ1 Now with same "salt" and $<id>$, and same 8 symbol prefix in password, but different password length we have different hashes. Signed-off-by: Sergey Popovich <sergey.popovich@ordnance.co>
2018-06-09 13:29:16 -05:00
else
$(warning PASSWORD given on command line: could be visible to other users)
endif
ifeq ($(SONIC_DEBUGGING_ON),y)
DEB_BUILD_OPTIONS_GENERIC := nostrip
endif
ifeq ($(SONIC_PROFILING_ON),y)
DEB_BUILD_OPTIONS_GENERIC := nostrip noopt
endif
ifeq ($(SONIC_BUILD_JOBS),)
override SONIC_BUILD_JOBS := $(SONIC_CONFIG_BUILD_JOBS)
endif
ifeq ($(VS_PREPARE_MEM),)
override VS_PREPARE_MEM := $(DEFAULT_VS_PREPARE_MEM)
endif
ifeq ($(KERNEL_PROCURE_METHOD),)
override KERNEL_PROCURE_METHOD := $(DEFAULT_KERNEL_PROCURE_METHOD)
endif
ifeq ($(BUILD_LOG_TIMESTAMP),)
override BUILD_LOG_TIMESTAMP := $(DEFAULT_BUILD_LOG_TIMESTAMP)
endif
MAKEFLAGS += -j $(SONIC_BUILD_JOBS)
export SONIC_CONFIG_MAKE_JOBS
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Routing stack related exports
###############################################################################
export SONIC_ROUTING_STACK
export FRR_USER_UID
export FRR_USER_GID
###############################################################################
## Dumping key config attributes associated to current building exercise
###############################################################################
ifndef SONIC_BUILD_QUIETER
$(info SONiC Build System)
$(info )
$(info Build Configuration)
$(info "CONFIGURED_PLATFORM" : "$(if $(PLATFORM),$(PLATFORM),$(CONFIGURED_PLATFORM))")
$(info "CONFIGURED_ARCH" : "$(if $(PLATFORM_ARCH),$(PLATFORM_ARCH),$(CONFIGURED_ARCH))")
$(info "SONIC_CONFIG_PRINT_DEPENDENCIES" : "$(SONIC_CONFIG_PRINT_DEPENDENCIES)")
$(info "SONIC_BUILD_JOBS" : "$(SONIC_BUILD_JOBS)")
$(info "SONIC_CONFIG_MAKE_JOBS" : "$(SONIC_CONFIG_MAKE_JOBS)")
$(info "SONIC_USE_DOCKER_BUILDKIT" : "$(SONIC_USE_DOCKER_BUILDKIT)")
[baseimage]: Improve password hashing for default user account (#1748) * [slave.mk]: Fix displaying username and password in build summary We display contents of DEFAULT_USERNAME and DEFAULT_PASSWORD, while image can be build with USERNAME and/or PASSWORD given on make(1) command line. For example: $ make USERNAME=adm PASSWORD=mypass target/sonic-broadcom.bin Fix by displaying USERNAME and PASSWORD variables in build summary. Signed-off-by: Sergey Popovich <sergey.popovich@ordnance.co> * [baseimage]: Improve default user account handling There are couple of issues with current implementation of default user account management in baseimage: 1) It uses DES to encrypt accounts password. Furthermore this effectively limits password length to 8 symbols, even if more provided with PASSWORD or DEFAULT_PASSWORD from rules/config. 2) Salt value for password is same on all builds even with different password increasing attack surface. 3) During the build process password passed as command line parameter either as plain text (if given to make(1) as "make PASSWORD=...") or DES encrypted (if given to build_debian.sh) can be seen by non-build users using /proc/<pid>/cmdline file that has group and world readable permissions. Both 1) and 2) come from: perl -e 'print crypt("$(PASSWORD)", "salt"),"\n"')" that by defalt uses DES if salt does not have format $<id>$<salt>$, where <id> is hashing function id. See crypt(3) for more details on valid <id> values. To address issues above we propose following changes: 1) Do not create password by hands (e.g. using perl snippet above): put this job to chpasswd(8) which is aware about system wide password hashing policy specified in /etc/login.defs with ENCRYPT_METHOD (by default it is SHA512 for Debian 8). 2) Now chpasswd(8) will take care about proper salt value. 3) This has two steps: 3.1) For compatibility reasons accept USERNAME and PASSWORD as make(1) parameters, but warn user that this is unsafe. 3.2) Use process environment to pass USERNAME and PASSWORD variables from Makefile to build_debian.sh as more secure alternative to passing via command line parameters: /proc/<pid>/environ readable only by user running process or privileged users like root. Before change: -------------- hash1 ----- # u='admin' # p="$(LANG=C perl -e 'print crypt("YourPaSs", "salt"),"\n"')" ^^^^^^^^ 8 symbols # echo "$u:$p" | chpasswd -e # getent shadow admin admin:sazQDkwgZPfSk:17680:0:99999:7::: ^^^^^^^^^^^^^ Note the hash (DES encrypted password) hash2 ----- # u='admin' # p="$(LANG=C perl -e 'print crypt("YourPaSsWoRd", "salt"),"\n"')" ^^^^^^^^^^^^ 12 symbols # echo "$u:$p" | chpasswd -e # getent shadow admin admin:sazQDkwgZPfSk:17680:0:99999:7::: ^^^^^^^^^^^^^ Hash is the same as for "YourPaSs" After change: ------------- hash1 ----- # echo "admin:YourPaSs" | chpasswd # getent shadow admin admin:$6$1Nho1jHC$T8YwK58FYToXMFuetQta7/XouAAN2q1IzWC3bdIg86woAs6WuTg\ ^^^^^^^^ Note salt here ksLO3oyQInax/wNVq.N4de6dyWZDsCAvsZ1:17681:0:99999:7::: hash2 ----- # echo "admin:YourPaSs" | chpasswd # getent shadow admin admin:$6$yKU5g7BO$kdT02Z1wHXhr1VCniKkZbLaMPZXK0WSSVGhSLGrNhsrsVxCJ.D9\ ^^^^^^^^ Here salt completely different from case above plFpd8ksGNpw/Vb92hvgYyCL2i5cfI8QEY/:17681:0:99999:7::: Since salt is different hashes for same password different too. hash1 ----- # LANG=C perl -e 'print crypt("YourPaSs", "\$6\$salt\$"),"\n"' ^^^^^ We want SHA512 hash $6$salt$qkwPvXqUeGpexO1vatnIQFAreOTXs6rnDX.OI.Sz2rcy51JrO8dFc9aGv82bB\ yd2ELrIMJ.FQLNjgSD0nNha7/ hash2 ----- # LANG=C perl -e 'print crypt("YourPaSsWoRd", "\$6\$salt\$"),"\n"' $6$salt$1JVndGzyy/dj7PaXo6hNcttlQoZe23ob8GWYWxVGEiGOlh6sofbaIvwl6Ho7N\ kYDI8zwRumRwga/A29nHm4mZ1 Now with same "salt" and $<id>$, and same 8 symbol prefix in password, but different password length we have different hashes. Signed-off-by: Sergey Popovich <sergey.popovich@ordnance.co>
2018-06-09 13:29:16 -05:00
$(info "USERNAME" : "$(USERNAME)")
$(info "PASSWORD" : "$(PASSWORD)")
$(info "ENABLE_DHCP_GRAPH_SERVICE" : "$(ENABLE_DHCP_GRAPH_SERVICE)")
$(info "SHUTDOWN_BGP_ON_START" : "$(SHUTDOWN_BGP_ON_START)")
$(info "ENABLE_PFCWD_ON_START" : "$(ENABLE_PFCWD_ON_START)")
[Dynamic buffer calc] Support dynamic buffer calculation (#6194) **- Why I did it** To support dynamic buffer calculation. This PR also depends on the following PRs for sub modules - [sonic-swss: [buffermgr/bufferorch] Support dynamic buffer calculation #1338](https://github.com/Azure/sonic-swss/pull/1338) - [sonic-swss-common: Dynamic buffer calculation #361](https://github.com/Azure/sonic-swss-common/pull/361) - [sonic-utilities: Support dynamic buffer calculation #973](https://github.com/Azure/sonic-utilities/pull/973) **- How I did it** 1. Introduce field `buffer_model` in `DEVICE_METADATA|localhost` to represent which buffer model is running in the system currently: - `dynamic` for the dynamic buffer calculation model - `traditional` for the traditional model in which the `pg_profile_lookup.ini` is used 2. Add the tables required for the feature: - ASIC_TABLE in platform/\<vendor\>/asic_table.j2 - PERIPHERAL_TABLE in platform/\<vendor\>/peripheral_table.j2 - PORT_PERIPHERAL_TABLE on a per-platform basis in device/\<vendor\>/\<platform\>/port_peripheral_config.j2 for each platform with gearbox installed. - DEFAULT_LOSSLESS_BUFFER_PARAMETER and LOSSLESS_TRAFFIC_PATTERN in files/build_templates/buffers_config.j2 - Add lossless PGs (3-4) for each port in files/build_templates/buffers_config.j2 3. Copy the newly introduced j2 files into the image and rendering them when the system starts 4. Update the CLI options for buffermgrd so that it can start with dynamic mode 5. Fetches the ASIC vendor name in orchagent: - fetch the vendor name when creates the docker and pass it as a docker environment variable - `buffermgrd` can use this passed-in variable 6. Clear buffer related tables from STATE_DB when swss docker starts 7. Update the src/sonic-config-engine/tests/sample_output/buffers-dell6100.json according to the buffer_config.j2 8. Remove buffer pool sizes for ingress pools and egress_lossy_pool Update the buffer settings for dynamic buffer calculation
2020-12-13 13:35:39 -06:00
$(info "SONIC_BUFFER_MODEL" : "$(SONIC_BUFFER_MODEL)")
$(info "INSTALL_DEBUG_TOOLS" : "$(INSTALL_DEBUG_TOOLS)")
$(info "ROUTING_STACK" : "$(SONIC_ROUTING_STACK)")
ifeq ($(SONIC_ROUTING_STACK),frr)
$(info "FRR_USER_UID" : "$(FRR_USER_UID)")
$(info "FRR_USER_GID" : "$(FRR_USER_GID)")
endif
$(info "ENABLE_SYNCD_RPC" : "$(ENABLE_SYNCD_RPC)")
$(info "ENABLE_ORGANIZATION_EXTENSIONS" : "$(ENABLE_ORGANIZATION_EXTENSIONS)")
$(info "HTTP_PROXY" : "$(HTTP_PROXY)")
$(info "HTTPS_PROXY" : "$(HTTPS_PROXY)")
$(info "NO_PROXY" : "$(NO_PROXY)")
$(info "ENABLE_ZTP" : "$(ENABLE_ZTP)")
$(info "SONIC_DEBUGGING_ON" : "$(SONIC_DEBUGGING_ON)")
$(info "SONIC_PROFILING_ON" : "$(SONIC_PROFILING_ON)")
$(info "KERNEL_PROCURE_METHOD" : "$(KERNEL_PROCURE_METHOD)")
$(info "BUILD_TIMESTAMP" : "$(BUILD_TIMESTAMP)")
$(info "BUILD_LOG_TIMESTAMP" : "$(BUILD_LOG_TIMESTAMP)")
$(info "SONIC_IMAGE_VERSION" : "$(SONIC_IMAGE_VERSION)")
$(info "BLDENV" : "$(BLDENV)")
$(info "VS_PREPARE_MEM" : "$(VS_PREPARE_MEM)")
$(info "INCLUDE_MGMT_FRAMEWORK" : "$(INCLUDE_MGMT_FRAMEWORK)")
$(info "INCLUDE_ICCPD" : "$(INCLUDE_ICCPD)")
$(info "INCLUDE_SYSTEM_TELEMETRY" : "$(INCLUDE_SYSTEM_TELEMETRY)")
$(info "ENABLE_HOST_SERVICE_ON_START" : "$(ENABLE_HOST_SERVICE_ON_START)")
$(info "INCLUDE_RESTAPI" : "$(INCLUDE_RESTAPI)")
$(info "INCLUDE_SFLOW" : "$(INCLUDE_SFLOW)")
$(info "INCLUDE_NAT" : "$(INCLUDE_NAT)")
[dhcp-relay] make DHCP relay an extension (#6531) - Why I did it Make DHCP relay docker an extension. DHCP relay now carries dhcp relay commands CLI plugin and has a complete manifest. It is installed as extension if INCLUDE_DHCP_REALY is set to y. DEPENDS on #5939 - How I did it Modify DHCP relay docker makefile and dockerfile. Make changes to sonic_debian_extension.j2 to install sonic packages. I moved DHCP related CLI tests from sonic-utilities to DHCP relay docker. This PR introduces a way to write a plugin as part of docker image and run the tests from cli-plugin-tests directory under docker directory. The test result is available in target/docker-dhcp-relay.gz.log: [ REASON ] : target/docker-dhcp-relay.gz does not exist NON-EXISTENT PREREQUISITES: docker-start target/docker-config-engine-buster.gz-load target/python-wheels/sonic_utilities-1.2-py3-none-any.whl-in stall target/debs/buster/python3-swsscommon_1.0.0_amd64.deb-install [ FLAGS FILE ] : [] [ FLAGS DEPENDS ] : [] [ FLAGS DIFF ] : [] ============================= test session starts ============================== platform linux -- Python 3.7.3, pytest-3.10.1, py-1.7.0, pluggy-0.8.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /sonic/dockers/docker-dhcp-relay/cli-plugin-tests, inifile: plugins: cov-2.6.0 collecting ... collected 10 items test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_plugin_registration PASSED [ 10%] test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_nonexist_vlanid PASSED [ 20%] test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_invalid_vlanid PASSED [ 30%] test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_invalid_ip PASSED [ 40%] test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_dhcp_relay_with_exist_ip PASSED [ 50%] test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_add_del_dhcp_relay_dest PASSED [ 60%] test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_remove_nonexist_dhcp_relay_dest PASSED [ 70%] test_config_dhcp_relay.py::TestConfigVlanDhcpRelay::test_config_vlan_remove_dhcp_relay_dest_with_nonexist_vlanid PASSED [ 80%] test_show_dhcp_relay.py::TestVlanDhcpRelay::test_plugin_registration PASSED [ 90%] test_show_dhcp_relay.py::TestVlanDhcpRelay::test_dhcp_relay_column_output PASSED [100%] =============================== warnings summary =============================== /usr/local/lib/python3.7/dist-packages/tabulate.py:7 /usr/local/lib/python3.7/dist-packages/tabulate.py:7: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import namedtuple, Iterable -- Docs: https://docs.pytest.org/en/latest/warnings.html ==================== 10 passed, 1 warnings in 0.35 seconds =====================
2021-07-15 12:35:56 -05:00
$(info "INCLUDE_DHCP_RELAY" : "$(INCLUDE_DHCP_RELAY)")
$(info "INCLUDE_KUBERNETES" : "$(INCLUDE_KUBERNETES)")
$(info "INCLUDE_MACSEC" : "$(INCLUDE_MACSEC)")
$(info "TELEMETRY_WRITABLE" : "$(TELEMETRY_WRITABLE)")
$(info "PDDF_SUPPORT" : "$(PDDF_SUPPORT)")
$(info "MULTIARCH_QEMU_ENVIRON" : "$(MULTIARCH_QEMU_ENVIRON)")
$(info "SONIC_VERSION_CONTROL_COMPONENTS": "$(SONIC_VERSION_CONTROL_COMPONENTS)")
$(info "DEFAULT_CONTAINER_REGISTRY" : "$(SONIC_DEFAULT_CONTAINER_REGISTRY)")
$(info )
else
$(info SONiC Build System for $(CONFIGURED_PLATFORM):$(CONFIGURED_ARCH))
endif
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Overwrite the buildinfo in slave container
$(shell sudo scripts/prepare_slave_container_buildinfo.sh $(SLAVE_DIR) $(CONFIGURED_ARCH) $(BLDENV))
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
include Makefile.cache
ifeq ($(SONIC_USE_DOCKER_BUILDKIT),y)
$(warning "Using SONIC_USE_DOCKER_BUILDKIT will produce larger installable SONiC image because of a docker bug (more details: https://github.com/moby/moby/issues/38903)")
export DOCKER_BUILDKIT=1
endif
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Generic rules section
## All rules must go after includes for propper targets expansion
###############################################################################
export kernel_procure_method=$(KERNEL_PROCURE_METHOD)
export vs_build_prepare_mem=$(VS_PREPARE_MEM)
###############################################################################
## Local targets
###############################################################################
# Copy debian packages from local directory
# Add new package for copy:
# SOME_NEW_DEB = some_new_deb.deb
# $(SOME_NEW_DEB)_PATH = path/to/some_new_deb.deb
# SONIC_COPY_DEBS += $(SOME_NEW_DEB)
$(addprefix $(DEBS_PATH)/, $(SONIC_COPY_DEBS)) : $(DEBS_PATH)/% : .platform
$(HEADER)
$(foreach deb,$* $($*_DERIVED_DEBS), \
{ cp $($(deb)_PATH)/$(deb) $(DEBS_PATH)/ $(LOG) || exit 1 ; } ; )
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(DEBS_PATH)/, $(SONIC_COPY_DEBS))
# Copy regular files from local directory
# Add new package for copy:
# SOME_NEW_FILE = some_new_file
# $(SOME_NEW_FILE)_PATH = path/to/some_new_file
# SONIC_COPY_FILES += $(SOME_NEW_FILE)
$(addprefix $(FILES_PATH)/, $(SONIC_COPY_FILES)) : $(FILES_PATH)/% : .platform
$(HEADER)
cp $($*_PATH)/$* $(FILES_PATH)/ $(LOG) || exit 1
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(FILES_PATH)/, $(SONIC_COPY_FILES))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Online targets
###############################################################################
# Download debian packages from online location
# Add new package for download:
# SOME_NEW_DEB = some_new_deb.deb
# $(SOME_NEW_DEB)_URL = https://url/to/this/deb.deb
# SONIC_ONLINE_DEBS += $(SOME_NEW_DEB)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(addprefix $(DEBS_PATH)/, $(SONIC_ONLINE_DEBS)) : $(DEBS_PATH)/% : .platform \
$(call dpkg_depend,$(DEBS_PATH)/%.dep)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(HEADER)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
$(foreach deb,$* $($*_DERIVED_DEBS), \
{ curl -L -f -o $(DEBS_PATH)/$(deb) $($(deb)_CURL_OPTIONS) $($(deb)_URL) $(LOG) || { exit 1 ; } } ; )
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(DEBS_PATH)/, $(SONIC_ONLINE_DEBS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
# Download regular files from online location
# Files are stored in deb packages directory for convenience
# Add new file for download:
# SOME_NEW_FILE = some_new_file
# $(SOME_NEW_FILE)_URL = https://url/to/this/file
# SONIC_ONLINE_FILES += $(SOME_NEW_FILE)
$(addprefix $(FILES_PATH)/, $(SONIC_ONLINE_FILES)) : $(FILES_PATH)/% : .platform
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(HEADER)
curl -L -f -o $@ $($*_CURL_OPTIONS) $($*_URL) $(LOG)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(FILES_PATH)/, $(SONIC_ONLINE_FILES))
###############################################################################
## Build targets
###############################################################################
# Build project using build.sh script
# They are essentially a one-time build projects that get sources from some URL
# and compile them
# Add new file for build:
# SOME_NEW_FILE = some_new_deb.deb
# $(SOME_NEW_FILE)_SRC_PATH = $(SRC_PATH)/project_name
# $(SOME_NEW_FILE)_DEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ...
# SONIC_MAKE_FILES += $(SOME_NEW_FILE)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(addprefix $(FILES_PATH)/, $(SONIC_MAKE_FILES)) : $(FILES_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
$$(addprefix $(DEBS_PATH)/,$$($$*_AFTER)) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(call dpkg_depend,$(FILES_PATH)/%.dep)
$(HEADER)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
# Remove target to force rebuild
rm -f $(addprefix $(FILES_PATH)/, $*)
# Apply series of patches if exist
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && quilt applied || QUILT_PATCHES=../$(notdir $($*_SRC_PATH)).patch quilt push -a; popd; fi
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Build project and take package
make DEST=$(shell pwd)/$(FILES_PATH) -C $($*_SRC_PATH) $(shell pwd)/$(FILES_PATH)/$* $(LOG)
# Clean up
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && quilt pop -a -f; [ -d .pc ] && rm -rf .pc; popd; fi
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
# Uninstall unneeded build dependency
$(call UNINSTALL_DEBS,$($*_UNINSTALLS))
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(FILES_PATH)/, $(SONIC_MAKE_FILES))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Debian package related targets
###############################################################################
# Build project using build.sh script
# They are essentially a one-time build projects that get sources from some URL
# and compile them
# Add new package for build:
# SOME_NEW_DEB = some_new_deb.deb
# $(SOME_NEW_DEB)_SRC_PATH = $(SRC_PATH)/project_name
# $(SOME_NEW_DEB)_DEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ...
# SONIC_MAKE_DEBS += $(SOME_NEW_DEB)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(addprefix $(DEBS_PATH)/, $(SONIC_MAKE_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
$$(addprefix $(DEBS_PATH)/,$$($$*_AFTER)) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(call dpkg_depend,$(DEBS_PATH)/%.dep)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(HEADER)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
# Remove target to force rebuild
rm -f $(addprefix $(DEBS_PATH)/, $* $($*_DERIVED_DEBS) $($*_EXTRA_DEBS))
# Apply series of patches if exist
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && QUILT_PATCHES=../$(notdir $($*_SRC_PATH)).patch quilt push -a; popd; fi
# Build project and take package
$(SETUP_OVERLAYFS_FOR_DPKG_ADMINDIR)
DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC}" make -j$(SONIC_CONFIG_MAKE_JOBS) DEST=$(shell pwd)/$(DEBS_PATH) -C $($*_SRC_PATH) $(shell pwd)/$(DEBS_PATH)/$* $(LOG)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Clean up
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && quilt pop -a -f; [ -d .pc ] && rm -rf .pc; popd; fi
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
# Uninstall unneeded build dependency
$(call UNINSTALL_DEBS,$($*_UNINSTALLS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(DEBS_PATH)/, $(SONIC_MAKE_DEBS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
# Build project with dpkg-buildpackage
# Add new package for build:
# SOME_NEW_DEB = some_new_deb.deb
# $(SOME_NEW_DEB)_SRC_PATH = $(SRC_PATH)/project_name
# $(SOME_NEW_DEB)_DEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ...
# SONIC_DPKG_DEBS += $(SOME_NEW_DEB)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
$$(addprefix $(DEBS_PATH)/,$$($$*_AFTER)) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(call dpkg_depend,$(DEBS_PATH)/%.dep )
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(HEADER)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
# Remove old build logs if they exist
rm -f $($*_SRC_PATH)/debian/*.debhelper.log
# Apply series of patches if exist
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && QUILT_PATCHES=../$(notdir $($*_SRC_PATH)).patch quilt push -a; popd; fi
# Build project
pushd $($*_SRC_PATH) $(LOG_SIMPLE)
if [ -f ./autogen.sh ]; then ./autogen.sh $(LOG); fi
$(SETUP_OVERLAYFS_FOR_DPKG_ADMINDIR)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(if $($*_DPKG_TARGET),
${$*_BUILD_ENV} DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC} ${$*_DEB_BUILD_OPTIONS}" dpkg-buildpackage -rfakeroot -b -us -uc -j$(SONIC_CONFIG_MAKE_JOBS) --as-root -T$($*_DPKG_TARGET) --admindir $$mergedir $(LOG),
${$*_BUILD_ENV} DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC} ${$*_DEB_BUILD_OPTIONS}" dpkg-buildpackage -rfakeroot -b -us -uc -j$(SONIC_CONFIG_MAKE_JOBS) --admindir $$mergedir $(LOG)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
)
popd $(LOG_SIMPLE)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Clean up
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && quilt pop -a -f; [ -d .pc ] && rm -rf .pc; popd; fi
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Take built package(s)
mv -f $(addprefix $($*_SRC_PATH)/../, $* $($*_DERIVED_DEBS) $($*_EXTRA_DEBS)) $(DEBS_PATH) $(LOG)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
# Uninstall unneeded build dependency
$(call UNINSTALL_DEBS,$($*_UNINSTALLS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
# Rules for derived debian packages (dev, dbg, etc.)
# All noise takes place in main deb recipe, so we are just telling that
# we depend on it and move our deb to other targets
# Add new dev package:
# $(eval $(call add_derived_package,$(ORIGINAL_DEB),derived_deb_file.deb))
$(addprefix $(DEBS_PATH)/, $(SONIC_DERIVED_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS)))
$(HEADER)
# All noise takes place in main deb recipe, so we are just telling that
# we depend on it
# Put newer timestamp
[ -f $@ ] && touch $@
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(DEBS_PATH)/, $(SONIC_DERIVED_DEBS))
# Rules for extra debian packages
# All noise takes place in main deb recipe, so we are just telling that
# we need to build the main deb and move our deb to other targets
# Add new dev package:
# $(eval $(call add_extra_package,$(ORIGINAL_DEB),extra_deb_file.deb))
$(addprefix $(DEBS_PATH)/, $(SONIC_EXTRA_DEBS)) : $(DEBS_PATH)/% : .platform $$(addprefix $(DEBS_PATH)/,$$($$*_MAIN_DEB))
$(HEADER)
# All noise takes place in main deb recipe, so we are just telling that
# we depend on it
# Put newer timestamp
[ -f $@ ] && touch $@
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(DEBS_PATH)/, $(SONIC_EXTRA_DEBS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
# Targets for installing debian packages prior to build one that depends on them
SONIC_INSTALL_DEBS = $(addsuffix -install,$(addprefix $(DEBS_PATH)/, \
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(SONIC_ONLINE_DEBS) \
$(SONIC_COPY_DEBS) \
$(SONIC_MAKE_DEBS) \
$(SONIC_DPKG_DEBS) \
$(SONIC_PYTHON_STDEB_DEBS) \
$(SONIC_DERIVED_DEBS) \
$(SONIC_EXTRA_DEBS)))
$(SONIC_INSTALL_DEBS) : $(DEBS_PATH)/%-install : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) $(DEBS_PATH)/$$*
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(HEADER)
[ -f $(DEBS_PATH)/$* ] || { echo $(DEBS_PATH)/$* does not exist $(LOG) && false $(LOG) }
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
while true; do
# wait for conflicted packages to be uninstalled
$(foreach deb, $($*_CONFLICT_DEBS), \
{ while dpkg -s $(firstword $(subst _, ,$(basename $(deb)))) | grep "^Version: $(word 2, $(subst _, ,$(basename $(deb))))" &> /dev/null; do echo "waiting for $(deb) to be uninstalled" $(LOG); sleep 1; done } )
# put a lock here because dpkg does not allow installing packages in parallel
if mkdir $(DEBS_PATH)/dpkg_lock &> /dev/null; then
{ sudo DEBIAN_FRONTEND=noninteractive dpkg -i $(DEBS_PATH)/$* $(LOG) && rm -d $(DEBS_PATH)/dpkg_lock && break; } || { rm -d $(DEBS_PATH)/dpkg_lock && sudo lsof /var/lib/dpkg/lock-frontend && ps aux && exit 1 ; }
fi
done
$(FOOTER)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Python packages
###############################################################################
# Build project with python setup.py --command-packages=stdeb.command
# Add new package for build:
# SOME_NEW_DEB = some_new_deb.deb
# $(SOME_NEW_DEB)_SRC_PATH = $(SRC_PATH)/project_name
# $(SOME_NEW_DEB)_DEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ...
# SONIC_PYTHON_STDEB_DEBS += $(SOME_NEW_DEB)
$(addprefix $(PYTHON_DEBS_PATH)/, $(SONIC_PYTHON_STDEB_DEBS)) : $(PYTHON_DEBS_PATH)/% : .platform \
$$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEBS_DEPENDS))) \
$$(addsuffix -install,$$(addprefix $(PYTHON_DEBS_PATH)/,$$($$*_DEPENDS))) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$$(addsuffix -install,$$(addprefix $(PYTHON_WHEELS_PATH)/,$$($$*_WHEEL_DEPENDS))) \
$(call dpkg_depend,$(PYTHON_DEBS_PATH)/%.dep)
$(HEADER)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
# Apply series of patches if exist
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && QUILT_PATCHES=../$(notdir $($*_SRC_PATH)).patch quilt push -a; popd; fi
# Build project
pushd $($*_SRC_PATH) $(LOG_SIMPLE)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
rm -rf deb_dist/* $(LOG)
python setup.py --command-packages=stdeb.command bdist_deb $(LOG)
popd $(LOG_SIMPLE)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Clean up
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && quilt pop -a -f; [ -d .pc ] && rm -rf .pc; popd; fi
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Take built package(s)
mv -f $(addprefix $($*_SRC_PATH)/deb_dist/, $* $($*_DERIVED_DEBS)) $(PYTHON_DEBS_PATH) $(LOG)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(PYTHON_DEBS_PATH)/, $(SONIC_PYTHON_STDEB_DEBS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
# Build project using python setup.py bdist_wheel
# Projects that generate python wheels
# Add new package for build:
# SOME_NEW_WHL = some_new_whl.whl
# $(SOME_NEW_WHL)_SRC_PATH = $(SRC_PATH)/project_name
# $(SOME_NEW_WHL)_PYTHON_VERSION = 2 (or 3)
# $(SOME_NEW_WHL)_DEPENDS = $(SOME_OTHER_WHL1) $(SOME_OTHER_WHL2) ...
# SONIC_PYTHON_WHEELS += $(SOME_NEW_WHL)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(addprefix $(PYTHON_WHEELS_PATH)/, $(SONIC_PYTHON_WHEELS)) : $(PYTHON_WHEELS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(PYTHON_WHEELS_PATH)/,$$($$*_DEPENDS))) \
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. (#3730) [sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. YANG models as per Guidelines. Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md [sonic-yang-models/tests]: YANG model test code and JSON input for testing. [sonic-yang-models/setup.py]: Build infra for yang models. **- What I did** Created Yang model for Sonic. Tables: PORT, VLAN, VLAN_INTERFACE, VLAN_MEMBER, ACL_RULE, ACL_TABLE, INTERFACE. Created build infra files using which a new package (sonic-yang-models) can be build and can be deployed on sonic switches. Yang models will be part of this new package. **- How I did it** Wrote yang models based on Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md and https://github.com/Azure/SONiC/wiki/Configuration. Wrote python wheel Package infra which runs test for these Yang models using a json files which consists configuration as per yang models. These configs are for negative tests, which means we want to test that most must condition, pattern and when condition works as expected. **- How to verify it** Build Logs and testing: ——————————————————————————————————— ``` /sonic/src/sonic-yang-models /sonic running test running egg_info writing top-level names to sonic_yang_models.egg-info/top_level.txt writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt writing sonic_yang_models.egg-info/PKG-INFO reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt' writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt' running build_ext ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK running bdist_wheel running build running build_py (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../libyang_1.0.73_amd64.deb ... Unpacking libyang (1.0.73) over (1.0.73) ... Setting up libyang (1.0.73) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... Processing triggers for man-db (2.7.6.1-2) ... (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../libyang-cpp_1.0.73_amd64.deb ... Unpacking libyang-cpp (1.0.73) over (1.0.73) ... Setting up libyang-cpp (1.0.73) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../python3-yang_1.0.73_amd64.deb ... Unpacking python3-yang (1.0.73) over (1.0.73) ... Setting up python3-yang (1.0.73) ... INFO:YANG-TEST:module: sonic-vlan is loaded successfully ERROR:YANG-TEST:Could not get module: sonic-head INFO:YANG-TEST:module: sonic-portchannel is loaded successfully INFO:YANG-TEST:module: sonic-acl is loaded successfully INFO:YANG-TEST:module: sonic-loopback-interface is loaded successfully ERROR:YANG-TEST:Could not get module: sonic-port INFO:YANG-TEST:module: sonic-interface is loaded successfully INFO:YANG-TEST: ------------------- Test 1: Configure a member port in VLAN_MEMBER table which does not exist.--------------------- libyang[0]: Leafref "/sonic-port:sonic-port/sonic-port:PORT/sonic-port:PORT_LIST/sonic-port:port_name" of value "Ethernet156" points to a non -existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlan_name='Vlan100'][port='Ethernet156']/port) INFO:YANG-TEST:Configure a member port in VLAN_MEMBER table which does not exist. Passed INFO:YANG-TEST: ------------------- Test 2: Configure non-existing ACL_TABLE in ACL_RULE.--------------------- libyang[0]: Leafref "/sonic-acl:sonic-acl/sonic-acl:ACL_TABLE/sonic-acl:ACL_TABLE_LIST/sonic-acl:ACL_TABLE_NAME" of value "NOT-EXIST" points to a non-existing leaf. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NOT-EXIST'][RULE_NAME='Rule_20']/ACL_TABLE_NAME) INFO:YANG-TEST:Configure non-existing ACL_TABLE in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 3: Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/ICMPV6_CODE) INFO:YANG-TEST:Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE. Passed INFO:YANG-TEST: INFO:YANG-TEST: ------------------- Test 4: Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_20']/SRC_IPV6) INFO:YANG-TEST:Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE. Passed ------------------- Test 5: Configure l4_src_port_range as 99999-99999 in ACL_RULE--------------------- libyang[0]: Value "99999-99999" does not satisfy the constraint "([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])-([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6'][RULE_NAME='Rule_20']/L4_SRC_PORT_RANGE) INFO:YANG-TEST:Configure l4_src_port_range as 99999-99999 in ACL_RULE Passed INFO:YANG-TEST: ------------------- Test 6: Configure empty string as ip-prefix in INTERFACE table.--------------------- libyang[0]: Invalid value "" in "ip-prefix" element. (path: /sonic-interface:sonic-interface/INTERFACE/INTERFACE_LIST[interface='Ethernet8'][ip-prefix='']/ip-prefix) INFO:YANG-TEST:Configure empty string as ip-prefix in INTERFACE table. Passed INFO:YANG-TEST: ------------------- Test 7: Configure Wrong family with ip-prefix for VLAN_Interface Table--------------------- libyang[0]: Must condition "(contains(../ip-prefix, ':') and current()='IPv6') or (contains(../ip-prefix, '.') and current()='IPv4')" not satisfied. (path: /sonic-vlan:sonic-vlan/VLAN_INTERFACE/VLAN_INTERFACE_LIST[vlanid='100'][ip-prefix='2a04:5555:66:7777::1/64']/family) INFO:YANG-TEST:Configure Wrong family with ip-prefix for VLAN_Interface Table Passed INFO:YANG-TEST: ------------------- Test 8: Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPV6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NS W-PACL-V6'][RULE_NAME='Rule_20']/DST_IPV6) INFO:YANG-TEST:Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 9: Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE.--------------------- libyang[0]: Value "0x080C" does not satisfy the constraint "(0x88CC|0x8100|0x8915|0x0806|0x0800|0x86DD|0x8847)" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/INNER_ETHER_TYPE) INFO:YANG-TEST:Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 10: Add dhcp_server which is not in correct ip-prefix format.--------------------- libyang[0]: Invalid value "10.186.72.566" in "dhcp_servers" element. (path: /sonic-vlan:sonic-vlan/VLAN/VLAN_LIST/dhcp_servers[.='10.186.72.566']) INFO:YANG-TEST:Add dhcp_server which is not in correct ip-prefix format. Passed INFO:YANG-TEST: ------------------- Test 11: Configure undefined acl_table_type in ACL_TABLE table.--------------------- libyang[0]: Invalid value "LAYER3V4" in "type" element. (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6']/type) INFO:YANG-TEST:Configure undefined acl_table_type in ACL_TABLE table. Passed INFO:YANG-TEST: ------------------- Test 12: Configure undefined packet_action in ACL_RULE table.--------------------- libyang[0]: Invalid value "SEND" in "PACKET_ACTION" element. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST/PACKET_ACTION) INFO:YANG-TEST:Configure undefined packet_action in ACL_RULE table. Passed INFO:YANG-TEST: ------------------- Test 13: Configure wrong value for tagging_mode.--------------------- libyang[0]: Invalid value "non-tagged" in "tagging_mode" element. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST/tagging_mode) INFO:YANG-TEST:Configure wrong value for tagging_mode. Passed INFO:YANG-TEST: ------------------- Test 14: Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table.--------------------- libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid) libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid) INFO:YANG-TEST:Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table. Passed INFO:YANG-TEST:All Test Passed ../../target/debs/stretch/libyang0.16_0.16.105-1_amd64.deb installtion failed ../../target/debs/stretch/libyang-cpp0.16_0.16.105-1_amd64.deb installtion failed ../../target/debs/stretch/python2-yang_0.16.105-1_amd64.deb installtion failed YANG Tests passed Passed: pyang -f tree ./yang-models/*.yang > ./yang-models/sonic_yang_tree copying tests/yangModelTesting.py -> build/lib/tests copying tests/test_sonic_yang_models.py -> build/lib/tests copying tests/__init__.py -> build/lib/tests running egg_info writing top-level names to sonic_yang_models.egg-info/top_level.txt writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt writing sonic_yang_models.egg-info/PKG-INFO reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt' writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt' installing to build/bdist.linux-x86_64/wheel running install running install_lib creating build/bdist.linux-x86_64/wheel creating build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/yangModelTesting.py -> build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/test_sonic_yang_models.py -> build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/wheel/tests running install_data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-head.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-acl.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-loopback-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-port.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-portchannel.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-vlan.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models ```
2020-04-14 17:36:02 -05:00
$(call dpkg_depend,$(PYTHON_WHEELS_PATH)/%.dep) \
$$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEBS_DEPENDS)))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(HEADER)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
pushd $($*_SRC_PATH) $(LOG_SIMPLE)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# apply series of patches if exist
if [ -f ../$(notdir $($*_SRC_PATH)).patch/series ]; then QUILT_PATCHES=../$(notdir $($*_SRC_PATH)).patch quilt push -a; fi
# Use pip instead of later setup.py to install dependencies into user home, but uninstall self
pip$($*_PYTHON_VERSION) install . && pip$($*_PYTHON_VERSION) uninstall --yes `python setup.py --name`
if [ ! "$($*_TEST)" = "n" ]; then python$($*_PYTHON_VERSION) setup.py test $(LOG); fi
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
python$($*_PYTHON_VERSION) setup.py bdist_wheel $(LOG)
# clean up
if [ -f ../$(notdir $($*_SRC_PATH)).patch/series ]; then quilt pop -a -f; [ -d .pc ] && rm -rf .pc; fi
popd $(LOG_SIMPLE)
mv -f $($*_SRC_PATH)/dist/$* $(PYTHON_WHEELS_PATH) $(LOG)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
# Uninstall unneeded build dependency
$(call UNINSTALL_DEBS,$($*_UNINSTALLS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(PYTHON_WHEELS_PATH)/, $(SONIC_PYTHON_WHEELS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
# Targets for installing python wheels.
# Autogenerated
SONIC_INSTALL_WHEELS = $(addsuffix -install, $(addprefix $(PYTHON_WHEELS_PATH)/, $(SONIC_PYTHON_WHEELS)))
$(SONIC_INSTALL_WHEELS) : $(PYTHON_WHEELS_PATH)/%-install : .platform $$(addsuffix -install,$$(addprefix $(PYTHON_WHEELS_PATH)/,$$($$*_DEPENDS))) $(PYTHON_WHEELS_PATH)/$$*
$(HEADER)
[ -f $(PYTHON_WHEELS_PATH)/$* ] || { echo $(PYTHON_WHEELS_PATH)/$* does not exist $(LOG) && exit 1; }
# put a lock here to avoid race conditions
while true; do
if mkdir $(PYTHON_WHEELS_PATH)/pip_lock &> /dev/null; then
{ sudo -E pip$($*_PYTHON_VERSION) install $(PYTHON_WHEELS_PATH)/$* $(LOG) && rm -d $(PYTHON_WHEELS_PATH)/pip_lock && break; } || { rm -d $(PYTHON_WHEELS_PATH)/pip_lock && exit 1 ; }
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
fi
done
$(FOOTER)
###############################################################################
## Docker images related targets
###############################################################################
# start docker daemon
docker-start :
@sudo sed -i '/http_proxy/d' /etc/default/docker
@sudo bash -c "{ echo \"export http_proxy=$$http_proxy\"; \
echo \"export no_proxy=$$no_proxy\"; } >> /etc/default/docker"
@test x$(SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD) != x"y" && sudo service docker status &> /dev/null || ( sudo service docker start &> /dev/null && ./scripts/wait_for_docker.sh 60 )
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
# targets for building simple docker images that do not depend on any debian packages
$(addprefix $(TARGET_PATH)/, $(SONIC_SIMPLE_DOCKER_IMAGES)) : $(TARGET_PATH)/%.gz : .platform docker-start $$(addsuffix -load,$$(addprefix $(TARGET_PATH)/,$$($$*.gz_LOAD_DOCKERS)))
$(HEADER)
# Apply series of patches if exist
if [ -f $($*.gz_PATH).patch/series ]; then pushd $($*.gz_PATH) && QUILT_PATCHES=../$(notdir $($*.gz_PATH)).patch quilt push -a; popd; fi
# Prepare docker build info
scripts/prepare_docker_buildinfo.sh $* $($*.gz_PATH)/Dockerfile $(CONFIGURED_ARCH) $(TARGET_DOCKERFILE)/Dockerfile.buildinfo
docker info $(LOG)
docker build --squash --no-cache \
--build-arg http_proxy=$(HTTP_PROXY) \
--build-arg https_proxy=$(HTTPS_PROXY) \
--build-arg no_proxy=$(NO_PROXY) \
--build-arg user=$(USER) \
--build-arg uid=$(UID) \
--build-arg guid=$(GUID) \
--build-arg docker_container_name=$($*.gz_CONTAINER_NAME) \
--label Tag=$(SONIC_IMAGE_VERSION) \
-f $(TARGET_DOCKERFILE)/Dockerfile.buildinfo \
-t $* $($*.gz_PATH) $(LOG)
scripts/collect_docker_version_files.sh $* $(TARGET_PATH)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
docker save $* | gzip -c > $@
# Clean up
if [ -f $($*.gz_PATH).patch/series ]; then pushd $($*.gz_PATH) && quilt pop -a -f; [ -d .pc ] && rm -rf .pc; popd; fi
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(TARGET_PATH)/, $(SONIC_SIMPLE_DOCKER_IMAGES))
DOCKER_IMAGES_FOR_INSTALLERS := $(sort $(foreach installer,$(SONIC_INSTALLERS),$($(installer)_DOCKERS)))
$(foreach DOCKER_IMAGE,$(SONIC_JESSIE_DOCKERS), $(eval $(DOCKER_IMAGE)_DEBS_PATH := $(JESSIE_DEBS_PATH)))
$(foreach DOCKER_IMAGE,$(SONIC_JESSIE_DOCKERS), $(eval $(DOCKER_IMAGE)_FILES_PATH := $(JESSIE_FILES_PATH)))
$(foreach DOCKER_IMAGE,$(SONIC_JESSIE_DBG_DOCKERS), $(eval $(DOCKER_IMAGE)_DEBS_PATH := $(JESSIE_DEBS_PATH)))
$(foreach DOCKER_IMAGE,$(SONIC_JESSIE_DBG_DOCKERS), $(eval $(DOCKER_IMAGE)_FILES_PATH := $(JESSIE_FILES_PATH)))
$(foreach DOCKER_IMAGE,$(SONIC_STRETCH_DOCKERS), $(eval $(DOCKER_IMAGE)_DEBS_PATH := $(STRETCH_DEBS_PATH)))
$(foreach DOCKER_IMAGE,$(SONIC_STRETCH_DOCKERS), $(eval $(DOCKER_IMAGE)_FILES_PATH := $(STRETCH_FILES_PATH)))
$(foreach DOCKER_IMAGE,$(SONIC_STRETCH_DBG_DOCKERS), $(eval $(DOCKER_IMAGE)_DEBS_PATH := $(STRETCH_DEBS_PATH)))
$(foreach DOCKER_IMAGE,$(SONIC_STRETCH_DBG_DOCKERS), $(eval $(DOCKER_IMAGE)_FILES_PATH := $(STRETCH_FILES_PATH)))
ifeq ($(BLDENV),jessie)
DOCKER_IMAGES := $(SONIC_JESSIE_DOCKERS)
DOCKER_DBG_IMAGES := $(SONIC_JESSIE_DBG_DOCKERS)
JESSIE_DOCKER_IMAGES = $(filter $(SONIC_JESSIE_DOCKERS),$(DOCKER_IMAGES_FOR_INSTALLERS) $(EXTRA_DOCKER_TARGETS))
JESSIE_DBG_DOCKER_IMAGES = $(filter $(SONIC_JESSIE_DBG_DOCKERS),$(DOCKER_IMAGES_FOR_INSTALLERS) $(EXTRA_DOCKER_TARGETS))
else
ifeq ($(BLDENV),stretch)
DOCKER_IMAGES := $(SONIC_STRETCH_DOCKERS)
DOCKER_DBG_IMAGES := $(SONIC_STRETCH_DBG_DOCKERS)
STRETCH_DOCKER_IMAGES = $(filter $(SONIC_STRETCH_DOCKERS),$(DOCKER_IMAGES_FOR_INSTALLERS) $(EXTRA_DOCKER_TARGETS))
STRETCH_DBG_DOCKER_IMAGES = $(filter $(SONIC_STRETCH_DBG_DOCKERS),$(DOCKER_IMAGES_FOR_INSTALLERS) $(EXTRA_DOCKER_TARGETS))
else
DOCKER_IMAGES = $(filter-out $(SONIC_JESSIE_DOCKERS) $(SONIC_STRETCH_DOCKERS),$(SONIC_DOCKER_IMAGES))
DOCKER_DBG_IMAGES = $(filter-out $(SONIC_JESSIE_DBG_DOCKERS) $(SONIC_STRETCH_DBG_DOCKERS), $(SONIC_DOCKER_DBG_IMAGES))
endif
endif
$(foreach IMAGE,$(DOCKER_IMAGES), $(eval $(IMAGE)_DEBS_PATH := $(DEBS_PATH)))
$(foreach IMAGE,$(DOCKER_IMAGES), $(eval $(IMAGE)_FILES_PATH := $(FILES_PATH)))
$(foreach IMAGE,$(DOCKER_DBG_IMAGES), $(eval $(IMAGE)_DEBS_PATH := $(DEBS_PATH)))
$(foreach IMAGE,$(DOCKER_DBG_IMAGES), $(eval $(IMAGE)_FILES_PATH := $(FILES_PATH)))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
# Targets for building docker images
$(addprefix $(TARGET_PATH)/, $(DOCKER_IMAGES)) : $(TARGET_PATH)/%.gz : .platform docker-start \
$$(addprefix $$($$*.gz_DEBS_PATH)/,$$($$*.gz_DEPENDS)) \
$$(addprefix $(TARGET_PATH)/,$$($$*.gz_AFTER)) \
$$(addprefix $$($$*.gz_FILES_PATH)/,$$($$*.gz_FILES)) \
$$(addprefix $(PYTHON_DEBS_PATH)/,$$($$*.gz_PYTHON_DEBS)) \
$$(addprefix $(PYTHON_WHEELS_PATH)/,$$($$*.gz_PYTHON_WHEELS)) \
$$(addsuffix -load,$$(addprefix $(TARGET_PATH)/,$$($$*.gz_LOAD_DOCKERS))) \
$$(addsuffix -install,$$(addprefix $(PYTHON_WHEELS_PATH)/,$$($$*.gz_INSTALL_PYTHON_WHEELS))) \
$$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*.gz_INSTALL_DEBS))) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$$($$*.gz_PATH)/Dockerfile.j2 \
$(call dpkg_depend,$(TARGET_PATH)/%.gz.dep)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(HEADER)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*.gz,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*.gz_CACHE_LOADED)' ] ; then
# Apply series of patches if exist
if [ -f $($*.gz_PATH).patch/series ]; then pushd $($*.gz_PATH) && QUILT_PATCHES=../$(notdir $($*.gz_PATH)).patch quilt push -a; popd; fi
mkdir -p $($*.gz_PATH)/debs $(LOG)
mkdir -p $($*.gz_PATH)/files $(LOG)
mkdir -p $($*.gz_PATH)/python-debs $(LOG)
mkdir -p $($*.gz_PATH)/python-wheels $(LOG)
sudo mount --bind $($*.gz_DEBS_PATH) $($*.gz_PATH)/debs $(LOG)
sudo mount --bind $($*.gz_FILES_PATH) $($*.gz_PATH)/files $(LOG)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
sudo mount --bind $(PYTHON_DEBS_PATH) $($*.gz_PATH)/python-debs $(LOG)
sudo mount --bind $(PYTHON_WHEELS_PATH) $($*.gz_PATH)/python-wheels $(LOG)
# Export variables for j2. Use path for unique variable names, e.g. docker_orchagent_debs
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_debs=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_DEPENDS),RDEPENDS))\n" | awk '!a[$$0]++'))
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_pydebs=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_PYTHON_DEBS)))\n" | awk '!a[$$0]++'))
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_whls=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_PYTHON_WHEELS)))\n" | awk '!a[$$0]++'))
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_dbgs=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_DBG_PACKAGES)))\n" | awk '!a[$$0]++'))
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_pkgs=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_APT_PACKAGES)))\n" | awk '!a[$$0]++'))
if [ -d $($*.gz_PATH)/cli-plugin-tests/ ]; then pushd $($*.gz_PATH)/cli-plugin-tests; pytest-$($(SONIC_UTILITIES_PY3)_PYTHON_VERSION) -v $(LOG); popd; fi
# Label docker image with componenets versions
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_labels=$(foreach component,\
$(call expand,$($*.gz_DEPENDS),RDEPENDS) \
$(call expand,$($*.gz_PYTHON_DEBS)) \
$(call expand,$($*.gz_PYTHON_WHEELS)),\
$(shell [[ ! -z "$($(component)_VERSION)" && ! -z "$($(component)_NAME)" ]] && \
echo "--label com.azure.sonic.versions.$($(component)_NAME)=$($(component)_VERSION)")))
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
j2 $($*.gz_PATH)/Dockerfile.j2 > $($*.gz_PATH)/Dockerfile
$(call generate_manifest,$*)
# Prepare docker build info
PACKAGE_URL_PREFIX=$(PACKAGE_URL_PREFIX) \
SONIC_ENFORCE_VERSIONS=$(SONIC_ENFORCE_VERSIONS) \
TRUSTED_GPG_URLS=$(TRUSTED_GPG_URLS) \
scripts/prepare_docker_buildinfo.sh $* $($*.gz_PATH)/Dockerfile $(CONFIGURED_ARCH)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
docker info $(LOG)
docker build --squash --no-cache \
--build-arg http_proxy=$(HTTP_PROXY) \
--build-arg https_proxy=$(HTTPS_PROXY) \
--build-arg no_proxy=$(NO_PROXY) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
--build-arg user=$(USER) \
--build-arg uid=$(UID) \
--build-arg guid=$(GUID) \
--build-arg docker_container_name=$($*.gz_CONTAINER_NAME) \
--build-arg frr_user_uid=$(FRR_USER_UID) \
--build-arg frr_user_gid=$(FRR_USER_GID) \
--build-arg image_version=$(SONIC_IMAGE_VERSION) \
--label com.azure.sonic.manifest="$$(cat $($*.gz_PATH)/manifest.json)" \
--label Tag=$(SONIC_IMAGE_VERSION) \
$($(subst -,_,$(notdir $($*.gz_PATH)))_labels) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
-t $* $($*.gz_PATH) $(LOG)
scripts/collect_docker_version_files.sh $* $(TARGET_PATH)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
docker save $* | gzip -c > $@
# Clean up
if [ -f $($*.gz_PATH).patch/series ]; then pushd $($*.gz_PATH) && quilt pop -a -f; [ -d .pc ] && rm -rf .pc; popd; fi
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*.gz,$@)
fi
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(TARGET_PATH)/, $(DOCKER_IMAGES))
# Targets for building docker images
$(addprefix $(TARGET_PATH)/, $(DOCKER_DBG_IMAGES)) : $(TARGET_PATH)/%-$(DBG_IMAGE_MARK).gz : .platform docker-start \
$$(addprefix $(TARGET_PATH)/,$$($$*.gz_AFTER)) \
$$(addprefix $$($$*.gz_DEBS_PATH)/,$$($$*.gz_DBG_DEPENDS)) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$$(addsuffix -load,$$(addprefix $(TARGET_PATH)/,$$*.gz)) \
$(call dpkg_depend,$(TARGET_PATH)/%-$(DBG_IMAGE_MARK).gz.dep)
$(HEADER)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*-$(DBG_IMAGE_MARK).gz,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*-$(DBG_IMAGE_MARK).gz_CACHE_LOADED)' ] ; then
mkdir -p $($*.gz_PATH)/debs $(LOG)
sudo mount --bind $($*.gz_DEBS_PATH) $($*.gz_PATH)/debs $(LOG)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Export variables for j2. Use path for unique variable names, e.g. docker_orchagent_debs
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_dbg_debs=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_DBG_DEPENDS),RDEPENDS))\n" | awk '!a[$$0]++'))
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_image_dbgs=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_DBG_IMAGE_PACKAGES)))\n" | awk '!a[$$0]++'))
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_dbg_pkgs=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_DBG_APT_PACKAGES),RDEPENDS))\n" | awk '!a[$$0]++'))
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
./build_debug_docker_j2.sh $* $(subst -,_,$(notdir $($*.gz_PATH)))_dbg_debs $(subst -,_,$(notdir $($*.gz_PATH)))_image_dbgs > $($*.gz_PATH)/Dockerfile-dbg.j2
j2 $($*.gz_PATH)/Dockerfile-dbg.j2 > $($*.gz_PATH)/Dockerfile-dbg
$(call generate_manifest,$*,dbg)
# Prepare docker build info
PACKAGE_URL_PREFIX=$(PACKAGE_URL_PREFIX) \
SONIC_ENFORCE_VERSIONS=$(SONIC_ENFORCE_VERSIONS) \
TRUSTED_GPG_URLS=$(TRUSTED_GPG_URLS) \
scripts/prepare_docker_buildinfo.sh $* $($*.gz_PATH)/Dockerfile-dbg $(CONFIGURED_ARCH)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
docker info $(LOG)
docker build \
$(if $($*.gz_DBG_DEPENDS), --squash --no-cache, --no-cache) \
--build-arg http_proxy=$(HTTP_PROXY) \
--build-arg http_proxy=$(HTTP_PROXY) \
--build-arg no_proxy=$(NO_PROXY) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
--build-arg docker_container_name=$($*.gz_CONTAINER_NAME) \
--label com.azure.sonic.manifest="$$(cat $($*.gz_PATH)/manifest.json)" \
--label Tag=$(SONIC_IMAGE_VERSION) \
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
--file $($*.gz_PATH)/Dockerfile-dbg \
-t $*-dbg $($*.gz_PATH) $(LOG)
scripts/collect_docker_version_files.sh $*-dbg $(TARGET_PATH)
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
docker save $*-dbg | gzip -c > $@
# Clean up
if [ -f $($*.gz_PATH).patch/series ]; then pushd $($*.gz_PATH) && quilt pop -a -f; [ -d .pc ] && rm -rf .pc; popd; fi
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*-$(DBG_IMAGE_MARK).gz,$@)
fi
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(TARGET_PATH)/, $(DOCKER_DBG_IMAGES))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
DOCKER_LOAD_TARGETS = $(addsuffix -load,$(addprefix $(TARGET_PATH)/, \
$(SONIC_SIMPLE_DOCKER_IMAGES) \
$(DOCKER_IMAGES) \
$(DOCKER_DBG_IMAGES)))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(DOCKER_LOAD_TARGETS) : $(TARGET_PATH)/%.gz-load : .platform docker-start $$(TARGET_PATH)/$$*.gz
$(HEADER)
docker load -i $(TARGET_PATH)/$*.gz $(LOG)
$(FOOTER)
###############################################################################
## Installers
###############################################################################
# targets for building installers with base image
$(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
.platform \
onie-image.conf \
2017-09-01 02:35:36 -05:00
build_debian.sh \
scripts/dbg_files.sh \
2017-09-01 02:35:36 -05:00
build_image.sh \
$$(addsuffix -install,$$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$$($$*_DEPENDS))) \
$$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$$($$*_INSTALLS)) \
$$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$$($$*_LAZY_INSTALLS)) \
$$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$$($$*_LAZY_BUILD_INSTALLS)) \
$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$(INITRAMFS_TOOLS) \
$(LINUX_KERNEL) \
$(SONIC_DEVICE_DATA) \
$(IFUPDOWN2) \
$(KDUMP_TOOLS) \
$(NTP) \
$(LIBPAM_RADIUS) \
$(LIBNSS_RADIUS) \
$(LIBPAM_TACPLUS) \
$(LIBNSS_TACPLUS) \
$(MONIT) \
$(OPENSSH_SERVER) \
$(PYTHON_SWSSCOMMON) \
$(PYTHON3_SWSSCOMMON) \
$(SONIC_UTILITIES_DATA) \
$(SONIC_HOST_SERVICES_DATA)) \
$$(addprefix $(TARGET_PATH)/,$$($$*_DOCKERS)) \
$$(addprefix $(TARGET_PATH)/,$$(SONIC_PACKAGES_LOCAL)) \
$$(addprefix $(FILES_PATH)/,$$($$*_FILES)) \
$(if $(findstring y,$(ENABLE_ZTP)),$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$(SONIC_ZTP))) \
[sonic-utilities] Update submodule; Build and install as a Python 3 wheel (#5926) Submodule updates include the following commits: * src/sonic-utilities 9dc58ea...f9eb739 (18): > Remove unnecessary calls to str.encode() now that the package is Python 3; Fix deprecation warning (#1260) > [generate_dump] Ignoring file/directory not found Errors (#1201) > Fixed porstat rate and util issues (#1140) > fix error: interface counters is mismatch after warm-reboot (#1099) > Remove unnecessary calls to str.decode() now that the package is Python 3 (#1255) > [acl-loader] Make list sorting compliant with Python 3 (#1257) > Replace hard-coded fast-reboot with variable. And some typo corrections (#1254) > [configlet][portconfig] Remove calls to dict.has_key() which is not available in Python 3 (#1247) > Remove unnecessary conversions to list() and calls to dict.keys() (#1243) > Clean up LGTM alerts (#1239) > Add 'requests' as install dependency in setup.py (#1240) > Convert to Python 3 (#1128) > Fix mock SonicV2Connector in python3: use decode_responses mode so caller code will be the same as python2 (#1238) > [tests] Do not trim from PATH if we did not append to it; Clean up/fix shebangs in scripts (#1233) > Updates to bgp config and show commands with BGP_INTERNAL_NEIGHBOR table (#1224) > [cli]: NAT show commands newline issue after migrated to Python3 (#1204) > [doc]: Update Command-Reference.md (#1231) > Added 'import sys' in feature.py file (#1232) * src/sonic-py-swsssdk 9d9f0c6...1664be9 (2): > Fix: no need to decode() after redis client scan, so it will work for both python2 and python3 (#96) > FieldValueMap `contains`(`in`) will also work when migrated to libswsscommon(C++ with SWIG wrapper) (#94) - Also fix Python 3-related issues: - Use integer (floor) division in config_samples.py (sonic-config-engine) - Replace print statement with print function in eeprom.py plugin for x86_64-kvm_x86_64-r0 platform - Update all platform plugins to be compatible with both Python 2 and Python 3 - Remove shebangs from plugins files which are not intended to be executable - Replace tabs with spaces in Python plugin files and fix alignment, because Python 3 is more strict - Remove trailing whitespace from plugins files
2020-11-25 12:28:36 -06:00
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_UTILITIES_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY2)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CONFIG_ENGINE_PY2)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CONFIG_ENGINE_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PLATFORM_COMMON_PY2)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PLATFORM_COMMON_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(REDIS_DUMP_LOAD_PY2)) \
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. (#3730) [sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. YANG models as per Guidelines. Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md [sonic-yang-models/tests]: YANG model test code and JSON input for testing. [sonic-yang-models/setup.py]: Build infra for yang models. **- What I did** Created Yang model for Sonic. Tables: PORT, VLAN, VLAN_INTERFACE, VLAN_MEMBER, ACL_RULE, ACL_TABLE, INTERFACE. Created build infra files using which a new package (sonic-yang-models) can be build and can be deployed on sonic switches. Yang models will be part of this new package. **- How I did it** Wrote yang models based on Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md and https://github.com/Azure/SONiC/wiki/Configuration. Wrote python wheel Package infra which runs test for these Yang models using a json files which consists configuration as per yang models. These configs are for negative tests, which means we want to test that most must condition, pattern and when condition works as expected. **- How to verify it** Build Logs and testing: ——————————————————————————————————— ``` /sonic/src/sonic-yang-models /sonic running test running egg_info writing top-level names to sonic_yang_models.egg-info/top_level.txt writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt writing sonic_yang_models.egg-info/PKG-INFO reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt' writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt' running build_ext ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK running bdist_wheel running build running build_py (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../libyang_1.0.73_amd64.deb ... Unpacking libyang (1.0.73) over (1.0.73) ... Setting up libyang (1.0.73) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... Processing triggers for man-db (2.7.6.1-2) ... (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../libyang-cpp_1.0.73_amd64.deb ... Unpacking libyang-cpp (1.0.73) over (1.0.73) ... Setting up libyang-cpp (1.0.73) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../python3-yang_1.0.73_amd64.deb ... Unpacking python3-yang (1.0.73) over (1.0.73) ... Setting up python3-yang (1.0.73) ... INFO:YANG-TEST:module: sonic-vlan is loaded successfully ERROR:YANG-TEST:Could not get module: sonic-head INFO:YANG-TEST:module: sonic-portchannel is loaded successfully INFO:YANG-TEST:module: sonic-acl is loaded successfully INFO:YANG-TEST:module: sonic-loopback-interface is loaded successfully ERROR:YANG-TEST:Could not get module: sonic-port INFO:YANG-TEST:module: sonic-interface is loaded successfully INFO:YANG-TEST: ------------------- Test 1: Configure a member port in VLAN_MEMBER table which does not exist.--------------------- libyang[0]: Leafref "/sonic-port:sonic-port/sonic-port:PORT/sonic-port:PORT_LIST/sonic-port:port_name" of value "Ethernet156" points to a non -existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlan_name='Vlan100'][port='Ethernet156']/port) INFO:YANG-TEST:Configure a member port in VLAN_MEMBER table which does not exist. Passed INFO:YANG-TEST: ------------------- Test 2: Configure non-existing ACL_TABLE in ACL_RULE.--------------------- libyang[0]: Leafref "/sonic-acl:sonic-acl/sonic-acl:ACL_TABLE/sonic-acl:ACL_TABLE_LIST/sonic-acl:ACL_TABLE_NAME" of value "NOT-EXIST" points to a non-existing leaf. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NOT-EXIST'][RULE_NAME='Rule_20']/ACL_TABLE_NAME) INFO:YANG-TEST:Configure non-existing ACL_TABLE in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 3: Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/ICMPV6_CODE) INFO:YANG-TEST:Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE. Passed INFO:YANG-TEST: INFO:YANG-TEST: ------------------- Test 4: Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_20']/SRC_IPV6) INFO:YANG-TEST:Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE. Passed ------------------- Test 5: Configure l4_src_port_range as 99999-99999 in ACL_RULE--------------------- libyang[0]: Value "99999-99999" does not satisfy the constraint "([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])-([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6'][RULE_NAME='Rule_20']/L4_SRC_PORT_RANGE) INFO:YANG-TEST:Configure l4_src_port_range as 99999-99999 in ACL_RULE Passed INFO:YANG-TEST: ------------------- Test 6: Configure empty string as ip-prefix in INTERFACE table.--------------------- libyang[0]: Invalid value "" in "ip-prefix" element. (path: /sonic-interface:sonic-interface/INTERFACE/INTERFACE_LIST[interface='Ethernet8'][ip-prefix='']/ip-prefix) INFO:YANG-TEST:Configure empty string as ip-prefix in INTERFACE table. Passed INFO:YANG-TEST: ------------------- Test 7: Configure Wrong family with ip-prefix for VLAN_Interface Table--------------------- libyang[0]: Must condition "(contains(../ip-prefix, ':') and current()='IPv6') or (contains(../ip-prefix, '.') and current()='IPv4')" not satisfied. (path: /sonic-vlan:sonic-vlan/VLAN_INTERFACE/VLAN_INTERFACE_LIST[vlanid='100'][ip-prefix='2a04:5555:66:7777::1/64']/family) INFO:YANG-TEST:Configure Wrong family with ip-prefix for VLAN_Interface Table Passed INFO:YANG-TEST: ------------------- Test 8: Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPV6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NS W-PACL-V6'][RULE_NAME='Rule_20']/DST_IPV6) INFO:YANG-TEST:Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 9: Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE.--------------------- libyang[0]: Value "0x080C" does not satisfy the constraint "(0x88CC|0x8100|0x8915|0x0806|0x0800|0x86DD|0x8847)" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/INNER_ETHER_TYPE) INFO:YANG-TEST:Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 10: Add dhcp_server which is not in correct ip-prefix format.--------------------- libyang[0]: Invalid value "10.186.72.566" in "dhcp_servers" element. (path: /sonic-vlan:sonic-vlan/VLAN/VLAN_LIST/dhcp_servers[.='10.186.72.566']) INFO:YANG-TEST:Add dhcp_server which is not in correct ip-prefix format. Passed INFO:YANG-TEST: ------------------- Test 11: Configure undefined acl_table_type in ACL_TABLE table.--------------------- libyang[0]: Invalid value "LAYER3V4" in "type" element. (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6']/type) INFO:YANG-TEST:Configure undefined acl_table_type in ACL_TABLE table. Passed INFO:YANG-TEST: ------------------- Test 12: Configure undefined packet_action in ACL_RULE table.--------------------- libyang[0]: Invalid value "SEND" in "PACKET_ACTION" element. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST/PACKET_ACTION) INFO:YANG-TEST:Configure undefined packet_action in ACL_RULE table. Passed INFO:YANG-TEST: ------------------- Test 13: Configure wrong value for tagging_mode.--------------------- libyang[0]: Invalid value "non-tagged" in "tagging_mode" element. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST/tagging_mode) INFO:YANG-TEST:Configure wrong value for tagging_mode. Passed INFO:YANG-TEST: ------------------- Test 14: Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table.--------------------- libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid) libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid) INFO:YANG-TEST:Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table. Passed INFO:YANG-TEST:All Test Passed ../../target/debs/stretch/libyang0.16_0.16.105-1_amd64.deb installtion failed ../../target/debs/stretch/libyang-cpp0.16_0.16.105-1_amd64.deb installtion failed ../../target/debs/stretch/python2-yang_0.16.105-1_amd64.deb installtion failed YANG Tests passed Passed: pyang -f tree ./yang-models/*.yang > ./yang-models/sonic_yang_tree copying tests/yangModelTesting.py -> build/lib/tests copying tests/test_sonic_yang_models.py -> build/lib/tests copying tests/__init__.py -> build/lib/tests running egg_info writing top-level names to sonic_yang_models.egg-info/top_level.txt writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt writing sonic_yang_models.egg-info/PKG-INFO reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt' writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt' installing to build/bdist.linux-x86_64/wheel running install running install_lib creating build/bdist.linux-x86_64/wheel creating build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/yangModelTesting.py -> build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/test_sonic_yang_models.py -> build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/wheel/tests running install_data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-head.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-acl.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-loopback-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-port.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-portchannel.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-vlan.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models ```
2020-04-14 17:36:02 -05:00
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PLATFORM_API_PY2)) \
$(if $(findstring y,$(PDDF_SUPPORT)),$(addprefix $(PYTHON_WHEELS_PATH)/,$(PDDF_PLATFORM_API_BASE_PY2))) \
$(if $(findstring y,$(PDDF_SUPPORT)),$(addprefix $(PYTHON_WHEELS_PATH)/,$(PDDF_PLATFORM_API_BASE_PY3))) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MODELS_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CTRMGRD)) \
$(addprefix $(FILES_PATH)/,$($(SONIC_CTRMGRD)_FILES)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MGMT_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SYSTEM_HEALTH)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_HOST_SERVICES_PY3))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(HEADER)
# Pass initramfs and linux kernel explicitly. They are used for all platforms
export debs_path="$(IMAGE_DISTRO_DEBS_PATH)"
export files_path="$(FILES_PATH)"
export python_debs_path="$(PYTHON_DEBS_PATH)"
export initramfs_tools="$(IMAGE_DISTRO_DEBS_PATH)/$(INITRAMFS_TOOLS)"
export linux_kernel="$(IMAGE_DISTRO_DEBS_PATH)/$(LINUX_KERNEL)"
export onie_recovery_image="$(FILES_PATH)/$(ONIE_RECOVERY_IMAGE)"
export kversion="$(KVERSION)"
export image_type="$($*_IMAGE_TYPE)"
export sonicadmin_user="$(USERNAME)"
export sonic_asic_platform="$(patsubst %-$(CONFIGURED_ARCH),%,$(CONFIGURED_PLATFORM))"
export enable_organization_extensions="$(ENABLE_ORGANIZATION_EXTENSIONS)"
export enable_dhcp_graph_service="$(ENABLE_DHCP_GRAPH_SERVICE)"
export enable_ztp="$(ENABLE_ZTP)"
export include_system_telemetry="$(INCLUDE_SYSTEM_TELEMETRY)"
export include_restapi="$(INCLUDE_RESTAPI)"
export include_nat="$(INCLUDE_NAT)"
export include_sflow="$(INCLUDE_SFLOW)"
export include_macsec="$(INCLUDE_MACSEC)"
export include_mgmt_framework="$(INCLUDE_MGMT_FRAMEWORK)"
export include_iccpd="$(INCLUDE_ICCPD)"
export pddf_support="$(PDDF_SUPPORT)"
export shutdown_bgp_on_start="$(SHUTDOWN_BGP_ON_START)"
[Dynamic buffer calc] Support dynamic buffer calculation (#6194) **- Why I did it** To support dynamic buffer calculation. This PR also depends on the following PRs for sub modules - [sonic-swss: [buffermgr/bufferorch] Support dynamic buffer calculation #1338](https://github.com/Azure/sonic-swss/pull/1338) - [sonic-swss-common: Dynamic buffer calculation #361](https://github.com/Azure/sonic-swss-common/pull/361) - [sonic-utilities: Support dynamic buffer calculation #973](https://github.com/Azure/sonic-utilities/pull/973) **- How I did it** 1. Introduce field `buffer_model` in `DEVICE_METADATA|localhost` to represent which buffer model is running in the system currently: - `dynamic` for the dynamic buffer calculation model - `traditional` for the traditional model in which the `pg_profile_lookup.ini` is used 2. Add the tables required for the feature: - ASIC_TABLE in platform/\<vendor\>/asic_table.j2 - PERIPHERAL_TABLE in platform/\<vendor\>/peripheral_table.j2 - PORT_PERIPHERAL_TABLE on a per-platform basis in device/\<vendor\>/\<platform\>/port_peripheral_config.j2 for each platform with gearbox installed. - DEFAULT_LOSSLESS_BUFFER_PARAMETER and LOSSLESS_TRAFFIC_PATTERN in files/build_templates/buffers_config.j2 - Add lossless PGs (3-4) for each port in files/build_templates/buffers_config.j2 3. Copy the newly introduced j2 files into the image and rendering them when the system starts 4. Update the CLI options for buffermgrd so that it can start with dynamic mode 5. Fetches the ASIC vendor name in orchagent: - fetch the vendor name when creates the docker and pass it as a docker environment variable - `buffermgrd` can use this passed-in variable 6. Clear buffer related tables from STATE_DB when swss docker starts 7. Update the src/sonic-config-engine/tests/sample_output/buffers-dell6100.json according to the buffer_config.j2 8. Remove buffer pool sizes for ingress pools and egress_lossy_pool Update the buffer settings for dynamic buffer calculation
2020-12-13 13:35:39 -06:00
export default_buffer_model="$(SONIC_BUFFER_MODEL)"
export include_kubernetes="$(INCLUDE_KUBERNETES)"
export enable_pfcwd_on_start="$(ENABLE_PFCWD_ON_START)"
export installer_debs="$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$($*_INSTALLS))"
export lazy_installer_debs="$(foreach deb, $($*_LAZY_INSTALLS),$(foreach device, $($(deb)_PLATFORM),$(addprefix $(device)@, $(IMAGE_DISTRO_DEBS_PATH)/$(deb))))"
export lazy_build_installer_debs="$(foreach deb, $($*_LAZY_BUILD_INSTALLS), $(addprefix $($(deb)_MACHINE)|,$(deb)))"
export installer_images="$(foreach docker, $($*_DOCKERS),\
$(addprefix $($(docker:-dbg.gz=.gz)_PACKAGE_NAME)|,\
$(addprefix $($(docker:-dbg.gz=.gz)_PATH)|,\
$(addprefix $($(docker:-dbg.gz=.gz)_MACHINE)|,\
$(addprefix $(TARGET_PATH)/,$(addsuffix :$($(docker:-dbg.gz=.gz)_VERSION),$(docker)))))))"
export sonic_packages="$(foreach package, $(SONIC_PACKAGES),\
$(addsuffix |$($(package)_DEFAULT_FEATURE_STATE_ENABLED),\
$(addsuffix |$($(package)_DEFAULT_FEATURE_OWNER),\
$(addsuffix |$($(package)_VERSION),\
$(addsuffix |$($(package)_REPOSITORY), $(package))))))"
export sonic_local_packages="$(foreach package, $(SONIC_PACKAGES_LOCAL),\
$(addsuffix |$($(package)_DEFAULT_FEATURE_STATE_ENABLED),\
$(addsuffix |$($(package)_DEFAULT_FEATURE_OWNER),\
$(addsuffix |$(addprefix $(TARGET_PATH)/, $(package)), $(package)))))"
export sonic_py_common_py2_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY2))"
export sonic_py_common_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY3))"
export config_engine_py2_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CONFIG_ENGINE_PY2))"
export config_engine_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CONFIG_ENGINE_PY3))"
export swsssdk_py2_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SWSSSDK_PY2))"
export swsssdk_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SWSSSDK_PY3))"
export platform_common_py2_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PLATFORM_COMMON_PY2))"
export platform_common_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PLATFORM_COMMON_PY3))"
export redis_dump_load_py2_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(REDIS_DUMP_LOAD_PY2))"
export redis_dump_load_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(REDIS_DUMP_LOAD_PY3))"
export install_debug_image="$(INSTALL_DEBUG_TOOLS)"
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. (#3730) [sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. YANG models as per Guidelines. Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md [sonic-yang-models/tests]: YANG model test code and JSON input for testing. [sonic-yang-models/setup.py]: Build infra for yang models. **- What I did** Created Yang model for Sonic. Tables: PORT, VLAN, VLAN_INTERFACE, VLAN_MEMBER, ACL_RULE, ACL_TABLE, INTERFACE. Created build infra files using which a new package (sonic-yang-models) can be build and can be deployed on sonic switches. Yang models will be part of this new package. **- How I did it** Wrote yang models based on Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md and https://github.com/Azure/SONiC/wiki/Configuration. Wrote python wheel Package infra which runs test for these Yang models using a json files which consists configuration as per yang models. These configs are for negative tests, which means we want to test that most must condition, pattern and when condition works as expected. **- How to verify it** Build Logs and testing: ——————————————————————————————————— ``` /sonic/src/sonic-yang-models /sonic running test running egg_info writing top-level names to sonic_yang_models.egg-info/top_level.txt writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt writing sonic_yang_models.egg-info/PKG-INFO reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt' writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt' running build_ext ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK running bdist_wheel running build running build_py (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../libyang_1.0.73_amd64.deb ... Unpacking libyang (1.0.73) over (1.0.73) ... Setting up libyang (1.0.73) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... Processing triggers for man-db (2.7.6.1-2) ... (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../libyang-cpp_1.0.73_amd64.deb ... Unpacking libyang-cpp (1.0.73) over (1.0.73) ... Setting up libyang-cpp (1.0.73) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... (Reading database ... 155852 files and directories currently installed.) Preparing to unpack .../python3-yang_1.0.73_amd64.deb ... Unpacking python3-yang (1.0.73) over (1.0.73) ... Setting up python3-yang (1.0.73) ... INFO:YANG-TEST:module: sonic-vlan is loaded successfully ERROR:YANG-TEST:Could not get module: sonic-head INFO:YANG-TEST:module: sonic-portchannel is loaded successfully INFO:YANG-TEST:module: sonic-acl is loaded successfully INFO:YANG-TEST:module: sonic-loopback-interface is loaded successfully ERROR:YANG-TEST:Could not get module: sonic-port INFO:YANG-TEST:module: sonic-interface is loaded successfully INFO:YANG-TEST: ------------------- Test 1: Configure a member port in VLAN_MEMBER table which does not exist.--------------------- libyang[0]: Leafref "/sonic-port:sonic-port/sonic-port:PORT/sonic-port:PORT_LIST/sonic-port:port_name" of value "Ethernet156" points to a non -existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlan_name='Vlan100'][port='Ethernet156']/port) INFO:YANG-TEST:Configure a member port in VLAN_MEMBER table which does not exist. Passed INFO:YANG-TEST: ------------------- Test 2: Configure non-existing ACL_TABLE in ACL_RULE.--------------------- libyang[0]: Leafref "/sonic-acl:sonic-acl/sonic-acl:ACL_TABLE/sonic-acl:ACL_TABLE_LIST/sonic-acl:ACL_TABLE_NAME" of value "NOT-EXIST" points to a non-existing leaf. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NOT-EXIST'][RULE_NAME='Rule_20']/ACL_TABLE_NAME) INFO:YANG-TEST:Configure non-existing ACL_TABLE in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 3: Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/ICMPV6_CODE) INFO:YANG-TEST:Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE. Passed INFO:YANG-TEST: INFO:YANG-TEST: ------------------- Test 4: Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_20']/SRC_IPV6) INFO:YANG-TEST:Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE. Passed ------------------- Test 5: Configure l4_src_port_range as 99999-99999 in ACL_RULE--------------------- libyang[0]: Value "99999-99999" does not satisfy the constraint "([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])-([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6'][RULE_NAME='Rule_20']/L4_SRC_PORT_RANGE) INFO:YANG-TEST:Configure l4_src_port_range as 99999-99999 in ACL_RULE Passed INFO:YANG-TEST: ------------------- Test 6: Configure empty string as ip-prefix in INTERFACE table.--------------------- libyang[0]: Invalid value "" in "ip-prefix" element. (path: /sonic-interface:sonic-interface/INTERFACE/INTERFACE_LIST[interface='Ethernet8'][ip-prefix='']/ip-prefix) INFO:YANG-TEST:Configure empty string as ip-prefix in INTERFACE table. Passed INFO:YANG-TEST: ------------------- Test 7: Configure Wrong family with ip-prefix for VLAN_Interface Table--------------------- libyang[0]: Must condition "(contains(../ip-prefix, ':') and current()='IPv6') or (contains(../ip-prefix, '.') and current()='IPv4')" not satisfied. (path: /sonic-vlan:sonic-vlan/VLAN_INTERFACE/VLAN_INTERFACE_LIST[vlanid='100'][ip-prefix='2a04:5555:66:7777::1/64']/family) INFO:YANG-TEST:Configure Wrong family with ip-prefix for VLAN_Interface Table Passed INFO:YANG-TEST: ------------------- Test 8: Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE.--------------------- libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPV6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NS W-PACL-V6'][RULE_NAME='Rule_20']/DST_IPV6) INFO:YANG-TEST:Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 9: Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE.--------------------- libyang[0]: Value "0x080C" does not satisfy the constraint "(0x88CC|0x8100|0x8915|0x0806|0x0800|0x86DD|0x8847)" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/INNER_ETHER_TYPE) INFO:YANG-TEST:Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE. Passed INFO:YANG-TEST: ------------------- Test 10: Add dhcp_server which is not in correct ip-prefix format.--------------------- libyang[0]: Invalid value "10.186.72.566" in "dhcp_servers" element. (path: /sonic-vlan:sonic-vlan/VLAN/VLAN_LIST/dhcp_servers[.='10.186.72.566']) INFO:YANG-TEST:Add dhcp_server which is not in correct ip-prefix format. Passed INFO:YANG-TEST: ------------------- Test 11: Configure undefined acl_table_type in ACL_TABLE table.--------------------- libyang[0]: Invalid value "LAYER3V4" in "type" element. (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6']/type) INFO:YANG-TEST:Configure undefined acl_table_type in ACL_TABLE table. Passed INFO:YANG-TEST: ------------------- Test 12: Configure undefined packet_action in ACL_RULE table.--------------------- libyang[0]: Invalid value "SEND" in "PACKET_ACTION" element. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST/PACKET_ACTION) INFO:YANG-TEST:Configure undefined packet_action in ACL_RULE table. Passed INFO:YANG-TEST: ------------------- Test 13: Configure wrong value for tagging_mode.--------------------- libyang[0]: Invalid value "non-tagged" in "tagging_mode" element. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST/tagging_mode) INFO:YANG-TEST:Configure wrong value for tagging_mode. Passed INFO:YANG-TEST: ------------------- Test 14: Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table.--------------------- libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid) libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid) INFO:YANG-TEST:Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN table. Passed INFO:YANG-TEST:All Test Passed ../../target/debs/stretch/libyang0.16_0.16.105-1_amd64.deb installtion failed ../../target/debs/stretch/libyang-cpp0.16_0.16.105-1_amd64.deb installtion failed ../../target/debs/stretch/python2-yang_0.16.105-1_amd64.deb installtion failed YANG Tests passed Passed: pyang -f tree ./yang-models/*.yang > ./yang-models/sonic_yang_tree copying tests/yangModelTesting.py -> build/lib/tests copying tests/test_sonic_yang_models.py -> build/lib/tests copying tests/__init__.py -> build/lib/tests running egg_info writing top-level names to sonic_yang_models.egg-info/top_level.txt writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt writing sonic_yang_models.egg-info/PKG-INFO reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt' writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt' installing to build/bdist.linux-x86_64/wheel running install running install_lib creating build/bdist.linux-x86_64/wheel creating build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/yangModelTesting.py -> build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/test_sonic_yang_models.py -> build/bdist.linux-x86_64/wheel/tests copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/wheel/tests running install_data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-head.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-acl.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-loopback-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-port.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-portchannel.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models copying ./yang-models/sonic-vlan.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models ```
2020-04-14 17:36:02 -05:00
export sonic_yang_models_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MODELS_PY3))"
export sonic_ctrmgmt_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CTRMGRD))"
export sonic_yang_mgmt_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_YANG_MGMT_PY3))"
export multi_instance="false"
export python_swss_debs="$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$($(LIBSWSSCOMMON)_RDEPENDS))"
export python_swss_debs+=" $(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$(LIBSWSSCOMMON)) $(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$(PYTHON_SWSSCOMMON)) $(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$(PYTHON3_SWSSCOMMON))"
[sonic-utilities] Update submodule; Build and install as a Python 3 wheel (#5926) Submodule updates include the following commits: * src/sonic-utilities 9dc58ea...f9eb739 (18): > Remove unnecessary calls to str.encode() now that the package is Python 3; Fix deprecation warning (#1260) > [generate_dump] Ignoring file/directory not found Errors (#1201) > Fixed porstat rate and util issues (#1140) > fix error: interface counters is mismatch after warm-reboot (#1099) > Remove unnecessary calls to str.decode() now that the package is Python 3 (#1255) > [acl-loader] Make list sorting compliant with Python 3 (#1257) > Replace hard-coded fast-reboot with variable. And some typo corrections (#1254) > [configlet][portconfig] Remove calls to dict.has_key() which is not available in Python 3 (#1247) > Remove unnecessary conversions to list() and calls to dict.keys() (#1243) > Clean up LGTM alerts (#1239) > Add 'requests' as install dependency in setup.py (#1240) > Convert to Python 3 (#1128) > Fix mock SonicV2Connector in python3: use decode_responses mode so caller code will be the same as python2 (#1238) > [tests] Do not trim from PATH if we did not append to it; Clean up/fix shebangs in scripts (#1233) > Updates to bgp config and show commands with BGP_INTERNAL_NEIGHBOR table (#1224) > [cli]: NAT show commands newline issue after migrated to Python3 (#1204) > [doc]: Update Command-Reference.md (#1231) > Added 'import sys' in feature.py file (#1232) * src/sonic-py-swsssdk 9d9f0c6...1664be9 (2): > Fix: no need to decode() after redis client scan, so it will work for both python2 and python3 (#96) > FieldValueMap `contains`(`in`) will also work when migrated to libswsscommon(C++ with SWIG wrapper) (#94) - Also fix Python 3-related issues: - Use integer (floor) division in config_samples.py (sonic-config-engine) - Replace print statement with print function in eeprom.py plugin for x86_64-kvm_x86_64-r0 platform - Update all platform plugins to be compatible with both Python 2 and Python 3 - Remove shebangs from plugins files which are not intended to be executable - Replace tabs with spaces in Python plugin files and fix alignment, because Python 3 is more strict - Remove trailing whitespace from plugins files
2020-11-25 12:28:36 -06:00
export sonic_utilities_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_UTILITIES_PY3))"
export sonic_host_services_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_HOST_SERVICES_PY3))"
export components="$(foreach component,$(notdir $^),\
$(shell [[ ! -z '$($(component)_VERSION)' && ! -z '$($(component)_NAME)' ]] && \
echo $($(component)_NAME)==$($(component)_VERSION)))"
$(foreach docker, $($*_DOCKERS),\
export docker_image="$(docker)"
export docker_image_name="$(basename $(docker))"
[build]: Build sonic-broadcom.bin using debug dockers for all stretch based dockers (#2833) * Updated Makefile infrastructure to build debug images. As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target. Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image. NOTE: If you don't specify NOSTRETcH=1, it implicitly calls "make stretch", which builds all stretch targets and that would include debug dockers too. This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside. "docker run -it --entrypoint=/bin/bash e47a8fb8ed38" You may map the core file path to this docker run. * Dropped the regular binary using DBG_PACKAGES and a small name change to help readability. * Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y. When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag. Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior. * 1) slave.mk - Dropped unused Docker build args 2) Debug template builder: renamed build_dbg_j2.sh to build_debug_docker_j2.sh 3) Dropped insignifcant statement CMD from debug Docker file, as base docker has Entrypoint. * Reverted some changes, per review comments. "User, uid, guid, frr-uid & frr-guid" are required for all docker images, with exception of debug images. * Get in sync with the new update that filters out dockers to be built (SONIC_STRETCH_DOCKERS_FOR_INSTALLERS) and build debug-dockers only for those to be built and debug target is available. * Mkae a template for each target that can be shared by all platforms. Where needed a platform entry can override the template. This avoids duplication, hence easier to maintain. * A small change, that can fit better with other targets too. Just take the platform code and do the rest in template. * Extended debug to all stretch based docker images * 1) Combined all orchagent makefiles into one platform independent make under rules/docker-orchagent.mk 2) Extened debug image to all stretch dockers * Changes per review comments: 1) Dropped LIBSAIREDIS_DBG from database, teamd, router-advertiser, telemetry, and platform-monitor docker*.mk files from _DBG_DEPENDS list 2) W.r.t docker make for syncd, moved DEPENDS from template to specific makefile and let the template has stuff that is applicable to all. * 1) Corrected a copy/paste mistake * Fixed a copy/paste bug * The base syncd dockers follow a template, which defines the base docker as DOCKER_SYNCD_BASE instead of DOCKER_SYNCD_<platform code>. Fix the docker-syncd-<mlnx, bfn>.mk to use the new one. [Yet to be tested locally] * Fixed spelling mistake * Enable build of dbg-sonic-broadcom.bin, which uses dbg-dockers in place of regular dockers, for dockers that build debug version. For dockers that do not build debug version, it uses the regular docker. This debug bin is installable and usable in a DUT, just like a regular bin. * Per review comments: 1) Share a single rule for final image for normal & debug flavors (e.g. sonic-broadcom.bin & sonic-broadcom-dbg.bin) 2) Put dbg as suffix in final image name. 3) Compared target/sonic-broadcom.bin.logs with & w/o fix to verify integrity of sonic-broadcom.bin 4) Compared target/sonic-broadcom.bin.logs with sonic-broadcom-dbg.bin.log for verification This fix takes care of ONIE image only. The next PR will cover the rest. The next PR, will also make debug image conditional with flag. * Updated per comments. Now that debug dockers are available, do not need a way to install debug symbols in regular dockers. With this commit, when INSTALL_DEBUG_TOOLS=y is set, it builds debug dockers (for dockers that enable debug build) and the final image uses debug dockers. For dockers that do not enable debug build, regular dockers get used in the final image. Note: The debug dockers are explicitly named as <docker name>-dbg.gz. But there is no "-dbg" suffix for image. Hence if you make two runs with and w/o INSTALL_DEBUG_TOOLS=y, you have complete set of regular dockers + debug dockers. But the image gets overwritten. Hence if both regular & debug images are needed, make two runs, as one with INSTALL_DEBUG_TOOLS=y and one w/o. Make sure to copy/rename the final image, before making the second run.
2019-06-12 03:36:21 -05:00
export docker_container_name="$($(docker:-dbg.gz=.gz)_CONTAINER_NAME)"
export mount_default_tmpfs="y"
[build]: Build sonic-broadcom.bin using debug dockers for all stretch based dockers (#2833) * Updated Makefile infrastructure to build debug images. As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target. Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image. NOTE: If you don't specify NOSTRETcH=1, it implicitly calls "make stretch", which builds all stretch targets and that would include debug dockers too. This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside. "docker run -it --entrypoint=/bin/bash e47a8fb8ed38" You may map the core file path to this docker run. * Dropped the regular binary using DBG_PACKAGES and a small name change to help readability. * Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y. When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag. Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior. * 1) slave.mk - Dropped unused Docker build args 2) Debug template builder: renamed build_dbg_j2.sh to build_debug_docker_j2.sh 3) Dropped insignifcant statement CMD from debug Docker file, as base docker has Entrypoint. * Reverted some changes, per review comments. "User, uid, guid, frr-uid & frr-guid" are required for all docker images, with exception of debug images. * Get in sync with the new update that filters out dockers to be built (SONIC_STRETCH_DOCKERS_FOR_INSTALLERS) and build debug-dockers only for those to be built and debug target is available. * Mkae a template for each target that can be shared by all platforms. Where needed a platform entry can override the template. This avoids duplication, hence easier to maintain. * A small change, that can fit better with other targets too. Just take the platform code and do the rest in template. * Extended debug to all stretch based docker images * 1) Combined all orchagent makefiles into one platform independent make under rules/docker-orchagent.mk 2) Extened debug image to all stretch dockers * Changes per review comments: 1) Dropped LIBSAIREDIS_DBG from database, teamd, router-advertiser, telemetry, and platform-monitor docker*.mk files from _DBG_DEPENDS list 2) W.r.t docker make for syncd, moved DEPENDS from template to specific makefile and let the template has stuff that is applicable to all. * 1) Corrected a copy/paste mistake * Fixed a copy/paste bug * The base syncd dockers follow a template, which defines the base docker as DOCKER_SYNCD_BASE instead of DOCKER_SYNCD_<platform code>. Fix the docker-syncd-<mlnx, bfn>.mk to use the new one. [Yet to be tested locally] * Fixed spelling mistake * Enable build of dbg-sonic-broadcom.bin, which uses dbg-dockers in place of regular dockers, for dockers that build debug version. For dockers that do not build debug version, it uses the regular docker. This debug bin is installable and usable in a DUT, just like a regular bin. * Per review comments: 1) Share a single rule for final image for normal & debug flavors (e.g. sonic-broadcom.bin & sonic-broadcom-dbg.bin) 2) Put dbg as suffix in final image name. 3) Compared target/sonic-broadcom.bin.logs with & w/o fix to verify integrity of sonic-broadcom.bin 4) Compared target/sonic-broadcom.bin.logs with sonic-broadcom-dbg.bin.log for verification This fix takes care of ONIE image only. The next PR will cover the rest. The next PR, will also make debug image conditional with flag. * Updated per comments. Now that debug dockers are available, do not need a way to install debug symbols in regular dockers. With this commit, when INSTALL_DEBUG_TOOLS=y is set, it builds debug dockers (for dockers that enable debug build) and the final image uses debug dockers. For dockers that do not enable debug build, regular dockers get used in the final image. Note: The debug dockers are explicitly named as <docker name>-dbg.gz. But there is no "-dbg" suffix for image. Hence if you make two runs with and w/o INSTALL_DEBUG_TOOLS=y, you have complete set of regular dockers + debug dockers. But the image gets overwritten. Hence if both regular & debug images are needed, make two runs, as one with INSTALL_DEBUG_TOOLS=y and one w/o. Make sure to copy/rename the final image, before making the second run.
2019-06-12 03:36:21 -05:00
$(eval $(docker:-dbg.gz=.gz)_RUN_OPT += $($(docker:-dbg.gz=.gz)_$($*_IMAGE_TYPE)_RUN_OPT))
export docker_image_run_opt="$($(docker:-dbg.gz=.gz)_RUN_OPT)"
[build]: Build sonic-broadcom.bin using debug dockers for all stretch based dockers (#2833) * Updated Makefile infrastructure to build debug images. As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target. Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image. NOTE: If you don't specify NOSTRETcH=1, it implicitly calls "make stretch", which builds all stretch targets and that would include debug dockers too. This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside. "docker run -it --entrypoint=/bin/bash e47a8fb8ed38" You may map the core file path to this docker run. * Dropped the regular binary using DBG_PACKAGES and a small name change to help readability. * Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y. When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag. Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior. * 1) slave.mk - Dropped unused Docker build args 2) Debug template builder: renamed build_dbg_j2.sh to build_debug_docker_j2.sh 3) Dropped insignifcant statement CMD from debug Docker file, as base docker has Entrypoint. * Reverted some changes, per review comments. "User, uid, guid, frr-uid & frr-guid" are required for all docker images, with exception of debug images. * Get in sync with the new update that filters out dockers to be built (SONIC_STRETCH_DOCKERS_FOR_INSTALLERS) and build debug-dockers only for those to be built and debug target is available. * Mkae a template for each target that can be shared by all platforms. Where needed a platform entry can override the template. This avoids duplication, hence easier to maintain. * A small change, that can fit better with other targets too. Just take the platform code and do the rest in template. * Extended debug to all stretch based docker images * 1) Combined all orchagent makefiles into one platform independent make under rules/docker-orchagent.mk 2) Extened debug image to all stretch dockers * Changes per review comments: 1) Dropped LIBSAIREDIS_DBG from database, teamd, router-advertiser, telemetry, and platform-monitor docker*.mk files from _DBG_DEPENDS list 2) W.r.t docker make for syncd, moved DEPENDS from template to specific makefile and let the template has stuff that is applicable to all. * 1) Corrected a copy/paste mistake * Fixed a copy/paste bug * The base syncd dockers follow a template, which defines the base docker as DOCKER_SYNCD_BASE instead of DOCKER_SYNCD_<platform code>. Fix the docker-syncd-<mlnx, bfn>.mk to use the new one. [Yet to be tested locally] * Fixed spelling mistake * Enable build of dbg-sonic-broadcom.bin, which uses dbg-dockers in place of regular dockers, for dockers that build debug version. For dockers that do not build debug version, it uses the regular docker. This debug bin is installable and usable in a DUT, just like a regular bin. * Per review comments: 1) Share a single rule for final image for normal & debug flavors (e.g. sonic-broadcom.bin & sonic-broadcom-dbg.bin) 2) Put dbg as suffix in final image name. 3) Compared target/sonic-broadcom.bin.logs with & w/o fix to verify integrity of sonic-broadcom.bin 4) Compared target/sonic-broadcom.bin.logs with sonic-broadcom-dbg.bin.log for verification This fix takes care of ONIE image only. The next PR will cover the rest. The next PR, will also make debug image conditional with flag. * Updated per comments. Now that debug dockers are available, do not need a way to install debug symbols in regular dockers. With this commit, when INSTALL_DEBUG_TOOLS=y is set, it builds debug dockers (for dockers that enable debug build) and the final image uses debug dockers. For dockers that do not enable debug build, regular dockers get used in the final image. Note: The debug dockers are explicitly named as <docker name>-dbg.gz. But there is no "-dbg" suffix for image. Hence if you make two runs with and w/o INSTALL_DEBUG_TOOLS=y, you have complete set of regular dockers + debug dockers. But the image gets overwritten. Hence if both regular & debug images are needed, make two runs, as one with INSTALL_DEBUG_TOOLS=y and one w/o. Make sure to copy/rename the final image, before making the second run.
2019-06-12 03:36:21 -05:00
if [ -f files/build_templates/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 ]; then
j2 files/build_templates/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 > $($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service
# Set the flag GLOBAL for all the global system-wide dockers.
$(if $(shell ls files/build_templates/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 2>/dev/null),\
$(eval $(docker:-dbg.gz=.gz)_GLOBAL = yes)
)
fi
# Any service template, inside instance directory, will be used to generate .service and @.service file.
if [ -f files/build_templates/per_namespace/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 ]; then
export multi_instance="true"
j2 files/build_templates/per_namespace/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 > $($(docker:-dbg.gz=.gz)_CONTAINER_NAME)@.service
$(if $(shell ls files/build_templates/per_namespace/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 2>/dev/null),\
$(eval $(docker:-dbg.gz=.gz)_TEMPLATE = yes)
)
export multi_instance="false"
j2 files/build_templates/per_namespace/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 > $($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service
fi
# Any service template, inside share_image directory, will be used to generate -chassis.service file.
# TODO: need better way to name the image-shared service
if [ -f files/build_templates/share_image/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 ]; then
j2 files/build_templates/share_image/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 > $($(docker:-dbg.gz=.gz)_CONTAINER_NAME)-chassis.service
$(if $(shell ls files/build_templates/share_image/$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service.j2 2>/dev/null),\
$(eval $(docker:-dbg.gz=.gz)_SHARE = yes)
)
fi
j2 files/build_templates/docker_image_ctl.j2 > $($(docker:-dbg.gz=.gz)_CONTAINER_NAME).sh
[build]: Build sonic-broadcom.bin using debug dockers for all stretch based dockers (#2833) * Updated Makefile infrastructure to build debug images. As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target. Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image. NOTE: If you don't specify NOSTRETcH=1, it implicitly calls "make stretch", which builds all stretch targets and that would include debug dockers too. This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside. "docker run -it --entrypoint=/bin/bash e47a8fb8ed38" You may map the core file path to this docker run. * Dropped the regular binary using DBG_PACKAGES and a small name change to help readability. * Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y. When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag. Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior. * 1) slave.mk - Dropped unused Docker build args 2) Debug template builder: renamed build_dbg_j2.sh to build_debug_docker_j2.sh 3) Dropped insignifcant statement CMD from debug Docker file, as base docker has Entrypoint. * Reverted some changes, per review comments. "User, uid, guid, frr-uid & frr-guid" are required for all docker images, with exception of debug images. * Get in sync with the new update that filters out dockers to be built (SONIC_STRETCH_DOCKERS_FOR_INSTALLERS) and build debug-dockers only for those to be built and debug target is available. * Mkae a template for each target that can be shared by all platforms. Where needed a platform entry can override the template. This avoids duplication, hence easier to maintain. * A small change, that can fit better with other targets too. Just take the platform code and do the rest in template. * Extended debug to all stretch based docker images * 1) Combined all orchagent makefiles into one platform independent make under rules/docker-orchagent.mk 2) Extened debug image to all stretch dockers * Changes per review comments: 1) Dropped LIBSAIREDIS_DBG from database, teamd, router-advertiser, telemetry, and platform-monitor docker*.mk files from _DBG_DEPENDS list 2) W.r.t docker make for syncd, moved DEPENDS from template to specific makefile and let the template has stuff that is applicable to all. * 1) Corrected a copy/paste mistake * Fixed a copy/paste bug * The base syncd dockers follow a template, which defines the base docker as DOCKER_SYNCD_BASE instead of DOCKER_SYNCD_<platform code>. Fix the docker-syncd-<mlnx, bfn>.mk to use the new one. [Yet to be tested locally] * Fixed spelling mistake * Enable build of dbg-sonic-broadcom.bin, which uses dbg-dockers in place of regular dockers, for dockers that build debug version. For dockers that do not build debug version, it uses the regular docker. This debug bin is installable and usable in a DUT, just like a regular bin. * Per review comments: 1) Share a single rule for final image for normal & debug flavors (e.g. sonic-broadcom.bin & sonic-broadcom-dbg.bin) 2) Put dbg as suffix in final image name. 3) Compared target/sonic-broadcom.bin.logs with & w/o fix to verify integrity of sonic-broadcom.bin 4) Compared target/sonic-broadcom.bin.logs with sonic-broadcom-dbg.bin.log for verification This fix takes care of ONIE image only. The next PR will cover the rest. The next PR, will also make debug image conditional with flag. * Updated per comments. Now that debug dockers are available, do not need a way to install debug symbols in regular dockers. With this commit, when INSTALL_DEBUG_TOOLS=y is set, it builds debug dockers (for dockers that enable debug build) and the final image uses debug dockers. For dockers that do not enable debug build, regular dockers get used in the final image. Note: The debug dockers are explicitly named as <docker name>-dbg.gz. But there is no "-dbg" suffix for image. Hence if you make two runs with and w/o INSTALL_DEBUG_TOOLS=y, you have complete set of regular dockers + debug dockers. But the image gets overwritten. Hence if both regular & debug images are needed, make two runs, as one with INSTALL_DEBUG_TOOLS=y and one w/o. Make sure to copy/rename the final image, before making the second run.
2019-06-12 03:36:21 -05:00
chmod +x $($(docker:-dbg.gz=.gz)_CONTAINER_NAME).sh
$(if $($(docker:-dbg.gz=.gz)_MACHINE),\
mv $($(docker:-dbg.gz=.gz)_CONTAINER_NAME).sh $($(docker:-dbg.gz=.gz)_MACHINE)_$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).sh
)
)
# Exported variables are used by sonic_debian_extension.sh
[build]: Build sonic-broadcom.bin using debug dockers for all stretch based dockers (#2833) * Updated Makefile infrastructure to build debug images. As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target. Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image. NOTE: If you don't specify NOSTRETcH=1, it implicitly calls "make stretch", which builds all stretch targets and that would include debug dockers too. This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside. "docker run -it --entrypoint=/bin/bash e47a8fb8ed38" You may map the core file path to this docker run. * Dropped the regular binary using DBG_PACKAGES and a small name change to help readability. * Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y. When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag. Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior. * 1) slave.mk - Dropped unused Docker build args 2) Debug template builder: renamed build_dbg_j2.sh to build_debug_docker_j2.sh 3) Dropped insignifcant statement CMD from debug Docker file, as base docker has Entrypoint. * Reverted some changes, per review comments. "User, uid, guid, frr-uid & frr-guid" are required for all docker images, with exception of debug images. * Get in sync with the new update that filters out dockers to be built (SONIC_STRETCH_DOCKERS_FOR_INSTALLERS) and build debug-dockers only for those to be built and debug target is available. * Mkae a template for each target that can be shared by all platforms. Where needed a platform entry can override the template. This avoids duplication, hence easier to maintain. * A small change, that can fit better with other targets too. Just take the platform code and do the rest in template. * Extended debug to all stretch based docker images * 1) Combined all orchagent makefiles into one platform independent make under rules/docker-orchagent.mk 2) Extened debug image to all stretch dockers * Changes per review comments: 1) Dropped LIBSAIREDIS_DBG from database, teamd, router-advertiser, telemetry, and platform-monitor docker*.mk files from _DBG_DEPENDS list 2) W.r.t docker make for syncd, moved DEPENDS from template to specific makefile and let the template has stuff that is applicable to all. * 1) Corrected a copy/paste mistake * Fixed a copy/paste bug * The base syncd dockers follow a template, which defines the base docker as DOCKER_SYNCD_BASE instead of DOCKER_SYNCD_<platform code>. Fix the docker-syncd-<mlnx, bfn>.mk to use the new one. [Yet to be tested locally] * Fixed spelling mistake * Enable build of dbg-sonic-broadcom.bin, which uses dbg-dockers in place of regular dockers, for dockers that build debug version. For dockers that do not build debug version, it uses the regular docker. This debug bin is installable and usable in a DUT, just like a regular bin. * Per review comments: 1) Share a single rule for final image for normal & debug flavors (e.g. sonic-broadcom.bin & sonic-broadcom-dbg.bin) 2) Put dbg as suffix in final image name. 3) Compared target/sonic-broadcom.bin.logs with & w/o fix to verify integrity of sonic-broadcom.bin 4) Compared target/sonic-broadcom.bin.logs with sonic-broadcom-dbg.bin.log for verification This fix takes care of ONIE image only. The next PR will cover the rest. The next PR, will also make debug image conditional with flag. * Updated per comments. Now that debug dockers are available, do not need a way to install debug symbols in regular dockers. With this commit, when INSTALL_DEBUG_TOOLS=y is set, it builds debug dockers (for dockers that enable debug build) and the final image uses debug dockers. For dockers that do not enable debug build, regular dockers get used in the final image. Note: The debug dockers are explicitly named as <docker name>-dbg.gz. But there is no "-dbg" suffix for image. Hence if you make two runs with and w/o INSTALL_DEBUG_TOOLS=y, you have complete set of regular dockers + debug dockers. But the image gets overwritten. Hence if both regular & debug images are needed, make two runs, as one with INSTALL_DEBUG_TOOLS=y and one w/o. Make sure to copy/rename the final image, before making the second run.
2019-06-12 03:36:21 -05:00
export installer_start_scripts="$(foreach docker, $($*_DOCKERS),$(addsuffix .sh, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME)))"
export feature_vs_image_names="$(foreach docker, $($*_DOCKERS), $(addsuffix :, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME):$(docker:-dbg.gz=.gz)))"
# Marks template services with an "@" according to systemd convention
# If the $($docker)_TEMPLATE) variable is set, the service will be treated as a template
# If the $($docker)_GLOBAL) and $($docker)_TEMPLATE) variables are set the service will be added both as a global and template service.
$(foreach docker, $($*_DOCKERS),\
$(if $($(docker:-dbg.gz=.gz)_TEMPLATE),\
$(if $($(docker:-dbg.gz=.gz)_GLOBAL),\
$(eval SERVICES += "$(addsuffix .service, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME))")\
)\
$(eval SERVICES += "$(addsuffix @.service, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME))"),\
$(eval SERVICES += "$(addsuffix .service, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME))")
)
$(if $($(docker:-dbg.gz=.gz)_SHARE),\
$(eval SERVICES += "$(addsuffix -chassis.service, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME))")
)
)
export installer_services="$(SERVICES)"
[build]: Build sonic-broadcom.bin using debug dockers for all stretch based dockers (#2833) * Updated Makefile infrastructure to build debug images. As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target. Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image. NOTE: If you don't specify NOSTRETcH=1, it implicitly calls "make stretch", which builds all stretch targets and that would include debug dockers too. This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside. "docker run -it --entrypoint=/bin/bash e47a8fb8ed38" You may map the core file path to this docker run. * Dropped the regular binary using DBG_PACKAGES and a small name change to help readability. * Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y. When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag. Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior. * 1) slave.mk - Dropped unused Docker build args 2) Debug template builder: renamed build_dbg_j2.sh to build_debug_docker_j2.sh 3) Dropped insignifcant statement CMD from debug Docker file, as base docker has Entrypoint. * Reverted some changes, per review comments. "User, uid, guid, frr-uid & frr-guid" are required for all docker images, with exception of debug images. * Get in sync with the new update that filters out dockers to be built (SONIC_STRETCH_DOCKERS_FOR_INSTALLERS) and build debug-dockers only for those to be built and debug target is available. * Mkae a template for each target that can be shared by all platforms. Where needed a platform entry can override the template. This avoids duplication, hence easier to maintain. * A small change, that can fit better with other targets too. Just take the platform code and do the rest in template. * Extended debug to all stretch based docker images * 1) Combined all orchagent makefiles into one platform independent make under rules/docker-orchagent.mk 2) Extened debug image to all stretch dockers * Changes per review comments: 1) Dropped LIBSAIREDIS_DBG from database, teamd, router-advertiser, telemetry, and platform-monitor docker*.mk files from _DBG_DEPENDS list 2) W.r.t docker make for syncd, moved DEPENDS from template to specific makefile and let the template has stuff that is applicable to all. * 1) Corrected a copy/paste mistake * Fixed a copy/paste bug * The base syncd dockers follow a template, which defines the base docker as DOCKER_SYNCD_BASE instead of DOCKER_SYNCD_<platform code>. Fix the docker-syncd-<mlnx, bfn>.mk to use the new one. [Yet to be tested locally] * Fixed spelling mistake * Enable build of dbg-sonic-broadcom.bin, which uses dbg-dockers in place of regular dockers, for dockers that build debug version. For dockers that do not build debug version, it uses the regular docker. This debug bin is installable and usable in a DUT, just like a regular bin. * Per review comments: 1) Share a single rule for final image for normal & debug flavors (e.g. sonic-broadcom.bin & sonic-broadcom-dbg.bin) 2) Put dbg as suffix in final image name. 3) Compared target/sonic-broadcom.bin.logs with & w/o fix to verify integrity of sonic-broadcom.bin 4) Compared target/sonic-broadcom.bin.logs with sonic-broadcom-dbg.bin.log for verification This fix takes care of ONIE image only. The next PR will cover the rest. The next PR, will also make debug image conditional with flag. * Updated per comments. Now that debug dockers are available, do not need a way to install debug symbols in regular dockers. With this commit, when INSTALL_DEBUG_TOOLS=y is set, it builds debug dockers (for dockers that enable debug build) and the final image uses debug dockers. For dockers that do not enable debug build, regular dockers get used in the final image. Note: The debug dockers are explicitly named as <docker name>-dbg.gz. But there is no "-dbg" suffix for image. Hence if you make two runs with and w/o INSTALL_DEBUG_TOOLS=y, you have complete set of regular dockers + debug dockers. But the image gets overwritten. Hence if both regular & debug images are needed, make two runs, as one with INSTALL_DEBUG_TOOLS=y and one w/o. Make sure to copy/rename the final image, before making the second run.
2019-06-12 03:36:21 -05:00
export installer_extra_files="$(foreach docker, $($*_DOCKERS), $(foreach file, $($(docker:-dbg.gz=.gz)_BASE_IMAGE_FILES), $($(docker:-dbg.gz=.gz)_PATH)/base_image_files/$(file)))"
j2 -f env files/initramfs-tools/union-mount.j2 onie-image.conf > files/initramfs-tools/union-mount
j2 -f env files/initramfs-tools/arista-convertfs.j2 onie-image.conf > files/initramfs-tools/arista-convertfs
$(if $($*_DOCKERS),
j2 files/build_templates/sonic_debian_extension.j2 > sonic_debian_extension.sh
chmod +x sonic_debian_extension.sh,
)
# Build images for the MACHINE, DEPENDENT_MACHINE defined.
$(foreach dep_machine, $($*_MACHINE) $($*_DEPENDENT_MACHINE), \
DEBUG_IMG="$(INSTALL_DEBUG_TOOLS)" \
DEBUG_SRC_ARCHIVE_DIRS="$(DBG_SRC_ARCHIVE)" \
DEBUG_SRC_ARCHIVE_FILE="$(DBG_SRC_ARCHIVE_FILE)" \
scripts/dbg_files.sh
DEBUG_IMG="$(INSTALL_DEBUG_TOOLS)" \
DEBUG_SRC_ARCHIVE_FILE="$(DBG_SRC_ARCHIVE_FILE)" \
USERNAME="$(USERNAME)" \
PASSWORD="$(PASSWORD)" \
TARGET_MACHINE=$(dep_machine) \
IMAGE_TYPE=$($*_IMAGE_TYPE) \
TARGET_PATH=$(TARGET_PATH) \
SONIC_ENFORCE_VERSIONS=$(SONIC_ENFORCE_VERSIONS) \
TRUSTED_GPG_URLS=$(TRUSTED_GPG_URLS) \
PACKAGE_URL_PREFIX=$(PACKAGE_URL_PREFIX) \
MULTIARCH_QEMU_ENVIRON=$(MULTIARCH_QEMU_ENVIRON) \
./build_debian.sh $(LOG)
USERNAME="$(USERNAME)" \
PASSWORD="$(PASSWORD)" \
TARGET_MACHINE=$(dep_machine) \
IMAGE_TYPE=$($*_IMAGE_TYPE) \
SONIC_ENABLE_IMAGE_SIGNATURE="$(SONIC_ENABLE_IMAGE_SIGNATURE)" \
SIGNING_KEY="$(SIGNING_KEY)" \
SIGNING_CERT="$(SIGNING_CERT)" \
CA_CERT="$(CA_CERT)" \
TARGET_PATH="$(TARGET_PATH)" \
./build_image.sh $(LOG)
)
$(foreach docker, $($*_DOCKERS), \
rm -f *$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).sh
[build]: Build sonic-broadcom.bin using debug dockers for all stretch based dockers (#2833) * Updated Makefile infrastructure to build debug images. As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target. Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image. NOTE: If you don't specify NOSTRETcH=1, it implicitly calls "make stretch", which builds all stretch targets and that would include debug dockers too. This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside. "docker run -it --entrypoint=/bin/bash e47a8fb8ed38" You may map the core file path to this docker run. * Dropped the regular binary using DBG_PACKAGES and a small name change to help readability. * Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y. When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag. Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior. * 1) slave.mk - Dropped unused Docker build args 2) Debug template builder: renamed build_dbg_j2.sh to build_debug_docker_j2.sh 3) Dropped insignifcant statement CMD from debug Docker file, as base docker has Entrypoint. * Reverted some changes, per review comments. "User, uid, guid, frr-uid & frr-guid" are required for all docker images, with exception of debug images. * Get in sync with the new update that filters out dockers to be built (SONIC_STRETCH_DOCKERS_FOR_INSTALLERS) and build debug-dockers only for those to be built and debug target is available. * Mkae a template for each target that can be shared by all platforms. Where needed a platform entry can override the template. This avoids duplication, hence easier to maintain. * A small change, that can fit better with other targets too. Just take the platform code and do the rest in template. * Extended debug to all stretch based docker images * 1) Combined all orchagent makefiles into one platform independent make under rules/docker-orchagent.mk 2) Extened debug image to all stretch dockers * Changes per review comments: 1) Dropped LIBSAIREDIS_DBG from database, teamd, router-advertiser, telemetry, and platform-monitor docker*.mk files from _DBG_DEPENDS list 2) W.r.t docker make for syncd, moved DEPENDS from template to specific makefile and let the template has stuff that is applicable to all. * 1) Corrected a copy/paste mistake * Fixed a copy/paste bug * The base syncd dockers follow a template, which defines the base docker as DOCKER_SYNCD_BASE instead of DOCKER_SYNCD_<platform code>. Fix the docker-syncd-<mlnx, bfn>.mk to use the new one. [Yet to be tested locally] * Fixed spelling mistake * Enable build of dbg-sonic-broadcom.bin, which uses dbg-dockers in place of regular dockers, for dockers that build debug version. For dockers that do not build debug version, it uses the regular docker. This debug bin is installable and usable in a DUT, just like a regular bin. * Per review comments: 1) Share a single rule for final image for normal & debug flavors (e.g. sonic-broadcom.bin & sonic-broadcom-dbg.bin) 2) Put dbg as suffix in final image name. 3) Compared target/sonic-broadcom.bin.logs with & w/o fix to verify integrity of sonic-broadcom.bin 4) Compared target/sonic-broadcom.bin.logs with sonic-broadcom-dbg.bin.log for verification This fix takes care of ONIE image only. The next PR will cover the rest. The next PR, will also make debug image conditional with flag. * Updated per comments. Now that debug dockers are available, do not need a way to install debug symbols in regular dockers. With this commit, when INSTALL_DEBUG_TOOLS=y is set, it builds debug dockers (for dockers that enable debug build) and the final image uses debug dockers. For dockers that do not enable debug build, regular dockers get used in the final image. Note: The debug dockers are explicitly named as <docker name>-dbg.gz. But there is no "-dbg" suffix for image. Hence if you make two runs with and w/o INSTALL_DEBUG_TOOLS=y, you have complete set of regular dockers + debug dockers. But the image gets overwritten. Hence if both regular & debug images are needed, make two runs, as one with INSTALL_DEBUG_TOOLS=y and one w/o. Make sure to copy/rename the final image, before making the second run.
2019-06-12 03:36:21 -05:00
rm -f $($(docker:-dbg.gz=.gz)_CONTAINER_NAME).service
rm -f $($(docker:-dbg.gz=.gz)_CONTAINER_NAME)@.service
)
$(if $($*_DOCKERS),
rm sonic_debian_extension.sh,
)
chmod a+x $@
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(FOOTER)
SONIC_TARGET_LIST += $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Clean targets
###############################################################################
SONIC_CLEAN_DEBS = $(addsuffix -clean,$(addprefix $(DEBS_PATH)/, \
$(SONIC_ONLINE_DEBS) \
$(SONIC_COPY_DEBS) \
$(SONIC_MAKE_DEBS) \
$(SONIC_DPKG_DEBS) \
$(SONIC_DERIVED_DEBS) \
$(SONIC_EXTRA_DEBS)))
SONIC_CLEAN_FILES = $(addsuffix -clean,$(addprefix $(FILES_PATH)/, \
$(SONIC_ONLINE_FILES) \
$(SONIC_COPY_FILES) \
$(SONIC_MAKE_FILES)))
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(SONIC_CLEAN_DEBS) :: $(DEBS_PATH)/%-clean : .platform $$(addsuffix -clean,$$(addprefix $(DEBS_PATH)/,$$($$*_MAIN_DEB)))
@# remove derived or extra targets if main one is removed, because we treat them
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
@# as part of one package
@rm -f $(addprefix $(DEBS_PATH)/, $* $($*_DERIVED_DEBS) $($*_EXTRA_DEBS))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(SONIC_CLEAN_FILES) :: $(FILES_PATH)/%-clean : .platform
@rm -f $(FILES_PATH)/$*
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
SONIC_CLEAN_TARGETS += $(addsuffix -clean,$(addprefix $(TARGET_PATH)/, \
$(SONIC_DOCKER_IMAGES) \
$(SONIC_DOCKER_DBG_IMAGES) \
$(SONIC_SIMPLE_DOCKER_IMAGES) \
$(SONIC_INSTALLERS)))
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(SONIC_CLEAN_TARGETS) :: $(TARGET_PATH)/%-clean : .platform
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
@rm -f $(TARGET_PATH)/$*
SONIC_CLEAN_STDEB_DEBS = $(addsuffix -clean,$(addprefix $(PYTHON_DEBS_PATH)/, \
$(SONIC_PYTHON_STDEB_DEBS)))
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(SONIC_CLEAN_STDEB_DEBS) :: $(PYTHON_DEBS_PATH)/%-clean : .platform
@rm -f $(PYTHON_DEBS_PATH)/$*
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
SONIC_CLEAN_WHEELS = $(addsuffix -clean,$(addprefix $(PYTHON_WHEELS_PATH)/, \
$(SONIC_PYTHON_WHEELS)))
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
$(SONIC_CLEAN_WHEELS) :: $(PYTHON_WHEELS_PATH)/%-clean : .platform
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
@rm -f $(PYTHON_WHEELS_PATH)/$*
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
clean-logs :: .platform
@rm -f $(TARGET_PATH)/*.log $(DEBS_PATH)/*.log $(FILES_PATH)/*.log $(PYTHON_DEBS_PATH)/*.log $(PYTHON_WHEELS_PATH)/*.log
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
clean :: .platform clean-logs $$(SONIC_CLEAN_DEBS) $$(SONIC_CLEAN_FILES) $$(SONIC_CLEAN_TARGETS) $$(SONIC_CLEAN_STDEB_DEBS) $$(SONIC_CLEAN_WHEELS)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## all
###############################################################################
all : .platform $$(addprefix $(TARGET_PATH)/,$$(SONIC_ALL))
stretch : $$(addprefix $(TARGET_PATH)/,$$(STRETCH_DOCKER_IMAGES)) \
$$(addprefix $(TARGET_PATH)/,$$(STRETCH_DBG_DOCKER_IMAGES))
jessie : $$(addprefix $(TARGET_PATH)/,$$(JESSIE_DOCKER_IMAGES)) \
$$(addprefix $(TARGET_PATH)/,$$(JESSIE_DBG_DOCKER_IMAGES))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Standard targets
###############################################################################
.PHONY : $(SONIC_CLEAN_DEBS) $(SONIC_CLEAN_FILES) $(SONIC_CLEAN_TARGETS) $(SONIC_CLEAN_STDEB_DEBS) $(SONIC_CLEAN_WHEELS) $(SONIC_PHONY_TARGETS) clean distclean configure
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
.INTERMEDIATE : $(SONIC_INSTALL_DEBS) $(SONIC_INSTALL_WHEELS) $(DOCKER_LOAD_TARGETS) docker-start .platform
## To build some commonly used libs. Some submodules depend on these libs.
## It is used in component pipelines. For example: swss needs libnl, libyang
lib-packages: $(addprefix $(DEBS_PATH)/,$(LIBNL3) $(LIBYANG))