sonic-buildimage/rules/functions

124 lines
3.4 KiB
Plaintext
Raw Normal View History

Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## FUNCTIONS
###############################################################################
###############################################################################
## Colored output
###############################################################################
# Enable colored output
ifeq ($(SONIC_CONFIG_ENABLE_COLORS),y)
ifeq ($(MAKE_TERMOUT),)
RED=\033[1;31m
PURPLE=\033[1;35m
CYAN=\033[1;36m
GREEN=\033[1;32m
GRAY=\033[0m
endif
endif
# Print red colored output
# call:
# log_red message
log_red = echo -e "$(RED)$(1)$(GRAY)"
# Print purple colored output
# call:
# log_purple message
log_purple = echo -e "$(PURPLE)$(1)$(GRAY)"
# Print blue colored output
# call:
# log_blue message
log_blue = echo -e "$(CYAN)$(1)$(GRAY)"
# Print green colored output
# call:
# log_green message
log_green = echo -e "$(GREEN)$(1)$(GRAY)"
###############################################################################
## Logging
###############################################################################
FLUSH_LOG = rm -f $@.log
LOG = &>> $(PROJECT_ROOT)/$@.log || { [ $$? -eq 0 ] || pushd $(PROJECT_ROOT) > /dev/null ; ./update_screen.sh -e $@ ; popd > /dev/null ; false ; }
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Header and footer for each target
###############################################################################
# Dump targets taht current depends on
ifeq ($(SONIC_CONFIG_PRINT_DEPENDENCIES),y)
PRINT_DEPENDENCIES = echo Dependencies for $@ are $^ $(LOG)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
endif
# header for each rule
define HEADER
@
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(PRINT_DEPENDENCIES)
$(FLUSH_LOG)
./update_screen.sh -a $@
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
endef
# footer for each rule
define FOOTER
./update_screen.sh -d $@
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
endef
###############################################################################
## Definition of derived target
###############################################################################
# call:
# add_derived_package some_deb.deb, some_derived_deb
define add_derived_package
$(2)_DEPENDS += $(1)
$(2)_RDEPENDS += $($(1)_RDEPENDS)
$(2)_MAIN_DEB = $(1)
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
$(1)_DERIVED_DEBS += $(2)
$(2)_URL = $($(1)_URL)
$(2)_SRC_PATH = $($(1)_SRC_PATH)
SONIC_DERIVED_DEBS += $(2)
endef
# call:
# add_extra_package some_deb.deb, some_extra_deb
define add_extra_package
$(2)_MAIN_DEB = $(1)
$(1)_EXTRA_DEBS += $(2)
SONIC_EXTRA_DEBS += $(2)
endef
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# call:
# add_dbg_docker some_docker.gz, some-docker-dbg.gz
define add_dbg_docker
$(2)_PATH = $($(1)_PATH)
$(2)_DBG_DEPENDS += $($(1)_DBG_DEPENDS)
$(2)_DBG_IMAGE_PACKAGES += $($(1)_DBG_IMAGE_PACKAGES)
$(2)_PYTHON_WHEELS += $($(1)_PYTHON_WHEELS)
$(2)_LOAD_DOCKERS += $($(1)_LOAD_DOCKERS)
$(2)_CACHE_MODE += $($(1)_CACHE_MODE)
$(2)_DEP_FLAGS += $($(1)_DEP_FLAGS)
$(2)_DEP_FILES += $($(1)_DEP_FILES)
$(2)_SMDEP_FILES += $($(1)_SMDEP_FILES)
$(2)_SMDEP_PATHS += $($(1)_SMDEP_PATHS)
$(2)_DEPENDS += $($(1)_DEPENDS)
$(2)_RDEPENDS += $($(1)_RDEPENDS)
$(2)_MAIN_DOCKER = $(1)
$(1)_DBG_DOCKER += $(2)
SONIC_DBG_DOCKERS += $(2)
endef
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
###############################################################################
## Utility functions
###############################################################################
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
# Recursive wildcard function
rwildcard=$(foreach d,$(wildcard $1*),$(call rwildcard,$d/,$2) $(filter $(subst *,%,$2),$d))
Build improvements (#80) * Build improvements Fix dependencies Add configuration options Automatically build sonic-slave * Set default number of jobs to 1 * Auto generate target/debs directory Signed-off-by: marian-pritsak <marianp@mellanox.com> * Automatically remove sonic-slave container after exit * Silence clean-logs * Add SONIC_CLEAN_TARGETS to clean * Use second expansion for clean dependencies * Avoid creating empty log files Remove log file on flush instead of writing empty string * Put dpkg install inside lock Use same lock as debian install targets do to avoid race condition in dpkg installation * Remove redirect to log from docker save * Add .platform dependency to all and clean targets * Remove header and footer from clean targets * Disable messages for SONIC_CLEAN_TARGETS * Exit with error if dpkg-buildpackage fails * Set new location for debs in build_debian.sh * Add recipe for docker-database * Update redis version to 3.2.4 * Add support for p4 platform * Add recipe for snmpd * Add slave targets to phony and make all target default * Remove build.sh from thrift * Add versioning to team, nl, hiredis and initramfs * Change sonic-slave to support snmpd build from sources * Remove src/tenjin * Add recipe for lldpd * Add recipe for mpdecimal * Remove hiredis directory on rebuild * Add recipe for Mellanox hw management * Remove generic image from all targets for Mellanox * Add support for python wheels * Add lldp and snmp dockers * Sync docker-database to include libjemalloc * Fix asyncsnmp variable name * Change default build configuration Redirect output to log files by default Set number of jobs to nproc value Do not print dependencies Fix logging to print log of failed job into console * Use docker inspect to check if sonic-slave image exists * Use config in slave.mk directly * Disable color output by default * Remove sswsdk dependency from lldp and snmp dockers * Fix comment in py wheels install targets * Add dependency between two versions of sswsdk * Add containers to mellanox platform lldp, snmp and database containers * Add recipe for team docker * Add team docker to mellanox platform * Encrypt password passed to build_debian.sh * Update mellanox SAI version Make version and revision setting only in main recipe * Fix error handling in makefiles As makefiles use .ONESHELL we should add -e option to shell options in order to exit after any command fails * Add recipe for platform monitor image * Add platfotm monitor to mellanox targets * Ignore submodules when building base image
2016-12-05 13:12:19 -06:00
expand = $(foreach d,$(1),$(call expand,$($(d)_$(2)),$(2))) $(1)