2018-08-16 17:54:31 -05:00
|
|
|
###############################################################################
|
|
|
|
|
## Wrapper for starting make inside sonic-slave container
|
|
|
|
|
#
|
|
|
|
|
# Supported parameters:
|
|
|
|
|
#
|
|
|
|
|
# * PLATFORM: Specific platform we wish to build images for.
|
|
|
|
|
# * BUILD_NUMBER: Desired version-number to pass to the building-system.
|
|
|
|
|
# * ENABLE_DHCP_GRAPH_SERVICE: Enables get-graph service to fetch minigraph files
|
|
|
|
|
# through http.
|
2019-12-04 06:50:56 -06:00
|
|
|
# * ENABLE_ZTP: Enables zero touch provisioning.
|
2018-08-16 17:54:31 -05:00
|
|
|
# * SHUTDOWN_BGP_ON_START: Sets admin-down state for all bgp peerings after restart.
|
2020-08-05 15:23:12 -05:00
|
|
|
# * INCLUDE_KUBERNETES: Allows including Kubernetes
|
2018-08-16 17:54:31 -05:00
|
|
|
# * ENABLE_PFCWD_ON_START: Enable PFC Watchdog (PFCWD) on server-facing ports
|
|
|
|
|
# * by default for TOR switch.
|
2018-10-21 19:20:27 -05:00
|
|
|
# * ENABLE_SYNCD_RPC: Enables rpc-based syncd builds.
|
|
|
|
|
# * INSTALL_DEBUG_TOOLS: Install debug tools and debug symbol packeages.
|
2018-08-16 17:54:31 -05:00
|
|
|
# * USERNAME: Desired username -- default at rules/config
|
|
|
|
|
# * PASSWORD: Desired password -- default at rules/config
|
2018-09-12 02:23:00 -05:00
|
|
|
# * KEEP_SLAVE_ON: Keeps slave container up and active after building process concludes.
|
|
|
|
|
# * Note that rm=true is still set, so once user quits from the docker
|
|
|
|
|
# * session, the docker will be removed.
|
|
|
|
|
# * Please note that with current Stretch build structure,
|
|
|
|
|
# * user of KEEP_SLAVE_ON feature will have to be conscious
|
|
|
|
|
# * about which docker to stay inside after build is done.
|
|
|
|
|
# * - If user desires to stay inside Jessie docker, please issue
|
2019-05-27 17:50:51 -05:00
|
|
|
# * make KEEP_SLAVE_ON=yes jessie
|
|
|
|
|
# * - If user desires to stay inside Stretch docker, please issue
|
|
|
|
|
# * make NOJESSIE=1 KEEP_SLAVE_ON=yes <any target>
|
2018-08-16 17:54:31 -05:00
|
|
|
# * SOURCE_FOLDER: host path to be mount as /var/$(USER)/src, only effective when KEEP_SLAVE_ON=yes
|
|
|
|
|
# * SONIC_BUILD_JOBS: Specifying number of concurrent build job(s) to run
|
2019-03-29 17:25:17 -05:00
|
|
|
# * VS_PREPARE_MEM: Prepare memory in VS build (drop cache and compact).
|
|
|
|
|
# * Default: yes
|
|
|
|
|
# * Values: yes, no
|
2018-08-16 17:54:31 -05:00
|
|
|
# * KERNEL_PROCURE_METHOD: Specifying method of obtaining kernel Debian package: download or build
|
2020-06-25 10:04:43 -05:00
|
|
|
# * TELEMETRY_WRITABLE: Enable write/config operations via the gNMI interface.
|
|
|
|
|
# * Default: unset
|
|
|
|
|
# * Values: y
|
[build]: support for DPKG local caching (#4117)
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.
The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.
This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.
- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=
where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
none: no caching
cache: cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.
For example: dependency files for Linux Kernel - src/sonic-linux-kernel,
SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST)
SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))
DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
$(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)
$(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA
$(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS)
$(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format
FORMAT:
<module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
Eg:
linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz
< 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
< 24-byte MOD SHA value > - the SHA value is derived from either of the following.
GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule
GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.
$(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
$(call dpkg_depend,$(DEBS_PATH)/%.dep )
$(HEADER)
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
.....
# Rules for Generating the target DEB file.
.....
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
$(FOOTER)
The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.
Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
It also updates the target-specific variable to indicate the cache is loaded or not.
The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it
The caching functionality is verified by enabling it in Linux kernel submodule.
It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)
DOCUMENT PR:
https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
|
|
|
# * SONIC_DPKG_CACHE_METHOD: Specifying method of obtaining the Debian packages from cache: none or cache
|
|
|
|
|
# * SONIC_DPKG_CACHE_SOURCE: Debian package cache location when cache enabled for debian packages
|
2020-03-23 11:45:43 -05:00
|
|
|
# * BUILD_LOG_TIMESTAMP: Set timestamp in the build log (simple/none)
|
2018-08-16 17:54:31 -05:00
|
|
|
#
|
|
|
|
|
###############################################################################
|
|
|
|
|
|
|
|
|
|
SHELL = /bin/bash
|
|
|
|
|
|
|
|
|
|
USER := $(shell id -un)
|
|
|
|
|
PWD := $(shell pwd)
|
2020-12-30 06:58:20 -06:00
|
|
|
USER_LC := $(shell echo $(USER) | tr A-Z a-z)
|
2018-08-16 17:54:31 -05:00
|
|
|
|
2020-08-01 23:59:06 -05:00
|
|
|
comma := ,
|
|
|
|
|
|
2018-08-16 17:54:31 -05:00
|
|
|
ifeq ($(USER), root)
|
|
|
|
|
$(error Add your user account to docker group and use your user account to make. root or sudo are not supported!)
|
|
|
|
|
endif
|
|
|
|
|
|
2019-07-26 15:13:33 -05:00
|
|
|
# Check for j2cli availability
|
|
|
|
|
J2_VER := $(shell j2 --version 2>&1 | grep j2cli | awk '{printf $$2}')
|
|
|
|
|
ifeq ($(J2_VER),)
|
2020-03-20 01:26:45 -05:00
|
|
|
$(error Please install j2cli (sudo pip install j2cli))
|
2019-07-26 15:13:33 -05:00
|
|
|
endif
|
|
|
|
|
|
2019-07-13 14:43:45 -05:00
|
|
|
# Check for minimum Docker version on build host
|
|
|
|
|
# Note: Using the greater of CE (17.05.0) and EE (17.06.1) versions that support ARG before FROM
|
|
|
|
|
docker_min := 17.06.1
|
|
|
|
|
docker_min_ver := $(shell echo "$(docker_min)" | awk -F. '{printf("%d%03d%03d\n",$$1,$$2,$$3);}' 2>/dev/null)
|
2019-07-23 09:01:21 -05:00
|
|
|
docker_ver := $(shell docker info 2>/dev/null | grep -i "server version" | rev | cut -d' ' -f1 | rev | awk -F. '{printf("%d%03d%03d\n",$$1,$$2,$$3);}' 2>/dev/null)
|
2019-07-31 17:26:00 -05:00
|
|
|
docker_is_valid := $(shell if [[ "$(docker_ver)" -lt $(docker_min_ver) ]] ; then echo "0"; else echo "1"; fi)
|
2019-07-13 14:43:45 -05:00
|
|
|
ifeq (0,$(docker_is_valid))
|
|
|
|
|
$(error SONiC requires Docker version $(docker_min) or later)
|
|
|
|
|
endif
|
|
|
|
|
|
2018-08-16 17:54:31 -05:00
|
|
|
# Remove lock file in case previous run was forcefully stopped
|
|
|
|
|
$(shell rm -f .screen)
|
|
|
|
|
|
|
|
|
|
MAKEFLAGS += -B
|
|
|
|
|
|
2019-07-26 00:06:41 -05:00
|
|
|
CONFIGURED_ARCH := $(shell [ -f .arch ] && cat .arch || echo $(PLATFORM_ARCH))
|
2019-07-26 15:13:33 -05:00
|
|
|
ifeq ($(CONFIGURED_ARCH),)
|
|
|
|
|
override CONFIGURED_ARCH = amd64
|
|
|
|
|
endif
|
2019-07-26 00:06:41 -05:00
|
|
|
ifeq ($(PLATFORM_ARCH),)
|
|
|
|
|
override PLATFORM_ARCH = $(CONFIGURED_ARCH)
|
|
|
|
|
endif
|
|
|
|
|
|
2020-02-02 14:03:34 -06:00
|
|
|
ifeq ($(BLDENV), buster)
|
|
|
|
|
SLAVE_DIR = sonic-slave-buster
|
|
|
|
|
else ifeq ($(BLDENV), stretch)
|
2018-08-16 17:54:31 -05:00
|
|
|
SLAVE_DIR = sonic-slave-stretch
|
|
|
|
|
else
|
2019-11-21 18:14:32 -06:00
|
|
|
SLAVE_DIR = sonic-slave-jessie
|
2018-08-16 17:54:31 -05:00
|
|
|
endif
|
2020-02-02 14:03:34 -06:00
|
|
|
|
2020-12-21 01:31:10 -06:00
|
|
|
include rules/config
|
|
|
|
|
|
2020-12-26 12:30:51 -06:00
|
|
|
ifeq ($(ENABLE_DOCKER_BASE_PULL),)
|
|
|
|
|
override ENABLE_DOCKER_BASE_PULL = n
|
|
|
|
|
endif
|
|
|
|
|
|
2020-12-25 04:21:57 -06:00
|
|
|
ifeq ($(CONFIGURED_ARCH),amd64)
|
2019-11-21 18:14:32 -06:00
|
|
|
SLAVE_BASE_IMAGE = $(SLAVE_DIR)
|
2020-12-25 04:21:57 -06:00
|
|
|
else
|
|
|
|
|
SLAVE_BASE_IMAGE = $(SLAVE_DIR)-$(CONFIGURED_ARCH)
|
|
|
|
|
endif
|
2020-12-30 06:58:20 -06:00
|
|
|
SLAVE_IMAGE = $(SLAVE_BASE_IMAGE)-$(USER_LC)
|
2018-08-16 17:54:31 -05:00
|
|
|
|
2020-12-21 01:31:10 -06:00
|
|
|
# Generate the version control build info
|
|
|
|
|
$(shell SONIC_VERSION_CONTROL_COMPONENTS=$(SONIC_VERSION_CONTROL_COMPONENTS) \
|
|
|
|
|
TRUSTED_GPG_URLS=$(TRUSTED_GPG_URLS) PACKAGE_URL_PREFIX=$(PACKAGE_URL_PREFIX) \
|
|
|
|
|
scripts/generate_buildinfo_config.sh)
|
|
|
|
|
|
|
|
|
|
# Generate the slave Dockerfile, and prepare build info for it
|
|
|
|
|
$(shell CONFIGURED_ARCH=$(CONFIGURED_ARCH) j2 $(SLAVE_DIR)/Dockerfile.j2 > $(SLAVE_DIR)/Dockerfile)
|
2020-12-25 04:21:57 -06:00
|
|
|
$(shell CONFIGURED_ARCH=$(CONFIGURED_ARCH) j2 $(SLAVE_DIR)/Dockerfile.user.j2 > $(SLAVE_DIR)/Dockerfile.user)
|
2020-12-21 01:31:10 -06:00
|
|
|
$(shell BUILD_SLAVE=y scripts/prepare_docker_buildinfo.sh $(SLAVE_BASE_IMAGE) $(SLAVE_DIR)/Dockerfile $(CONFIGURED_ARCH) "" $(BLDENV))
|
|
|
|
|
|
|
|
|
|
# Add the versions in the tag, if the version change, need to rebuild the slave
|
2021-02-04 03:01:06 -06:00
|
|
|
SLAVE_BASE_TAG = $(shell cat $(SLAVE_DIR)/Dockerfile $(SLAVE_DIR)/buildinfo/versions/versions-* src/sonic-build-hooks/hooks/* | sha1sum | awk '{print substr($$1,0,11);}')
|
2020-12-21 01:31:10 -06:00
|
|
|
SLAVE_TAG = $(shell cat $(SLAVE_DIR)/Dockerfile.user $(SLAVE_DIR)/Dockerfile $(SLAVE_DIR)/buildinfo/versions/versions-* | sha1sum | awk '{print substr($$1,0,11);}')
|
|
|
|
|
|
2019-06-20 11:20:34 -05:00
|
|
|
OVERLAY_MODULE_CHECK := \
|
|
|
|
|
lsmod | grep -q "^overlay " &>/dev/null || \
|
|
|
|
|
zgrep -q 'CONFIG_OVERLAY_FS=y' /proc/config.gz &>/dev/null || \
|
|
|
|
|
grep -q 'CONFIG_OVERLAY_FS=y' /boot/config-$(shell uname -r) &>/dev/null || \
|
|
|
|
|
(echo "ERROR: Module 'overlay' not loaded. Try running 'sudo modprobe overlay'."; exit 1)
|
2018-09-05 00:32:46 -05:00
|
|
|
|
2018-10-04 23:20:01 -05:00
|
|
|
BUILD_TIMESTAMP := $(shell date +%Y%m%d\.%H%M%S)
|
|
|
|
|
|
2018-11-05 12:44:06 -06:00
|
|
|
ifeq ($(DOCKER_BUILDER_MOUNT),)
|
|
|
|
|
override DOCKER_BUILDER_MOUNT := "$(PWD):/sonic"
|
|
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
ifeq ($(DOCKER_BUILDER_WORKDIR),)
|
|
|
|
|
override DOCKER_BUILDER_WORKDIR := "/sonic"
|
|
|
|
|
endif
|
|
|
|
|
|
2020-07-29 20:27:44 -05:00
|
|
|
DOCKER_RUN := docker run --rm=true --privileged --init \
|
2018-11-05 12:44:06 -06:00
|
|
|
-v $(DOCKER_BUILDER_MOUNT) \
|
|
|
|
|
-w $(DOCKER_BUILDER_WORKDIR) \
|
2018-08-16 17:54:31 -05:00
|
|
|
-e "http_proxy=$(http_proxy)" \
|
|
|
|
|
-e "https_proxy=$(https_proxy)" \
|
2020-12-30 06:57:44 -06:00
|
|
|
-i$(shell { if [ -t 0 ]; then echo t; fi }) \
|
2020-09-10 03:21:47 -05:00
|
|
|
$(SONIC_BUILDER_EXTRA_CMDLINE)
|
2018-08-16 17:54:31 -05:00
|
|
|
|
2020-08-01 23:59:06 -05:00
|
|
|
ifneq ($(DOCKER_BUILDER_USER_MOUNT),)
|
|
|
|
|
DOCKER_RUN += $(foreach mount,$(subst $(comma), ,$(DOCKER_BUILDER_USER_MOUNT)), $(addprefix -v , $(mount)))
|
|
|
|
|
endif
|
|
|
|
|
|
2020-09-10 03:21:47 -05:00
|
|
|
ifdef SONIC_BUILD_QUIETER
|
|
|
|
|
DOCKER_RUN += -e "SONIC_BUILD_QUIETER=$(SONIC_BUILD_QUIETER)"
|
|
|
|
|
endif
|
|
|
|
|
|
[build]: support for DPKG local caching (#4117)
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.
The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.
This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.
- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=
where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
none: no caching
cache: cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.
For example: dependency files for Linux Kernel - src/sonic-linux-kernel,
SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST)
SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))
DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
$(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)
$(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA
$(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS)
$(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format
FORMAT:
<module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
Eg:
linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz
< 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
< 24-byte MOD SHA value > - the SHA value is derived from either of the following.
GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule
GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.
$(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
$(call dpkg_depend,$(DEBS_PATH)/%.dep )
$(HEADER)
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
.....
# Rules for Generating the target DEB file.
.....
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
$(FOOTER)
The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.
Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
It also updates the target-specific variable to indicate the cache is loaded or not.
The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it
The caching functionality is verified by enabling it in Linux kernel submodule.
It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)
DOCUMENT PR:
https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
|
|
|
ifneq ($(SONIC_DPKG_CACHE_SOURCE),)
|
|
|
|
|
DOCKER_RUN += -v "$(SONIC_DPKG_CACHE_SOURCE):/dpkg_cache:rw"
|
|
|
|
|
endif
|
|
|
|
|
|
2018-11-01 20:00:00 -05:00
|
|
|
ifeq ($(SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD), y)
|
|
|
|
|
DOCKER_RUN += -v /var/run/docker.sock:/var/run/docker.sock
|
|
|
|
|
endif
|
2020-12-25 02:10:08 -06:00
|
|
|
|
2019-07-26 00:06:41 -05:00
|
|
|
ifneq (,$(filter $(CONFIGURED_ARCH), armhf arm64))
|
2020-12-25 02:10:08 -06:00
|
|
|
ifeq ($(DOCKER_DATA_ROOT_FOR_MULTIARCH),)
|
|
|
|
|
DOCKER_DATA_ROOT_FOR_MULTIARCH := /var/lib/march/docker
|
|
|
|
|
endif
|
2019-07-26 00:06:41 -05:00
|
|
|
# Multiarch docker cannot start dockerd service due to iptables cannot run over different arch kernel
|
|
|
|
|
SONIC_SERVICE_DOCKERD_FOR_MULTIARCH=y
|
|
|
|
|
SONIC_NATIVE_DOCKERD_FOR_MUTLIARCH := dockerd --experimental=true --storage-driver=vfs \
|
2020-12-25 02:10:08 -06:00
|
|
|
--data-root=$(DOCKER_DATA_ROOT_FOR_MULTIARCH) --exec-root=/var/run/march/docker/ \
|
|
|
|
|
-H unix:///var/run/march/docker.sock -p /var/run/march/docker.pid
|
2019-07-26 00:06:41 -05:00
|
|
|
DOCKER_RUN += -v /var/run/march/docker.sock:/var/run/docker.sock
|
|
|
|
|
DOCKER_RUN += -v /var/run/march/docker.pid:/var/run/docker.pid
|
|
|
|
|
DOCKER_RUN += -v /var/run/march/docker:/var/run/docker
|
2020-12-25 02:10:08 -06:00
|
|
|
DOCKER_RUN += -v $(DOCKER_DATA_ROOT_FOR_MULTIARCH):/var/lib/docker
|
2019-07-26 00:06:41 -05:00
|
|
|
SONIC_USERFACL_DOCKERD_FOR_MUTLIARCH := setfacl -m user:$(USER):rw /var/run/march/docker.sock
|
|
|
|
|
|
|
|
|
|
#Override Native config to prevent docker service
|
|
|
|
|
SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y
|
|
|
|
|
|
|
|
|
|
DOCKER_MULTIARCH_CHECK := docker inspect --type image multiarch/qemu-user-static:register &> /dev/null || (echo "multiarch docker not found ..."; docker run --rm --privileged multiarch/qemu-user-static:register --reset --credential yes)
|
|
|
|
|
|
2019-10-24 15:50:45 -05:00
|
|
|
DOCKER_SERVICE_SAFE_KILLER := (MARCH_PID=`ps -eo pid,cmd | grep "[0-9] dockerd.*march" | awk '{print $$1}'`; echo "Killing march docker $$MARCH_PID"; [ -z "$$MARCH_PID" ] || sudo kill -9 "$$MARCH_PID";)
|
|
|
|
|
DOCKER_SERVICE_MULTIARCH_CHECK := ($(DOCKER_SERVICE_SAFE_KILLER); sudo rm -fr /var/run/march/; (echo "Starting docker march service..."; sudo $(SONIC_NATIVE_DOCKERD_FOR_MUTLIARCH) &) &>/dev/null ; sleep 2; sudo $(SONIC_USERFACL_DOCKERD_FOR_MUTLIARCH);)
|
2019-07-26 00:06:41 -05:00
|
|
|
|
2020-01-23 18:50:17 -06:00
|
|
|
# Docker service to load the compiled dockers-*.gz
|
|
|
|
|
# docker 19.0 version above has path/length restriction, so replaced it with soft link in /tmp/
|
|
|
|
|
# Also dockerd does mkdir on the provided softlink, so used two level path "d/d"
|
|
|
|
|
D_ROOT=/tmp/d/d
|
|
|
|
|
SONIC_NATIVE_DOCKERD_FOR_DOCKERFS := rm -fr $(PWD)/dockerfs; mkdir -p $(PWD)/dockerfs; sudo rm -fr /tmp/d; mkdir -p /tmp/d; ln -s -f $(PWD)/dockerfs $(D_ROOT); \
|
|
|
|
|
sudo dockerd --storage-driver=overlay2 --iptables=false \
|
|
|
|
|
--data-root $(D_ROOT)/var/lib/docker/ --exec-root=$(D_ROOT)/var/run/docker/ \
|
|
|
|
|
-H unix://$(D_ROOT)/var/run/docker.sock -p $(D_ROOT)/var/run/docker.pid &
|
|
|
|
|
SONIC_USERFACL_DOCKERD_FOR_DOCKERFS := setfacl -m user:$(USER):rw $(D_ROOT)/var/run/docker.sock
|
|
|
|
|
DOCKER_SERVICE_DOCKERFS_CHECK := (sudo docker -H unix://$(D_ROOT)/var/run/docker.sock info &> /dev/null && sudo kill -9 `sudo cat $(D_ROOT)/var/run/docker.pid` && false) || (echo "Starting docker build service..."; (sudo $(SONIC_NATIVE_DOCKERD_FOR_DOCKERFS) ) &> /tmp/dockerfs.log ; sleep 1; sudo $(SONIC_USERFACL_DOCKERD_FOR_DOCKERFS);)
|
2019-07-26 00:06:41 -05:00
|
|
|
|
|
|
|
|
endif
|
2018-11-01 20:00:00 -05:00
|
|
|
|
2018-08-16 17:54:31 -05:00
|
|
|
DOCKER_BASE_BUILD = docker build --no-cache \
|
2019-07-13 14:43:45 -05:00
|
|
|
-t $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) \
|
2018-08-16 17:54:31 -05:00
|
|
|
--build-arg http_proxy=$(http_proxy) \
|
|
|
|
|
--build-arg https_proxy=$(https_proxy) \
|
2019-07-13 14:43:45 -05:00
|
|
|
$(SLAVE_DIR)
|
2018-08-16 17:54:31 -05:00
|
|
|
|
2020-12-26 12:30:51 -06:00
|
|
|
DOCKER_BASE_PULL = docker pull \
|
|
|
|
|
$(REGISTRY_SERVER):$(REGISTRY_PORT)/$(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG)
|
|
|
|
|
|
2018-08-16 17:54:31 -05:00
|
|
|
DOCKER_BUILD = docker build --no-cache \
|
|
|
|
|
--build-arg user=$(USER) \
|
|
|
|
|
--build-arg uid=$(shell id -u) \
|
|
|
|
|
--build-arg guid=$(shell id -g) \
|
|
|
|
|
--build-arg hostname=$(shell echo $$HOSTNAME) \
|
2019-07-13 14:43:45 -05:00
|
|
|
--build-arg slave_base_tag_ref=$(SLAVE_BASE_TAG) \
|
|
|
|
|
-t $(SLAVE_IMAGE):$(SLAVE_TAG) \
|
2018-08-16 17:54:31 -05:00
|
|
|
-f $(SLAVE_DIR)/Dockerfile.user \
|
2019-07-13 14:43:45 -05:00
|
|
|
$(SLAVE_DIR)
|
2018-08-16 17:54:31 -05:00
|
|
|
|
|
|
|
|
SONIC_BUILD_INSTRUCTION := make \
|
|
|
|
|
-f slave.mk \
|
|
|
|
|
PLATFORM=$(PLATFORM) \
|
2019-07-26 00:06:41 -05:00
|
|
|
PLATFORM_ARCH=$(PLATFORM_ARCH) \
|
2018-08-16 17:54:31 -05:00
|
|
|
BUILD_NUMBER=$(BUILD_NUMBER) \
|
2018-10-04 23:20:01 -05:00
|
|
|
BUILD_TIMESTAMP=$(BUILD_TIMESTAMP) \
|
2020-09-16 12:47:26 -05:00
|
|
|
SONIC_IMAGE_VERSION=$(SONIC_IMAGE_VERSION) \
|
2018-08-16 17:54:31 -05:00
|
|
|
ENABLE_DHCP_GRAPH_SERVICE=$(ENABLE_DHCP_GRAPH_SERVICE) \
|
2019-12-04 06:50:56 -06:00
|
|
|
ENABLE_ZTP=$(ENABLE_ZTP) \
|
2018-08-16 17:54:31 -05:00
|
|
|
SHUTDOWN_BGP_ON_START=$(SHUTDOWN_BGP_ON_START) \
|
2020-08-05 15:23:12 -05:00
|
|
|
INCLUDE_KUBERNETES=$(INCLUDE_KUBERNETES) \
|
2020-04-13 10:41:18 -05:00
|
|
|
KUBERNETES_VERSION=$(KUBERNETES_VERSION) \
|
2020-07-24 12:14:24 -05:00
|
|
|
KUBERNETES_CNI_VERSION=$(KUBERNETES_CNI_VERSION) \
|
2020-04-13 10:41:18 -05:00
|
|
|
K8s_GCR_IO_PAUSE_VERSION=$(K8s_GCR_IO_PAUSE_VERSION) \
|
2018-08-16 17:54:31 -05:00
|
|
|
SONIC_ENABLE_PFCWD_ON_START=$(ENABLE_PFCWD_ON_START) \
|
2018-10-15 15:49:35 -05:00
|
|
|
SONIC_ENABLE_SYNCD_RPC=$(ENABLE_SYNCD_RPC) \
|
2018-10-21 19:20:27 -05:00
|
|
|
SONIC_INSTALL_DEBUG_TOOLS=$(INSTALL_DEBUG_TOOLS) \
|
[build]: support for DPKG local caching (#4117)
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.
The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.
This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.
- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=
where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
none: no caching
cache: cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.
For example: dependency files for Linux Kernel - src/sonic-linux-kernel,
SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST)
SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))
DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
$(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)
$(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA
$(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS)
$(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format
FORMAT:
<module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
Eg:
linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz
< 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
< 24-byte MOD SHA value > - the SHA value is derived from either of the following.
GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule
GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.
$(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
$(call dpkg_depend,$(DEBS_PATH)/%.dep )
$(HEADER)
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
.....
# Rules for Generating the target DEB file.
.....
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
$(FOOTER)
The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.
Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
It also updates the target-specific variable to indicate the cache is loaded or not.
The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it
The caching functionality is verified by enabling it in Linux kernel submodule.
It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)
DOCUMENT PR:
https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
|
|
|
MDEBUG=$(MDEBUG) \
|
2018-08-16 17:54:31 -05:00
|
|
|
PASSWORD=$(PASSWORD) \
|
|
|
|
|
USERNAME=$(USERNAME) \
|
|
|
|
|
SONIC_BUILD_JOBS=$(SONIC_BUILD_JOBS) \
|
2019-06-20 11:19:33 -05:00
|
|
|
SONIC_USE_DOCKER_BUILDKIT=$(SONIC_USE_DOCKER_BUILDKIT) \
|
2019-03-29 17:25:17 -05:00
|
|
|
VS_PREPARE_MEM=$(VS_PREPARE_MEM) \
|
2018-08-16 17:54:31 -05:00
|
|
|
KERNEL_PROCURE_METHOD=$(KERNEL_PROCURE_METHOD) \
|
[build]: support for DPKG local caching (#4117)
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.
The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.
This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.
- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=
where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
none: no caching
cache: cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.
For example: dependency files for Linux Kernel - src/sonic-linux-kernel,
SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST)
SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))
DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
$(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)
$(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA
$(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS)
$(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format
FORMAT:
<module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
Eg:
linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz
< 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
< 24-byte MOD SHA value > - the SHA value is derived from either of the following.
GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule
GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.
$(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
$(call dpkg_depend,$(DEBS_PATH)/%.dep )
$(HEADER)
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
.....
# Rules for Generating the target DEB file.
.....
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
$(FOOTER)
The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.
Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
It also updates the target-specific variable to indicate the cache is loaded or not.
The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it
The caching functionality is verified by enabling it in Linux kernel submodule.
It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)
DOCUMENT PR:
https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
|
|
|
SONIC_DPKG_CACHE_METHOD=$(SONIC_DPKG_CACHE_METHOD) \
|
|
|
|
|
SONIC_DPKG_CACHE_SOURCE=$(SONIC_DPKG_CACHE_SOURCE) \
|
2018-08-16 17:54:31 -05:00
|
|
|
HTTP_PROXY=$(http_proxy) \
|
|
|
|
|
HTTPS_PROXY=$(https_proxy) \
|
2020-08-05 15:23:12 -05:00
|
|
|
SONIC_INCLUDE_SYSTEM_TELEMETRY=$(INCLUDE_SYSTEM_TELEMETRY) \
|
|
|
|
|
SONIC_INCLUDE_RESTAPI=$(INCLUDE_RESTAPI) \
|
2020-06-25 10:04:43 -05:00
|
|
|
TELEMETRY_WRITABLE=$(TELEMETRY_WRITABLE) \
|
2020-02-02 14:04:18 -06:00
|
|
|
EXTRA_DOCKER_TARGETS=$(EXTRA_DOCKER_TARGETS) \
|
2020-03-23 11:45:43 -05:00
|
|
|
BUILD_LOG_TIMESTAMP=$(BUILD_LOG_TIMESTAMP) \
|
2020-06-09 02:25:17 -05:00
|
|
|
SONIC_ENABLE_IMAGE_SIGNATURE=$(ENABLE_IMAGE_SIGNATURE) \
|
2020-12-21 01:31:10 -06:00
|
|
|
SLAVE_DIR=$(SLAVE_DIR) \
|
2019-03-27 20:32:25 -05:00
|
|
|
$(SONIC_OVERRIDE_BUILD_VARS)
|
2018-08-16 17:54:31 -05:00
|
|
|
|
|
|
|
|
.PHONY: sonic-slave-build sonic-slave-bash init reset
|
|
|
|
|
|
|
|
|
|
.DEFAULT_GOAL := all
|
|
|
|
|
|
|
|
|
|
%::
|
2019-07-26 00:06:41 -05:00
|
|
|
ifneq (,$(filter $(CONFIGURED_ARCH), armhf arm64))
|
|
|
|
|
@$(DOCKER_MULTIARCH_CHECK)
|
2019-10-10 17:11:26 -05:00
|
|
|
ifneq ($(BLDENV), )
|
2019-07-26 00:06:41 -05:00
|
|
|
@$(DOCKER_SERVICE_MULTIARCH_CHECK)
|
|
|
|
|
@$(DOCKER_SERVICE_DOCKERFS_CHECK)
|
2019-10-10 17:11:26 -05:00
|
|
|
endif
|
2019-07-26 00:06:41 -05:00
|
|
|
endif
|
2018-09-05 00:32:46 -05:00
|
|
|
@$(OVERLAY_MODULE_CHECK)
|
2020-04-13 12:35:25 -05:00
|
|
|
|
2020-12-21 01:31:10 -06:00
|
|
|
@pushd src/sonic-build-hooks; TRUSTED_GPG_URLS=$(TRUSTED_GPG_URLS) make all; popd
|
2020-12-25 02:10:08 -06:00
|
|
|
@cp src/sonic-build-hooks/buildinfo/sonic-build-hooks* $(SLAVE_DIR)/buildinfo
|
2018-08-16 17:54:31 -05:00
|
|
|
@docker inspect --type image $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) &> /dev/null || \
|
2020-12-26 12:30:51 -06:00
|
|
|
{ [ $(ENABLE_DOCKER_BASE_PULL) == y ] && { echo Image $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) not found. Pulling...; } && \
|
|
|
|
|
$(DOCKER_BASE_PULL) && \
|
2020-12-31 02:46:21 -06:00
|
|
|
{ docker tag $(REGISTRY_SERVER):$(REGISTRY_PORT)/$(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) && \
|
2020-12-26 12:30:51 -06:00
|
|
|
scripts/collect_docker_version_files.sh $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) target ; } } || \
|
2018-08-16 17:54:31 -05:00
|
|
|
{ echo Image $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) not found. Building... ; \
|
2020-12-26 12:30:51 -06:00
|
|
|
$(DOCKER_BASE_BUILD) ; \
|
|
|
|
|
scripts/collect_docker_version_files.sh $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) target ; }
|
2018-08-16 17:54:31 -05:00
|
|
|
@docker inspect --type image $(SLAVE_IMAGE):$(SLAVE_TAG) &> /dev/null || \
|
|
|
|
|
{ echo Image $(SLAVE_IMAGE):$(SLAVE_TAG) not found. Building... ; \
|
|
|
|
|
$(DOCKER_BUILD) ; }
|
|
|
|
|
ifeq "$(KEEP_SLAVE_ON)" "yes"
|
|
|
|
|
ifdef SOURCE_FOLDER
|
2020-12-21 01:31:10 -06:00
|
|
|
@$(DOCKER_RUN) -v $(SOURCE_FOLDER):/var/$(USER)/src $(SLAVE_IMAGE):$(SLAVE_TAG) bash -c "$(SONIC_BUILD_INSTRUCTION) $@; scripts/collect_build_version_files.sh \$$?; /bin/bash"
|
2018-08-16 17:54:31 -05:00
|
|
|
else
|
2020-12-21 01:31:10 -06:00
|
|
|
@$(DOCKER_RUN) $(SLAVE_IMAGE):$(SLAVE_TAG) bash -c "$(SONIC_BUILD_INSTRUCTION) $@; scripts/collect_build_version_files.sh \$$?; /bin/bash"
|
2018-08-16 17:54:31 -05:00
|
|
|
endif
|
|
|
|
|
else
|
2020-12-21 01:31:10 -06:00
|
|
|
@$(DOCKER_RUN) $(SLAVE_IMAGE):$(SLAVE_TAG) bash -c "$(SONIC_BUILD_INSTRUCTION) $@; scripts/collect_build_version_files.sh \$$?"
|
2018-08-16 17:54:31 -05:00
|
|
|
endif
|
|
|
|
|
|
2020-12-21 01:31:10 -06:00
|
|
|
sonic-build-hooks:
|
|
|
|
|
@pushd src/sonic-build-hooks; TRUSTED_GPG_URLS=$(TRUSTED_GPG_URLS) make all; popd
|
2020-12-25 02:10:08 -06:00
|
|
|
@cp src/sonic-build-hooks/buildinfo/sonic-build-hooks* $(SLAVE_DIR)/buildinfo
|
2020-12-21 01:31:10 -06:00
|
|
|
|
|
|
|
|
sonic-slave-base-build : sonic-build-hooks
|
2018-09-05 00:32:46 -05:00
|
|
|
@$(OVERLAY_MODULE_CHECK)
|
2020-12-25 13:29:44 -06:00
|
|
|
@echo Checking sonic-slave-base image: $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG)
|
2018-08-16 17:54:31 -05:00
|
|
|
@docker inspect --type image $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) &> /dev/null || \
|
2020-12-26 12:30:51 -06:00
|
|
|
{ [ $(ENABLE_DOCKER_BASE_PULL) == y ] && { echo Image $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) not found. Pulling...; } && \
|
|
|
|
|
$(DOCKER_BASE_PULL) && \
|
2020-12-31 02:46:21 -06:00
|
|
|
{ docker tag $(REGISTRY_SERVER):$(REGISTRY_PORT)/$(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) && \
|
2020-12-26 12:30:51 -06:00
|
|
|
scripts/collect_docker_version_files.sh $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) target ; } } || \
|
2018-08-16 17:54:31 -05:00
|
|
|
{ echo Image $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) not found. Building... ; \
|
2020-12-26 12:30:51 -06:00
|
|
|
$(DOCKER_BASE_BUILD) ; \
|
|
|
|
|
scripts/collect_docker_version_files.sh $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) target ; }
|
2020-12-03 18:03:43 -06:00
|
|
|
|
|
|
|
|
sonic-slave-build : sonic-slave-base-build
|
2020-12-25 13:29:44 -06:00
|
|
|
@echo Checking sonic-slave image: $(SLAVE_IMAGE):$(SLAVE_TAG)
|
2018-08-16 17:54:31 -05:00
|
|
|
@docker inspect --type image $(SLAVE_IMAGE):$(SLAVE_TAG) &> /dev/null || \
|
|
|
|
|
{ echo Image $(SLAVE_IMAGE):$(SLAVE_TAG) not found. Building... ; \
|
|
|
|
|
$(DOCKER_BUILD) ; }
|
2020-12-03 18:03:43 -06:00
|
|
|
|
|
|
|
|
sonic-slave-bash : sonic-slave-build
|
2018-08-16 17:54:31 -05:00
|
|
|
@$(DOCKER_RUN) -t $(SLAVE_IMAGE):$(SLAVE_TAG) bash
|
|
|
|
|
|
2020-12-03 18:03:43 -06:00
|
|
|
sonic-slave-run : sonic-slave-build
|
2020-04-30 00:58:03 -05:00
|
|
|
@$(DOCKER_RUN) $(SLAVE_IMAGE):$(SLAVE_TAG) bash -c "$(SONIC_RUN_CMDS)"
|
2020-04-25 12:52:38 -05:00
|
|
|
|
2018-08-16 17:54:31 -05:00
|
|
|
showtag:
|
|
|
|
|
@echo $(SLAVE_IMAGE):$(SLAVE_TAG)
|
|
|
|
|
@echo $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG)
|
|
|
|
|
|
|
|
|
|
init :
|
|
|
|
|
@git submodule update --init --recursive
|
|
|
|
|
@git submodule foreach --recursive '[ -f .git ] && echo "gitdir: $$(realpath --relative-to=. $$(cut -d" " -f2 .git))" > .git'
|
|
|
|
|
|
2019-08-28 11:33:03 -05:00
|
|
|
.ONESHELL : reset
|
2018-08-16 17:54:31 -05:00
|
|
|
reset :
|
|
|
|
|
@echo && echo -n "Warning! All local changes will be lost. Proceed? [y/N]: "
|
2019-08-28 11:33:03 -05:00
|
|
|
@read ans && (
|
|
|
|
|
if [ $$ans == y ]; then
|
|
|
|
|
echo "Resetting local repository. Please wait...";
|
|
|
|
|
$(DOCKER_RUN) $(SLAVE_IMAGE):$(SLAVE_TAG) sudo rm -rf fsroot;
|
|
|
|
|
if [[ "$(CONFIGURED_ARCH)" == "armhf" || "$(CONFIGURED_ARCH)" == "arm64" ]]; then
|
|
|
|
|
echo "Stopping march $(CONFIGURED_ARCH) docker"
|
|
|
|
|
sudo kill -9 `sudo cat /var/run/march/docker.pid` || true
|
|
|
|
|
sudo rm -f /var/run/march/docker.pid || true
|
|
|
|
|
fi
|
|
|
|
|
git clean -xfdf;
|
|
|
|
|
git reset --hard;
|
2020-06-01 15:28:03 -05:00
|
|
|
git submodule foreach --recursive 'git clean -xfdf || true';
|
|
|
|
|
git submodule foreach --recursive 'git reset --hard || true';
|
2020-10-09 10:53:05 -05:00
|
|
|
git submodule foreach --recursive 'git remote update || true';
|
2019-08-28 11:33:03 -05:00
|
|
|
git submodule update --init --recursive;
|
|
|
|
|
echo "Reset complete!";
|
|
|
|
|
else
|
|
|
|
|
echo "Reset aborted";
|
|
|
|
|
fi )
|
