This change allows the recursive `git clean` and `git reset` commands to continue even if they encounter an error in one of the submodules. Previously, if an error was encountered, the operation would terminate with a message similar to the following:
Stopping at 'src/sonic-mgmt-framework'; script returned non-zero status.
- build SONIC_STRETCH_DOCKERS in sonic-slave-stretch docker
- build image related module in sonic-slave-buster docker.
This includes all kernels modules and some packages
Signed-off-by: Guohan Lu <lguohan@gmail.com>
- add sonic-slave-buster docker
- remove debhelper from buster backports
debhelper in buster is enough to build smartmontools 6.6
Signed-off-by: Guohan Lu <lguohan@gmail.com>
* Install kubernetes worker node packages, if enabled.
* Minor updates
* Added some comments
* Updates per review comments.
Built a private image to test to work fine.
* Remove the removed file.
* Update per comments
Make a fix, as kubeadm no demands a higher version of kubelet & kubectl.
As kubeadm auto install kubectl & kubelet, removing explicit install is an easier/robust fix.
* Changes per review comments.
* Updates per comments.
1) Dropped helper & pod scripts
2) Made install verbose
* Drop creation of pods subdir, as this PR does not use them.
* From comments to 'n' per review comments.
* 1) kubeadm.conf is created as part of kubeadm package install. Hence dropped explicit copy.
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.
The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.
This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.
- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=
where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
none: no caching
cache: cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.
For example: dependency files for Linux Kernel - src/sonic-linux-kernel,
SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST)
SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))
DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
$(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)
$(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA
$(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS)
$(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
$(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format
FORMAT:
<module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
Eg:
linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz
< 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
< 24-byte MOD SHA value > - the SHA value is derived from either of the following.
GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule
GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.
$(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
$(call dpkg_depend,$(DEBS_PATH)/%.dep )
$(HEADER)
# Load the target deb from DPKG cache
$(call LOAD_CACHE,$*,$@)
# Skip building the target if it is already loaded from cache
if [ -z '$($*_CACHE_LOADED)' ] ; then
.....
# Rules for Generating the target DEB file.
.....
# Save the target deb into DPKG cache
$(call SAVE_CACHE,$*,$@)
fi
$(FOOTER)
The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.
Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
It also updates the target-specific variable to indicate the cache is loaded or not.
The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it
The caching functionality is verified by enabling it in Linux kernel submodule.
It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)
DOCUMENT PR:
https://github.com/Azure/SONiC/pull/559
* Build sonic-ztp package
- Add changes in make rules to conditionally include sonic-ztp package
Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
[Makefile] Kill arm march docker service before every run to avoid stale entries
[Platform] Marvell - add dtb deb pkg
Signed-off-by: Antony Rheneus <arheneus@marvell.com>
* Makefile: ARM kernel support from sonic-linux-kernel
* Fix for multiarch build docker spawn
Platform: Install the DTB deb for the platform
Signed-off-by: Antony Rheneus <arheneus@marvell.com>
* [Makefile] make reset target changed to ONESHELL and multiarch docker
stop has been added in reset
Signed-off-by: Antony Rheneus <arheneus@marvell.com>
ARM Architecture support in SONIC
make configure platform=[ASIC_VENDOR_ARCH] PLATFORM_ARCH=[ARM_ARCH]
SONIC_ARCH: default amd64
armhf - arm32bit
arm64 - arm64bit
Signed-off-by: Antony Rheneus <arheneus@marvell.com>
Define slave_base_tag_ref variable in Makefile.work containing
specific base image tag to use, rather than always defaulting
to :latest.
Add an ARG command before FROM statement in Dockerfile.user
for sonic-slave and sonic-slave-stretch. ARG variable defaults
to latest if slave_base_tag_ref not specified in Makefile.work.
The presumption to always refer to the :latest tagged Docker base
image when creating the user image causes problems in a shared
build server environment, where the most recently created base
image (i.e. the current :latest tag) may not be compatible with
the current build. For example, different users working in
different branches may all be sharing the same build server.
Signed-off-by: Greg Paussa greg.paussa@broadcom.com
- What I did
Added a DOCKER_AVOID_BASE_TAG_LATEST build option to rules/config that forces the Docker user image creation to refer to its base image by a specific tag rather than rely on the :latest tag. This is needed in a shared build server environment where builds from different developers and/or different SONiC branches all converge on the same Docker daemon instance running on the build server. The :latest tag is always assigned to the most recent base image built, which might not correspond to the base image needed for a particular build, thus causing various build errors that mostly manifest as missing Debian packages or package version mismatches.
NOTE TO REVIEWERS: This PR relies on Docker support of "ARG before FROM," which was first introduced in Docker version 17.05.1-ce. Although there is no mention of a minimum required Docker version for the build server in the SONiC Building Guide pages, please consider whether it is reasonable to assume that Docker 17.05.1-ce or later must be used for SONiC build hosts before approving this PR.
- How I did it
Added an ARG before the FROM statement at the top of the sonic-slave/Dockerfile.user and sonic-slave-stretch/Dockerfile.user files. The ARG variable defaults to latest, but can be overridden in Makefile.work to reference the SLAVE_BASE_TAG so that it refers to the specific, matching base image for the build. This override is activated by un-commenting the DOCKER_AVOID_BASE_TAG_LATEST = y line in rules/config.
Some kernels are built with overlayfs as a builtin and not a module.
For these the check via lsmod currently fails.
This improvement now checks the kernel configuration for the
CONFIG_OVERLAY_FS entry. Depending on the OS and kernel version the
build configuration can be in multiple places.
* [vm build] force Linux to drop cache before calling kvm
KVM need to allocate 2G memory for this build. The system memory might
be occupied by cache at the moment and doesn't have 2G chunk to give
out. Forcing Kernel to drop cache to boost the chance of getting 2G
memory.
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
* [make] add option to enable/disable VS build memory preparation
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
* [build]: put stretch debian packages under target/debs/stretch/
* in stretch build phase, all debian packages built in that stage are placed under target/debs/stretch directory.
* for python-based debian packages, since they are really the same for jessie and stretch, they are placed under target/python-debs directory.
Signed-off-by: Guohan Lu <gulv@microsoft.com>
This can be used when sonic-buildimage repo is a git submodule of other
repo.
e.g) if sonic-buildimage is a git submodule of "parent" repo and
sonic-buildimage is just under the top directory of "parent" repo, we
can build sonic-buildimage like below
$ cd parent/sonic-buildimage
$ DOCKER_BUILDER_MOUNT=`realpath ../`:`realpath ../` \
DOCKER_BUILDER_WORKDIR=`pwd` make all
Signed-off-by: Wataru Ishida <ishida@nel-america.com>
