sonic-buildimage/platform/broadcom/rules.dep

40 lines
1.7 KiB
Plaintext
Raw Permalink Normal View History

[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
#DPKG FRK
include $(PLATFORM_PATH)/sai-modules.dep
include $(PLATFORM_PATH)/sai.dep
# TODO: re-enable once the following have been updated for bookworm:
# * Nokia
# * Juniper
# * Ragile
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
include $(PLATFORM_PATH)/platform-modules-s6000.dep
include $(PLATFORM_PATH)/platform-modules-dell.dep
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
include $(PLATFORM_PATH)/platform-modules-arista.dep
include $(PLATFORM_PATH)/platform-modules-ingrasys.dep
include $(PLATFORM_PATH)/platform-modules-accton.dep
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
include $(PLATFORM_PATH)/platform-modules-alphanetworks.dep
include $(PLATFORM_PATH)/platform-modules-inventec.dep
[PR:16737, PR:16739] platform-modules: pddf, broadcom/cel: adapt for kernel 6.1 and bookworm (#16954) * sonic-platform-modules-cel: broadcom: adapt for kernel 6.1 and bookworm The i2c_driver->remove API declaration has been updated to return void instead of int, as part of cleanup patches in 6.1. More details can be referred from here: [1]. Update the remove API definition in the modules accordingly and cleanup variables that go unused from the remove API. Update python build commands for bookworm. The packaging based on calling setup.py is deprecated and using build module/pip utility is the recommended method for python packaging/installation. Further details can be referred to from here: [2], [3]. The build module is picky about the package information file, which needs to be either setup.py or pyproject.toml. Additionally, fix formatting inconsistencies in debian/changelog reported by `dh_installchangelogs` during the build. Tested the changes by compiling the changes as below: make sonic-slave-bash NOBUSTER=1 NOBULLSEYE=1 sudo dpkg -i target/debs/bookworm/linux-headers-6.1.0-11-2-*.deb cd platform/broadcom/sonic-platform-modules-cel KVERSION=6.1.0-11-2-amd64 dpkg-buildpackage Also verified the python scripts under the sonic-platform-modules-cel with pyflakes to ensure no new errors are flagged (with exception of unused modules). References: [1] - https://github.com/torvalds/linux/commit/ed5c2f5f [2] - https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.htm [3] - 0b20a4863 (Update Python build commands for Bookworm, 2023-09-07) Signed-off-by: Ramasamy Chandramouli <rachandr@celestica.com> * platform/pddf: i2c: adapt for kernel 6.1 and bookworm * Fixup i2c_driver->remove API due to changes in the function prototype (ref: [1]). * Cleanup `MODULE_SUPPORTED_DEVICE` macros that were cleaned up in the upstream (ref: [2]). * Sanitize python packaging and installation using the `build` module instead of calling the setup.py directly (ref: [3]. [4]). Tested the changes by compiling pddf module as below: make sonic-slave-bash NOBUSTER=1 NOBULLSEYE=1 sudo dpkg -i target/debs/bookworm/linux-headers-6.1.0-11-2-*.deb cd platform/pddf/i2c KVERSION=6.1.0-11-2-amd64 dpkg-buildpackage References: [1] - https://github.com/torvalds/linux/commit/ed5c2f5f [2] - https://github.com/torvalds/linux/commit/6417f031 [2] - https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.htm [3] - 0b20a4863 (Update Python build commands for Bookworm, 2023-09-07) Signed-off-by: Ramasamy Chandramouli <rachandr@celestica.com> * platform/broadcom: include platform-modules-cel in builds With pddf modules patched for 6.1, platform-modules-cel can be compiled and included in the final image. Testing by building sonic-broadcom.bin/sonic-broadcom-dnx.bin. Signed-off-by: Ramasamy Chandramouli <rachandr@celestica.com> * pddf/i2c: revert correct rootdir for pip install The pip install directory has been set to test-pkg1/ for testing the build and incorrectly retained as is. Revert this to the correct path $(PACKAGE_PRE_NAME). Signed-off-by: Ramasamy Chandramouli <rachandr@celestica.com> * platform/broadcom: include pddf/modules-cel in the base package Without this change, the modules were built but not packaged in the final .bin. The final sonic-broadcom.bin has been tested for bootup on Celestica's Silverstone platform. admin@sonic:~$ uname -a Linux sonic 6.1.0-11-2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4 (2023-08-08) x86_64 GNU/Linux admin@sonic:~$ show platform summary Platform: x86_64-cel_silverstone-r0 HwSKU: Silverstone ASIC: broadcom ASIC Count: 1 Serial Number: R4009B2F062504LK200024 Model Number: N/A Hardware Revision: N/A admin@sonic:~$ show version | head SONiC Software Version: SONiC.g0aad6c67c-rachandr SONiC OS Version: 12 Distribution: Debian 12.2 Kernel: 6.1.0-11-2-amd64 Build commit: 0aad6c67c Build date: Thu Oct 26 07:13:47 UTC 2023 Built by: rachandr@AZUHPS14 Platform: x86_64-cel_silverstone-r0 Signed-off-by: Ramasamy Chandramouli <rachandr@celestica.com> --------- Signed-off-by: Ramasamy Chandramouli <rachandr@celestica.com>
2023-09-08 17:59:48 -05:00
include $(PLATFORM_PATH)/platform-modules-cel.dep
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
include $(PLATFORM_PATH)/platform-modules-delta.dep
include $(PLATFORM_PATH)/platform-modules-quanta.dep
#include $(PLATFORM_PATH)/platform-modules-mitac.dep
#include $(PLATFORM_PATH)/platform-modules-juniper.dep
#include $(PLATFORM_PATH)/platform-modules-ragile.dep
include $(PLATFORM_PATH)/platform-modules-ruijie.dep
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
include $(PLATFORM_PATH)/platform-modules-brcm-xlr-gts.dep
include $(PLATFORM_PATH)/docker-syncd-brcm.dep
include $(PLATFORM_PATH)/docker-syncd-brcm-rpc.dep
include $(PLATFORM_PATH)/docker-saiserver-brcm.dep
ifeq ($(INCLUDE_PDE), y)
include $(PLATFORM_PATH)/docker-pde.dep
include $(PLATFORM_PATH)/sonic-pde-tests.dep
endif
[build]: support for DPKG local caching (#4117) DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker. The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package. This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files. - How I did it It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache. SONIC_DPKG_CACHE_METHOD=cache SONIC_DPKG_CACHE_SOURCE= where SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching none: no caching cache: cache from local directory Dependency file tracking: Dependency files are tracked for each target in two levels. 1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc. 2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc. For example: dependency files for Linux Kernel - src/sonic-linux-kernel, SPATH := $($(LINUX_HEADERS_COMMON)_SRC_PATH) DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files)) DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \ $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH) $(LINUX_HEADERS_COMMON)_CACHE_MODE := GIT_CONTENT_SHA $(LINUX_HEADERS_COMMON)_DEP_FLAGS := $(DEP_FLAGS) $(LINUX_HEADERS_COMMON)_DEP_FILES := $(DEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES) $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH) Cache file tracking: The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files. The cache filename is formed with the following format FORMAT: <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz Eg: linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages. < 24-byte MOD SHA value > - the SHA value is derived from either of the following. GIT_COMMIT_SHA - SHA value of the last git commit ID if it is a submodule GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files. Target Specific rules: Caching can be enabled/disabled on a global level and also on the per-target level. $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \ $(call dpkg_depend,$(DEBS_PATH)/%.dep ) $(HEADER) # Load the target deb from DPKG cache $(call LOAD_CACHE,$*,$@) # Skip building the target if it is already loaded from cache if [ -z '$($*_CACHE_LOADED)' ] ; then ..... # Rules for Generating the target DEB file. ..... # Save the target deb into DPKG cache $(call SAVE_CACHE,$*,$@) fi $(FOOTER) The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target. Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents. The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed. It also updates the target-specific variable to indicate the cache is loaded or not. The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock. - How to verify it The caching functionality is verified by enabling it in Linux kernel submodule. It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build. - Description for the changelog The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files. If the module's dependency files are not changed, it restores the module deb files from the cache storage. - Description for the changelog - A picture of a cute animal (not mandatory but encouraged) DOCUMENT PR: https://github.com/Azure/SONiC/pull/559
2020-03-11 22:04:52 -05:00
include $(PLATFORM_PATH)/one-image.dep
include $(PLATFORM_PATH)/raw-image.dep
include $(PLATFORM_PATH)/one-aboot.dep
include $(PLATFORM_PATH)/libsaithrift-dev.dep
include $(PLATFORM_PATH)/docker-syncd-brcm-dnx.dep
include $(PLATFORM_PATH)/docker-syncd-brcm-dnx-rpc.dep
ifeq ($(INCLUDE_GBSYNCD), y)
include $(PLATFORM_PATH)/../components/docker-gbsyncd-credo.dep
include $(PLATFORM_PATH)/../components/docker-gbsyncd-broncos.dep
endif