Commit Graph

291 Commits

Author SHA1 Message Date
xumia
30f935c994 [Build] Fix some of the patches not applied issue (#15660)
Why I did it
Fix some of the patches in .patches folder not applied issue.
The command "quilt applied" only lists the applied patches, if some of the patches have issues, then the patches will not be applied when you run the build command again.

Work item tracking
Microsoft ADO (number only): 24410730
How I did it
Run the command to apply the patches without any conditions.
If failed, check if the failure reason is "series fully applied".
How to verify it
2023-08-07 16:33:01 +08:00
DavidZagury
5acec530bb
Add SECURE_UPGRADE_PROD_TOOL_ARGS flag to make it possible for vendors to pass their own arguments on the prod signing script (#14581)
- Why I did it
Since the prod signing tool is vendor specific, and each vendor may have different arguments they would like to use in the script, we would need a way to inject those arguments to the script.

- How I did it
Add a compilation flag SECURE_UPGRADE_PROD_TOOL_ARGS which vendors can use to inject any flag they would want to the prod signing script.

- How to verify it
Build SONiC using your own prod script
2023-05-16 08:36:13 +03:00
Yevhen Fastiuk
ef2d08b2ef
Enable missed cache for local debs (#14794)
- Why I did it
To be able to cache, and then retrieve cached "copied" debs

- How I did it
Add missed caching and cache retrieval steps

- How to verify it
Build with cache and then clean and rebuild again. The targets added to SONIC_COPY_DEBS should be taken from a cache.

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>
2023-05-03 13:24:42 +03:00
DavidZagury
2d0a12af6d
Fix issue with prod script not found, change the prod signing to work with flags to align to the dev script (#14580)
- Why I did it
Fix issue with signing tool not running due to being call with the path from the host and not the path it is mounted on inside the docker-slave

- How I did it
Modified the path on the SECURE_UPGRADE_PROD_SIGNING_TOOL flag to the path where it is mounted inside the slave docker

- How to verify it
Build SONiC using your own prod script
2023-05-02 09:13:16 +03:00
DavidZagury
90f45d9774
Change SECURE_UPGRADE_DEV_SIGNING_CERT to SECURE_UPGRADE_SIGNING_CERT (#14591)
Depends on https://github.com/sonic-net/sonic-linux-kernel/pull/315

#### Why I did it
The name SECURE_UPGRADE_DEV_SIGNING_CERT is misleading, this flag is relevant to both to dev and prod signing.

#### How I did it
Rename all mentions of name SECURE_UPGRADE_DEV_SIGNING_CERT to SECURE_UPGRADE_SIGNING_CERT - this is also done with PR in sonic-linux-kernel repository

#### How to verify it
Build SONiC using your own prod script
2023-04-24 11:17:51 -07:00
Justin Sherman
737c42d47d
SONIC_ONLINE_DEBS no longer overwrites local build outputs (#14698)
Closes #14697

Why I did it
When using the dpkg cache feature, debians referenced under SONIC_ONLINE_DEBS always get downloaded, even if the expected debian package already exists under target/. The runs contrary to the design of Makefiles (where presence of the output file indicates it is already built).

This is also counter to the behavior of the SONiC build when dpkg cache is not enabled, causing further confusion.

This behavior also causes problems when doing local development, where we may want to modify the local debian files when evaluating which changes to push to the HTTP repository storing them (Artifactory). With the current behavior, our local changes are always overwritten.

Work item tracking
Microsoft ADO (number only):
How I did it
The SONIC_ONLINE_DEBS rule now skips downloading debians if they already exist under target/.

How to verify it
Populate target/ with locally modified debian packages. Perform the build. Ensure the local modifications remain intact, and are not overwritten.
2023-04-20 21:31:40 +08:00
Sudharsan Dhamal Gopalarathnam
2804998766
[config reload]Config Reload Enhancement (#13969)
#### Why I did it
Implementing code changes for https://github.com/sonic-net/SONiC/pull/1203

#### How I did it
Removed the timers and delayed target since the delayed services would start based on event driven approach.
Cleared port table during config reload and cold reboot scenario.
Modified yang model, init_cfg.json to change has_timer to delayed

#### How to verify it
Running regression
2023-04-12 11:20:03 -07:00
xumia
f1fd42558a
Support to add SONiC OS Version in device info (#14601)
Why I did it
Support to add SONiC OS Version in device info.
It will be used to display the version info in the SONiC command "show version". The version is used to do the FIPS certification. We do not do the FIPS certification on a specific release, but on the SONiC OS Version.

SONiC Software Version: SONiC.master-13812.218661-7d94c0c28
SONiC OS Version: 11
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
How I did it
2023-04-12 09:20:08 +08:00
Konstantin Vasin
d7d6445abf
[Build] disable DOCKER_BUILDKIT explicitly (#14405)
Why I did it
Fix #14081
By default DOCKER_BUILDKIT is enabled after docker version 23.0.0
So we need to disable it explicitly if SONIC_USE_DOCKER_BUILDKIT is not set.
Otherwise it will produce larger installable images.

How I did it
set DOCKER_BUILDKIT=0 in slave.mk

How to verify it
2023-04-11 08:06:07 +00:00
xumia
9b769244d5
[Build] Fix the SLAVE_DRI not defined issue in the slave container issue (#14297)
Why I did it
It is to fix the issue #13773

It only has impact on the build triggered manually inside of the slave container. Developers can go to the slave container do a build, it will print a skippable error message complaining the variable not found.

How I did it
Add the default value for variable SLAVE_DRI.

How to verify it
2023-04-06 16:42:59 +08:00
davidpil2002
8098bc4bf5
Add Secure Boot Support (#12692)
- Why I did it
Add Secure Boot support to SONiC OS.
Secure Boot (SB) is a verification mechanism for ensuring that code launched by a computer's UEFI firmware is trusted. It is designed to protect a system against malicious code being loaded and executed early in the boot process before the operating system has been loaded.

- How I did it
Added a signing process to sign the following components:
shim, grub, Linux kernel, and kernel modules when doing the build, and when feature is enabled in build time according to the HLD explanations (the feature is disabled by default).

- How to verify it
There are self-verifications of each boot component when building the image, in addition, there is an existing end-to-end test in sonic-mgmt repo that checks that the boot succeeds when loading a secure system (details below).

How to build a sonic image with secure boot feature: (more description in HLD)

Required to use the following build flags from rules/config:
SECURE_UPGRADE_MODE="dev"
SECURE_UPGRADE_DEV_SIGNING_KEY="/path/to/private/key.pem"
SECURE_UPGRADE_DEV_SIGNING_CERT="/path/to/cert/key.pem"
After setting those flags should build the sonic-buildimage.
Before installing the image, should prepared the setup (switch device) with the follow:
check that the device support UEFI
stored pub keys in UEFI DB

enabled Secure Boot flag in UEFI
How to run a test that verify the Secure Boot flow:
The existing test "test_upgrade_path" under "sonic-mgmt/tests/upgrade_path/test_upgrade_path", is enough to validate proper boot
You need to specify the following arguments:
Base_image_list your_secure_image
Taget_image_list your_second_secure_image
Upgrade_type cold
And run the test, basically the test will install the base image given in the parameter and then upgrade to target image by doing cold reboot and validates all the services are up and working correctly
2023-03-14 14:55:22 +02:00
Andriy Dobush
c1dd94f368
Add California-SB237 feature. Requires to change default user password (#12678)
#### Why I did it
Add support of California-SB237 conformance.
https://github.com/sonic-net/SONiC/tree/master/doc/California-SB237

#### How I did it
Expire user passwords during build

#### How to verify it
Enable build flag and check if default user is prompted for a new password
2023-02-23 15:36:37 -08:00
Liu Shilong
c7d6b8ddbb
[build] Check if patches are applied before applying patches. (#13566)
Why I did it
If make fails, we can't rerun the make process, because existing patches can't apply again.

#13386 missed some change.
2023-02-06 17:08:31 +08:00
Liu Shilong
d55913a679
[build] Check if patches are applied before applying patches (#13386)
Why I did it
If make fails, we can't rerun the make process, because existing patches can't apply again.

How I did it
Check if patches are applied. if yes, don't apply patches again.

How to verify it
2023-01-18 13:35:11 +08:00
xumia
5e4a866e33
[Build] Support Debian snapshot mirror to improve build stability (#13097)
Why I did it
[Build] Support Debian snapshot mirror to improve build stability

It is to enhance the reproducible build, supports the Debian snapshot mirror. It guarantees all the docker images using the same Debian mirror snapshot and fixes the temporary build failure which is caused by remote Debain mirror indexes changed during the build. It is also to fix the version conflict issue caused by no fixed versions of some of the Debian packages.

How I did it
Add a new feature to support the Debian snapshot mirror.

How to verify it
2023-01-13 16:16:35 +08:00
Oleksandr Ivantsiv
127d60f9b8
[build] Adjust teamd and radv features configuration according to the compilation options. (#13139)
- Why I did it
The followup to #12920 PR.
If the feature compilation is disabled its configuration should not be included into init_cfg.json.

- How I did it
Update init_cfg.json.j2 template to include teamd and radv features configuration only if their compilation is enabled.

- How to verify it
The default behavior is preserved. To verify the changes compile the image without overriding INCLUDE_TEAMD and INCLUDE_ROUTER_ADVERTISER options. The generated /etc/sonic/init_cfg.json should remain with no changes. Install the image and verify that both teamd and radv containers are present and running. Verify that feature state returned by show feature status command is enabled.
Change the INCLUDE_TEAMD or INCLUDE_ROUTER_ADVERTISER value to "n". Compile and install the image. Verify that feature configuration is not included in generated /etc/sonic/init_cfg.json file. Verify that show feature status output doesn't include the feature.
2022-12-27 13:55:37 +02:00
Liu Shilong
25d9ebbaf4
[build] Fix unexpected warnings: No names found, cannot describe anything. (#12963)
Why I did it
In PR check pipelines, there are too many duplicated warnings:
fatal: No names found, cannot describe anything.
SONIC_IMAGE_VERSION will not change in one build. We don't need to calculate in every reference. We just need calculate one time, then record it.
In Makefile, '=' will calculate again and again when it is referred.

How I did it
Fix it in Makefile.

How to verify it
Check this PR's check pipeline result.
2022-12-20 16:40:10 +08:00
Konstantin Vasin
bf90b498a4
[Build] use pigz to speed up a build (#12825)
Why I did it

It's possible to speed up some parts of a build using parallel compression/decompression.
This is especially important for build_debian.sh.

How I did it
pigz is a parallel implementation of gzip: https://zlib.net/pigz/
Some programs like docker and mkinitramfs can automatically detect and use it instead of gzip.
For tar we need to select it directly.

To enable this feature you need to set GZ_COMPRESS_PROGRAM=pigz
2022-12-17 14:38:31 -08:00
xumia
14a5ec7914
[Build] Fix the docker image docker-dhcp-relay:latest not found issue (#13048)
Why I did it
It is to fix the broadcom build failure, it is caused by the build image docker-dhcp-relay:latest not found.

2022-12-14T00:09:57.5464893Z [ FAIL LOG START ] [ target/docker-dhcp-relay.gz-load ]
2022-12-14T00:09:57.5466036Z Attempting docker image lock for docker-dhcp-relay load
2022-12-14T00:09:57.5467113Z Obtained docker image lock for docker-dhcp-relay load
2022-12-14T00:09:57.5468206Z Loading docker image target/docker-dhcp-relay.gz
2022-12-14T00:09:57.5469361Z Loaded image: docker-dhcp-relay:internal.65852159-11ad82a07a
2022-12-14T00:09:57.5470686Z Tagging docker image docker-dhcp-relay:latest as docker-dhcp-relay-sonic:latest
2022-12-14T00:09:57.5471997Z Error response from daemon: No such image: docker-dhcp-relay:latest
2022-12-14T00:09:57.5473122Z [  FAIL LOG END  ] [ target/docker-dhcp-relay.gz-load ]
2022-12-14T00:09:57.5539792Z make: *** [slave.mk:1180: target/docker-dhcp-relay.gz-load] Error 1
2022-12-14T00:09:57.5540958Z make: *** Waiting for unfinished jobs....
The image had been built succeeded

2022-12-13T17:01:59.9046935Z [ finished ] [ target/docker-eventd.gz ] 
2022-12-13T17:02:00.4947165Z [ building ] [ target/docker-dhcp-relay.gz ] 
2022-12-13T17:02:00.6688627Z /sonic/dockers/docker-dhcp-relay/cli-plugin-tests /sonic
2022-12-13T17:02:41.1123955Z /sonic
2022-12-13T17:07:04.1786069Z [ finished ] [ target/docker-dhcp-relay.gz ] 
But it was tagged by another value:

Obtained docker image lock for docker-dhcp-relay save
Tagging docker image docker-dhcp-relay-sonic:latest as docker-dhcp-relay:internal.65852159-11ad82a07a
Saving docker image docker-dhcp-relay:internal.65852159-11ad82a07a
Released docker image lock for docker-dhcp-relay save
Removing docker image docker-dhcp-relay-sonic:latest
Untagged: docker-dhcp-relay-sonic:latest
target/docker-dhcp-relay.gz
File /dpkg_cache/docker-dhcp-relay.gz-2ddfa01a109ca69b7621f1a-450bae36026d9dee62646f2.tgz saved in cache 
[ CACHE::SAVED ] /dpkg_cache/docker-dhcp-relay.gz-2ddfa01a109ca69b7621f1a-450bae36026d9dee62646f2.tgz
How I did it
When the feature SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD not enabled, always save as the latest tag, not use the specify version.
The version is dynamic, it is changed when a new commit checked in, but the image of docker-dhcp-relay is not necessary to change.
2022-12-15 21:03:21 +08:00
Oleksandr Ivantsiv
9988ff888b
[build] Add the possibility to disable compilation of teamd and radv containers. (#12920)
- Why I did it
This optimization is needed for DPU SONiC. DPU SONiC runs a limited set of containers and teamd and radv containers are not part of them. Unlike the other containers, there was no possibility to disable teamd and radv containers compilation.
To reduce DPU SONiC compilation time and reduce the image size this commit adds the possibility to disable their compilation.

- How I did it
Two new configuration options are added to rules/config file:

INCLUDE_TEAMD
INCLUDE_ROUTER_ADVERTISER
By default to preserve the existing behavior both options are enabled. There are two ways to override them:

To change option value to "n" in rules/config file.
To override their value using SONIC_OVERRIDE_BUILD_VARS env variable:
SONIC_OVERRIDE_BUILD_VARS="SONIC_INCLUDE_TEAMD=y SONIC_INCLUDE_ROUTER_ADVERTISER=n"

- How to verify it
The default behavior is preserved. To verify it compile the image without overriding new options. Install the image and verify that both teamd and radv containers are present and running.
To verify the new options override them with "n" value. Compile and install image. Verify that no docker containers are present. Verify that SWSS can start without errors.
2022-12-13 12:06:30 +02:00
Kalimuthu-Velappan
0dc22bd27c
05.Version cache - docker dpkg caching support (#12005)
This feature caches all the deb files during docker build and stores them
into version cache.

It loads the cache file if already exists in the version cache and copies the extracted
deb file from cache file into Debian cache path( /var/cache/apt/archives).

The apt-install always installs the deb file from the cache if exists, this
avoid unnecessary package download from the repo and speeds up the overall build.

The cache file is selected based on the SHA value of version dependency
files.

Why I did it
How I did it
How to verify it


* 03.Version-cache - framework environment settings

It defines and passes the necessary version cache environment variables
to the caching framework.

It adds the utils script for shared cache file access.

It also adds the post-cleanup logic for cleaning the unwanted files from
the docker/image after the version cache creation.

* 04.Version cache - debug framework

Added DBGOPT Make variable to enable the cache framework
scripts in trace mode. This option takes the part name of the script to
enable the particular shell script in trace mode.

Multiple shell script names can also be given.

	Eg: make DBGOPT="image|docker"

Added verbose mode to dump the version merge details during
build/dry-run mode.
	Eg: scripts/versions_manager.py freeze -v \
		'dryrun|cmod=docker-swss|cfile=versions-deb|cname=all|stage=sub|stage=add'

* 05.Version cache - docker dpkg caching support

This feature caches all the deb files during docker build and stores them
into version cache.

It loads the cache file if already exists in the version cache and copies the extracted
deb file from cache file into Debian cache path( /var/cache/apt/archives).

The apt-install always installs the deb file from the cache if exists, this
avoid unnecessary package download from the repo and speeds up the overall build.

The cache file is selected based on the SHA value of version dependency
files.
2022-12-12 09:20:56 +08:00
Saikrishna Arcot
61536028f8
[build]: Fix docker load image tag not being the expected tag (#12959)
PR #12829 modified the docker tagging scheme such that optional docker
containers would be tagged with the SONiC image version. However, the
docker-image-load macro wasn't updated for these changes. Update it
here.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2022-12-06 23:36:00 -08:00
LuiSzee
5281f6c3f6
[build][arm64] disable p4rt compile on arm64 for bazel not work (#12798)
pre-compiled bazel is not work in arm64 docker container

shil@2f910d8d37b2:/sonic/src/sonic-p4rt/sonic-pins$ uname -a
Linux 2f910d8d37b2 5.4.0-132-generic #148-Ubuntu SMP Mon Oct 17 16:02:06 UTC 2022 aarch64 GNU/Linux
shil@2f910d8d37b2:/sonic/src/sonic-p4rt/sonic-pins$ bazel
Opening zip "/proc/self/exe": lseek(): Bad file descriptor
FATAL: Failed to open '/proc/self/exe' as a zip file: (error: 9): Bad file descriptor
shil@2f910d8d37b2:/sonic/src/sonic-p4rt/sonic-pins$
2022-12-03 23:07:12 -08:00
Kalimuthu-Velappan
aaeafa8411
02.Version cache - docker cache build framework (#12001)
During docker build, host files can be passed to the docker build through
docker context files. But there is no straightforward way to transfer
the files from docker build to host.

This feature provides a tricky way to pass the cache contents from docker
build to host. It tar's the cached content and encodes them as base64 format
and passes it through a log file with a special tag as 'VCSTART and VCENT'.

Slave.mk in the host, it extracts the cache contents from the log and stores them
in the cache folder. Cache contents are encoded as base64 format for
easy passing.

<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it

#### How I did it

#### How to verify it
2022-12-02 08:28:45 +08:00
Stepan Blyshchak
d22cf46ceb
[dockers] save extension dockers with an image tag (#12829)
Fixes: #11521

- Why I did it
When build SONiC dockers, SONiC build system tags all of them with latest tag. This is Ok for all built-in dockers because we will also tag them with image version tag in sonic_debian_extension.j2 script. On the other hand, some of these dockers are SONiC packages and they are installed by sonic-package-manager which creates a only one tag whcih is recorded in the corresponding .gz file. This leads to having these dockers tagged only with latest tag. This change saves the tag as an image version string in .gz file, so that these dockers have version identification in their tag.

- How I did it
I modified slave.mk to save the version tag instead of latest tag.

- How to verify it
I verified this change by running show version

Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
2022-11-30 19:49:34 +02:00
ganglv
c16b8dbcc5
[sonic-gnmi] Support GNMI native write (#10948)
Why I did it
Provide GNMI native write interface for configuration.

How I did it
Add configuration parameters for GNMI native write.

How to verify it
Check build pipeline.
2022-11-29 16:58:27 +08:00
xumia
ac5d89c6ac
[Build] Support j2 template for debian sources (#12557)
Why I did it
Unify the Debian mirror sources
Make easy to upgrade to the next Debian release, not source url code change required.
Support to customize the Debian mirror sources during the build
Relative issue: #12523
2022-11-09 08:09:53 +08:00
xumia
61246b62c8
[Build] Fix the docker-sync not found issue (#12593)
Why I did it
[Build] Fix the docker-sync not found issue

How I did it
When SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD not enabled, not to remove the docker-sync tag.
2022-11-07 10:04:43 +08:00
ntoorchi
45d174663a
Enable P4RT at build time and disable at startup (#10499)
#### Why I did it
Currently at the Azure build system, the P4RT container is disabled by default at the build time. Here the goal is to include the P4RT container at the build time while disabling it at the runtime. The user can enable/disable the p4rt app through the config based on the preference. 

#### How I did it
Changed the config in rules/config and init-cfg.json.j2
2022-10-31 16:18:42 -07:00
Vivek
5d83d424b1
Added BUILD flags to provision for building the kernel with non-upstream patches (#12428)
* Added ENV vars for non-upstream patches

Signed-off-by: Vivek Reddy <vkarri@nvidia.com>

* Made MLNX_PATCH_LOC an absolute path

Signed-off-by: Vivek Reddy <vkarri@nvidia.com>

* Added non-upstream-patches dir

Signed-off-by: Vivek Reddy <vkarri@nvidia.com>

* Update README.md

* Addressed comments

* Env vars updated

Signed-off-by: Vivek Reddy <vkarri@nvidia.com>

* Readme updated

Signed-off-by: Vivek Reddy <vkarri@nvidia.com>

Signed-off-by: Vivek Reddy <vkarri@nvidia.com>
2022-10-31 12:16:05 -07:00
xumia
078608e7f0
Add the original docker tag without username (#12472)
Why I did it
Add the original docker tag without username to support some of the docker file not changed build broken issue.
The username suffix only required when the native build feature enabled, but if not enabled, the docker file not necessary to change, the build should be succeeded.
It is to support cisco 202205 build.
2022-10-25 15:31:18 +08:00
Hua Liu
257cc96d7c
Remove swsssdk from sonic OS image and docker container image (#12323)
Remove swsssdk from sonic OS image and docker image

#### Why I did it
swsssdk is deprecated, so need remove from image.

#### How I did it
Update config file to remove swsssdk from image.

#### How to verify it
Pass all test case.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

#### Description for the changelog
Remove swsssdk from sonic OS image and docker image

#### Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
2022-10-12 13:04:14 +08:00
Kalimuthu-Velappan
c691b73959
01.Version-cache - restructuring of Makefile.work (#12000)
- The Makefile.work becomes complex and it is very difficult to manage the changes across branches.
- Restructured the Makefile.work and it becomes more readable.
- Added $(QUIET) option to turn on command echo mode through command line option.
- Exported the SONIC_BUILD_VARS variable, through which make options can be set dynamically.
	Eg: make SONIC_BUILD_VARS='INCLUDE_NAT=y'
2022-10-04 14:13:40 -07:00
Zain Budhwani
fd6a1b0ce2
Add events to host and create rsyslog_plugin deb pkg (#12059)
Why I did it

Create rsyslog plugin deb for other containers/host to install
Add events for bgp and host events
2022-09-21 09:20:53 -07:00
ganglv
c1d2e88de9
Replace configuration parameter for gnmi write (#11780)
Why I did it
Replace configuration parameter for gnmi write, and we will add other gnmi write features in the future.

How I did it
Update rules/config and other Makefile.

How to verify it
Build sonic image.
2022-09-19 14:54:08 +08:00
Zain Budhwani
6a54bc439a
Streaming structured events implementation (#11848)
With this PR in, you flap BGP and use events_tool to see the published events.
With telemetry PR #111 in and corresponding submodule update done in buildimage, one could run gnmi_cli to capture BGP flap events.
2022-09-03 07:33:25 -07:00
Saikrishna Arcot
de54eece46
Fix error handling when failing to install a deb package (#11846)
The current error handling code for when a deb package fails to be
installed currently has a chain of commands linked together by && and
ends with `exit 1`. The assumption is that the commands would succeed,
and the last `exit 1` would end it with a non-zero return code, thus
fully failing the target and causing the build to stop because of bash's
-e flag.

However, if one of the commands prior to `exit 1` returns a non-zero
return code, then bash won't actually treat it as a terminating error.
From bash's man page:

```
-e      Exit immediately if a pipeline (which may consist of a single simple
	command), a list, or a compound command (see SHELL GRAMMAR above),
        exits with a non-zero status.  The shell does not exit if the
        command that fails is part of the  command  list  immediately
        following a while or until keyword, part of the test following the
        if or elif reserved words, part of any command executed in a && or
        || list except the command following the final && or ||, any
        command in a pipeline but the last, or if the command's return
        value is being inverted with !.  If a compound command other than a
        subshell returns a non-zero status because a command failed while
        -e was being ignored, the shell does not exit.
```

The part `part of any command executed in a && or || list except the
command following the final && or ||` says that if the failing command
is not the `exit 1` that we have at the end, then bash doesn't treat it
as an error and exit immediately. Additionally, since this is a compound
command, but isn't in a subshell (subshell are marked by `(` and `)`,
whereas `{` and `}` just tells bash to run the commands in the current
environment), bash doesn't exist. The result of this is that in the
deb-install target, if a package installation fails, it may be
infinitely stuck in that while-loop.

There are two fixes for this: change to using a subshell, or use `;`
instead of `&&`. Using a subshell would, I think, require exporting any
shell variables used in the subshell, so I chose to change the `&&` to
`;`. In addition, at the start of the subshell, `set +e` is added in,
which removes the exit-on-error handling of bash. This makes sure that
all commands are run (the output of which may help for debugging) and
that it still exits with 1, which will then fully fail the target.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2022-08-29 11:35:07 -07:00
lixiaoyuner
8d6431e754
Add k8s master feature (#11637)
* Add k8s master feature

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Update kubernetes version mistake and make variable passing clear

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Add CRI-dockerd package

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Update version variable passing logic

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Upgrade the worker kubernetes version

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Install xml file parse tool

Signed-off-by: Yun Li <yunli1@microsoft.com>

Signed-off-by: Yun Li <yunli1@microsoft.com>
2022-08-13 23:01:35 +08:00
gregshpit
5df09490dc
Ported Marvell armhf build on amd64 host for debian buster to use cross-comp… (#8035)
* Ported Marvell armhf build on x86 for debian buster to use cross-compilation instead of qemu emulation

Current armhf Sonic build on amd64 host uses qemu emulation. Due to the
nature of the emulation it takes a very long time, about 22-24 hours to
complete the build. The change I did to reduce the building time by
porting Sonic armhf build on amd64 host for Marvell platform for debian
buster to use cross-compilation on arm64 host for armhf target. The
overall Sonic armhf building time using cross-compilation reduced to
about 6 hours.

Signed-off-by: marvell <marvell@cpss-build3.marvell.com>

* Fixed final Sonic image build with dockers inside

* Update Dockerfile.j2

Fixed qemu-user-static:x86_64-aarch64-5.0.0-2 .

* Update cross-build-arm-python-reqirements.sh

Added support for both armhf and arm64 cross-build platform using $PY_PLAT environment variable.

* Update Makefile

Added TARGET=<cross-target> for armhf/arm64 cross-compilation.

* Reviewer's @qiluo-msft requests done

Signed-off-by: marvell <marvell@cpss-build3.marvell.com>

* Added new radius/pam patch for arm64 support

* Update slave.mk

Added missing back tick.

* Added libgtest-dev: libgmock-dev: to the buster Dockerfile.j2. Fixed arm perl version to be generic

* Added missing armhf/arm64 entries in /etc/apt/sources.list

* fix libc-bin core dump issue from xumia:fix-libc-bin-install-issue commit

* Removed unnecessary 'apt-get update' from sonic-slave-buster/Dockerfile.j2

* Fixed saiarcot895 reviewer's requests

* Fixed README and replaced 'sed/awk' with patches

* Fixed ntp build to use openssl

* Unuse sonic-slave-buster/cross-build-arm-python-reqirements.sh script (put all prebuilt python packages cross-compilation/install inside Dockerfile.j2). Fixed src/snmpd/Makefile to use -j1 in all cases

* Clean armhf cross-compilation build fixes

* Ported cross-compilation armhf build to bullseye

* Additional change for bullseye

* Set CROSS_BUILD_ENVIRON default value n

* Removed python2 references

* Fixes after merge with the upstream

* Deleted unused sonic-slave-buster/cross-build-arm-python-reqirements.sh file

* Fixed 2 @saiarcot895 requests

* Fixed @saiarcot895 reviewer's requests

* Removed use of prebuilt python wheels

* Incorporated saiarcot895 CC/CXX and other simplification/generalization changes

Signed-off-by: marvell <marvell@cpss-build3.marvell.com>

* Fixed saiarcot895 reviewer's  additional requests

* src/libyang/patch/debian-packaging-files.patch

* Removed --no-deps option when installing wheels. Removed unnecessary lazy_object_proxy arm python3 package instalation

Co-authored-by: marvell <marvell@cpss-build3.marvell.com>
Co-authored-by: marvell <marvell@cpss-build2.marvell.com>
2022-07-21 14:15:16 -07:00
Alexander Allen
429254cb2d
[arm] Refactor installer and build to allow arm builds targeted at grub platforms (#11341)
Refactors the SONiC Installer to support greater flexibility in building for a given architecture and bootloader. 

#### Why I did it

Currently the SONiC installer assumes that if a platform is ARM based that it uses the `uboot` bootloader and uses the `grub` bootloader otherwise. This is not a correct assumption to make as ARM is not strictly tied to uboot and x86 is not strictly tied to grub. 

#### How I did it

To implement this I introduce the following changes:
* Remove the different arch folders from the `installer/` directory
* Merge the generic components of the ARM and x86 installer into `installer/installer.sh`
* Refactor x86 + grub specific functions into `installer/default_platform.conf` 
* Modify installer to call `default_platform.conf` file and also call `platform/[platform]/patform.conf` file as well to override as needed
* Update references to the installer in the `build_image.sh` script
* Add `TARGET_BOOTLOADER` variable that is by default `uboot` for ARM devices and `grub` for x86 unless overridden in `platform/[platform]/rules.mk`
* Update bootloader logic in `build_debian.sh` to be based on `TARGET_BOOTLOADER` instead of `TARGET_ARCH` and to reference the grub package in a generic manner

#### How to verify it

This has been tested on a ARM test platform as well as on Mellanox amd64 switches as well to ensure there was no impact. 

#### Description for the changelog
[arm] Refactor installer and build to allow arm builds targeted at grub platforms

#### Link to config_db schema for YANG module changes

N/A
2022-07-12 15:00:57 -07:00
Stepan Blyshchak
ef8675d7ab
[sonic_debian_extension] install systemd-bootchart (#11047)
- Why I did it
Implemented sonic-net/SONiC#1001

- How I did it
Install systemd-bootchart tool and provide default config for it.

- How to verify it
Run build and verify systemd-bootchart is installed.

Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
2022-07-06 14:03:31 +03:00
Vadym Hlushko
87425a5b2b
[sflow + dropmon] added the ENABLE_SFLOW_DROPMON build flag. Added patches for sflow repo. (#10370)
* [sflow + dropmon] added INCLUDE_SFLOW_DROPMON flag, added patches for hsflowd
*Added a capability of monitoring dropped packets for the sFlow daemon in order to improve network - monitoring, diagnostic, and troubleshooting. The drop monitor service allows the sFlow daemon to export another type of sample - dropped packets as Discard samples alongside Counter samples and Packet Flow samples.

Signed-off-by: Vadym Hlushko <vadymh@nvidia.com>
2022-06-20 17:07:02 -07:00
xumia
b6811a58cf
[Build] Improve docker build performance (#11111)
Why I did it
The docker storage driver vfs is not a good option for build, it uses the “deep copy” when building a new layer, leads to lower performance and more space used on disk than other storage drivers.
A better docker storage driver is the default one overlay2, it is a modern union filesystem.
2022-06-16 14:13:01 +08:00
vdahiya12
0b8e463db4
[sonic-platform-common][sonic-platform-daemons] submodule update; Remove python2 sonic-platform-common wheel (#10994)
* [sonic-platform-common][sonic-platform-daemons] submodule update

vdahiya@vdahiya-dev3:~/sonic-buildimage8/sonic-buildimage/src/sonic-platform-daemons$
git log --oneline 9ac12bf..master
0d90023 (HEAD -> master, origin/master, origin/HEAD, origin/202205) grpc
client implementation for active-active dualtor (#248)
6b8bf69 [ycabled] Fix some syntax warnings in ycabled (#263)
2bcf936 [ycabled] fix the posting for mux_cable_static_info per downlink
when ycabled is spawned; synchronizing executing Telemetry API (#257)
ce217c0 Include changes from xcvr_api in transceiver_info table (#253)
e0f8a35 Fix checkReplyType failed issue via recreating xcvr_table_helper
on forking subprocess (#255)

f575a40 (origin/master, origin/HEAD, origin/202205, master)
[Credo][Ycable] changes for synchronizing executing Telemetry API's when
mux toggle is inprogress (#280)
b043372 [sonic_ssd] Nokia-7215: "show platform ssdhealth" not showing
health percent (#279)
d62d3d6 [CMIS]Fix low-power to high power mode transition (#268)
f918125 [syseeprom] Enable display of vendor extension TLV content
(#270)
4e08440 [Credo][Ycable] improve logging for Server Powered off/Faulty
cables (#272)

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

* remove python2 wheel for sonic-platform-common

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

* remove python2 platform_common definitions

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>
2022-06-04 07:41:15 -07:00
Hua Liu
96954f0134
[swsscommon] Add c++ version sonic-db-cli from sonic-swss-common (#10825)
#### Why I did it
    Fix sonic-db-cli high CPU usage on SONiC startup issue: https://github.com/Azure/sonic-buildimage/issues/10218
    ETA of this issue will be 2022/05/31

#### How I did it
    Re-write sonic-cli with c++ in sonic-swss-common: https://github.com/Azure/sonic-swss-common/pull/607
    Modify swss-common rules and slave.mk to install c++ version sonic-db-cli.
    

#### How to verify it
    Pass all E2E test scenario.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111

#### Description for the changelog
    Build and install c++ version sonic-db-cli from swss-common.

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/SONiC/wiki/Configuration.
-->

#### A picture of a cute animal (not mandatory but encouraged)
2022-06-01 08:05:53 +08:00
Shilong Liu
b313563986
[ci] Add arm artifacts in common lib azure pipeline (#10817)
* [ci] Add arm artifacts in common lib azure pipeline
2022-05-23 17:52:56 +08:00
vmittal-msft
f7882b3885
Fix for libsaithrift build for BRCM image (#10852)
Updated libsaibcm to fix libsaithrift compile issue on BRCM image
2022-05-19 11:15:34 -07:00
Saikrishna Arcot
05c6488029
Fix setting the HTTPS proxy (#10739)
Some places were not correctly setting the HTTPS proxy, and were only
setting the HTTP proxy. This was fine until Docker 20.10.10, which then
started using `https_proxy` for HTTPS connections.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2022-05-05 22:48:58 -07:00
vmittal-msft
9ae17e66a3
[sonic-sairedis update] Support for SAI header v1.10.2 with BRCM SAI v7.1.0.0 and MLNX SAI v1.21.1.0 (#10583) 2022-05-05 20:27:29 -07:00
xumia
8ec8900d31
Support SONiC OpenSSL FIPS 140-3 based on SymCrypt engine (#9573)
Why I did it
Support OpenSSL FIPS 140-3, see design doc: https://github.com/Azure/SONiC/blob/master/doc/fips/SONiC-OpenSSL-FIPS-140-3.md.

How I did it
Install the fips packages.
To build the fips packages, see https://github.com/Azure/sonic-fips
Azure pipelines: https://dev.azure.com/mssonic/build/_build?definitionId=412

How to verify it
Validate the SymCrypt engine:

admin@sonic:~$ dpkg-query -W | grep openssl
openssl 1.1.1k-1+deb11u1+fips
symcrypt-openssl        0.1

admin@sonic:~$ openssl engine -v | grep -i symcrypt
(symcrypt) SCOSSL (SymCrypt engine for OpenSSL)
admin@sonic:~$
2022-05-06 07:21:30 +08:00