Commit Graph

5262 Commits

Author SHA1 Message Date
xumia
0171c0e278
[Build] Add the missing debian security mirrors in slave images (#11305) (#11324)
Why I did it
The build below was broken, it was caused by one of the required debian mirror missing.
https://dev.azure.com/mssonic/build/_build/results?buildId=116719&view=logs&j=88ce9a53-729c-5fa9-7b6e-3d98f2488e3f&t=88f376cf-c35d-5783-0a48-9ad83a873284

 libpci-dev : Depends: libudev-dev (>= 196) but it is not going to be installed
 libsystemd-dev : Depends: libsystemd0 (= 232-25+deb9u14) but 232-25+deb9u13 is to be installed
How I did it
Add the missing mirrors for buster and stretch.
2022-07-05 09:23:16 +08:00
Liu Shilong
9fd9409590
[build] Add version files to docker image dependencies (#11193)
* [build] Add version files to docker image dependencies

* [ci] Support to skip vstest using include/exclude config file. (#11086)

example:
├── folderA
│  ├──  fileA (skip vstest)
│  ├──  fileB
│  └──  fileC
If we want to skip vstest when changing /folderA/fileA, and not skip vstest when changing fileB or fileC.

vstest-include:
^folderA/fileA

vstest-exclude:
^folderA
2022-06-22 14:13:08 +08:00
mssonicbld
8e331e077b
[ci/build]: Upgrade SONiC package versions (#10721)
Co-authored-by: mssonicbld <vsts@fv-az105-917.rmmgkgolfqletehdpj5nkpzfpa.ex.internal.cloudapp.net>
2022-06-19 12:46:22 +08:00
Shilong Liu
6ce393e3c0
[build] Add dependencies for sonic-config-engine to block bad PR. (#10770) (#11055)
Add dependencies device/* to sonic-config-engine
2022-06-17 15:44:53 +08:00
xumia
f9005e3dc0
[Build]: Support to use the base image version when a package version not specified (#10971) (#10975)
Why I did it
It is to fix issue: #10952
[Build]: Support to use the base image version when a package version not specified
2022-05-31 10:27:41 +08:00
Arun Saravanan Balachandran
6ebe5e6d50
[DellEMC S5248f] Remove duplicate ipmihelper.py (#10455)
Why I did it
To remove the ipmihelper.py in S5248f directory to prevent the image label being marked 'dirty', due to the file being replaced by the ipmihelper.py in common folder during build.

How I did it
Remove ipmihelper.py in S5248f directory.

How to verify it
Build a broadcom image and verify that the tracked files are not modified.

Which release branch to backport (provide reason below if selected)
 201811
 201911
 202006
 202012
 202106
 202111
Description for the changelog
DellEMC S5248f : Remove duplicate ipmihelper.py

Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)
2022-05-26 12:39:42 -07:00
Shilong Liu
a0ef7459f2
[ci] Add arm artifacts in common lib azure pipeline (#10891)
cherry-pick PR from master:
https://github.com/Azure/sonic-buildimage/pull/10817
2022-05-24 14:58:19 +08:00
Shilong Liu
76feaac0b6 [build] docker-sonic-mgmt replace USER by whoami (#9702) 2022-05-11 14:09:43 +08:00
Shilong Liu
0bbd384c9b Fix docker-sonic-mgmt reproducible related issue. (#9647)
Reproducible build script breaks docker-sonic-mgmt build.
2022-05-11 14:09:43 +08:00
Shilong Liu
7d5e7119f7 [ci] Support multi tags when pushing docker image (#10771) 2022-05-11 14:09:43 +08:00
mssonicbld
2b0bfe766f
[ci/build]: Upgrade SONiC package versions (#10661) 2022-04-30 19:06:09 +00:00
xumia
3e29919b3a
[Build]: Fix the buster slave not built issue when building configure (#10696)
Why I did it
[Build]: Fix the buster slave not built issue when building configure

When running build command "make configure", we can expect the buster slave will be built.
2022-04-28 16:13:21 +08:00
Shilong Liu
7a3e2c19b1
[submodule] Update submodule for sonic-mgmt-common (#10672) 2022-04-28 13:41:04 +08:00
xumia
32dbd3da04 [Build]: Fix host image debian package version issue (#10358)
Why I did it
Fix host image debian package version issue.
The package dependencies may have issue, when some of debian packages of the base image are upgraded. For example, libc is installed in base image, but if the mirror has new version, when running "apt-get upgrade", the package will be upgraded unexpected. To avoid such issue, need to add the versions when building the host image.

How I did it
The package versions of host-image should contain host-base-image.
2022-04-28 03:26:49 +00:00
xumia
e54cc71a16 Fix armhf version issue (#9382)
Why I did it
Fix some of the version files not used issue.
One of example version file version-py3-all-armhf, when building marvell-armhf, the version is used as expected, but it not use.
2022-04-28 03:26:40 +00:00
xumia
0cc75ce5f4
[Submodule]: update submodule for sonic-restapi (#10681)
Why I did it
Update submodule sonic-restapi
e83e0e8 Fix Ctype_char larger than address space issue in 32-bit armhf (#107)
2022-04-27 07:15:52 +08:00
Shilong Liu
56bccb6478
[ci] Add azure pipeline to build common libs. (#10367) (#10688)
Why I did it
To remove reference on Azure.sonic-buildimage artifacts.
Azure.sonic-buildimage has a higher failure rate.
2022-04-27 00:13:16 +08:00
xumia
f9f2235827
[Submodule]: Update submodule for sonic-telemetry (#10667)
Why I did it
Update sonic-telemetry submodules (to fix CVE-2021-3121 alert).

e56e9b4 Fix CVE-2021-3121 warning (#96)
bf1be4f [ci]: Support code diff coverage threshold 50% (#94)
64e516c Ported Marvell armhf build on x86 for debian buster to use cross-compilation instead of qemu emulation (#80)
e426388 [ci]: Support azp code coverage (#87)
0443e66 Remove DB Directory removal as part of make clean (#84)
085f29d Fix unhandled nil err check to prevent rpc causing a crash (#78)
2022-04-26 07:28:18 +08:00
xumia
cb449477e7
[Build] Upgrade pip3 package docker to 5.0.3 (#10523) (#10659)
* Upgrade pip3 package docker to 5.0.3 (#10523)

Why I did it
In sonic-utilities repo, it is required to install docker>=4.4.4
f70dc27827/setup.py (L187)

* Update the docker version to 5.0.3
2022-04-24 19:47:06 +08:00
mssonicbld
217514196b
[ci/build]: Upgrade SONiC package versions (#10657) 2022-04-23 13:39:52 +00:00
Shilong Liu
c929c08672
[ci] Fix PR checker archieve artifacts step (#9357) (#10651)
Why I did it
When a failed job retry. Publish artifact will fail for duplicated name
2022-04-23 13:57:35 +08:00
shlomibitton
d8fa3eb92f
Add a offset bound check during DHCPv6 header inspection to prevent segmentation fault issue (#10560)
Signed-off-by: Shlomi Bitton <shlomibi@nvidia.com>
2022-04-19 09:07:19 -07:00
xumia
331bc53c6f [Build]: Support reproducible build for release branches (#9426)
[Build]: Support reproducible build for release branches #9426
2022-04-18 08:19:29 +08:00
Saikrishna Arcot
b069e0921b Check to see that the py2 and py3 version files exist before trying to sort them (#10325)
For Bullseye, Python 2 isn't present at all. This means that in certain
build cases (such as building something only for Bullseye), the version
file may not exist, and so the sort command would fail.

For most normal build commands, this probably won't be an issue, because
the SONiC build will start with Buster (which has both Python 2 and
Python 3 wheels built), and so the py2 and py3 files will be present
even during the Bullseye builds.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2022-04-18 08:19:29 +08:00
xumia
10b93d9281 [Build]: Fix pip version constraint conflict issue (#10525)
Why I did it
[Build]: Fix pip version constraint conflict issue
When a version is specified in the constraint file, if upgrading the version in build script, it will have conflict issue.

How I did it
If a specified version has specified in pip command line, then the version constraint will be skipped.
2022-04-18 08:19:29 +08:00
xumia
6b88f4b33d [Bug][Build]: fix the file not found issue caused by the relative pat… (#9443)
Fix the nodesource.list cannot read issue, it is cased by the full path not used.

```
2021-12-03T06:59:26.0019306Z Removing intermediate container 77cfe980cd36
2021-12-03T06:59:26.0020872Z  ---> 528fd40e60f6
2021-12-03T06:59:26.0021457Z Step 81/81 : RUN post_run_buildinfo
2021-12-03T06:59:26.0841136Z  ---> Running in d804bd7e1b06
2021-12-03T06:59:29.1626594Z DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
2021-12-03T06:59:34.2960105Z /usr/bin/sed: can't read nodesource.list: No such file or directory
2021-12-03T06:59:34.5094880Z The command '/bin/sh -c post_run_buildinfo' returned a non-zero code: 2
```

Co-authored-by: Ubuntu <xumia@xumia-vm1.jqzc3g5pdlluxln0vevsg3s20h.xx.internal.cloudapp.net>
2022-04-18 08:19:29 +08:00
xumia
72fea76ef7
Support to build armhf/arm64 platforms on arm based system (#7731) (#10577)
Why I did it
Support to build armhf/arm64 platforms on arm based system without qemu simulator.
When building the armhf/arm64 on arm based system, it is not necessary to use qemu simulator.

How I did it
Build armhf on armhf system, or build arm64 on arm64 system, by default, qemu simulator will not be used.
When building armhf on arm64, and you have enabled armhf docker, then it will build images without simulator automatically. It is based how the docker service is run.

Docker base image change:
For amd64, change from debian:to amd64/debian:
For arm64, change from multiarch/debian-debootstrap:arm64- to arm64v8/debian:
For armhf, change from multiarch/debian-debootstrap:armhf- to arm32v7/debian:
See https://github.com/docker-library/official-images#architectures-other-than-amd64
The mapping relations:
arm32v6 --- armel
arm32v7 --- armhf
arm64v8 --- arm64

Docker image armhf deprecated info: https://hub.docker.com/r/armhf/debian, using arm32v7 instead.
2022-04-15 18:00:14 +08:00
Shilong Liu
2058147cca
[ci] Fix PR validation issue involved by cherry-pick enabling ACR.
involved un-expectec lines.
2022-04-13 11:59:57 +08:00
xumia
56be714935
[Submodule]: update sonic-restapi (#10295)
[Submodule]: update sonic-restapi
bd97dfe Fix urllib3 CVE-2021-33503 issue (#104)
f159bfa Upgrade the containers to be based on Debian Buster (#103)
a1830c1 Fix OpenAPI spec to be readable by autorest (#101)
94805a3 Identify and report Vnet GUID for conflicting VNI (#99)
4832dfd Static route expiry if not specified as persistent (#98)
5cc4358 Add support for overlay ECMP (#96)
6822a46 [CI] Set diff cover threshold to 50% (#97)
dcc826a Add PR diff coverage (#95)
e842c5f Generate code coverage for Unit Tests (#94)
f9bbed3 Convert Unit Tests from unittest framework to pytest framework (#93)
e466ac2 Reject incorrect CIDR addresses while configuring routes (#91)
07cf59d [CI] Set up Azure pipeline (#89)
47476ac Upgrade Go version to 1.14 (#90)
ee887c7 Added flag to make go/pkg/ folder writable (removable) (#87)
6cfc8fe (origin/shilongliu/azp) Default Vrf static route support (#85)
d3e8b8e Add BFD config to subinterface and neighbor (#68)
8d4fd4a Fix WS-2018-0594 potential non-random UUIDs security issue (#78)
fad87fa Update API description for default vrf and ecmp routes (#82)

Some of them have been updated in 202012 branch, should be added in 202106 as well.
2022-04-08 17:42:46 -07:00
Shilong Liu
ac9c0b5269 [ci] Set default ACR in UpgrateVersion/PR/official pipeline. (#10341)
Why I did it
docker hub will limit the pull rate.
Use ACR instead to pull debian related docker image.

How I did it
Set DEFAULT_CONTAINER_REGISTRY in pipeline.
2022-04-08 13:09:58 +08:00
xumia
b1105af83c [Build]: support to collect version when purging debian package (#10287)
Why I did it
support to collect version when purging debian package
Support to collect version multiple times

How I did it
Add the collection action before purging.
2022-04-07 09:16:02 +08:00
xumia
b93080d246 [Build][Bug]: fix the warning message not printed as expected issue (#10278)
Fix the warning message not printed as expected issue
2022-04-07 09:16:02 +08:00
xumia
92f97bddb9 [Build]: Fix installing dpkg packages in parallel issue (#10175)
Why I did it
Fix the debian packages installing in parallel issue.
Add apt hook command to support apt to print no version control info.
2022-04-07 09:16:02 +08:00
xumia
2177a68b36 [Build]: Cleanup the reproducible mirrors when build complete (#9132)
Why I did it
The reproducible build mirrors are only used during the build, the mirrors can be removed after that.
2022-04-07 09:16:02 +08:00
xumia
0cecab27fa [Build]: Fix the version not found issue (#9331)
When we update the a sai package downing from a remote server, we need to update the version file as well currently, but the reproducible build feature is not enabled in master, it can only be detected when merging the code into the release branches, such as 202106, 202012, etc.
The reproducible feature is to reduce the build failure, not need to break the build when the version not specified. If version not specified, the best choice is to accept the version from remote server.

Co-authored-by: Ubuntu <xumia@xumia-vm1.jqzc3g5pdlluxln0vevsg3s20h.xx.internal.cloudapp.net>
2022-04-07 09:16:02 +08:00
mssonicbld
04e24c0f29
[ci/build]: Upgrade SONiC package versions (#10452)
Upgrade SONiC Versions #10452
2022-04-06 19:27:26 +08:00
xumia
d354cd84d6 [ci] Fix the generating version file failure issue caused by artifacts folder change (#7464) (#10451)
Fix the generating version file failure issue caused by artifacts folder change.
When changing to use the same template for PR build, official build and packages version upgrade, the artifacts folder adding a "target" folder, the version upgrade task should be changed accordingly.
2022-04-02 14:41:13 +00:00
xumia
a4b26ce298 Update azure-pipelines-UpgrateVersion.yml for Azure Pipelines
Remove duplicate resources define in template
2022-04-02 08:29:02 +08:00
xumia
ad21d1ca74 Support dpkg cache for marvell-armhf (#9381)
Why I did it
Support marvell-armhf dpkg cache
2022-04-02 00:14:51 +00:00
Shilong Liu
e28c30d4e4 [build] Fix issues found in reproducible build. (#10407) 2022-04-01 23:58:48 +00:00
Shilong Liu
1366e0bb63 [ci] Use template from master branch in UpgrateVersion/sonic-slave pipeline (#10380) 2022-04-01 23:58:29 +00:00
xumia
32c00fb014 [build][Bug]: Fix the command set_reproducible_mirrors not found issue (#10398)
Why I did it
Fix the command set_reproducible_mirrors not found issue during the build.
2022-03-31 06:24:44 +00:00
Shilong Liu
ab91b607f2
[ci] Fix remove sonic-slave-* docker image issue when building sonic-slave* (#10296) (#10381) 2022-03-30 16:08:52 +08:00
xumia
ca85c6be5e
[Unit Test]: Fix sonic config engine test not stable issue(#10147) (#10355)
Why I did it
Fix sonic config engine test not stable issue
2022-03-29 10:41:11 +08:00
Alexander Allen
b10833425c
[202106] [Mellanox] Fix DPB supported breakout modes (#10129)
Cherry pick of #10072

- Why I did it
Removing DPB breakout modes that require adjacent ports to be disabled as that is not supported by the current DPB infrastructure.

Correspondingly had to remove the hwsku.json file from any SKUs which utilized these removed modes such that the system will fall back to ports_config.ini and DPB will not be supported for those SKUs.

- How I did it
Modified the platform.json files of Mellanox devices.

- How to verify it
Execute show int break [Ethernet] on the affected platforms and ensure there are no modes present that would require an adjacent port to be disabled to function.
2022-03-27 09:15:23 +03:00
Oleksandr Ivantsiv
40265e8dde [sonic-config-engine]: Improve comparison between default and supported breakout modes. (#9278)
Closes #7958 

#### Why I did it
The previous implementation of sonic-cfggen did a simple comparison between default breakout mode in
hwsku.json and supported modes in platform.json. To set a different default speed in hwsku.json
it was required to add one more entry to supported modes in platfrom.json file:

1x10G[100G,50G] vs 1x100G[50G,10G]

The new implementation does more intelligent parsing and analysis of supported and default modes. It
allows changing default speed without adding a new entry to platform.json.

#### How I did it
Add more intelligent parsing and analysis of supported and default modes.

#### How to verify it
Run sonic-config-engine unit tests from sonic-config-engine/tests directory
2022-03-24 05:15:11 +00:00
Alexander Allen
b5849c0918 [pmon] Clean up supervisord chassis_db_init entry and fix startsecs (#10071)
Why I did it
Code review was still in progress when #9858 was merged and upon further testing I have arrived at a better solution.

How I did it
Modified supervisord configuration j2 template for pmon to require no minimum uptime for chassisd_db_init and to remove the redundant exit_codes directive

How to verify it
Boot switch and verify in syslog that there are no errors related to chassis_db_init
2022-03-24 05:14:38 +00:00
Sudharsan Dhamal Gopalarathnam
4da20e7ff9 [containerd]Fixing container commands when mode is local and state is disabled (#9986)
Why I did it
During warm-reboot and fast-reboot the below error logs appear
Feb 3 22:05:15.187408 r-lionfish-13 ERR container: docker cmd: kill for nat failed with 404 Client Error for http+docker://localhost/v1.41/containers/nat/json: Not Found ("No such container: nat")

The container command when called for local mode doesn't check if it is enabled before calling docker kill which throws the above errors.
b6ca76b482/scripts/fast-reboot (L699)

How I did it
Checking feature state if local mode and returning error exit code along with valid debug message.

How to verify it
Manually tested with warm-reboot and fast-reboot
Added UT to verify it.
2022-03-24 05:14:01 +00:00
xumia
43c32dfa5d
[Build]: Use one debian mirror config (#10274) (#10300)
Why I did it
Use one debian mirror config.
The empty config in https://github.com/Azure/sonic-buildimage/blob/master/files/image_config/apt/sources.list overrides the file https://github.com/Azure/sonic-buildimage/blob/master/files/apt/sources.list.amd64 (armhf/arm64), it does not make sense.
All the content in files/image_config/apt is no use, any one wants to add mirror config, please add in files/apt.

How I did it
Remove files/image_config/apt and the reference.
2022-03-23 22:42:52 +08:00
xumia
37aaed44bd [Build]: Support to set jobFilters (#10280)
[Build]: Support to set jobFilters
2022-03-21 02:28:54 +00:00