Commit Graph

129 Commits

Author SHA1 Message Date
Liu Shilong
e0c29aabe0
[ci] Fix docker sonic slave tag issue. (#13179) 2022-12-28 10:56:50 +08:00
Liu Shilong
550cd6649f
Fix sonic slave pipeline to set correct tag on sonic slave image. (#13177)
Why I did it
Currently sonic-slave-* tag is confusing. Set correct tag on sonic-slave-* image.
Fix job name to fit the build.

How I did it
build amd image in amd64:
sonic-slave-bullseye:cfe29bff67c
sonic-slave-bullseye:latest
sonic-slave-bullseye:master

build armhf image in amd64:
sonic-slave-bullseye-march-armhf:33614806dc3
sonic-slave-bullseye-march-armhf:latest
sonic-slave-bullseye-march-armhf:master

build arm64 image in amd64:
sonic-slave-bullseye-march-arm64:f3b1b16c801
sonic-slave-bullseye-march-arm64:latest
sonic-slave-bullseye-march-arm64:master

build arm64 image in arm64:
sonic-slave-bullseye:75cb326c9a7
sonic-slave-bullseye-arm64:latest
sonic-slave-bullseye:master

build armhf image in armhf:
sonic-slave-bullseye:64d178951fc
sonic-slave-bullseye-armhf:latest
sonic-slave-bullseye:master

How to verify it
2022-12-27 15:40:15 +08:00
Liu Shilong
449ae1332f
[ci] Add branch tag on sonic-slave-bullseye image. (#13151)
Why I did it
tag sonic-slave-* image with branch.
Only keep sonic-slave-* latest tag when it is master branch and amd64 arch.

How I did it
How to verify it
2022-12-26 10:35:15 +08:00
Liu Shilong
364045cfe5
[build] Add retry when make SONiC image to improve success rate. (#12325)
Why I did it
Makefile needs some dependencies from the Internet. It will fail for network related issue.
Retries will fix most of these issues.

How I did it
Add retries when running commands which maybe related with networking.

How to verify it
2022-12-19 12:18:36 +08:00
Liu Shilong
d9eec94c18
[build] Remove sonic-build-hooks in slave base tag. (#12966) 2022-12-09 16:12:59 +08:00
Liu Shilong
82cbe6b029
[ci] Disable Shallow fetch in pipeline build. (#12964) 2022-12-09 16:12:43 +08:00
wenyiz2021
5073dc0f8b
[MASIC] [azp] remove official-build-multi-asic.yml (#12973)
remove ..multi-asic.yml file as original purpose was to test stability before multi_asic PR check were brought up
2022-12-07 15:12:12 -08:00
Yutong Zhang
cb354a5af2
[TestbedV2][master] Remove timeout in each step. (#12915)
Previously, we set timeout in each step such as Lock testbed, Prepare testbed, Run test and KVM dump. When some issue suck like retry happens in one step, it will cause timeout error, but actually, it only needs more time to success. In this pr, we remove the timeout limit in each step and control the timeout outside in each job. When the job runs more than four hours, it will be cancelled.

Why I did it
Previously, we set timeout in each step such as Lock testbed, Prepare testbed, Run test and KVM dump. When some issue suck like retry happens in one step, it will cause timeout error, but actually, it only needs more time to success. In this pr, we remove the timeout limit in each step and control the timeout outside in each job. When the job runs more than four hours, it will be cancelled.

How I did it
Remove the timeout parameter in each step, and control the timeout outside in each job.

How to verify it
Set the timeout of one job to 4 hours, and when timeout happens, azure pipeline will cancel this job.
2022-12-03 22:30:03 -08:00
zitingguo-ms
b774ebfdc2
[SAI-PTF] Publish docker saiserverv2 in master branch (#12842)
Why I did it
Publish docker saiserverv2 in the build pipeline.

How I did it
Add docker saiserverv2 target in the build template.

How to verify it
Test this by running this pipeline: https://dev.azure.com/mssonic/build/_build/results?buildId=182134&view=results
2022-11-30 22:26:54 -08:00
Yutong Zhang
df4312f7ef
Support passing the instance numbers of a testplan. (#12879)
Previously, we hard code the min and max numbers of instance in a plan. In this pr, we support passing the instance numbers of a testplan.

Why I did it
Previously, we hard code the min and max numbers of instance in a plan. In this pr, we support passing the instance numbers of a testplan.

How I did it
Use a variable to set the instance number.
2022-11-30 22:36:55 +08:00
Richard.Yu
47d63bcd06
[SAI PTF] SAI PTF docker support sai-ptf v2 (#12719)
* [SAI PTF] SAI PTF docker support sai-ptf v2

Publish the sai-ptf docker.

Take part of the change from previous PR #11610 (already reverted as some cache issue)
Cause in #11610, added two new target in it, one is sai-ptf another one is syncd-rpc with sai-ptf v2, to make the upgrade with more clear target, use this one take the sai-ptf one.

Test one:
NOSTRETCH=y NOJESSIE=y make configure PLATFORM=vs
NOSTRETCH=y NOJESSIE=y NOBULLSEYE=y SAITHRIFT_V2=y make target/docker-ptf-sai.gz

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove useless change

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove useless parameters

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove useless change

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* Update azure-pipelines-build.yml

remove a useless option

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
2022-11-17 04:42:51 -08:00
Liu Shilong
6d78199d6f
Revert "[SAI PTF]Syncd-rpc and PTF docker support sai ptf v2 (#11610)" (#12677)
This reverts commit f0873f29d8.
2022-11-14 09:56:10 +08:00
Yutong Zhang
5aa03246fc
[master][TestbedV2] Migrate multi-asic test jobs to TestbedV2. (#12668)
Migrate multi-asic test jobs to TestbedV2.

Why I did it
Migrate multi-asic test jobs to TestbedV2.

How I did it
Add one parameter num_asic to create testplan.
Modify azure-pipelines.yml to run multi-asic on tbv2.

Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>
2022-11-11 10:54:37 +08:00
Yutong Zhang
7c746e67d2
[master][TestbedV2] migrate t0-sonic test jobs to TestbedV2. (#12651)
Migrate t0-sonic test jobs to TestbedV2.

Why I did it
Migrate t0-sonic test jobs to TestbedV2.

How I did it
Add two parameters to create testplan.
Modify azure-pipelines.yml to run t0-sonic on tbv2.

Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>
2022-11-10 15:27:31 +08:00
Richard.Yu
f0873f29d8
[SAI PTF]Syncd-rpc and PTF docker support sai ptf v2 (#11610)
* support sai-ptf-v2 in libsaithrift vs

* add build target docker-ptf-sai syncd-rpcv2 and saiserverv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add docker ptf sai

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add build condition for broadcom

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add docker syncd dbg and add debug symbol to docker-saiserverv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* correct the build option

* change the azure pipeline build template

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* change build option for docker-ptf-sai

* enable ptf-sai docker build

* remove the build for syncd-rpcv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* fix issue in build tempalte

* ignore useless package build when build sai-ptf

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove scapy version contraint

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove duplicated target docker-ptf

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* change template for testing the pipeline

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove duplicated target

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* fix error in make script

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add shel to setup env

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* replace with certain platform name

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* disable cache for syncd-rpcv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* test without cache

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* disable cache

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* testing: disable the cache for build syncd-rpcv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add cache back and get the code ready for testing

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* refactor code

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add workaround for issue in rules/sairedis.dep

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* refactor code

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
2022-11-07 21:47:52 +08:00
Yutong Zhang
dc0ceaa500
[TestbedV2] Add dualtor test using TestbedV2. (#12601)
Add dualtor test using TestbedV2 in buildimage repo.

Why I did it
Add dualtor test using TestbedV2 in buildimage repo.

How I did it
Add dualtor test using TestbedV2 in buildimage repo.

Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>
2022-11-07 19:48:06 +08:00
xumia
d7a9f18d18
[ci] Upload the debian packages (#12582)
Why I did it
[ci] Upload the debian packages
2022-11-04 15:40:16 +08:00
Liu Shilong
3df031c9b1
[ci] Add azp trigger for future release branches. (#12508) 2022-10-27 17:36:43 +08:00
xumia
158371de38
[Ci] clean up the old artifacts in the agent before downloading the new artifacts (#12391)
Ci] clean up the old artifacts in the agent before downloading the new artifacts
2022-10-25 15:45:01 +08:00
Ye Jianquan
d7b9c64757
Fix the issue that test plan can't be canceled by KVM dump stage (#12469)
Why I did it
Fix the issue that test plan can't be canceled by KVM dump stage

How I did it
Fix the issue that test plan can't be canceled by KVM dump stage
2022-10-22 01:04:17 +08:00
Liu Shilong
1dec49005b
[ci] Fix test job issue on checkout step (#12445) 2022-10-20 13:06:10 +08:00
Ye Jianquan
ef0559c030
[TestbedV2]Migrate t0 and t1-lag to TestbedV2 (#12383)
co-authorized by: jianquanye@microsoft.com

Migrate the t0 and t1-lag test jobs in buildimage repo to TestbedV2.

Why I did it
Migrate the t0 and t1-lag test jobs in buildimage repo to TestbedV2.

How I did it
Migrate the t0 and t1-lag test jobs in buildimage repo to TestbedV2.
2022-10-19 13:33:17 +08:00
Ye Jianquan
2bf2e02719
Enable to cancel pipeline jobs during checkout code and tests (#12436)
co-authorized by: jianquanye@microsoft.com

Why I did it
Now, checkout code step and KVM test job can't be cancelled even though the whole build is cancelled.
That's because by using Azure Pipeline Conditions, we customized the running condition, and we need to react to the Cancel action explicitly by asserting 'succeeded'
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#succeeded
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml#ive-got-a-conditional-step-that-runs-even-when-a-job-is-canceled-how-do-i-manage-to-cancel-all-jobs-at-once

How I did it
Assert 'succeeded' condition explicitly.

How to verify it
Verified by cancelling and rerunning the azure pipeline.
2022-10-19 12:26:50 +08:00
Liu Shilong
c968114a36
[ci] Use absolute template file path in docker-sonic-slave pipeline. (#12153) 2022-09-23 12:54:57 +08:00
Liu Shilong
8211c850f1
[ci] Update docker sonic slave pipeline to build slave base docker (#11908)
* [ci] Update docker sonic slave pipeline to build slave base docker
2022-09-21 15:50:30 +08:00
Ye Jianquan
750e1b3017
Define whether a test is required by code (#11921)
* Define whether a test is required by code

Why I did it
Define whether a test job is required before merging by code.
Let the failure of multi-asic and t0-sonic don't block pr merge.
The 'required' configuration can be modified by the owner in the future.

How I did it
Required:
t1-lag, t0
Not required:
multi-asic, t0-sonic

How to verify it
AZP itself verifies it.

Signed-off-by: jianquanye@microsoft.com
2022-09-03 06:53:54 +08:00
Liu Shilong
cf69206d02
[ci] Fix bug involved by PR 11810 which affect official build pipeline (#11891)
Why I did it
Fix the official build not triggered correctly issue, caused by the azp template path not existing.

How I did it
Change the azp template path.
2022-08-30 14:23:09 +08:00
Liu Shilong
35945c9015
[ci] Update reproducible build related pipeline. (#11810) 2022-08-29 11:26:15 +08:00
Saikrishna Arcot
adffbd4643
Build the Broadcom DNX RPC container as part of the official build (#11829)
With the Broadcom syncd containers getting upgraded to Bullseye, the DNX
RPC container is no longer automatically built. Explicitly add a make
command to build it.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2022-08-24 11:42:15 -07:00
Liu Shilong
9ff2e2cff3
[ci] Update azp reference to support transfering organization from Azure to sonic-net (#11601)
Why I did it
When transfer repo to another organization, azp reference also need change.
Change azp reference to avoid pipeline failure.
2022-08-02 16:24:14 +08:00
xumia
6f323b3e83
[ci] Support the cross build for armhf/arm64 (#11587)
Why I did it
[ci] Support the cross build for armhf/arm64
2022-08-01 11:31:02 +08:00
SuvarnaMeenakshi
6e7616b504
[multi-asic]: Add job to run multi-asic VS tests in sonic-buildimage pipeline (#11517)
Why I did it
Require multi-asic VS tests to be run during PR checks and merges in master branch.

How I did it
Add job to run multi-asic VS tests in sonic-buildimage pipeline. Currently pipeline will run basic bgp fact test to ensure the testbed comes up, load minigraph works and bgp sessions are up.
Use new multi-asic VS agent pool sonictest-ma in official-multi-asic-vs pipeline
Make multi-asic VS test optional for now. There are two known issues:
Announce route failure during refresh-dut in setup testbed stage.
bgp sessions not getting established.
How to verify it
Tested using test azure pipelines.
2022-07-27 11:47:00 -07:00
xumia
af2dea9c7f
[ci] Fix some not sai package removed issue (#11544)
Only replace the file name starts with "cisco-".
2022-07-27 09:58:04 -07:00
Liu Shilong
d7101963f1
[ci] Disable reproducible build in master branch official build (#11452) 2022-07-18 12:45:05 +08:00
Liu Shilong
77c224b0fd
[ci] Add vstest skip folder sonic-slave-* (#11295) 2022-07-15 13:13:58 +08:00
Liu Shilong
5223f4917f
[ci] Update azp template variable for reproducible build. (#11419) 2022-07-13 17:30:23 +08:00
Yakiv Huryk
1568667011
[ci] add docker-sonic-vs with ASAN build (#11297)
The asan-enabled docker image will be used in other CIs that run DVS tests.

Why I did it
To add a possibility to run DVS tests with ASAN for other CIs (e.g. swss).

How I did it
Added the 'asan_image' flag to the vs job group.

How to verify it
Run the CI and check the docker-sonic-vs-asan.gz artifact.
2022-07-04 08:40:04 +08:00
Ze Gan
0ae4903ef9
[azurepipeline]: Add t0-sonic pool back to Azp checker (#11181)
Why I did it
The t0-sonic pool has been fixed, so add it back to azp checker.

How I did it
Remove continueOnError in run-test-template.yml.

Signed-off-by: Ze Gan <ganze718@gmail.com>
2022-06-21 15:47:06 +08:00
Shilong Liu
2851884c8b
[ci] Support to skip vstest using include/exclude config file. (#11086)
example:
├── folderA
│  ├──  fileA (skip vstest)
│  ├──  fileB
│  └──  fileC
If we want to skip vstest when changing /folderA/fileA, and not skip vstest when changing fileB or fileC.

vstest-include:
^folderA/fileA

vstest-exclude:
^folderA
2022-06-17 15:39:41 +08:00
xumia
b6811a58cf
[Build] Improve docker build performance (#11111)
Why I did it
The docker storage driver vfs is not a good option for build, it uses the “deep copy” when building a new layer, leads to lower performance and more space used on disk than other storage drivers.
A better docker storage driver is the default one overlay2, it is a modern union filesystem.
2022-06-16 14:13:01 +08:00
xumia
5072315c89
[Ci]: Fix the target directory not empty issue when publishing artifacts #10972
Why I did it
Fix the target directory not empty issue when publishing artifacts.
Some of the artifacts are published to $(Build.ArtifactStagingDirectory)/target/ before source code checked out.
2022-05-30 16:50:06 +08:00
Shilong Liu
650b00e41e
[ci] Publish logs when building image job is canceled by timeout. (#10919) 2022-05-30 16:02:27 +08:00
Ze Gan
2faa69eed2
[Azp]: Ignore the result of t0-sonic pool (#10905)
Signed-off-by: Ze Gan <ganze718@gmail.com>
2022-05-24 10:42:09 +08:00
Shilong Liu
b313563986
[ci] Add arm artifacts in common lib azure pipeline (#10817)
* [ci] Add arm artifacts in common lib azure pipeline
2022-05-23 17:52:56 +08:00
xumia
f6927606b3
[Ci]: Fix to trigger the publish pipeline in failure build issue (#10847)
Why I did it
It is not necessary to trigger the publish pipeline when build is failed.

How I did it
Remove the condition in the azp task, change to use template condition.
2022-05-20 14:32:53 +08:00
xumia
3567888b03
[Ci] Support to trigger a pipeline to download and publish artifacts to storage (#10820)
Why I did it
Support to trigger a pipeline to download and publish artifacts to storage and container registry.
Support to specify the patterns which docker images to upload.

How I did it
Pass the pipeline information and the artifact information by pipeline parameters to the pipeline which will be triggered a new build. It is to decouple the artifacts generation and the publish logic, how and where the artifacts/docker images will be published, depends on the triggered pipeline.

How to verify it
2022-05-16 08:57:26 +08:00
Ze Gan
992d233090
Add pool: sonictest-sonic-t0 (#10635)
Signed-off-by: Ze Gan <ganze718@gmail.com>

Existing pools cannot test MACsec scenario, So I add sonictest-sonic-t0 pool that can run MACsec testcases.
2022-05-12 13:21:25 +08:00
xumia
508dda6ad3
[Ci]: Support to sign image for cisco-8000 uefi secure boot (#10616)
Why I did it
[Ci]: Support to sign image for cisco-8000 uefi secure boot
2022-04-21 12:13:09 +08:00
Shilong Liu
0a99f87bec
[ci] Update common lib pipeline to build more distribute packages. (#10576) 2022-04-19 10:32:01 +08:00
xumia
f24a04fa3d
[Ci]: check if there is a sonic dirty version issue (#10445)
Why I did it
[Ci]: check if there is a sonic dirty version issue
If there is a dirty version issue in PR build, the build will be failed.
2022-04-08 10:49:21 +08:00