Commit Graph

136 Commits

Author SHA1 Message Date
Chun'ang Li
a915ee1c1e
Refine PR test template (#15640)
Why I did it
Refine PR test template.

How I did it
Refine PR test template.

How to verify it
PR test executed normally.

Signed-off-by: Chun'ang Li <chunangli@microsoft.com>
2023-07-10 10:49:22 +08:00
Ye Jianquan
6cb077f52a
[CI/CD] Refine t0 sonic and remove SPECIFIED_PARAMS (#15627)
Why I did it
t0-sonic's specific params has been set on sonic-mgmt repo, remove useless SPECIFIED_PARAMS usage
2023-06-27 11:01:56 +08:00
Ye Jianquan
dd989a64d7
[CI/CD] Refine pr test definition, remove old test jobs and testbedv2 flags (#15305) 2023-06-02 16:33:41 +08:00
Ye Jianquan
69d61047c4
[CI/CD] Refine PR test templates and test_plan.py to be ready to migrate to Elastictest (#15259) 2023-05-31 09:37:38 +08:00
xumia
30d919945d [Ci] Support marvell/marvell-arm64 build (#14875)
Why I did it
Support marvell/marvell-arm64 build

Work item tracking
Microsoft ADO (number only): 19995559
How I did it
2023-05-16 00:36:28 +08:00
xumia
61fe9b924f [Ci] Fix the wrong SONIC_BUILD_JOBS build variable used issue in Azp (#14071)
Why I did it
[Ci] Fix the no parallel jobs in some of the platforms issue
We observed some of the pipelines running more time than expected. The issue is the SONIC_BUILD_JOBS using the wrong value 1. It is caused by the runtime variable issue, there is additional single quota mark character added in the make command line.

make 'SONIC_BUILD_JOBS=$(nproc)' targe/xxxx
Need to change to

make SONIC_BUILD_JOBS=$(nproc) targe/xxxx
It is to improve the build performance for some of the platforms using the variable SONIC_BUILD_JOBS=1.
Good one vs: https://dev.azure.com/mssonic/build/_build/results?buildId=227986&view=logs&j=cef3d8a9-152e-5193-620b-567dc18af272&t=cf595088-5c84-5cf1-9d7e-03331f31d795

"SONIC_BUILD_JOBS"                : "8"
Bad one barefoot: https://dev.azure.com/mssonic/build/_build/results?buildId=227379&view=logs&j=993d6e22-aeec-5c03-fa19-35ecba587dd9&t=7be0d2ec-661f-5569-462c-2d9b7ca4ca5d

"SONIC_BUILD_JOBS"                : "1"
How I did it
Expand the BUILD_OPTIONS variable for all platforms.
2023-04-21 04:32:47 +08:00
Liu Shilong
e5dcae8a11
[ci] Fix build issue for vs vhdx image. #14424
Why I did it
sonic-slave-stretch build failed for mmh3 version update to 3.10 on Mar 24.

How I did it
Enable reproducible build for vhdx image.

How to verify it
2023-03-27 22:34:23 +08:00
mssonicbld
2a1c29b7d4
[ci] Fix docker hang issue and change template reference branch (#13894) (#13919) 2023-02-22 14:56:03 +08:00
mssonicbld
1748332684
[Ci] Support to use the same snapshot for all platform builds (#13913) (#13917) 2023-02-22 12:58:56 +08:00
Liu Shilong
785f9c3f0c
[ci] Kill hanged docker build process to avoid build timeout issue. (#13726) (#13728)
Why I did it
Docker build has a low rate of hanging up.
It hangs on different steps. So, it looks like a bug in docker daemon.

How I did it
Start a daemon process to scan running time more than 1 hours, and kill the process.

How to verify it
2023-02-20 18:16:44 +08:00
Liu Shilong
cd84a1f26e [ci] Disable Shallow fetch in pipeline build. (#12964) 2023-02-04 02:36:55 +08:00
Yutong Zhang
c199814fb1
Improve the display of pipeline. (#13127)
The display of azure pipeline is not specific now, such as when the step Run test fails, the display of itself shows successful, but the display of step Kvmdump shows fails, but actually, the step Kvmdump doesn't fail. I improve the display of azure pipeline in this pr, each step has its own success or failure, and is shown in azure pipeline.

Why I did it
The display of azure pipeline is not specific now, such as when the step Run test fails, the display of itself shows successful, but the display of step Kvmdump shows fails, but actually, the step Kvmdump doesn't fail. I improve the display of azure pipeline in this pr, each step has its own success or failure, and is shown in azure pipeline.

How I did it
Each step has its own signature of success or failure.
Using the chain of responsibility pattern to manage all status.
Modify the expected-state in each step.
2023-01-04 17:12:23 +08:00
Liu Shilong
60365bfcb7
Fix sonic slave pipeline to set correct tag on sonic slave image. (#13177) (#13186)
Why I did it
Currently sonic-slave-* tag is confusing. Set correct tag on sonic-slave-* image.
Fix job name to fit the build.

How I did it
build amd image in amd64:
sonic-slave-bullseye:cfe29bff67c
sonic-slave-bullseye:latest
sonic-slave-bullseye:master

build armhf image in amd64:
sonic-slave-bullseye-march-armhf:33614806dc3
sonic-slave-bullseye-march-armhf:latest
sonic-slave-bullseye-march-armhf:master

build arm64 image in amd64:
sonic-slave-bullseye-march-arm64:f3b1b16c801
sonic-slave-bullseye-march-arm64:latest
sonic-slave-bullseye-march-arm64:master

build arm64 image in arm64:
sonic-slave-bullseye:75cb326c9a7
sonic-slave-bullseye-arm64:latest
sonic-slave-bullseye:master

build armhf image in armhf:
sonic-slave-bullseye:64d178951fc
sonic-slave-bullseye-armhf:latest
sonic-slave-bullseye:master

How to verify it
2022-12-28 16:33:47 +08:00
Liu Shilong
9bec8bd198 [build] Add retry when make SONiC image to improve success rate. (#12325)
Why I did it
Makefile needs some dependencies from the Internet. It will fail for network related issue.
Retries will fix most of these issues.

How I did it
Add retries when running commands which maybe related with networking.

How to verify it
2022-12-21 16:36:18 +08:00
Yutong Zhang
1500c2d41c fix merge conflict 2022-12-10 10:33:21 +08:00
Yutong Zhang
bb0f738d78
[TestbedV2][202211]Sync pipeline in 202211 branch with master. (#12942)
Sync pipeline in 202211 branch with master.

Why I did it
Sync pipeline in 202211 branch with master.

How I did it
1. Remove the timeout in each step of a job, and control the timout outside.
2. Set all job mandatory.

Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>
2022-12-06 10:17:15 +08:00
zitingguo-ms
94d56b83ac
Publish docker saiserverv2 in 202211 branch (#12883)
Why I did it
Publish docker saiserverv2 in the build pipeline.

How I did it
Add docker saiserverv2 target in the build template.

How to verify it
Run test in #12837 and it has been built out successfully.
2022-11-30 22:25:17 -08:00
Richard.Yu
47d63bcd06
[SAI PTF] SAI PTF docker support sai-ptf v2 (#12719)
* [SAI PTF] SAI PTF docker support sai-ptf v2

Publish the sai-ptf docker.

Take part of the change from previous PR #11610 (already reverted as some cache issue)
Cause in #11610, added two new target in it, one is sai-ptf another one is syncd-rpc with sai-ptf v2, to make the upgrade with more clear target, use this one take the sai-ptf one.

Test one:
NOSTRETCH=y NOJESSIE=y make configure PLATFORM=vs
NOSTRETCH=y NOJESSIE=y NOBULLSEYE=y SAITHRIFT_V2=y make target/docker-ptf-sai.gz

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove useless change

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove useless parameters

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove useless change

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* Update azure-pipelines-build.yml

remove a useless option

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
2022-11-17 04:42:51 -08:00
Liu Shilong
6d78199d6f
Revert "[SAI PTF]Syncd-rpc and PTF docker support sai ptf v2 (#11610)" (#12677)
This reverts commit f0873f29d8.
2022-11-14 09:56:10 +08:00
Yutong Zhang
5aa03246fc
[master][TestbedV2] Migrate multi-asic test jobs to TestbedV2. (#12668)
Migrate multi-asic test jobs to TestbedV2.

Why I did it
Migrate multi-asic test jobs to TestbedV2.

How I did it
Add one parameter num_asic to create testplan.
Modify azure-pipelines.yml to run multi-asic on tbv2.

Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>
2022-11-11 10:54:37 +08:00
Yutong Zhang
7c746e67d2
[master][TestbedV2] migrate t0-sonic test jobs to TestbedV2. (#12651)
Migrate t0-sonic test jobs to TestbedV2.

Why I did it
Migrate t0-sonic test jobs to TestbedV2.

How I did it
Add two parameters to create testplan.
Modify azure-pipelines.yml to run t0-sonic on tbv2.

Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>
2022-11-10 15:27:31 +08:00
Richard.Yu
f0873f29d8
[SAI PTF]Syncd-rpc and PTF docker support sai ptf v2 (#11610)
* support sai-ptf-v2 in libsaithrift vs

* add build target docker-ptf-sai syncd-rpcv2 and saiserverv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add docker ptf sai

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add build condition for broadcom

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add docker syncd dbg and add debug symbol to docker-saiserverv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* correct the build option

* change the azure pipeline build template

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* change build option for docker-ptf-sai

* enable ptf-sai docker build

* remove the build for syncd-rpcv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* fix issue in build tempalte

* ignore useless package build when build sai-ptf

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove scapy version contraint

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove duplicated target docker-ptf

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* change template for testing the pipeline

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* remove duplicated target

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* fix error in make script

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add shel to setup env

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* replace with certain platform name

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* disable cache for syncd-rpcv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* test without cache

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* disable cache

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* testing: disable the cache for build syncd-rpcv2

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add cache back and get the code ready for testing

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* refactor code

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* add workaround for issue in rules/sairedis.dep

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* refactor code

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
2022-11-07 21:47:52 +08:00
Yutong Zhang
dc0ceaa500
[TestbedV2] Add dualtor test using TestbedV2. (#12601)
Add dualtor test using TestbedV2 in buildimage repo.

Why I did it
Add dualtor test using TestbedV2 in buildimage repo.

How I did it
Add dualtor test using TestbedV2 in buildimage repo.

Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>
2022-11-07 19:48:06 +08:00
xumia
d7a9f18d18
[ci] Upload the debian packages (#12582)
Why I did it
[ci] Upload the debian packages
2022-11-04 15:40:16 +08:00
Liu Shilong
3df031c9b1
[ci] Add azp trigger for future release branches. (#12508) 2022-10-27 17:36:43 +08:00
xumia
158371de38
[Ci] clean up the old artifacts in the agent before downloading the new artifacts (#12391)
Ci] clean up the old artifacts in the agent before downloading the new artifacts
2022-10-25 15:45:01 +08:00
Ye Jianquan
d7b9c64757
Fix the issue that test plan can't be canceled by KVM dump stage (#12469)
Why I did it
Fix the issue that test plan can't be canceled by KVM dump stage

How I did it
Fix the issue that test plan can't be canceled by KVM dump stage
2022-10-22 01:04:17 +08:00
Liu Shilong
1dec49005b
[ci] Fix test job issue on checkout step (#12445) 2022-10-20 13:06:10 +08:00
Ye Jianquan
ef0559c030
[TestbedV2]Migrate t0 and t1-lag to TestbedV2 (#12383)
co-authorized by: jianquanye@microsoft.com

Migrate the t0 and t1-lag test jobs in buildimage repo to TestbedV2.

Why I did it
Migrate the t0 and t1-lag test jobs in buildimage repo to TestbedV2.

How I did it
Migrate the t0 and t1-lag test jobs in buildimage repo to TestbedV2.
2022-10-19 13:33:17 +08:00
Ye Jianquan
2bf2e02719
Enable to cancel pipeline jobs during checkout code and tests (#12436)
co-authorized by: jianquanye@microsoft.com

Why I did it
Now, checkout code step and KVM test job can't be cancelled even though the whole build is cancelled.
That's because by using Azure Pipeline Conditions, we customized the running condition, and we need to react to the Cancel action explicitly by asserting 'succeeded'
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#succeeded
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml#ive-got-a-conditional-step-that-runs-even-when-a-job-is-canceled-how-do-i-manage-to-cancel-all-jobs-at-once

How I did it
Assert 'succeeded' condition explicitly.

How to verify it
Verified by cancelling and rerunning the azure pipeline.
2022-10-19 12:26:50 +08:00
Liu Shilong
c968114a36
[ci] Use absolute template file path in docker-sonic-slave pipeline. (#12153) 2022-09-23 12:54:57 +08:00
Liu Shilong
8211c850f1
[ci] Update docker sonic slave pipeline to build slave base docker (#11908)
* [ci] Update docker sonic slave pipeline to build slave base docker
2022-09-21 15:50:30 +08:00
Ye Jianquan
750e1b3017
Define whether a test is required by code (#11921)
* Define whether a test is required by code

Why I did it
Define whether a test job is required before merging by code.
Let the failure of multi-asic and t0-sonic don't block pr merge.
The 'required' configuration can be modified by the owner in the future.

How I did it
Required:
t1-lag, t0
Not required:
multi-asic, t0-sonic

How to verify it
AZP itself verifies it.

Signed-off-by: jianquanye@microsoft.com
2022-09-03 06:53:54 +08:00
Liu Shilong
cf69206d02
[ci] Fix bug involved by PR 11810 which affect official build pipeline (#11891)
Why I did it
Fix the official build not triggered correctly issue, caused by the azp template path not existing.

How I did it
Change the azp template path.
2022-08-30 14:23:09 +08:00
Liu Shilong
35945c9015
[ci] Update reproducible build related pipeline. (#11810) 2022-08-29 11:26:15 +08:00
Saikrishna Arcot
adffbd4643
Build the Broadcom DNX RPC container as part of the official build (#11829)
With the Broadcom syncd containers getting upgraded to Bullseye, the DNX
RPC container is no longer automatically built. Explicitly add a make
command to build it.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2022-08-24 11:42:15 -07:00
Liu Shilong
9ff2e2cff3
[ci] Update azp reference to support transfering organization from Azure to sonic-net (#11601)
Why I did it
When transfer repo to another organization, azp reference also need change.
Change azp reference to avoid pipeline failure.
2022-08-02 16:24:14 +08:00
xumia
6f323b3e83
[ci] Support the cross build for armhf/arm64 (#11587)
Why I did it
[ci] Support the cross build for armhf/arm64
2022-08-01 11:31:02 +08:00
SuvarnaMeenakshi
6e7616b504
[multi-asic]: Add job to run multi-asic VS tests in sonic-buildimage pipeline (#11517)
Why I did it
Require multi-asic VS tests to be run during PR checks and merges in master branch.

How I did it
Add job to run multi-asic VS tests in sonic-buildimage pipeline. Currently pipeline will run basic bgp fact test to ensure the testbed comes up, load minigraph works and bgp sessions are up.
Use new multi-asic VS agent pool sonictest-ma in official-multi-asic-vs pipeline
Make multi-asic VS test optional for now. There are two known issues:
Announce route failure during refresh-dut in setup testbed stage.
bgp sessions not getting established.
How to verify it
Tested using test azure pipelines.
2022-07-27 11:47:00 -07:00
xumia
af2dea9c7f
[ci] Fix some not sai package removed issue (#11544)
Only replace the file name starts with "cisco-".
2022-07-27 09:58:04 -07:00
Liu Shilong
d7101963f1
[ci] Disable reproducible build in master branch official build (#11452) 2022-07-18 12:45:05 +08:00
Liu Shilong
77c224b0fd
[ci] Add vstest skip folder sonic-slave-* (#11295) 2022-07-15 13:13:58 +08:00
Liu Shilong
5223f4917f
[ci] Update azp template variable for reproducible build. (#11419) 2022-07-13 17:30:23 +08:00
Yakiv Huryk
1568667011
[ci] add docker-sonic-vs with ASAN build (#11297)
The asan-enabled docker image will be used in other CIs that run DVS tests.

Why I did it
To add a possibility to run DVS tests with ASAN for other CIs (e.g. swss).

How I did it
Added the 'asan_image' flag to the vs job group.

How to verify it
Run the CI and check the docker-sonic-vs-asan.gz artifact.
2022-07-04 08:40:04 +08:00
Ze Gan
0ae4903ef9
[azurepipeline]: Add t0-sonic pool back to Azp checker (#11181)
Why I did it
The t0-sonic pool has been fixed, so add it back to azp checker.

How I did it
Remove continueOnError in run-test-template.yml.

Signed-off-by: Ze Gan <ganze718@gmail.com>
2022-06-21 15:47:06 +08:00
Shilong Liu
2851884c8b
[ci] Support to skip vstest using include/exclude config file. (#11086)
example:
├── folderA
│  ├──  fileA (skip vstest)
│  ├──  fileB
│  └──  fileC
If we want to skip vstest when changing /folderA/fileA, and not skip vstest when changing fileB or fileC.

vstest-include:
^folderA/fileA

vstest-exclude:
^folderA
2022-06-17 15:39:41 +08:00
xumia
b6811a58cf
[Build] Improve docker build performance (#11111)
Why I did it
The docker storage driver vfs is not a good option for build, it uses the “deep copy” when building a new layer, leads to lower performance and more space used on disk than other storage drivers.
A better docker storage driver is the default one overlay2, it is a modern union filesystem.
2022-06-16 14:13:01 +08:00
xumia
5072315c89
[Ci]: Fix the target directory not empty issue when publishing artifacts #10972
Why I did it
Fix the target directory not empty issue when publishing artifacts.
Some of the artifacts are published to $(Build.ArtifactStagingDirectory)/target/ before source code checked out.
2022-05-30 16:50:06 +08:00
Shilong Liu
650b00e41e
[ci] Publish logs when building image job is canceled by timeout. (#10919) 2022-05-30 16:02:27 +08:00
Ze Gan
2faa69eed2
[Azp]: Ignore the result of t0-sonic pool (#10905)
Signed-off-by: Ze Gan <ganze718@gmail.com>
2022-05-24 10:42:09 +08:00