Commit Graph

286 Commits

Author SHA1 Message Date
Saikrishna Arcot
34a1ac1a0f Migrate from ntp to ntpsec
Debian Bookworm no longer uses NTP, and instead uses NTPsec. Modify our
files to update/replace the NTPsec files instead.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2023-11-21 18:53:15 -08:00
Samuel Angebault
e4a497183a
Add build option to reduce final image size (#16729)
* Reduce SONiC image filesystem size

Add a build option to reduce the image size.
The image reduction process is affecting the builds in 2 ways:
 - change some packages that are installed in the rootfs
 - apply a rootfs reduction script

The script itself will perform a few steps:
 - remove file duplication by leveraging hardlinks
   - under /usr/share/sonic since the symlinks under the device folder are lost during the build.
   - under /var/lib/docker since the files there will only be mounted ro
 - remove some extra files (man, docs, licenses, ...)
 - some image specific space reduction (only for aboot images currently)

The script can later be improved but for now it's reducing the rootfs
size by ~30%.

* restore fully featured vim package
2023-10-24 10:01:58 +08:00
Yakiv Huryk
6cb8893180
[build] add support for 2 stage rootfs build (#15924)
This adds optimization for the SONiC image build by splitting the final build step into two stages. It allows running the first stage in parallel, improving build time.

The optimization is enabled via new rules/config flag ENABLE_RFS_SPLIT_BUILD (disabled by default)

- Why I did it
To improve a build time.

- How I did it
Added a logic to run build_debian.sh in two stages, transferring the progress via a new build artifact.

- How to verify it
make ENABLE_RFS_SPLIT_BUILD=y SONIC_BUILD_JOBS=32 target/<IMAGE_NAME>.bin

Signed-off-by: Yakiv Huryk <yhuryk@nvidia.com>
2023-10-11 09:33:17 +03:00
lixiaoyuner
bca2ce25ef
[k8master]: Install nc cmd for k8s master network issue debug (#16745) 2023-09-30 01:16:51 -07:00
Konstantin Vasin
a12b42f94c
[build]: remove 15s sleep in build_debian.sh (#16672)
Remove 15s unconditional sleep.
Instead check every second that /proc is not mounted.
Go to the next step if /proc is not mounted anymore or after 15s.
2023-09-28 09:00:50 -07:00
Yevhen Fastiuk
52f6dd65a3
Improve remote fetch (#12795)
### Why I did it
To fix those errors:
One:
```
Connecting to urm.nvidia.com (urm.nvidia.com)|*.*.*.*|:443... connected.
GnuTLS: Error in the pull function.
Unable to establish SSL connection.
Error 4
make[1]: Leaving directory '/sonic/src/smartmontools'
[ target/debs/bullseye/smartmontools_6.6-1_amd64.deb ]
```
Second:
```
Get:90 https://debian-mirror-url buster/main amd64 librrd-dev amd64 1.7.1-2 [284 kB]
Get:91 https://debian-mirror-url buster/main amd64 psmisc amd64 23.2-1+deb10u1 [126 kB]
Get:92 https://debian-mirror-url buster/main amd64 python-smbus amd64 4.1-1 [12.2 kB]
Get:93 https://debian-mirror-url buster/main amd64 python3.7-dev amd64 3.7.3-2+deb10u3 [510 kB]
Get:94 https://debian-mirror-url buster/main amd64 python3-dev amd64 3.7.3-1 [1264 B]
Get:95 https://debian-mirror-url buster/main amd64 python3-smbus amd64 4.1-1 [12.5 kB]
Get:96 https://debian-mirror-url buster/main amd64 rrdtool amd64 1.7.1-2 [485 kB]
Fetched 122 MB in 12s (9976 kB/s)
E: Failed to fetch https://debian-mirror-url/pool/main/p/python-defaults/python2-minimal_2.7.16-1_amd64.deb  500  Internal Server Error [IP: *.*.*.* 443]
E: Failed to fetch https://debian-mirror-url/pool/main/f/fontconfig/fontconfig-config_2.13.1-2_all.deb  500  Internal Server Error [IP: *.*.*.* 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get update &&       apt-get install -y          build-essential         python3-dev             ipmitool                librrd8                 librrd-dev              rrdtool                 python-smbus            python3-smbus           dmidecode               i2c-tools               psmisc                  libpci3' returned a non-zero code: 100
[ target/docker-platform-monitor.gz ]
Error 1
```

#### How I did it
Add retry mechanism to apt, wget, and curl hooks
2023-09-23 18:07:04 -07:00
Hua Liu
11f5a75425
[tacacs]: Fix tcpdump report error when tacacs enabled (#16372)
Fix tcpdump report error when tacacs enabled.

Why I did it
Fix tcpdump report error when tacacs enabled:
Sep 1 09:25:18.189395 vlab-01 ERR tcpdump: nss_tacplus: /etc/tacplus_nss.conf fopen failed
Sep 1 09:25:18.189606 vlab-01 ERR tcpdump: nss_tacplus: bad config or server line for nss_tacplus

This is because debian add a patch create AppArmor profile for resource access control. The profile need update to allow tcpdump access /etc/tacplus_nss.conf.

Work item tracking
Microsoft ADO: 17667308

How I did it
Modify tcpdump AppArmor profile, add new line to allow tcpdump access TACACS config file:

/etc/tacplus_nss.conf r,
2023-09-23 00:07:53 -07:00
Saikrishna Arcot
d62ad707bc
Update to Linux 5.10.179 (#15926)
## How I did it

Depends on sonic-net/sonic-linux-kernel#328 and sonic-net/saibcm-modules#12.

#### How to verify it

Verified that the image boots up, BGP comes up, and a basic warm-reboot works on VS, broadcom, and mellanox.
2023-09-20 15:24:39 -07:00
lixiaoyuner
4f53819efa
Install parted package for k8s master (#16484)
### Why I did it
Need a tool to extend disk size
##### Work item tracking
- Microsoft ADO **(number only)**: 25094467
#### How I did it
Install parted package
#### How to verify it
Use apt list parted command to check if it's installed
2023-09-07 23:22:47 -07:00
xumia
a0b3ec2df6
Support FIPS DB configuration (#15632)
Why I did it
Support FIPS DB configuration
Design Doc: sonic-net/SONiC#1372

Work item tracking
Microsoft ADO (number only): 24411148
How I did it
Add the FIPS Yang model to make FIPS configurable in ConfigDB.

How to verify it
See TestPlan: sonic-net/sonic-mgmt#9092
Build the image and run the tests: sonic-net/sonic-mgmt#9091
2023-07-28 16:54:02 +08:00
lixiaoyuner
10b65d9826
Add k8s master code new (#15716)
Why I did it
Currently, k8s master image is generated from a separate branch which we created by ourselves, not release ones. We need to commit these k8s master related code to master branch for a better way to do k8s master image build out.

Work item tracking
Microsoft ADO (number only):
19998138
How I did it
Install k8s dashboard docker images
Install geneva mds and mdsd and fluentd docker images and tag them as latest, tagging latest will help create container always with the latest version
Install azure-storage-blob and azure-identity, this will help do etcd backup and restore.
Install kubernetes python client packages, this will help read worker and container state, we can send these metric to Geneva.
Remove mdm debian package, will replace it with the mdm docker image
Add k8s master entrance script, this script will be called by rc-local service when system startup. we have some master systemd services in compute-move repo, when VMM service create master VM, VMM will copy all master service files inside VM, the entrance script will setup all services according to the service files.
When the entrance script content changed, the PR build will set include_kubernetes_master=y to help do validation for k8s master related code change. The default value of include_kubernetes_master should be always n for public master branch. We will generate master image from internal master branch
How to verify it
Build with INCLUDE_KUBERNETES_MASTER = y
2023-07-25 07:44:59 +08:00
xumia
bdef73ea96
[Build] Fix the PyYang python package installation issue (#15890)
Why I did it
Fix the armhf build failure.
How to reproduce the issue:

docker run -it debain:bullseye bash
apt-get update && apt-get install -y python3-pip
pip3 install PyYAML==5.4.1
Error message:

Collecting PyYAML==5.4.1
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /tmp/tmp6xabslgb_in_process.py get_requires_for_build_wheel /tmp/tmp_er01ztl
....
      raise AttributeError(attr)
  AttributeError: cython_sources
  ----------------------------------------
WARNING: Discarding d63f2d7597/PyYAML-5.4.1.tar.gz (sha256)=607774cbba28732bfa802b54baa7484215f530991055bb562efbed5b2f20a45e (from https://pypi.org/simple/pyyaml/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*). Command errored out with exit status 1: /usr/bin/python3 /tmp/tmp6xabslgb_in_process.py get_requires_for_build_wheel /tmp/tmp_er01ztl Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement PyYAML==5.4.1
ERROR: No matching distribution found for PyYAML==5.4.1
root@fa2fa92edcfd:/# 
But if adding the option --no-build-isolation, then it is good, see fix.

install "PyYAML==5.4.1" --no-build-isolation
The same error can be found in the multiple builds.

Work item tracking
Microsoft ADO (number only): 24567457

How I did it
Add a build option --no-build-isolation.
2023-07-19 06:33:49 +08:00
ycoheNvidia
05bbf72c86
Reduced root directory privileges (#15147)
#### Why I did it
Reduced root directory privileges

#### How I did it
During build_debian - called chroot to reduce root directory and its subdirectories privileges to 744
#### How to verify it
After image build and upgrade - check /root privileges by calling "ls -a /root"

#### Description for the changelog
reduced /root directory privileges
2023-07-16 11:06:29 -07:00
Mohammedz93
28b9299445
Support Reset factory (#14105)
#### Why I did it
Support reset factory in Sonic OS
[Reset Factory HLD](https://github.com/sonic-net/SONiC/pull/1231)
[Sonic-mgmt tests](https://github.com/sonic-net/sonic-mgmt/pull/7652)

#### How I did it
- Added new script "/usr/bin/reset-factory"
   * It generates a new config_db.json files with factory configurations
   * It clears system files and logs
   * It removes all docker containers on system except database
   * It clears non-default users and restores default users password
- Dump the default users info to a new file during build "/etc/sonic/default_users.json"
- Supported new type "Keep-basic" in "config-setup factory"
- Add new conf file for config-setup "/etc/config-setup/config-setup.conf

#### How to verify it
- Run reset-factory script with all types: < none | keep-all-config | only-config | keep-basic >
- Run config-setup factory with parameters < none | keep-basic >

#### Description for the changelog
Support reset factory in Sonic OS

#### Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
2023-07-11 16:14:17 -07:00
Saikrishna Arcot
e46be54f14
Update the docker daemon to 24.0.2 (#15652)
Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2023-07-06 14:44:29 -07:00
Junchao-Mellanox
b07957bdad
Fix issue: systemctl daemon-reload would sporadically cause udev handler fail (#15253)
#### Why I did it

A workaround to back port the fix for a systemd issue.

The systemd issue: https://github.com/systemd/systemd/issues/24668
The systemd PR to fix the issue: https://github.com/systemd/systemd/pull/24673/files

The formal solution should upgrade systemd to a version that contains the fix. But, systemd is a very basic service, upgrading systemd requires heavy test. 

#### How I did it
Copy the correct systemd-udevd.service file in build time 

#### Tested branch (Please provide the tested image version)

- [x] 202211
- [ ] <!-- image version 2 -->

```
SONiC Software Version: SONiC.fix-udev.3-b65c7bdec_Internal
SONiC OS Version: 11
Distribution: Debian 11.7
Kernel: 5.10.0-18-2-amd64
Build commit: b65c7bdec
Build date: Mon Jun 19 10:54:50 UTC 2023
Built by: sw-r2d2-bot@r-build-sonic-ci02-241

Platform: x86_64-mlnx_msn4700-r0
HwSKU: ACS-MSN4700
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2022X08597
Model Number: MSN4700-WS2FO
Hardware Revision: A1
Uptime: 08:10:11 up 1 min,  1 user,  load average: 1.81, 0.67, 0.24
Date: Sun 25 Jun 2023 08:10:11

Docker images:
REPOSITORY                    TAG                             IMAGE ID       SIZE
docker-fpm-frr                fix-udev.3-b65c7bdec_Internal   a7b911e7cb6f   346MB
docker-fpm-frr                latest                          a7b911e7cb6f   346MB
docker-platform-monitor       fix-udev.3-b65c7bdec_Internal   94c5178cf80b   731MB
docker-platform-monitor       latest                          94c5178cf80b   731MB
docker-orchagent              fix-udev.3-b65c7bdec_Internal   46b393e0ace8   328MB
docker-orchagent              latest                          46b393e0ace8   328MB
docker-syncd-mlnx             fix-udev.3-b65c7bdec_Internal   1f5c6c23e33a   734MB
docker-syncd-mlnx             latest                          1f5c6c23e33a   734MB
docker-sflow                  fix-udev.3-b65c7bdec_Internal   7e45992c8c59   317MB
docker-sflow                  latest                          7e45992c8c59   317MB
docker-teamd                  fix-udev.3-b65c7bdec_Internal   e4d905592cda   316MB
docker-teamd                  latest                          e4d905592cda   316MB
docker-nat                    fix-udev.3-b65c7bdec_Internal   7fe799367580   319MB
docker-nat                    latest                          7fe799367580   319MB
docker-macsec                 latest                          d702a5554171   318MB
docker-snmp                   fix-udev.3-b65c7bdec_Internal   3bce8fcf71cd   338MB
docker-snmp                   latest                          3bce8fcf71cd   338MB
docker-sonic-telemetry        fix-udev.3-b65c7bdec_Internal   f13949cbc817   597MB
docker-sonic-telemetry        latest                          f13949cbc817   597MB
docker-dhcp-relay             latest                          153d9072805d   306MB
docker-router-advertiser      fix-udev.3-b65c7bdec_Internal   aed642b9a6bc   299MB
docker-router-advertiser      latest                          aed642b9a6bc   299MB
docker-sonic-p4rt             fix-udev.3-b65c7bdec_Internal   a3cae5ca65a7   870MB
docker-sonic-p4rt             latest                          a3cae5ca65a7   870MB
docker-mux                    fix-udev.3-b65c7bdec_Internal   b81f0401b9a8   347MB
docker-mux                    latest                          b81f0401b9a8   347MB
docker-eventd                 fix-udev.3-b65c7bdec_Internal   c5917d0e801f   298MB
docker-eventd                 latest                          c5917d0e801f   298MB
docker-lldp                   fix-udev.3-b65c7bdec_Internal   fd5dc14a7976   341MB
docker-lldp                   latest                          fd5dc14a7976   341MB
docker-database               fix-udev.3-b65c7bdec_Internal   438c2715a1dd   299MB
docker-database               latest                          438c2715a1dd   299MB
docker-sonic-mgmt-framework   fix-udev.3-b65c7bdec_Internal   5c50b115fbcd   414MB
docker-sonic-mgmt-framework   latest  
```
2023-06-25 16:58:14 -07:00
Oleksandr Ivantsiv
475fe27c0b
[dns] Add support for static DNS configuration. (#14549)
- Why I did it
Add support for static DNS configuration. According to sonic-net/SONiC#1262 HLD.

- How I did it
Add a new resolv-config.service that is responsible for transferring configuration from Config DB into /etc/resolv.conf file that is consumed by various subsystems in Linux to resolve domain names into IP addresses.

- How to verify it
Run the image compilation. Each component related to the static DNS feature is covered with the unit tests.
Run sonic-mgmt tests. Static DNS feature will be covered with the system tests.
Install the image and run manual tests.
2023-06-22 19:12:30 +03:00
DavidZagury
5acec530bb
Add SECURE_UPGRADE_PROD_TOOL_ARGS flag to make it possible for vendors to pass their own arguments on the prod signing script (#14581)
- Why I did it
Since the prod signing tool is vendor specific, and each vendor may have different arguments they would like to use in the script, we would need a way to inject those arguments to the script.

- How I did it
Add a compilation flag SECURE_UPGRADE_PROD_TOOL_ARGS which vendors can use to inject any flag they would want to the prod signing script.

- How to verify it
Build SONiC using your own prod script
2023-05-16 08:36:13 +03:00
Konstantin Vasin
bba4fb86c7
[Build] update python package docker in host image to 6.1.1 (#14993)
Fix #14974
Refs: https://github.com/docker/docker-py/pull/3116
2023-05-10 07:43:35 -07:00
DavidZagury
2d0a12af6d
Fix issue with prod script not found, change the prod signing to work with flags to align to the dev script (#14580)
- Why I did it
Fix issue with signing tool not running due to being call with the path from the host and not the path it is mounted on inside the docker-slave

- How I did it
Modified the path on the SECURE_UPGRADE_PROD_SIGNING_TOOL flag to the path where it is mounted inside the slave docker

- How to verify it
Build SONiC using your own prod script
2023-05-02 09:13:16 +03:00
Ying Xie
72c52bc677
Revert "Clear DNS configuration received from DHCP during networking reconfiguration in Linux. (#13516)" (#14902)
This reverts commit c7ecd92c54.
2023-05-01 17:12:38 -07:00
Ying Xie
6fc357604e
Revert "[Ragile]Import defusedxml packet (#14815)" (#14830)
This reverts commit c422cab5eb.
2023-04-24 14:10:38 -07:00
DavidZagury
90f45d9774
Change SECURE_UPGRADE_DEV_SIGNING_CERT to SECURE_UPGRADE_SIGNING_CERT (#14591)
Depends on https://github.com/sonic-net/sonic-linux-kernel/pull/315

#### Why I did it
The name SECURE_UPGRADE_DEV_SIGNING_CERT is misleading, this flag is relevant to both to dev and prod signing.

#### How I did it
Rename all mentions of name SECURE_UPGRADE_DEV_SIGNING_CERT to SECURE_UPGRADE_SIGNING_CERT - this is also done with PR in sonic-linux-kernel repository

#### How to verify it
Build SONiC using your own prod script
2023-04-24 11:17:51 -07:00
pettershao-ragilenetworks
c422cab5eb
[Ragile]Import defusedxml packet (#14815)
Why I did it
Import defusedxml packet to fix semgrep error "using defusedxml instead of xml"

How I did it
Add "pip3 install defusedxml" in build_debian.sh

Signed-off-by: pettershao-ragilenetworks <pettershao@ragilenetworks.com>
2023-04-24 08:45:57 -07:00
xumia
f1fd42558a
Support to add SONiC OS Version in device info (#14601)
Why I did it
Support to add SONiC OS Version in device info.
It will be used to display the version info in the SONiC command "show version". The version is used to do the FIPS certification. We do not do the FIPS certification on a specific release, but on the SONiC OS Version.

SONiC Software Version: SONiC.master-13812.218661-7d94c0c28
SONiC OS Version: 11
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
How I did it
2023-04-12 09:20:08 +08:00
lixiaoyuner
935f5dc5f0
Install kubernetes-cni for kubelet (#14163)
Why I did it
Find a new bug on kubelet side. The kubernetes-cni plug-in was removed in #12997, the reason is that the plug-in will be auto installed when install kubeadm, and will report error if we don't remove the install code. But after removal, the version auto installed is different from what we installed before. This will affect the kubelet action in some scenarios we don't find before. Need to install it by another way.

How I did it
Install kubernetes-cni==0.8.7-00 before install kubeadm

How to verify it
Flannel binary will be installed under /opt/cni/bin/ folder
2023-03-16 17:21:37 -07:00
davidpil2002
8098bc4bf5
Add Secure Boot Support (#12692)
- Why I did it
Add Secure Boot support to SONiC OS.
Secure Boot (SB) is a verification mechanism for ensuring that code launched by a computer's UEFI firmware is trusted. It is designed to protect a system against malicious code being loaded and executed early in the boot process before the operating system has been loaded.

- How I did it
Added a signing process to sign the following components:
shim, grub, Linux kernel, and kernel modules when doing the build, and when feature is enabled in build time according to the HLD explanations (the feature is disabled by default).

- How to verify it
There are self-verifications of each boot component when building the image, in addition, there is an existing end-to-end test in sonic-mgmt repo that checks that the boot succeeds when loading a secure system (details below).

How to build a sonic image with secure boot feature: (more description in HLD)

Required to use the following build flags from rules/config:
SECURE_UPGRADE_MODE="dev"
SECURE_UPGRADE_DEV_SIGNING_KEY="/path/to/private/key.pem"
SECURE_UPGRADE_DEV_SIGNING_CERT="/path/to/cert/key.pem"
After setting those flags should build the sonic-buildimage.
Before installing the image, should prepared the setup (switch device) with the follow:
check that the device support UEFI
stored pub keys in UEFI DB

enabled Secure Boot flag in UEFI
How to run a test that verify the Secure Boot flow:
The existing test "test_upgrade_path" under "sonic-mgmt/tests/upgrade_path/test_upgrade_path", is enough to validate proper boot
You need to specify the following arguments:
Base_image_list your_secure_image
Taget_image_list your_second_secure_image
Upgrade_type cold
And run the test, basically the test will install the base image given in the parameter and then upgrade to target image by doing cold reboot and validates all the services are up and working correctly
2023-03-14 14:55:22 +02:00
Andriy Dobush
c1dd94f368
Add California-SB237 feature. Requires to change default user password (#12678)
#### Why I did it
Add support of California-SB237 conformance.
https://github.com/sonic-net/SONiC/tree/master/doc/California-SB237

#### How I did it
Expire user passwords during build

#### How to verify it
Enable build flag and check if default user is prompted for a new password
2023-02-23 15:36:37 -08:00
Saikrishna Arcot
14012cfcf6
Add lsof and sysstat packages to the base system for debugging purposes (#13741)
The lsof and sysstat packages make determining what files/sockets a
program has open a bit easier. This helps if, for example, some
application has a file open that's been deleted from disk.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2023-02-10 04:22:44 +00:00
Oleksandr Ivantsiv
c7ecd92c54
Clear DNS configuration received from DHCP during networking reconfiguration in Linux. (#13516)
- Why I did it
fixes #12907

When the management interface IP address configuration changes from dynamic to static the DNS configuration (retrieved from the DHCP server) in /etc/resolv.conf remains uncleared. This leads to a DNS configuration pointing to the wrong nameserver. To make the behavior clear DNS configuration received from DHCP should be cleared.

- How I did it
Use resolvconf package for managing DNS configuration. It is capable of tracking the source of DNS configuration and puts the configuration retrieved from the DHCP servers into a separate file. This allows the implementation of DNS configuration cleanup retrieved from DHCP during networking reconfiguration.

- How to verify it
Ensure that the management interface has no static configuration.
Check that /etc/resolv.conf has DNS configuration.
Configure a static IP address on the management interface.
Verify that /etc/resolv.conf has no DNS configuration.
Remove the static IP address from the management interface.
Verify that /etc/resolv.conf has DNS configuration retrieved form DHCP server.
2023-01-30 22:13:10 +02:00
lixiaoyuner
bc7b35473e
Add k8s support feature set and Add platform label for scheduler usage (#12997)
Why I did it
We plan to pilot k8s feature, need to fix several bugs including enable telemetry feature and add platform label.

How I did it
Add support feature set, only enable telemetry container upgrade for now
Add platform label for scheduler usage
Remove CNI installation code, it would be auto installed when install kubeadm
How to verify it
After sonic device join k8s cluster, show node labels to check if platform label is visible.

Signed-off-by: Yun Li yunli1@microsoft.com
2023-01-10 07:56:44 -08:00
Liu Shilong
6e1df20104
[build] Add environment when using dpkg hook to avoid lock loop. (#13150)
Why I did it
In some cases, dpkg will call dpkg to validate version.
dpkg hook will get stuck in a loop to lock.

How I did it
Use an env variable to skip duplicated lock.
2023-01-05 13:16:49 +08:00
Konstantin Vasin
bf90b498a4
[Build] use pigz to speed up a build (#12825)
Why I did it

It's possible to speed up some parts of a build using parallel compression/decompression.
This is especially important for build_debian.sh.

How I did it
pigz is a parallel implementation of gzip: https://zlib.net/pigz/
Some programs like docker and mkinitramfs can automatically detect and use it instead of gzip.
For tar we need to select it directly.

To enable this feature you need to set GZ_COMPRESS_PROGRAM=pigz
2022-12-17 14:38:31 -08:00
Kalimuthu-Velappan
0dc22bd27c
05.Version cache - docker dpkg caching support (#12005)
This feature caches all the deb files during docker build and stores them
into version cache.

It loads the cache file if already exists in the version cache and copies the extracted
deb file from cache file into Debian cache path( /var/cache/apt/archives).

The apt-install always installs the deb file from the cache if exists, this
avoid unnecessary package download from the repo and speeds up the overall build.

The cache file is selected based on the SHA value of version dependency
files.

Why I did it
How I did it
How to verify it


* 03.Version-cache - framework environment settings

It defines and passes the necessary version cache environment variables
to the caching framework.

It adds the utils script for shared cache file access.

It also adds the post-cleanup logic for cleaning the unwanted files from
the docker/image after the version cache creation.

* 04.Version cache - debug framework

Added DBGOPT Make variable to enable the cache framework
scripts in trace mode. This option takes the part name of the script to
enable the particular shell script in trace mode.

Multiple shell script names can also be given.

	Eg: make DBGOPT="image|docker"

Added verbose mode to dump the version merge details during
build/dry-run mode.
	Eg: scripts/versions_manager.py freeze -v \
		'dryrun|cmod=docker-swss|cfile=versions-deb|cname=all|stage=sub|stage=add'

* 05.Version cache - docker dpkg caching support

This feature caches all the deb files during docker build and stores them
into version cache.

It loads the cache file if already exists in the version cache and copies the extracted
deb file from cache file into Debian cache path( /var/cache/apt/archives).

The apt-install always installs the deb file from the cache if exists, this
avoid unnecessary package download from the repo and speeds up the overall build.

The cache file is selected based on the SHA value of version dependency
files.
2022-12-12 09:20:56 +08:00
Konstantin Vasin
6448afd338
[Build] set apt Acquire::Retries to 3 for bullseye (#12758)
Why I did it
There were some changes in apt source code in version 2.1.9.
As a result apt used in bullseye (2.2.4) is intolerant to network issues.
This was fixed in 10631550f1 Already fixed version is used in bookworm (2.5.4)
And not yet affected version is used in buster (1.8.2.3)

How I did it
Set Acquire::Retries to 3 for sonic-slave-bullseye, docker-base-bullseye and final Debian image.

Ref: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1876035

Signed-off-by: Konstantin Vasin k.vasin@yadro.com
2022-11-21 08:05:16 +08:00
Hua Liu
4be5f1ed31
Revert SSHD config change. (#12729)
Revert SSHD config change.

#### Why I did it
Some test case and code may impact by SSHD config change.


#### How I did it
Revert following change in build_debian.sh script:
ClientAliveInterval change back to 900.
MaxAuthTries change back to default value.
Banner change to disabled.

#### How to verify it
Pass all E2E test case.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

#### Description for the changelog
Revert SSHD config change.

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
2022-11-19 15:30:34 +08:00
Saikrishna Arcot
672367c33e
Update Linux kernel from 5.10.103 to 5.10.140 (#12660)
Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2022-11-14 16:33:34 -08:00
Devesh Pathak
0ea4f4d00e
Clear /etc/resolv.conf before building image (#12592)
Why I did it
nameserver and domain entries from build system fsroot gets into sonic image.

How I did it
Clear /etc/resolv.conf before building image

How to verify it
Built image with it and verified with install that /etc/resolv.conf is empty
2022-11-09 16:54:56 -08:00
xumia
ac5d89c6ac
[Build] Support j2 template for debian sources (#12557)
Why I did it
Unify the Debian mirror sources
Make easy to upgrade to the next Debian release, not source url code change required.
Support to customize the Debian mirror sources during the build
Relative issue: #12523
2022-11-09 08:09:53 +08:00
Saikrishna Arcot
0dd4d5dda9
[openssh]: Restore behavior of ClientAliveCountMax=0 (#12549)
* [openssh]: Restore behavior of ClientAliveCountMax=0

OpenSSH 8.2 changed the behavior of ClientAliveCountMax=0 such that
setting it to 0 disables connection-killing entirely when the connection
is idle. Revert that change.

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

* Remove build-dep command that should not be there

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
2022-10-31 16:00:05 -07:00
Hua Liu
f8494d10ad
Improve SSHD config to use more secure settings (#12109)
Improve SSHD config to use more secure settings

#### Why I did it
According to Sonic OS review result, SSHD config file /etc/ssh/sshd_config using insecure settings.


#### How I did it
Change build_debian.sh script to set following settings to /etc/ssh/sshd_config:
ClientAliveInterval is set to 300
MaxAuthTries is set to default of 3
Banner set to /etc/issue
LogLevel is set to VERBOSE

#### How to verify it
Pass all E2E test case.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

#### Description for the changelog
Improve SSHD config to use more secure settings

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
2022-09-22 09:25:29 +08:00
Junchao-Mellanox
46292d71be
Add linux perf tool to sonic image (#11906) 2022-08-31 13:09:36 -07:00
Ying Xie
946bc3b969
[arping] install arping utility in the base image (#11791)
Signed-off-by: Ying Xie <ying.xie@microsoft.com>

Signed-off-by: Ying Xie <ying.xie@microsoft.com>
2022-08-20 18:09:04 -07:00
lixiaoyuner
8d6431e754
Add k8s master feature (#11637)
* Add k8s master feature

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Update kubernetes version mistake and make variable passing clear

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Add CRI-dockerd package

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Update version variable passing logic

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Upgrade the worker kubernetes version

Signed-off-by: Yun Li <yunli1@microsoft.com>

* Install xml file parse tool

Signed-off-by: Yun Li <yunli1@microsoft.com>

Signed-off-by: Yun Li <yunli1@microsoft.com>
2022-08-13 23:01:35 +08:00
Junchao-Mellanox
736c739bf4
Fix issue: rsyslog rate limit does not work on version 8.2110.0 (#11588)
#### Why I did it

The default stable version of rsyslog on bullseye has a bug about rate limit. It causes rate limit not work. The bug has been fixed on backport version 8.2206.0-1~bpo11+1.

Buster has no such issue.

#### How I did it

Upgrade rsyslog from 8.2110.0 to 8.2206.0-1~bpo11+1

#### How to verify it

Manual test
2022-08-04 15:10:34 -07:00
gregshpit
5df09490dc
Ported Marvell armhf build on amd64 host for debian buster to use cross-comp… (#8035)
* Ported Marvell armhf build on x86 for debian buster to use cross-compilation instead of qemu emulation

Current armhf Sonic build on amd64 host uses qemu emulation. Due to the
nature of the emulation it takes a very long time, about 22-24 hours to
complete the build. The change I did to reduce the building time by
porting Sonic armhf build on amd64 host for Marvell platform for debian
buster to use cross-compilation on arm64 host for armhf target. The
overall Sonic armhf building time using cross-compilation reduced to
about 6 hours.

Signed-off-by: marvell <marvell@cpss-build3.marvell.com>

* Fixed final Sonic image build with dockers inside

* Update Dockerfile.j2

Fixed qemu-user-static:x86_64-aarch64-5.0.0-2 .

* Update cross-build-arm-python-reqirements.sh

Added support for both armhf and arm64 cross-build platform using $PY_PLAT environment variable.

* Update Makefile

Added TARGET=<cross-target> for armhf/arm64 cross-compilation.

* Reviewer's @qiluo-msft requests done

Signed-off-by: marvell <marvell@cpss-build3.marvell.com>

* Added new radius/pam patch for arm64 support

* Update slave.mk

Added missing back tick.

* Added libgtest-dev: libgmock-dev: to the buster Dockerfile.j2. Fixed arm perl version to be generic

* Added missing armhf/arm64 entries in /etc/apt/sources.list

* fix libc-bin core dump issue from xumia:fix-libc-bin-install-issue commit

* Removed unnecessary 'apt-get update' from sonic-slave-buster/Dockerfile.j2

* Fixed saiarcot895 reviewer's requests

* Fixed README and replaced 'sed/awk' with patches

* Fixed ntp build to use openssl

* Unuse sonic-slave-buster/cross-build-arm-python-reqirements.sh script (put all prebuilt python packages cross-compilation/install inside Dockerfile.j2). Fixed src/snmpd/Makefile to use -j1 in all cases

* Clean armhf cross-compilation build fixes

* Ported cross-compilation armhf build to bullseye

* Additional change for bullseye

* Set CROSS_BUILD_ENVIRON default value n

* Removed python2 references

* Fixes after merge with the upstream

* Deleted unused sonic-slave-buster/cross-build-arm-python-reqirements.sh file

* Fixed 2 @saiarcot895 requests

* Fixed @saiarcot895 reviewer's requests

* Removed use of prebuilt python wheels

* Incorporated saiarcot895 CC/CXX and other simplification/generalization changes

Signed-off-by: marvell <marvell@cpss-build3.marvell.com>

* Fixed saiarcot895 reviewer's  additional requests

* src/libyang/patch/debian-packaging-files.patch

* Removed --no-deps option when installing wheels. Removed unnecessary lazy_object_proxy arm python3 package instalation

Co-authored-by: marvell <marvell@cpss-build3.marvell.com>
Co-authored-by: marvell <marvell@cpss-build2.marvell.com>
2022-07-21 14:15:16 -07:00
Alexander Allen
429254cb2d
[arm] Refactor installer and build to allow arm builds targeted at grub platforms (#11341)
Refactors the SONiC Installer to support greater flexibility in building for a given architecture and bootloader. 

#### Why I did it

Currently the SONiC installer assumes that if a platform is ARM based that it uses the `uboot` bootloader and uses the `grub` bootloader otherwise. This is not a correct assumption to make as ARM is not strictly tied to uboot and x86 is not strictly tied to grub. 

#### How I did it

To implement this I introduce the following changes:
* Remove the different arch folders from the `installer/` directory
* Merge the generic components of the ARM and x86 installer into `installer/installer.sh`
* Refactor x86 + grub specific functions into `installer/default_platform.conf` 
* Modify installer to call `default_platform.conf` file and also call `platform/[platform]/patform.conf` file as well to override as needed
* Update references to the installer in the `build_image.sh` script
* Add `TARGET_BOOTLOADER` variable that is by default `uboot` for ARM devices and `grub` for x86 unless overridden in `platform/[platform]/rules.mk`
* Update bootloader logic in `build_debian.sh` to be based on `TARGET_BOOTLOADER` instead of `TARGET_ARCH` and to reference the grub package in a generic manner

#### How to verify it

This has been tested on a ARM test platform as well as on Mellanox amd64 switches as well to ensure there was no impact. 

#### Description for the changelog
[arm] Refactor installer and build to allow arm builds targeted at grub platforms

#### Link to config_db schema for YANG module changes

N/A
2022-07-12 15:00:57 -07:00
xumia
fdef1f0342
[Build]: Support to use symbol links for lazy installation targets to reduce the image size (#10923)
Why I did it
Support to use symbol links in platform folder to reduce the image size.
The current solution is to copy each lazy installation targets (xxx.deb files) to each of the folders in the platform folder. The size will keep growing when more and more packages added in the platform folder. For cisco-8000 as an example, the size will be up to 2G, while most of them are duplicate packages in the platform folder.

How I did it
Create a new folder in platform/common, all the deb packages are copied to the folder, any other folders where use the packages are the symbol links to the common folder.

Why platform.tar?
We have implemented a patch for it, see #10775, but the problem is the the onie use really old unzip version, cannot support the symbol links.
The current solution is similar to the PR 10775, but make the platform folder into a tar package, which can be supported by onie. During the installation, the package.tar will be extracted to the original folder and removed.
2022-06-21 13:03:55 +08:00
Hua Liu
1dbfc66cef
[SSHD] Enable SSHD keepalive timeout feature (#11115)
#### Why I did it
SSHD keepalive timeout feature not enabled on sonic.

#### How I did it
Enable SSHD keepalive timeout feature by set ClientAliveCountMax to 1.

#### How to verify it
Pass All E2E test case.
Manually test with following steps:

1. Change config and restart sshd
2. Connect a ssh with -vvv option to show debug message
3. Get running ssh by command and stop it:

```
azureuser@liuh-dev-vm-02:~$ ps -auxww | grep vvv
azureus+ 1614153  0.0  0.0  12244  6004 pts/1S+   15:48   0:00 ssh admin@10.250.0.101 -vvv
azureus+ 1615570  0.0  0.0   8168  2424 pts/3S+   15:49   0:00 grep --color=auto vvv
azureuser@liuh-dev-vm-02:~$ kill -Stop 1614153
```

4. Check TCP status from server side with ss command:
https://man7.org/linux/man-pages/man8/ss.8.html

```
admin@vlab-01:~$ ss | grep -i ssh
tcp   ESTAB  0  010.250.0.101:ssh 10.250.0.1:58150
tcp   FIN-WAIT-2 0  010.250.0.101:ssh 10.250.0.1:58164
tcp   ESTAB  0  010.250.0.101:ssh 10.250.0.1:57978
```

FIN-WAIT-2 means server already terminate the connection and wait for client response:
https://kb.iu.edu/d/ajmi
.  FIN-WAIT-2  <-- <SEQ=300><ACK=101><CTL=ACK>  <-- CLOSE-WAIT

5. Check again later will show the session been complete closed:

```
admin@vlab-01:~$ ss | grep -i ssh
tcp   ESTAB  0  010.250.0.101:ssh 10.250.0.1:58150
tcp   ESTAB  0  010.250.0.101:ssh 10.250.0.1:57978
```
2022-06-15 22:27:07 -07:00
xumia
f0dfd398a6
Revert "Reduce image size for lazy installation packages (#10775)" (#10916)
This reverts commit 15cf9b0d70.
Why I did it
Revert the PR #10775, for it has impact on onie installation.
It is caused by the symbol links not supported in some of the onie unzip.
We will enable after fixing the issue, see #10914
2022-05-26 09:39:48 +08:00