Commit Graph

291 Commits

Author SHA1 Message Date
Stepan Blyshchak
1f7d9e2698
[docker_img_ctl.j2] make tmpfs mounts optional and add ability to run container by image id (#6439)
- Why I did it
I made the docker_img_ctl.j2 applicable for more dockers (including application extensions dockers) by adding an option not to mount tmpfs on /tmp/ and /var/tmp/. In some applications /tmp/ is a different docker volume which can't be tmpfs.
Also, I added and ability to pass REPO[:TAG]|[@digest]/IMAGE_ID instead of just REPO name.

- How I did it
Modified docker_img_ctl.j2 and docker makefiles.

- How to verify it
Run it on the switch.
2021-03-16 17:03:12 +02:00
Ze Gan
4068944202
[MACsec]: Set MACsec feature to be auto-start (#6678)
1. Add supervisord as the entrypoint of docker-macsec
2. Add wpa_supplicant conf into docker-macsec
3. Set the macsecmgrd as the critical_process
4. Configure supervisor to monitor macsecmgrd
5. Set macsec in the features list
6. Add config variable `INCLUDE_MACSEC`
7. Add macsec.service

**- How to verify it**

Change the `/etc/sonic/config_db.json` as follow
```
{
    "PORT": {
        "Ethernet0": {
            ...
            "macsec": "test"
         }
    }
    ...
    "MACSEC_PROFILE": {
        "test": {
            "priority": 64,
            "cipher_suite": "GCM-AES-128",
            "primary_cak": "0123456789ABCDEF0123456789ABCDEF",
            "primary_ckn": "6162636465666768696A6B6C6D6E6F707172737475767778797A303132333435",
            "policy": "security"
        }
    }
}
```
To execute `sudo config reload -y`, We should find the following new items were inserted in app_db of redis
```
127.0.0.1:6379> keys *MAC*
1) "MACSEC_EGRESS_SC_TABLE:Ethernet0:72152375678227538"
2) "MACSEC_PORT_TABLE:Ethernet0"
127.0.0.1:6379> hgetall "MACSEC_EGRESS_SC_TABLE:Ethernet0:72152375678227538"
1) "ssci"
2) ""
3) "encoding_an"
4) "0"
127.0.0.1:6379> hgetall "MACSEC_PORT_TABLE:Ethernet0"
 1) "enable"
 2) "false"
 3) "cipher_suite"
 4) "GCM-AES-128"
 5) "enable_protect"
 6) "true"
 7) "enable_encrypt"
 8) "true"
 9) "enable_replay_protect"
10) "false"
11) "replay_window"
12) "0"
```

Signed-off-by: Ze Gan <ganze718@gmail.com>
2021-02-23 13:22:45 -08:00
Joe LeVeque
d6f236cb4b
[slave.mk] Remove PYTHON_CLICK target (#6763)
We stopped building our own custom python-click package via #5549. This target should have been removed at that time but was missed.
2021-02-10 23:25:05 -08:00
Arun Saravanan Balachandran
3015de1dd0
[sonic-host-service] Move to sonic-host-services package (#6273)
- Why I did it

To move ‘sonic-host-service’ which is currently built as a separate package to ‘sonic-host-services' package. 

- How I did it

- Moved 'sonic-host-server' to 'src/sonic-host-services' and included it as part of the python3 wheel.
- Other files were moved to 'src/sonic-host-services-data' and included as part of the deb package.
- Changed build option ‘INCLUDE_HOST_SERVICE’ to ‘ENABLE_HOST_SERVICE_ON_START’ for enabling sonic-hostservice at boot-up by default.
2021-02-08 19:35:08 -08:00
Qi Luo
0e7287856c
[build]: stop prompt during build (#6585)
Some commands used during build will prompt user interactively, but this is not expected during build. Since most output is collected into log file, user could not see the prompt and feel the build process hangs.

- How I did it

Use mv command in non interactive mode
Redirect stdin to null if command output is collected into log file.
2021-01-28 02:21:38 -08:00
Guohan Lu
044efe72ca [build]: add _BUILD_ENV to specify env for dpkg-buildpackage
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2021-01-27 08:28:21 -08:00
lguohan
ab2ae41212
[build]: fix dpkg admindir corruption issue in parallel build (#6408)
Fix #119

when parallel build is enable, multiple dpkg-buildpackage
instances are running at the same time. /var/lib/dpkg is shared
by all instances and the /var/lib/dpkg/updates could be corrupted
and cause the build failure.

the fix is to use overlay fs to mount separate /var/lib/dpkg
for each dpkg-buildpackage instance so that they are not affecting
each other.

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2021-01-12 06:03:12 -08:00
guxianghong
c64052bb28
[Centec ARM64]Upgrade Centec syncd docker to buster and Enable Telemetry on ARM64 (#6386)
* Enable telemetry for ARM64 by default

* [Centec]Upgrade Centec syncd docker to buster; libjemalloc2 have been installed in docker-base-buster, remove libjemalloc1 from docker-syncd-centec's Dockerfile.j2

Co-authored-by: Gu Xianghong <xgu@centecnetworks.com>
2021-01-09 08:07:30 -08:00
Joe LeVeque
e52581e919
[PDDF] Build and install Python 3 package (#6286)
- Make PDDF code compliant with both Python 2 and Python 3
- Align code with PEP8 standards using autopep8
- Build and install both Python 2 and Python 3 PDDF packages
2021-01-07 10:03:29 -08:00
carl-nokia
f99dbff722
[Nokia]: Enable Telemetry for armhf and provide required qos files (#6364)
* [platform][Nokia]: Add buffers and qos files for config qos reload

   - providing required files

* [platform][armhf]: remove hardcoded disable for Telemetry on armhf

Co-authored-by: Carl Keene <keene@nokia.com>
2021-01-06 15:09:13 -08:00
Guohan Lu
ae2cb47091 [build]: add artifical dependency between libyang and frr
frr build requires libyang 1.0.184 which conflicts with
libyang 1.0.73. Solution here is to compile frr and libyang 1.0.184
first, and then uninstall libyang 1.0.184 after frr build.
Then, compile libyang 1.0.73 and all packages depend on it later.

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2021-01-02 12:45:32 -08:00
Guohan Lu
d365d0d99e [build]: check if package with given version is installed or not
in case conflicting packages have same name but different version,
the install step needs to wait till package with the conflicting
version is uninstalled

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2021-01-02 12:45:32 -08:00
lguohan
aa1cc848e2
[sonic-yang-mgmt-py2]: remove sonic-yang-mgmt py2 (#6262)
No longer needed as sonic-utilties has been moved python3

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-12-22 21:05:33 -08:00
Renuka Manavalan
ba02209141
First cut image update for kubernetes support. (#5421)
* First cut image update for kubernetes support.
With this,
    1)  dockers dhcp_relay, lldp, pmon, radv, snmp, telemetry are enabled
        for kube management
        init_cfg.json configure set_owner as kube for these

    2)  Each docker's start.sh updated to call container_startup.py to register going up
          As part of this call, it registers the current owner as local/kube and its version
          The images are built with its version ingrained into image during build

    3)  Update all docker's bash script to call 'container start/stop/wait' instead of 'docker start/stop/wait'.
         For all locally managed containers, it calls docker commands, hence no change for locally managed.
        
    4)  Introduced a new ctrmgrd service, that helps with transition between owners as  kube & local and carry over any labels update from STATE-DB to API server

    5)  hostcfgd updated to handle owner change

    6) Reboot scripts are updatd to tag kube running images as local, so upon reboot they run the same image.

   7) Added kube_commands.py to handle all updates with Kubernetes API serrver -- dedicated for k8s interaction only.
2020-12-22 08:01:33 -08:00
xumia
0a36de3a89
Recover "Support SONiC Reproduceable Build-debian/pip/web packages (#6255)
* Revert "Revert "Support SONiC Reproduceable Build-debian/pip/web packages (#5718)""

This reverts commit 17497a65e3.

* Revert "Revert "Remove unnecessary sudo authority in build Makefile (#6237)""

This reverts commit 163b7111b5.
2020-12-21 15:31:10 +08:00
Guohan Lu
17497a65e3 Revert "Support SONiC Reproduceable Build-debian/pip/web packages (#5718)"
This reverts commit 55a707586b.
2020-12-18 23:37:27 -08:00
xumia
55a707586b
Support SONiC Reproduceable Build-debian/pip/web packages (#5718)
* Support SONiC reproduceable build for deb/py2/py3/web

* Remove j2 files

* Fix bug

* Fix some issues

1. Change some code format issues
2. Fix curl calling wget command, pip2 calling pip3 issue
3. Fix wget/curl downloading multiple urls issue

* Fix some code format issue

* Fix bug

* Fix bug

* Fix command path hard code in build info scripts issue

* Add debian package sonic-build-tools

* Fix auto debian package removed issue

* Change build debian package name, and change the folder

* Collect the pre-versions and post-versions

* Change to use debian:buster

* Remove apt-mark and improve code

* Remove set_build_hooks

* Change docker trusted gpg files

* Fix docker build COPY directory name issue

* Move the trusted gpg files into the sonic-build-hooks package
2020-12-17 13:06:53 +08:00
Stephen Sun
e010d83fc3
[Dynamic buffer calc] Support dynamic buffer calculation (#6194)
**- Why I did it**
To support dynamic buffer calculation.
This PR also depends on the following PRs for sub modules
- [sonic-swss: [buffermgr/bufferorch] Support dynamic buffer calculation #1338](https://github.com/Azure/sonic-swss/pull/1338)
- [sonic-swss-common: Dynamic buffer calculation #361](https://github.com/Azure/sonic-swss-common/pull/361)
- [sonic-utilities: Support dynamic buffer calculation #973](https://github.com/Azure/sonic-utilities/pull/973)

**- How I did it**
1. Introduce field `buffer_model` in `DEVICE_METADATA|localhost` to represent which buffer model is running in the system currently:
    - `dynamic` for the dynamic buffer calculation model
    - `traditional` for the traditional model in which the `pg_profile_lookup.ini` is used
2. Add the tables required for the feature:
   - ASIC_TABLE in platform/\<vendor\>/asic_table.j2
   - PERIPHERAL_TABLE in platform/\<vendor\>/peripheral_table.j2
   - PORT_PERIPHERAL_TABLE on a per-platform basis in device/\<vendor\>/\<platform\>/port_peripheral_config.j2 for each platform with gearbox installed.
   - DEFAULT_LOSSLESS_BUFFER_PARAMETER and LOSSLESS_TRAFFIC_PATTERN in files/build_templates/buffers_config.j2
   - Add lossless PGs (3-4) for each port in files/build_templates/buffers_config.j2
3. Copy the newly introduced j2 files into the image and rendering them when the system starts
4. Update the CLI options for buffermgrd so that it can start with dynamic mode
5. Fetches the ASIC vendor name in orchagent:
   - fetch the vendor name when creates the docker and pass it as a docker environment variable
   - `buffermgrd` can use this passed-in variable
6. Clear buffer related tables from STATE_DB when swss docker starts
7. Update the src/sonic-config-engine/tests/sample_output/buffers-dell6100.json according to the buffer_config.j2
8. Remove buffer pool sizes for ingress pools and egress_lossy_pool
   Update the buffer settings for dynamic buffer calculation
2020-12-13 11:35:39 -08:00
lguohan
4812953468
[ntp]: build ntp with various fixes (#6037)
- NTP Bug 1970 (UNLINK_EXPR_SLIST empty list) Fix
- ENOBUFS log message level set to WARN
- Fix audit message seen on console apparmor
- add force-confold option when install ntp

Signed-off-by: Guohan Lu <lguohan@gmail.com>
Co-authored-by: Prabhu Sreenivasan <prabhu.sreenivasan@broadcom>
2020-12-02 15:02:50 -08:00
Joe LeVeque
7f4ab8fbd8
[sonic-utilities] Update submodule; Build and install as a Python 3 wheel (#5926)
Submodule updates include the following commits:

* src/sonic-utilities 9dc58ea...f9eb739 (18):
  > Remove unnecessary calls to str.encode() now that the package is Python 3; Fix deprecation warning (#1260)
  > [generate_dump] Ignoring file/directory not found Errors (#1201)
  > Fixed porstat rate and util issues (#1140)
  > fix error: interface counters is mismatch after warm-reboot (#1099)
  > Remove unnecessary calls to str.decode() now that the package is Python 3 (#1255)
  > [acl-loader] Make list sorting compliant with Python 3 (#1257)
  > Replace hard-coded fast-reboot with variable. And some typo corrections (#1254)
  > [configlet][portconfig] Remove calls to dict.has_key() which is not available in Python 3 (#1247)
  > Remove unnecessary conversions to list() and calls to dict.keys() (#1243)
  > Clean up LGTM alerts (#1239)
  > Add 'requests' as install dependency in setup.py (#1240)
  > Convert to Python 3 (#1128)
  > Fix mock SonicV2Connector in python3: use decode_responses mode so caller code will be the same as python2 (#1238)
  > [tests] Do not trim from PATH if we did not append to it; Clean up/fix shebangs in scripts (#1233)
  > Updates to bgp config and show commands with BGP_INTERNAL_NEIGHBOR table (#1224)
  > [cli]: NAT show commands newline issue after migrated to Python3 (#1204)
  > [doc]: Update Command-Reference.md (#1231)
  > Added 'import sys' in feature.py file (#1232)

* src/sonic-py-swsssdk 9d9f0c6...1664be9 (2):
  > Fix: no need to decode() after redis client scan, so it will work for both python2 and python3 (#96)
  > FieldValueMap `contains`(`in`)  will also work when migrated to libswsscommon(C++ with SWIG wrapper) (#94)

- Also fix Python 3-related issues:
    - Use integer (floor) division in config_samples.py (sonic-config-engine)
    - Replace print statement with print function in eeprom.py plugin for x86_64-kvm_x86_64-r0 platform
    - Update all platform plugins to be compatible with both Python 2 and Python 3
    - Remove shebangs from plugins files which are not intended to be executable
    - Replace tabs with spaces in Python plugin files and fix alignment, because Python 3 is more strict
    - Remove trailing whitespace from plugins files
2020-11-25 10:28:36 -08:00
fk410167
a3dd3f55f9
Platform Driver Developement Framework (PDDF) (#4756)
This change introduces PDDF which is described here: https://github.com/Azure/SONiC/pull/536

Most of the platform bring up effort goes in developing the platform device drivers, SONiC platform APIs and validating them. Typically each platform vendor writes their own drivers and platform APIs which is very tailor made to that platform. This involves writing code, building, installing it on the target platform devices and testing. Many of the details of the platform are hard coded into these drivers, from the HW spec. They go through this cycle repetitively till everything works fine, and is validated before upstreaming the code.
PDDF aims to make this platform driver and platform APIs development process much simpler by providing a data driven development framework. This is enabled by:

JSON descriptor files for platform data
Generic data-driven drivers for various devices
Generic SONiC platform APIs
Vendor specific extensions for customisation and extensibility

Signed-off-by: Fuzail Khan <fuzail.khan@broadcom.com>
2020-11-12 10:22:38 -08:00
Praveen Chaudhary
6156cb2805
[sonic-yang-mgmt] Build PY3 & PY2 packages (#5559)
Moving sonic-yang-mgmt to PY3 to support move of sonic-utilities to PY3.

Signed-off-by: Praveen Chaudhary<pchaudhary@linkedin.com>
2020-11-07 13:03:41 -08:00
Blueve
698b5544c9
[openssh] Introduce custom openssh-server package for supporting reverse console SSH (#5717)
* Build and install openssh from source
* Copy openssh deb package to dest folder
* Update make rule
* Update sonic debian extension
* Append empty line before EOF
* Update openssh patch
* Add openssh-server to base image dependency
* Fix indent type
* Fix comments
* Use commit id instead of tag id and add comment

Signed-off-by: Jing Kan jika@microsoft.com
2020-11-02 10:31:15 +08:00
Joe LeVeque
3a4435eb53
Add sonic-host-services and sonic-host-services-data packages (#5694)
**- Why I did it**

Install all host services and their data files in package format rather than file-by-file

**- How I did it**

- Create sonic-host-services Python wheel package, currently including procdockerstatsd
  - Also add the framework for unit tests by adding one simple procdockerstatsd test case
- Create sonic-host-services-data Debian package which is responsible for installing the related systemd unit files to control the services in the Python wheel. This package will also be responsible for installing any Jinja2 templates and other data files needed by the host services.
2020-10-23 09:52:29 -07:00
BrynXu
a2e3d2fcea
[ChassisDB]: bring up ChassisDB service (#5283)
bring up chassisdb service on sonic switch according to the design in
Distributed Forwarding in VoQ Arch HLD

Signed-off-by: Honggang Xu <hxu@arista.com>

**- Why I did it**
To bring up new ChassisDB service in sonic as designed in ['Distributed forwarding in a VOQ architecture HLD' ](90c1289eaf/doc/chassis/architecture.md). 

**- How I did it**
Implement the section 2.3.1 Global DB Organization of the VOQ architecture HLD.

**- How to verify it**
ChassisDB service won't start without chassisdb.conf file on the existing platforms.
ChassisDB service is accessible with global.conf file in the distributed arichitecture.

Signed-off-by: Honggang Xu <hxu@arista.com>
2020-10-14 15:15:24 -07:00
Junchao-Mellanox
1c97a03b81
[system-health] Add support for monitoring system health (#4835)
* system health first commit

* system health daemon first commit

* Finish healthd

* Changes due to lower layer logic change

* Get ASIC temperature from TEMPERATURE_INFO table

* Add system health make rule and service files

* fix bugs found during manual test

* Change make file to install system-health library to host

* Set system LED to blink on bootup time

* Caught exceptions in system health checker to make it more robust

* fix issue that fan/psu presence will always be true

* fix issue for external checker

* move system-health service to right after rc-local service

* Set system-health service start after database service

* Get system up time via /proc/uptime

* Provide more information in stat for CLI to use

* fix typo

* Set default category to External for external checker

* If external checker reported OK, save it to stat too

* Trim string for external checker output

* fix issue: PSU voltage check always return OK

* Add unit test cases for system health library

* Fix LGTM warnings

* fix demo comments: 1. get boot up timeout from monit configuration file; 2. set system led in library instead of daemon

* Remove boot_timeout configuration because it will get from monit config file

* Fix argument miss

* fix unit test failure

* fix issue: summary status is not correct

* Fix format issues found in code review

* rename th to threshold to make it clearer

* Fix review comment: 1. add a .dep file for system health; 2. deprecated daemon_base and uses sonic-py-common instead

* Fix unit test failure

* Fix LGTM alert

* Fix LGTM alert

* Fix review comments

* Fix review comment

* 1. Add relevant comments for system health; 2. rename external_checker to user_define_checker

* Ignore check for unknown service type

* Fix unit test issue

* Rename user define checker to user defined checker

* Rename user_define_checkers to user_defined_checkers for configuration file

* Renmae file user_define_checker.py -> user_defined_checker.py

* Fix typo

* Adjust import order for config.py

Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>

* Adjust import order for src/system-health/health_checker/hardware_checker.py

Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>

* Adjust import order for src/system-health/scripts/healthd

Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>

* Adjust import orders in src/system-health/tests/test_system_health.py

* Fix typo

* Add new line after import

* If system health configuration file not exist, healthd should exit

* Fix indent and enable pytest coverage

* Fix typo

* Fix typo

* Remove global logger and use log functions inherited from super class

* Change info level logger to notice level

Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>
2020-10-12 11:12:49 +03:00
Renuka Manavalan
48c089b9bb
Add Python3-swss-common package to image. (#5547)
Make Python3-swsscommon available in image.

```
admin@str-s6000-acs-13:~$ sudo apt-cache search swss
libswsscommon - This package contains Switch State Service common library.
python-swsscommon - This package contains Switch State Service common Python2 library.
python3-swsscommon- This package contains Switch State Service common Python3 library.
admin@str-s6000-acs-13:~$ 
```
2020-10-06 06:23:24 -07:00
Tamer Ahmed
110f7b7817 [cfggen] Build Python 2 And Python 3 Wheel Packages
This builds Python 2&3 wheel packages for sonic-cfggen script.

singed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
2020-09-30 07:07:43 -07:00
Volodymyr Boiko
d71a4efe3b
[sonic-platform-common] Install Python 3 package in host OS and PMon container (#5461)
Signed-off-by: Volodymyr Boyko <volodymyrx.boiko@intel.com>
2020-09-29 13:57:54 -07:00
Joe LeVeque
3987cbd80a
[sonic-utilities] Build and install as a Python wheel package (#5409)
We are moving toward building all Python packages for SONiC as wheel packages rather than Debian packages. This will also allow us to more easily transition to Python 3.

Python files are now packaged in "sonic-utilities" Pyhton wheel. Data files are now packaged in "sonic-utilities-data" Debian package.

**- How I did it**
- Build and install sonic-utilities as a Python package
- Remove explicit installation of wheel dependencies, as these will now get installed implicitly by pip when installing sonic-utilities as a wheel
- Build and install new sonic-utilities-data package to install data files required by sonic-utilities applications
- Update all references to sonic-utilities scripts/entrypoints to either reference the new /usr/local/bin/ location or remove absolute path entirely where applicable

Submodule updates:

* src/sonic-utilities aa27dd9...2244d7b (5):
  > Support building sonic-utilities as a Python wheel package instead of a Debian package (#1122)
  > [consutil] Display remote device name in show command (#1120)
  > [vrf] fix check state_db error when vrf moving (#1119)
  > [consutil] Fix issue where the ConfigDBConnector's reference is missing (#1117)
  > Update to make config load/reload backward compatible. (#1115)

* src/sonic-ztp dd025bc...911d622 (1):
  > Update paths to reflect new sonic-utilities install location, /usr/local/bin/ (#19)
2020-09-20 20:16:42 -07:00
Stepan Blyshchak
6de9390bb0
[build] Add a parameter to specify sonic version during build (#5278)
Introduced a new build parameter 'SONIC_IMAGE_VERSION' that allows build
system users to build SONiC image with a specific version string. If
'SONIC_IMAGE_VERSION' was not passed by the user, SONIC_IMAGE_VERSION will be
set to the output of functions.sh:sonic_get_version function.

Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
2020-09-16 10:47:26 -07:00
shi-su
339cfbf9af
Remove the configuration of synchronous mode from init_cfg.json (#5308)
Remove the configuration of synchronous mode from init_cfg.json
2020-09-10 01:26:10 -07:00
Samuel Angebault
7e2fa7d997
[build]: Add a user defined rules/config file not tracked by git (#5325)
This change introduce the rules/config.user file.
It gets loaded after rules/config as a mean to override default values.
If the configuration file does not exists this step is ignored.
Since this path is tracked by .gitignore, it provides a mean to change
the configuration of sonic build without dirtying the git tree.

Signed-off-by: Samuel Angebault <staphylo@arista.com>
2020-09-10 01:24:12 -07:00
Samuel Angebault
8bac4904b8
[build]: Allow user to silence variable dump during make (#5327)
Most of the devs know how they configured their build.
Instead of generating a huge header for every make operation, allow the
user to silence it via an environment variable.

SONIC_BUILD_QUIETER=y make ...

Also add SONIC_BUILDER_EXTRA_ENV variable which allows the user to
inject arbitrary options to the docker run cmdline for the builder.

SONIC_BUILDER_EXTRA_CMDLINE="-e test=var -e var='with spaces'" make ...
2020-09-10 01:21:47 -07:00
shi-su
f3feb56c8a
Add switch for synchronous mode (#5237)
Add a master switch so that the sync/async mode can be configured.
Example usage of the switch:
1.  Configure mode while building an image
    `make ENABLE_SYNCHRONOUS_MODE=y <target>`
2. Configure when the device is running 
    Change CONFIG_DB with `sonic-cfggen -a '{"DEVICE_METADATA":{"localhost": {"synchronous_mode": "enable"}}}' --write-to-db`
    Restart swss with `systemctl restart swss`
2020-08-24 14:04:10 -07:00
nirenjan
bb57ccecd4
[sonic-host-service]: Add SONiC Host Services infrastructure (#4840)
- Why I did it

When SONiC is configured with the management framework and/or telemetry services, the applications running inside those containers need to access some functionality on the host system. The following is a non-exhaustive list of such functionality:

Image management
Configuration save and load
ZTP enable/disable and status
Show tech support
- How I did it

The host service is a Python process that listens for requests via D-Bus. It will then service those requests and send a response back to the requestor.

This PR only introduces the host service infrastructure. Applications that need access to the host services must add applets that will register on D-Bus endpoints to service the appropriate functionality.

- How to verify it

- Description for the changelog

Add SONiC Host Service for container to execute select commands in host

Signed-off-by: Nirenjan Krishnan <Nirenjan.Krishnan@dell.com>
2020-08-21 15:34:14 -07:00
abdosi
74d8b4a6be
[caclmgrd] Add support for multi-ASIC platforms (#5022)
* Support for Control Plane ACL's for Multi-asic Platforms.
Following changes were done:
 1) Moved from using blocking listen() on Config DB to the select() model
 via python-swsscommon since we have to wait on event from multiple
 config db's
 2) Since  python-swsscommon is not available on host added libswsscommon and python-swsscommon
    and dependent packages in the base image (host enviroment)
 3) Made iptables programmed in all namespace using ip netns exec

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* Address Review Comments

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* Fix Review Comments

* Fix Comments

* Added Change for Multi-asic to have iptables
rules to accept internal docker tcp/udp traffic
needed for syslog and redis-tcp connection.

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* Fix Review Comments

* Added more comments on logic.

* Fixed all warning/errors reported by http://pep8online.com/
other than line > 80 characters.

* Fix Comment
Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* Verified with swsscommon package. Fix issue for single asic platforms.

* Moved to new python package

* Address Review Comments.

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* Address Review Comments.
2020-08-20 15:11:42 -07:00
Joe LeVeque
d370ba7ca2
[build] Clean up any .pc/ directories left by quilt (#5196)
`quilt` may leave behind a `.pc/` directory, which will show up in `git status`. Delete it if it is present as part of cleaning up after `quilt` operations.
2020-08-16 11:40:21 -07:00
lguohan
082c26a27d
[build]: combine feature and container feature table (#5081)
1. remove container feature table
2. do not generate feature entry if the feature is not included
   in the image
3. rename ENABLE_* to INCLUDE_* for better clarity
4. rename feature status to feature state
5. [submodule]: update sonic-utilities

* 9700e45 2020-08-03 | [show/config]: combine feature and container feature cli (#1015) (HEAD, origin/master, origin/HEAD) [lguohan]
* c9d3550 2020-08-03 | [tests]: fix drops_group_test failure on second run (#1023) [lguohan]
* dfaae69 2020-08-03 | [lldpshow]: Fix input device is not a TTY error (#1016) [Arun Saravanan Balachandran]
* 216688e 2020-08-02 | [tests]: rename sonic-utilitie-tests to tests (#1022) [lguohan]

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-08-05 13:23:12 -07:00
lguohan
8dfc824d0b
[build]: auto-detect bldenv in the sonic-slave docker (#5087)
use lsb_release -cs to get build env within the docker

for jessie:  jessie
for stretch: stretch
for buster:  buster

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-08-01 22:54:30 -07:00
lguohan
e1ac3cfc6a
[build]: wait for conflicts package to be uninstalled (#5039)
when parallel build is enabled, both docker-fpm-frr and docker-syncd-brcm
is built at the same time, docker-fpm-frr requires swss which requires to
install libsaivs-dev. docker-syncd-brcm requires syncd package which requires
to install libsaibcm-dev.

since libsaivs-dev and libsaibcm-dev install the sai header in the same
location, these two packages cannot be installed at the same time. Therefore,
we need to serialize the build between these two packages. Simply uninstall
the conflict package is not enough to solve this issue. The correct solution
is to have one package wait for another package to be uninstalled.

For example, if syncd is built first, then it will install libsaibcm-dev.
Meanwhile, if the swss build job starts and tries to install libsaivs-dev,
it will first try to query if libsaibcm-dev is installed or not. if it is
installed, then it will wait until libsaibcm-dev is uninstalled. After syncd
job is finished, it will uninstall libsaibcm-dev and swss build job will be
unblocked.

To solve this issue, _UNINSTALLS is introduced to uninstall a package that
is no longer needed and to allow blocked job to continue.

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-07-27 10:46:20 -07:00
Joe LeVeque
c0d1616f89
Introduce sonic-py-common package (#5003)
Consolidate common SONiC Python-language functionality into one shared package (sonic-py-common) and eliminate duplicate code.

The package currently includes three modules:

- daemon_base
- device_info
- logger
2020-07-26 23:15:41 -07:00
lguohan
58632e6e83 [docker-orchagent]: make build depends only on sairedis package (#4880)
make swss build depends only on libsairedis instead of syncd. This allows to build swss without depending
on vendor sai library.

Currently, libsairedis build also buils syncd which requires vendor SAI lib. This makes difficult to build
swss docker in buster while still keeping syncd docker in stretch, as swss requires libsairedis which also
build syncd and requires vendor to provide SAI for buster. As swss docker does not really contain syncd
binary, so it is not necessary to build syncd for swss docker.

* [submodule]: update sonic-sairedis

* ccbb3bc 2020-06-28 | add option to build without syncd (HEAD, origin/master, origin/HEAD) [Guohan Lu]
* 4247481 2020-06-28 | install saidiscovery into syncd package [Guohan Lu]
* 61b8e8e 2020-06-26 | Revert "sonic-sairedis: Add support to sonic-sairedis for gearbox phys (#624)" (#630) [Danny Allen]
* 85e543c 2020-06-26 | add a README to tests directory to describe how to run 'make check' (#629) [Syd Logan]
* 2772f15 2020-06-26 | sonic-sairedis: Add support to sonic-sairedis for gearbox phys (#624) [Syd Logan]

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-07-12 18:08:51 +00:00
Praveen Chaudhary
94d448e9bd
[slave.mk]: Adding support to specify debs dependencies for python-debs package. (#4849)
**- How I did it**
added below line:
$$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEBS_DEPENDS))) \

**- How to verify it**
Added below dependencies in sonic-utils
```
SONIC_UTILS = python-sonic-utilities_1.2-1_all.deb
$(SONIC_UTILS)_SRC_PATH = $(SRC_PATH)/sonic-utilities
$(SONIC_UTILS)_DEBS_DEPENDS = $(LIBYANG) $(LIBYANG_CPP) $(LIBYANG_PY2) \  <<<<<<<<<<<
                                     $(LIBYANG_PY3)
$(SONIC_UTILS)_WHEEL_DEPENDS = $(SONIC_CONFIG_ENGINE) $(SONIC_YANG_MGMT_PY)
SONIC_PYTHON_STDEB_DEBS += $(SONIC_UTILS)
```
Build the PKGs successfully.

Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
2020-06-26 11:32:35 -07:00
Eric Seifert
de07029352
Add TELEMETRY_WRITABLE build option to enable telemetry write mode (default read-only) (#4209)
Add build flag TELEMETRY_WRITABLE. When set to "y" it will add a go build flag in the telemetry build that will enable telemetry write mode to allow configuration via gNMI Set RPC as well as operations via the gNOI RPC's. The default for TELEMETRY_WRITABLE is unset in which case telemetry is read-only. In read-only mode the Set RPC and all gNOI RPC's are disabled and will return an "Unsupported" error when called.

authored-by: Eric Seifert <eric@seifert.casa>
2020-06-25 08:04:43 -07:00
xumia
e28cb8f58e
[Secure Boot] Support to sign swi image (#4627)
* [secure boot] Support to sign swi image

* Fix build issue

* fix tab format issue

* Fix typing issue

* Change the sign_image.sh command line

* Remove SONIC_CETIFICATE_PATH

* Fix bugs
2020-06-09 15:25:17 +08:00
Qi Luo
65e7a84509
[baseimage]: Build and install redis-dump-load Python 3 package in host image (#4661)
Fix #4656
2020-05-30 05:52:27 -07:00
Praveen Chaudhary
0ccdd70671
[sonic-yang-mgmt]: sonic-yang-mgmt package for configuration validation. (#3861)
**- What I did**

#### wheel package Makefiles

- wheel package Makefiles for sonic-yang-mgmt package.

#### libyang Python APIs:
- python APIs based on libyang
- functions to load/merge yang models and Yang data files
- function to validate data trees based on Yang models
- functions to merge yang data files/trees
- add/set/delete node in schema and data trees
- find data/schema nodes from xpath from the Yang data/schema tree in memory
- find dependencies
- dump the data tree in json/xml

#### Extension of libyang Python APIs:
-- Cropping input config based on Yang Model.
-- Translate input config based on Yang Model.
-- rev Translate input config based on Yang Model.
-- Find xpath of port, portleaf and a yang list.
-- Find if node is key of a list while deletion if yes, then delete the parent.

Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com
Signed-off-by: Ping Mao pmao@linkedin.com
2020-05-21 16:27:57 -07:00
rajendra-dendukuri
9c7105b5f3
Install swsssdk-py3 in the base Debian image for python3 based apps (#4542)
Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
2020-05-19 11:15:05 -07:00
lguohan
087e5fbf78
[build]: build missing stretch dockers when debug build is enabled (#4494)
$DOCKER_IMAGES_FOR_INSTALLERS contains all dockers to be included
in the image. If INSTALL_DEBUG_TOOLS=y, then it contains debug docker
images as well non-debug docker images when there is no debug version
of that docker image.

$STRETCH_DOCKER_IMAGES will have the non-debug docker images needed
by the image. $STRETCH_DBG_DOCKER_IMAGES should contain the debug
version of the docker images.

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-29 11:07:08 -07:00
joyas-joseph
fc55329b22
[mgmt-framework]: convert mgmt-framework to use buster docker (#4480)
This change adds support to build dockers using buster as base.
- Define docker-base-buster using docker-base-stretch as starting point
- Define docker-config-engine-buster using docker-config-engine-stretch as starting point.
- sonic-mgmt-framework docker is updated to build using buster as base

Signed-off-by: Joyas Joseph <joyas_joseph@dell.com>
2020-04-28 15:57:13 -07:00
Guohan Lu
67cc385e36 [baseimage]: remove ixgbe.ko as 4.19 has new ixgbe driver
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-17 04:51:51 +00:00
Guohan Lu
fc0d4d1bab [build]: setup correct debs files dependency for docker targets
Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-17 04:51:51 +00:00
Guohan Lu
01cb7934b0 [build]: add buster docker as the last step of the build proces
- build SONIC_STRETCH_DOCKERS in sonic-slave-stretch docker
- build image related module in sonic-slave-buster docker.
  This includes all kernels modules and some packages

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-04-16 10:26:18 +00:00
Praveen Chaudhary
a02255e2f4
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL. (#3730)
[sonic-yang-models]: First version of yang models for Port, VLan, Interface, PortChannel, loopback and ACL.

YANG models as per Guidelines.

Guideline doc: https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md

[sonic-yang-models/tests]: YANG model test code and JSON input for testing.

[sonic-yang-models/setup.py]: Build infra for yang models.

**- What I did**
Created Yang model for Sonic.
Tables:  PORT, VLAN, VLAN_INTERFACE, VLAN_MEMBER, ACL_RULE, ACL_TABLE, INTERFACE.

Created build infra files using which a new package (sonic-yang-models) can be build and can be deployed on sonic switches. Yang models will be part of this new package.

**- How I did it**
Wrote yang models based on Guideline doc: 
https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md
and 
https://github.com/Azure/SONiC/wiki/Configuration.

Wrote python wheel Package infra which runs test for these Yang models using a json files which consists configuration as per yang models. These configs are for negative tests, which means we want to test that most must condition, pattern and when condition works as expected.

**- How to verify it**
Build Logs and testing:
———————————————————————————————————

```
/sonic/src/sonic-yang-models /sonic
running test
running egg_info
writing top-level names to sonic_yang_models.egg-info/top_level.txt
writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt
writing sonic_yang_models.egg-info/PKG-INFO
reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
running bdist_wheel
running build
running build_py
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../libyang_1.0.73_amd64.deb ...
Unpacking libyang (1.0.73) over (1.0.73) ...
Setting up libyang (1.0.73) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Processing triggers for man-db (2.7.6.1-2) ...
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../libyang-cpp_1.0.73_amd64.deb ...
Unpacking libyang-cpp (1.0.73) over (1.0.73) ...
Setting up libyang-cpp (1.0.73) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
(Reading database ... 155852 files and directories currently installed.)
Preparing to unpack .../python3-yang_1.0.73_amd64.deb ...
Unpacking python3-yang (1.0.73) over (1.0.73) ...
Setting up python3-yang (1.0.73) ...
INFO:YANG-TEST:module: sonic-vlan is loaded successfully
ERROR:YANG-TEST:Could not get module: sonic-head
INFO:YANG-TEST:module: sonic-portchannel is loaded successfully
INFO:YANG-TEST:module: sonic-acl is loaded successfully
INFO:YANG-TEST:module: sonic-loopback-interface is loaded successfully
ERROR:YANG-TEST:Could not get module: sonic-port
INFO:YANG-TEST:module: sonic-interface is loaded successfully
INFO:YANG-TEST:
------------------- Test 1: Configure a member port in VLAN_MEMBER table which does not exist.---------------------
libyang[0]: Leafref "/sonic-port:sonic-port/sonic-port:PORT/sonic-port:PORT_LIST/sonic-port:port_name" of value "Ethernet156" points to a non
-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlan_name='Vlan100'][port='Ethernet156']/port)
INFO:YANG-TEST:Configure a member port in VLAN_MEMBER table which does not exist. Passed

INFO:YANG-TEST:
------------------- Test 2: Configure non-existing ACL_TABLE in ACL_RULE.---------------------
libyang[0]: Leafref "/sonic-acl:sonic-acl/sonic-acl:ACL_TABLE/sonic-acl:ACL_TABLE_LIST/sonic-acl:ACL_TABLE_NAME" of value "NOT-EXIST" points
to a non-existing leaf. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NOT-EXIST'][RULE_NAME='Rule_20']/ACL_TABLE_NAME)
INFO:YANG-TEST:Configure non-existing ACL_TABLE in ACL_RULE. Passed

INFO:YANG-TEST:
------------------- Test 3: Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU
LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/ICMPV6_CODE)
INFO:YANG-TEST:Configure IP_TYPE as ARP and ICMPV6_CODE in ACL_RULE. Passed
INFO:YANG-TEST:

INFO:YANG-TEST:
------------------- Test 4: Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPv6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RU
LE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_20']/SRC_IPV6)
INFO:YANG-TEST:Configure IP_TYPE as ipv4any and SRC_IPV6 in ACL_RULE. Passed

------------------- Test 5: Configure l4_src_port_range as 99999-99999 in ACL_RULE---------------------
libyang[0]: Value "99999-99999" does not satisfy the constraint "([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])-([0-9]{1,4}|[0-5][0-9]{4}|[6][0-4][0-9]{3}|[6][5][0-2][0-9]{2}|[6][5][3][0-5]{2}|[6][5][3][6][0-5])" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6'][RULE_NAME='Rule_20']/L4_SRC_PORT_RANGE)
INFO:YANG-TEST:Configure l4_src_port_range as 99999-99999 in ACL_RULE Passed

INFO:YANG-TEST:
------------------- Test 6: Configure empty string as ip-prefix in INTERFACE table.---------------------
libyang[0]: Invalid value "" in "ip-prefix" element. (path: /sonic-interface:sonic-interface/INTERFACE/INTERFACE_LIST[interface='Ethernet8'][ip-prefix='']/ip-prefix)
INFO:YANG-TEST:Configure empty string as ip-prefix in INTERFACE table. Passed

INFO:YANG-TEST:
------------------- Test 7: Configure Wrong family with ip-prefix for VLAN_Interface Table---------------------
libyang[0]: Must condition "(contains(../ip-prefix, ':') and current()='IPv6') or                               (contains(../ip-prefix, '.') and current()='IPv4')" not satisfied. (path: /sonic-vlan:sonic-vlan/VLAN_INTERFACE/VLAN_INTERFACE_LIST[vlanid='100'][ip-prefix='2a04:5555:66:7777::1/64']/family)
INFO:YANG-TEST:Configure Wrong family with ip-prefix for VLAN_Interface Table Passed

INFO:YANG-TEST:
------------------- Test 8: Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE.---------------------
libyang[0]: When condition "boolean(IP_TYPE[.='ANY' or .='IP' or .='IPV6' or .='IPV6ANY'])" not satisfied. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NS
W-PACL-V6'][RULE_NAME='Rule_20']/DST_IPV6)
INFO:YANG-TEST:Configure IP_TYPE as ARP and DST_IPV6 in ACL_RULE. Passed

INFO:YANG-TEST:
------------------- Test 9: Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE.---------------------
libyang[0]: Value "0x080C" does not satisfy the constraint "(0x88CC|0x8100|0x8915|0x0806|0x0800|0x86DD|0x8847)" (range, length, or pattern). (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V4'][RULE_NAME='Rule_40']/INNER_ETHER_TYPE)
INFO:YANG-TEST:Configure INNER_ETHER_TYPE as 0x080C in ACL_RULE. Passed

INFO:YANG-TEST:
------------------- Test 10: Add dhcp_server which is not in correct ip-prefix format.---------------------
libyang[0]: Invalid value "10.186.72.566" in "dhcp_servers" element. (path: /sonic-vlan:sonic-vlan/VLAN/VLAN_LIST/dhcp_servers[.='10.186.72.566'])
INFO:YANG-TEST:Add dhcp_server which is not in correct ip-prefix format. Passed

INFO:YANG-TEST:
------------------- Test 11: Configure undefined acl_table_type in ACL_TABLE table.---------------------
libyang[0]: Invalid value "LAYER3V4" in "type" element. (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='NO-NSW-PACL-V6']/type)
INFO:YANG-TEST:Configure undefined acl_table_type in ACL_TABLE table. Passed

INFO:YANG-TEST:
------------------- Test 12: Configure undefined packet_action in ACL_RULE table.---------------------
libyang[0]: Invalid value "SEND" in "PACKET_ACTION" element. (path: /sonic-acl:sonic-acl/ACL_RULE/ACL_RULE_LIST/PACKET_ACTION)
INFO:YANG-TEST:Configure undefined packet_action in ACL_RULE table. Passed

INFO:YANG-TEST:
------------------- Test 13: Configure wrong value for tagging_mode.---------------------
libyang[0]: Invalid value "non-tagged" in "tagging_mode" element. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST/tagging_mode)
INFO:YANG-TEST:Configure wrong value for tagging_mode. Passed

INFO:YANG-TEST:
------------------- Test 14: Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN  table.---------------------
libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid)
libyang[0]: Leafref "../../../VLAN/VLAN_LIST/vlanid" of value "200" points to a non-existing leaf. (path: /sonic-vlan:sonic-vlan/VLAN_MEMBER/VLAN_MEMBER_LIST[vlanid='200'][port='Ethernet0']/vlanid)
INFO:YANG-TEST:Configure vlan-id in VLAN_MEMBER table which does not exist in VLAN  table. Passed

INFO:YANG-TEST:All Test Passed
../../target/debs/stretch/libyang0.16_0.16.105-1_amd64.deb installtion failed
../../target/debs/stretch/libyang-cpp0.16_0.16.105-1_amd64.deb installtion failed
../../target/debs/stretch/python2-yang_0.16.105-1_amd64.deb installtion failed
YANG Tests passed
Passed: pyang -f tree ./yang-models/*.yang > ./yang-models/sonic_yang_tree
copying tests/yangModelTesting.py -> build/lib/tests
copying tests/test_sonic_yang_models.py -> build/lib/tests
copying tests/__init__.py -> build/lib/tests
running egg_info
writing top-level names to sonic_yang_models.egg-info/top_level.txt
writing dependency_links to sonic_yang_models.egg-info/dependency_links.txt
writing sonic_yang_models.egg-info/PKG-INFO
reading manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_models.egg-info/SOURCES.txt'
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/yangModelTesting.py -> build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/test_sonic_yang_models.py -> build/bdist.linux-x86_64/wheel/tests
copying build/lib/tests/__init__.py -> build/bdist.linux-x86_64/wheel/tests
running install_data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data
creating build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-head.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-acl.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-loopback-interface.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-port.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-portchannel.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
copying ./yang-models/sonic-vlan.yang -> build/bdist.linux-x86_64/wheel/sonic_yang_models-1.0.data/data/yang-models
```
2020-04-14 15:36:02 -07:00
Renuka Manavalan
f128153706
[baseimage]: Install Kubernetes packages if enabled in image (#4374)
* Install kubernetes worker node packages, if enabled.

* Minor updates

* Added some comments

* Updates per review comments.
Built a private image to test to work fine.

* Remove the removed file.

* Update per comments
Make a fix, as kubeadm no demands a higher version of kubelet & kubectl.
As kubeadm auto install kubectl & kubelet, removing explicit install is an easier/robust fix.

* Changes per review comments.

* Updates per comments.
1) Dropped helper & pod scripts
2) Made install verbose

* Drop creation of pods subdir, as this PR does not use them.

* From comments to 'n' per review comments.

* 1) kubeadm.conf is created as part of kubeadm package install. Hence dropped explicit copy.
2020-04-13 08:41:18 -07:00
Renuka Manavalan
768d18d45b
Pass image_type to build_debian.sh, so it can do any required customizations. (#4401) 2020-04-08 14:24:08 -07:00
SuvarnaMeenakshi
4b8067e913
Multi-ASIC implementation (#3888)
Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.
2020-03-31 10:06:19 -07:00
lguohan
760e763935
[build]: allow to specify timestamp format in the build log (#4311)
only simple/none are supported currently

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-03-23 09:45:43 -07:00
lguohan
20260ceb1d
[build]: add SONIC_CONFIG_BUILD_LOG_TIMESTAMP to add timestamp in build log (#4269)
add timestamp in each job build log

example:

   [01:39:21] dh clean  --with autotools-dev
   [01:39:22]    dh_auto_clean
   [01:39:27]      make -j16 distclean

Signed-off-by: Guohan Lu <lguohan@gmail.com>
2020-03-21 14:21:26 -07:00
Kalimuthu-Velappan
7d2ebf8116
[build]: support for DPKG local caching (#4117)
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.

The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.

This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.

- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=

    where  SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
                            none:     no caching
                            cache:    cache from local directory
Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.

    For example: dependency files for Linux Kernel - src/sonic-linux-kernel,

            SPATH       := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
            DEP_FILES   := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
            DEP_FILES   += $(SONIC_COMMON_BASE_FILES_LIST)
            SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))

            DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
                         $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)

            $(LINUX_HEADERS_COMMON)_CACHE_MODE  := GIT_CONTENT_SHA
            $(LINUX_HEADERS_COMMON)_DEP_FLAGS   := $(DEP_FLAGS)
            $(LINUX_HEADERS_COMMON)_DEP_FILES   := $(DEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)
Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format

    FORMAT:
            <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
            Eg:
              linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz

            < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
            < 24-byte MOD SHA value > - the SHA value is derived from either of the following.
                    GIT_COMMIT_SHA  - SHA value of the last git commit ID if it is a submodule
                    GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.
Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.

            $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
                    $(call dpkg_depend,$(DEBS_PATH)/%.dep )
            $(HEADER)


            # Load the target deb from DPKG cache
            $(call LOAD_CACHE,$*,$@)


            # Skip building the target if it is already loaded from cache
            if [ -z '$($*_CACHE_LOADED)' ] ; then

                  .....
                 # Rules for Generating the target DEB file.
                  .....

                  # Save the target deb into DPKG cache
                  $(call SAVE_CACHE,$*,$@)
            fi


            $(FOOTER)


    The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.

    Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
    The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
    It also updates the target-specific variable to indicate the cache is loaded or not.
    The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.
- How to verify it

    The caching functionality is verified by enabling it in Linux kernel submodule.
    It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.
- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

DOCUMENT PR:

           https://github.com/Azure/SONiC/pull/559
2020-03-11 20:04:52 -07:00
abdosi
f5ab24d8f1
Made Change to have Configurable option to enable/disable docker image (#4171)
* Made Change to have Configurable option to enable/disable docker image

* [Submodule-Update] sonic-utilities to latest master
2020-02-21 10:20:07 -08:00
SuvarnaMeenakshi
c9483796dc [baseimage]: support building multi-asic component (#3856)
- move single instance services into their own folder
- generate Systemd templates for any multi-instance service files in slave.mk
- detect single or multi-instance platform in systemd-sonic-generator based on asic.conf platform specific file.
- update container hostname after creation instead of during creation (docker_image_ctl)
- run Docker containers in a network namespace if specified
- add a service to create a simulated multi-ASIC topology on the virtual switch platform

Signed-off-by: Lawrence Lee <t-lale@microsoft.com>
Signed-off-by: Suvarna Meenakshi <Suvarna.Meenaksh@microsoft.com>
2020-01-26 13:56:42 -08:00
Sumukha Tumkur Vani
ebb07d9942 [docker-restapi]: Avoid building RestAPI docker by default (#4018)
Provide build option for RESTAPI docker
2020-01-14 17:13:45 -08:00
Stepan Blyshchak
e45ff4edb4 [build] make debian package install noninteractive (#3980)
Otherwise, build process gets stuck because isc-dhcp-relay gives promt:

```
$ make SONIC_PROFILING_ON=y INSTALL_DEBUG_TOOLS=y SONIC_BUILD_JOBS=12
target/debs/stretch/isc-dhcp-relay_4.3.5-2_amd64.deb-install
+++ --- Making
target/debs/stretch/isc-dhcp-relay_4.3.5-2_amd64.deb-install --- +++
EXTRA_JESSIE_TARGETS=isc-dhcp-relay_4.3.5-2_amd64.deb-install make -f
Makefile.work jessie
make[1]: Entering directory `/build2/stepanb/hitless/sonic-buildimage'
SONiC Build System

Build Configuration
"CONFIGURED_PLATFORM"             : "mellanox"
"CONFIGURED_ARCH"                 : "amd64"
"SONIC_CONFIG_PRINT_DEPENDENCIES" : ""
"SONIC_BUILD_JOBS"                : "12"
"SONIC_CONFIG_MAKE_JOBS"          : "12"
"SONIC_USE_DOCKER_BUILDKIT"       : ""
"USERNAME"                        : "admin"
"PASSWORD"                        : "YourPaSsWoRd"
"ENABLE_DHCP_GRAPH_SERVICE"       : ""
"SHUTDOWN_BGP_ON_START"           : ""
"ENABLE_PFCWD_ON_START"           : ""
"INSTALL_DEBUG_TOOLS"             : ""
"ROUTING_STACK"                   : "frr"
"FRR_USER_UID"                    : "300"
"FRR_USER_GID"                    : "300"
"ENABLE_SYNCD_RPC"                : ""
"ENABLE_ORGANIZATION_EXTENSIONS"  : "y"
"HTTP_PROXY"                      : ""
"HTTPS_PROXY"                     : ""
"ENABLE_SYSTEM_TELEMETRY"         : "y"
"ENABLE_ZTP"                      : ""
"SONIC_DEBUGGING_ON"              : ""
"SONIC_PROFILING_ON"              : ""
"KERNEL_PROCURE_METHOD"           : "build"
"BUILD_TIMESTAMP"                 : "20200106.124851"
"BLDENV"                          : ""
"VS_PREPARE_MEM"                  : "yes"
"ENABLE_SFLOW"                    : "y"

make: Nothing to be done for 'jessie'.
make[1]: Leaving directory `/build2/stepanb/hitless/sonic-buildimage'
BLDENV=stretch make -f Makefile.work
target/debs/stretch/isc-dhcp-relay_4.3.5-2_amd64.deb-install
make[1]: Entering directory `/build2/stepanb/hitless/sonic-buildimage'
SONiC Build System

Build Configuration
"CONFIGURED_PLATFORM"             : "mellanox"
"CONFIGURED_ARCH"                 : "amd64"
"SONIC_CONFIG_PRINT_DEPENDENCIES" : ""
"SONIC_BUILD_JOBS"                : "12"
"SONIC_CONFIG_MAKE_JOBS"          : "12"
"SONIC_USE_DOCKER_BUILDKIT"       : ""
"USERNAME"                        : "admin"
"PASSWORD"                        : "YourPaSsWoRd"
"ENABLE_DHCP_GRAPH_SERVICE"       : ""
"SHUTDOWN_BGP_ON_START"           : ""
"ENABLE_PFCWD_ON_START"           : ""
"INSTALL_DEBUG_TOOLS"             : ""
"ROUTING_STACK"                   : "frr"
"FRR_USER_UID"                    : "300"
"FRR_USER_GID"                    : "300"
"ENABLE_SYNCD_RPC"                : ""
"ENABLE_ORGANIZATION_EXTENSIONS"  : "y"
"HTTP_PROXY"                      : ""
"HTTPS_PROXY"                     : ""
"ENABLE_SYSTEM_TELEMETRY"         : "y"
"ENABLE_ZTP"                      : ""
"SONIC_DEBUGGING_ON"              : ""
"SONIC_PROFILING_ON"              : ""
"KERNEL_PROCURE_METHOD"           : "build"
"BUILD_TIMESTAMP"                 : "20200106.124855"
"BLDENV"                          : "stretch"
"VS_PREPARE_MEM"                  : "yes"
"ENABLE_SFLOW"                    : "y"

[ 01 ] [ target/debs/stretch/isc-dhcp-relay_4.3.5-2_amd64.deb-install ]
Servers the DHCP relay should forward requests to:
Interfaces the DHCP relay should listen on:
```

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2020-01-10 12:55:29 -08:00
Joe LeVeque
24a0c46464
[monit] Build from source and patch to use MemAvailable value if available on system (#3875) 2019-12-30 18:25:57 -08:00
Stepan Blyshchak
baea7e2a24 [slave.mk] make curl follow HTTP redirects (#3947)
Fix an issue with SONIC_ONLINE_DEBS with curl by instructing curl to follow http redirects

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2019-12-27 09:26:51 -08:00
Stepan Blyshchak
183c945ab1 [slave.mk] use curl instead of wget (#3939)
wget creates empty file on failure which makes subsequent make runs
think that file is already there and won't try to download it again.

e.g.:

$ make target/files/stretch/fw-SPC-rel-13_2000_2602-EVB.mfa
...
Fails to download
...
$ ls target/files/stretch/fw-SPC-rel-13_2000_2602-EVB.mfa
target/files/stretch/fw-SPC-rel-13_2000_2602-EVB.mfa
$ make target/files/stretch/fw-SPC-rel-13_2000_2602-EVB.mfa
make: `target/files/stretch/fw-SPC-rel-13_2000_2602-EVB.mfa' is up to
date.

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2019-12-23 10:34:22 -08:00
Stepan Blyshchak
4ba0ff25d2 [services] make snmp.timer work again and delay telemetry.service (#3742)
Delay CPU intensive services at boot

- How I did it
Made snmp.timer work and add telemetry.timer.
But this is not enough because it breaks the existing snmp dependency on swss.
So, in this solution snmp timer is a wanted by swss service, but since OnBootSec timer expires only once it will not trigger snmp service, so I added line "OnUnitActiveSec=0 sec" which will start snmp service based on the last time it was active. On boot only OnBootSec will expire, on swss start/restarts only second timer will expire immediately and trigger snmp service.
However, snmp service will not stop after "systemctl stop snmp" because of the second timer which will always expire when snmp service because unavailable.
So there is a conflict which will be handled by systemd if we add "Conflicts=" line to both snmp.service and snmp.timer.

So during boot:

snmp does not start by default
swss starts and starts snmp timer
OnUnitActiveSec=0 does not expire since there is no snmp active
OnBootSec expires and starts snmp service and snmp timer gets stopped
During "systemctl restart swss"

snmp stops because of Requisite on swss
snmp unblocks snmp timer from running
swss starts and starts snmp timer
OnUnitActiveSec=0 expires imidiately and start snmp which stops snmp timer
During "systemctl stop snmp"

stop of snmp service unblocks snmp timer but no one starts the timer so it is not started by "OnUnitActiveSec=0"
2019-12-16 09:07:05 -08:00
rajendra-dendukuri
eec594adf2 [sonic-ztp]: Build sonic-ztp package (#3299)
* Build sonic-ztp package

- Add changes in make rules to conditionally include sonic-ztp package

Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
2019-12-04 04:50:56 -08:00
Olivier Singla
c70d8bca9f [baseimage]: kdump support (#3722)
* In the event of a kernel crash, we need to gather as much information
as possible to understand and identify the root cause of the crash.
Currently, the kernel does not provide much information, which make
kernel crash investigation difficult and time consuming.

Fortunately, there is a way in the kernel to provide more information
in the case of a kernel crash. kdump is a feature of the Linux kernel
that creates crash dumps in the event of a kernel crash. This PR
will add kermel kdump support.

An extension to the CLI utilities config and show is provided to
configure and manage kdump:
 - enable / disable kdump functionality
 - configure kdump (how many kernel crash logs can be saved, memory
   allocated for capture kernel)
 - view kernel crash logs
2019-11-08 23:08:42 -08:00
Stephen Sun
c34a4783e0 [build] install new platform api on host (#3282)
slave.mk: add SONIC_PLATFORM_API_PY2 as dependency of host
sonic_debian_extension.j2: install sonic_daemon_base and Mellanox-specific sonic_platform on host
mlnx-platform-api.mk: export mlnx_platform_api_py2_wheel_path for sonic_debian_extension.j2
sonic-daemon-base.mk: export daemon_base_py2_wheel_path for sonic_debian_extension.j2
daemon_base.py: hind unnecessary dependency of swss_common on host
2019-09-25 11:00:24 -07:00
padmanarayana
75104bb35d [sflow]: Build infrastructure changes to support sflow docker and utilities (#3251)
Introduce a new "sflow" container (if ENABLE_SFLOW is set). The new docker will include:
hsflowd : host-sflow based daemon is the sFlow agent
psample : Built from libpsample repository. Useful in debugging sampled packets/groups.
sflowtool : Locally dump sflow samples (e.g. with a in-unit collector)

In case of SONiC-VS, enable psample & act_sample kernel modules.

VS' syncd needs iproute2=4.20.0-2~bpo9+1 & libcap2-bin=1:2.25-1 to support tc-sample

tc-syncd is provided as a convenience tool for debugging (e.g. tc-syncd filter show ...)
2019-09-14 20:27:09 -07:00
Renuka Manavalan
14458b79ba
Create the src archive in target/ when INSTALL_DEBUG_TOOLS=y (#3323)
* Create the src archive in target/ when INSTALL_DEBUG_TOOLS=y

* Dropped -v (verbose flag) from tar command
2019-08-28 09:29:48 -07:00
Greg Paussa
f65c7969b6 [build] Fix cleaning SONIC_PYTHON_STDEB_DEBS (#3343)
* slave.mk
* .gitignore

The SONIC_PYTHON_STDEB_DEBS group is part of SONIC_CLEAN_DEBS
for cleaning with 'make clean', but this group is overlooked
since they reside in PYTHON_DEBS_PATH, not DEBS_PATH. This
fix creates a separate clean rule for SONIC_PYTHON_STDEB_DEBS.

Also noticed the .arch file showing up on 'git status' so
added it to .gitignore.

Signed-off-by: Greg Paussa <greg.paussa@broadcom.com>
2019-08-15 00:34:34 -07:00
arheneus@marvell.com
8de26b7bb9 [Makefile/slave docker] ARM arch doesn't support few packages (#3273)
* [Makefile/slave docker] ARM arch doesn't support few packages
  iproute2 is missing for ARM sonic-slave docker

Signed-off-by: Antony Rheneus <arheneus@marvell.com>
2019-08-06 21:33:14 -07:00
arheneus@marvell.com
d80d3d60db [Makefile/Telemtry] Removing Telemetry build for ARM arch as qemu (#3287)
crashes.

Raised go lang support
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/golang-nuts/1txPOGa4aGc

Also raised qemu bug for support as golang rejected it.
https://bugs.launchpad.net/qemu/+bug/1838946

In parallel, debugging issue to resolve it

Signed-off-by: Antony Rheneus <arheneus@marvell.com>
2019-08-06 07:38:52 -07:00
arheneus@marvell.com
e9d3858633 [build] Fixes for multi arch in Dockerfile (#3237)
Fix for rules/config as it is overridden by Make infra

Signed-off-by: Antony Rheneus <arheneus@marvell.com>
2019-07-31 15:26:00 -07:00
arheneus@marvell.com
50fe458592 [build]: SONiC buildimage ARM arch support (#2980)
ARM Architecture support in SONIC

make configure platform=[ASIC_VENDOR_ARCH] PLATFORM_ARCH=[ARM_ARCH]
SONIC_ARCH: default amd64
armhf - arm32bit
arm64 - arm64bit

Signed-off-by: Antony Rheneus <arheneus@marvell.com>
2019-07-25 22:06:41 -07:00
rajendra-dendukuri
40c8bc14cd [baseimage]: Upgrade ifupdown2 to version 1.2.8 (#3180)
* Upgrade ifupdown2 to version 1.2.8

Required by ZTP to support ZTP over IPv6 transport

Signed-off-by: Rajendra Dendukuri <rajendra.dendukuri@broadcom.com>
2019-07-19 23:09:14 -07:00
Lawrence Lee
34b841e7dd [slave.mk]: remove updategraph.service generation in slave.mk (#3153) 2019-07-17 12:13:35 -07:00
Renuka Manavalan
76bf5a0bc4 [build]: Added debug symbols to many debug dockers. (#3098)
* Added debug symbols to many debug dockers.

* For debug images *only*:
1) Archive source files into debug image
2) Archived source is copied into /src
3) Created an empty dir /debug
4) Mount both /src as ro & /debug as rw into every docker
5) Login banner will give some details on /src & /debug
6) Devs can copy core file into /debug and view it from inside a container.
7) Dev may create all gdb logs and other data directly into /debug.

* Dropped redundant REDIS_TOOLS per review comments.

* Added debug symbols to frr package and hence FRR based BGP docker.

* 1) Moved dbg_files.sh to scripts/
2) Src directories to archive are now collected from individual Makefiles.
3) Added few more debug symbols
4) Added few more debug dockers.

Here after no more changes except per review comments.

To debug:
Install required version of debug image in Switch or VM.
Copy core file into /debug of host
Get into Docker
gdb /usr/bin/<daemon> -c /debug/<your core file>
set directory /src/... <-- inside gdb to get the source

For non-in-depth debugging:

Download corresponding debug Docker image (docker-...-dbg.gz) to your VM
Load the image
Run image with entrypoint as 'bash' with dir containing core mapped in.
Run gdb on the core.
2019-07-03 22:13:55 -07:00
lguohan
40330bcbdf
[build]: wait up to 60 seconds for docker engine to start (#3067)
* [build]: wait 60 seconds for docker engine to start

On some platforms, it can take more than 1 second for docker
engine to start.

Signed-off-by: Guohan Lu <gulv@microsoft.com>
2019-06-22 14:40:05 -07:00
Stepan Blyshchak
1bcd54d38c [build]: Upgrade docker to 18.09 in stretch slave (#3049)
Also add an option to use docker buildkit for image build

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2019-06-20 09:19:33 -07:00
Renuka Manavalan
cdca062693 [build]: Build sonic-broadcom.bin using debug dockers for all stretch based dockers (#2833)
* Updated Makefile infrastructure to build debug images.
As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target.

Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image.

NOTE: If you don't specify NOSTRETcH=1, it implicitly calls "make stretch", which builds all stretch targets and that would include debug dockers too.

This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside.

"docker run -it --entrypoint=/bin/bash e47a8fb8ed38"

You may map the core file path to this docker run.

* Dropped the regular binary using DBG_PACKAGES and a small name change to help readability.

* Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y.

When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag.

Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior.

* 1) slave.mk - Dropped unused Docker build args
2) Debug template builder: renamed build_dbg_j2.sh to build_debug_docker_j2.sh
3) Dropped insignifcant statement CMD from debug Docker file, as base docker has Entrypoint.

* Reverted some changes, per review comments.
"User, uid, guid, frr-uid & frr-guid" are required for all docker images, with exception of debug images.

* Get in sync with the new update that filters out dockers to be built (SONIC_STRETCH_DOCKERS_FOR_INSTALLERS) and build debug-dockers only for those to be built and debug target is available.

* Mkae a template for each target that can be shared by all platforms.
Where needed a platform entry can override the template.
This avoids duplication, hence easier to maintain.

* A small change, that can fit better with other targets too.
Just take the platform code and do the rest in template.

* Extended debug to all stretch based docker images

* 1) Combined all orchagent makefiles into one platform independent make under rules/docker-orchagent.mk
2) Extened debug image to all stretch dockers

* Changes per review comments:
1) Dropped LIBSAIREDIS_DBG from database, teamd, router-advertiser, telemetry, and platform-monitor docker*.mk files from _DBG_DEPENDS list
2) W.r.t docker make for syncd, moved DEPENDS from template to specific makefile and let the template has stuff that is applicable to all.

* 1) Corrected a copy/paste mistake

* Fixed a copy/paste bug

* The base syncd dockers follow a template, which defines the base docker as DOCKER_SYNCD_BASE instead of DOCKER_SYNCD_<platform code>. Fix the docker-syncd-<mlnx, bfn>.mk to use the new one.

[Yet to be tested locally]

* Fixed spelling mistake

* Enable build of dbg-sonic-broadcom.bin, which uses dbg-dockers in place of regular dockers, for dockers that build debug version. For dockers that do not build debug version, it uses the regular docker.

This debug bin is installable and usable in a DUT, just like a regular bin.

* Per review comments:
  1) Share a single rule for final image for normal & debug flavors (e.g. sonic-broadcom.bin & sonic-broadcom-dbg.bin)
  2) Put dbg as suffix in final image name.
  3) Compared target/sonic-broadcom.bin.logs with & w/o fix to verify integrity of sonic-broadcom.bin
  4) Compared target/sonic-broadcom.bin.logs with sonic-broadcom-dbg.bin.log for verification

This fix takes care of ONIE image only. The next PR will cover the rest.
The next PR, will also make debug image conditional with flag.

* Updated per comments.
Now that debug dockers are available, do not need a way to install debug symbols in regular dockers.

With this commit, when INSTALL_DEBUG_TOOLS=y is set, it builds debug dockers (for dockers that enable debug build) and the final image uses debug dockers. For dockers that do not enable debug build, regular dockers get used in the final image.

Note:
The debug dockers are explicitly named as <docker name>-dbg.gz. But there is no "-dbg" suffix for image.
Hence if you make two runs with and w/o INSTALL_DEBUG_TOOLS=y, you have complete set of regular dockers + debug dockers. But the image gets overwritten.
Hence if both regular & debug images are needed, make two runs, as one with INSTALL_DEBUG_TOOLS=y and one w/o. Make sure to copy/rename the final image, before making the second run.
2019-06-12 01:36:21 -07:00
Joe LeVeque
42591718b5 [slave.mk] Fix typo for STRETCH_DEBS_PATH for linux-kernel (#2971) 2019-06-05 09:31:29 -07:00
lguohan
30b37ec6fb
[build]: make sonic-slave-stretch as the default build docker (#2921)
Signed-off-by: Guohan Lu <gulv@microsoft.com>
2019-05-27 15:50:51 -07:00
lguohan
8080695ecf
[docker-{sonic,syncd}-vs]: upgrade {sonic,syncd}-vs docker to stretch (#2865)
* [docker-{sonic,syncd}-vs]: upgrade sonic-vs and syncd-vs docker to stretch

* remove python-click 6.6

Signed-off-by: Guohan Lu <gulv@microsoft.com>
2019-05-06 07:19:36 -07:00
Renuka Manavalan
ba0ca01ee0 [build]: Makefile: Extend to build debug docker images for all stretch dockers (#2789)
Overall goal: Build debug images for every stretch docker.

An earlier PR (#2789) made the first cut, by transforming broadcom/orchagent to build target/docker-orhagent-dbg.gz.

Changes in this PR:

Made docker-orchagent build to be platform independent.
1.1) Created rules/docker_orchagent.mk
1.2) Removed platform//docker-orchagent-*.mk
1.3) Removed the corresponding entry from platform//rules.mk

Extended the debug docker image build to stretch based syncd dockers.
2.1) For now, only mellanox & barefoot are stretch based.
2.2) All the common variable definitions are put in one place platform/template/docker-syncd-base.mk
2.3) platform/[mellanox, bfn]/docker-syncd-[mlnx, bfn].mk are updated as detailed below.
2.3.1) Set platform code and include template base file
2.3.2) Add the dependencies & debug dependencies and any update over what base template offers.

Extended all stretch based non-platform dockers to build debug dockers too.
3.1) Affected are:
docker-database.mk,
docker-platform-monitor.mk,
docker-router-advertiser.mk,
docker-teamd.mk,
docker-telemetry.mk

Next: Build debug flavor of final images with regular dockers replaced with debug dockers where available.
2019-04-19 18:49:21 -07:00
Renuka Manavalan
c846fb9ead [build]: Updated Makefile infrastructure to build debug images. (#2753)
* Updated Makefile infrastructure to build debug images.
As a sample, platform/broadcom/docker-orchagent-brcm.mk is updated to add a docker-orchagent-brcm-dbg.gz target.

Now "BLDENV=stretch make target/docker-orchagent-brcm-dbg.gz" will build the debug image.

This debug image can be used in any linux box to inspect core file. If your module's external dependency can be suitably mocked, you my even manually run it inside.

"docker run -it --entrypoint=/bin/bash e47a8fb8ed38"

You may map the core file path to this docker run.

* Dropped the regular binary using DBG_PACKAGES and a small name change to help readability.

* Tweaked the changes to retain the existing behavior w.r.t INSTALL_DEBUG_TOOLS=y.

When this change ('building debug docker image transparently') is extended to all dockers, this flag would become redundant. Yet, there can be some test based use cases that rely on this flag.

Until after all the dockers gets their debug images by default and we switch all use cases of this flag to use the newly built debug images, we need to maintain the existing behavior.
2019-04-13 17:05:18 -07:00
lguohan
158531b48d
[build]: only build dockers needed by installers (#2771)
Signed-off-by: Guohan Lu <gulv@microsoft.com>
2019-04-11 19:28:41 -07:00
Ying Xie
9d6bac567b [build] force Linux to drop cache before calling kvm (#2717)
* [vm build] force Linux to drop cache before calling kvm

KVM need to allocate 2G memory for this build. The system memory might
be occupied by cache at the moment and doesn't have 2G chunk to give
out. Forcing Kernel to drop cache to boost the chance of getting 2G
memory.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* [make] add option to enable/disable VS build memory preparation

Signed-off-by: Ying Xie <ying.xie@microsoft.com>
2019-03-29 15:25:17 -07:00
Stepan Blyshchak
c0caf0c372 [dockers] Build stretch dockers only once and only in stretch slave docker (#2678)
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2019-03-19 09:16:33 -07:00
lguohan
f682e7b131
[docker-radvd]: upgrade docker radvd to stretch based (#2524)
* [docker-radvd]: upgrade docker radvd to stretch based

* install jinja>=2.10

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* install pip packages for testing sonic-utilities

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* set storage driver to vfs

Signed-off-by: Guohan Lu <gulv@microsoft.com>
2019-02-06 21:28:07 -08:00
lguohan
f20665008c
[build]: put stretch debian packages under target/debs/stretch/ (#2519)
* [build]: put stretch debian packages under target/debs/stretch/

* in stretch build phase, all debian packages built in that stage are placed under target/debs/stretch directory.
* for python-based debian packages, since they are really the same for jessie and stretch, they are placed under target/python-debs directory.

Signed-off-by: Guohan Lu <gulv@microsoft.com>
2019-02-04 22:06:37 -08:00
stepanblyschak
c998dc81d2 [slave.mk] Fix DEB_BUILD_OPTIONS passing (#2473)
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
2019-01-23 18:49:44 -08:00
Nikos
7056b49af7 Routing application split config support (#2286)
* Routing application split config support

Signed-off-by: nikos <ntriantafillis@gmail.com>

* Routing application split config support
Routing application split config support

Signed-off-by: nikos <Nikos Triantafillis>
2018-11-26 18:19:12 -08:00
lguohan
64a2b1ce99
[vs]: build sonic vs kvm image (#2269)
Signed-off-by: Guohan Lu <gulv@microsoft.com>
2018-11-20 22:32:40 -08:00
Qi Luo
465ebbafff
Build patched redis-dump-load (#2277)
* Build patched redis-dump-load
* Fix build
* Add build rule
2018-11-20 19:27:56 -08:00
lguohan
8de5aadea6
[fix]: various minor fixes (#2246)
Signed-off-by: Guohan Lu <gulv@microsoft.com>
2018-11-10 13:39:30 -08:00
Wataru Ishida
4db98b46e4 [build]: add build option to use native docker instead of dind for build (#2215)
using native docker is faster than dind dockerd with vfs storage driver

https://github.com/Azure/sonic-buildimage/pull/2016

https://github.com/Azure/draft/issues/181

Signed-off-by: Wataru Ishida <ishida@nel-america.com>
2018-11-01 18:00:00 -07:00
Jipan Yang
c64f105a3e [build]: Add missing timestamp and build_number in docker meta label tag value (#2194)
* Add missing timestamp in docker meta label tag value

Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>

* add BUILD_NUMBER as well

Signed-off-by: Guohan Lu <gulv@microsoft.com>
2018-10-29 12:06:57 -07:00
Joe LeVeque
1e1add90f9
Remove Arista-specific service ACL solution; All platforms now use caclmgrd (#2202) 2018-10-29 10:25:18 -07:00
lguohan
9078377028
[build]: install debug symbols packages when INSTALL_DEBUG_TOOLS=y (#2175)
[build]: install debug symbols packages when INSTALL_DEBUG_TOOLS=y
2018-10-21 17:20:27 -07:00
lguohan
a401ee7fa8
[build]: do not override ENABLE_SYNCD_RPC in rules/config (#2151)
Signed-off-by: Guohan Lu <gulv@microsoft.com>
2018-10-15 13:49:35 -07:00
lguohan
0a8e0c1238
[build]: generate build timestamp at the beginning and use it for the… (#2118)
* [build]: generate build timestamp at the beginning and use it for the rest of build

Signed-off-by: Guohan Lu <gulv@microsoft.com>
2018-10-04 21:20:01 -07:00
Jipan Yang
0e85986088 Add Tag info to docker label meta data (#2058)
* Add Tag info to docker label meta data

Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>

* Move DIRTY_SUFFIX to sonic_get_version()

Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>
2018-09-21 10:44:28 -07:00
lguohan
be9f3ad7c1
[build]: use vfs storage driver to build dockers (#2016)
seen issues to build dockers using aufs in ubuntu 18.04

Immedidate dockers are exported to docker file and then
imported into sonic image. Thus, whether using vfs or aufs
as the storage driver does not matter for the immediate build.

Signed-off-by: Guohan Lu <gulv@microsoft.com>
2018-09-05 15:28:32 -07:00
Joe LeVeque
98082d56a0 [baseimage]: Download picocom version 3.1-2 from stretch-backports; No longer build from source (#1946) 2018-08-17 17:38:20 -07:00
lguohan
c059d9982a
[baseimage]: install picocom 3.1 in base image (#1943)
* [baseimage]: install picocom 3.1 in base image

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* add picocom to stretch build

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* fix slave.mk bug

Signed-off-by: Guohan Lu <gulv@microsoft.com>
2018-08-17 09:06:05 -07:00
Guohan Lu
66ffca2832 [kernel]: set default option to build kernel
Signed-off-by: Guohan Lu <gulv@microsoft.com>
2018-08-11 23:46:13 +00:00
Guohan Lu
0edb3802b0 [build]: add stretch target to build in stretch docker 2018-08-11 09:09:03 +00:00
Guohan Lu
2449fafae0 [kernel]: update kernel submodule and remove standalone igb driver 2018-08-11 09:07:59 +00:00
Guohan Lu
ff1f508f33 [baseimage]: use debian 4.9.0-3 kernel 2018-08-11 09:07:59 +00:00
Rodny Molina
c3c8f7fd7f Fix for bash's memory-leak (#1879)
* Fix for bash's memory-leak

Memory leak is observed during the execution of scripts that make use of bash-arrays. In scenarios where the offending script is executed on a regular basis (e.g. fancontrol), the leaking process may end up consuming most of the system resources.

In this PR i'm replacing bash in all the contexts where it executes (both host and dockers). The official patch for this issue is here: https://ftp.gnu.org/gnu/bash/bash-4.3-patches/bash43-040

* Fixing minor issue during code-merge

Signed-off-by: Rodny Molina <rmolina@linkedin.com>
2018-07-27 17:46:33 -07:00
Ying Xie
3c33737d58
[make] introducing new build option KERNEL_PROCURE_METHOD (#1837)
* [make] introducing new build option KERNEL_BUILD_METHOD

- Kernel could be built from source files with method 'build'
- Kernel could be downloaded from Azure storage with method 'download'

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* Replace BUILD with PROCURE

* fix typo
2018-07-25 08:14:18 -07:00
Sagar Balani
93905d3d82 [barefoot]: Support for platforms based on Barefoot Networks' device (#1796)
* Initial commit

* Add Ingrasys S9180-32X platform dirver.

Signed-off-by: Wade He <chihen.he@gmail.com>

* Add bfn.service for init barefoot.

Signed-off-by: Wade He <chihen.he@gmail.com>

* [Barefoot Beta] Add some functions and fixed some bugs.

1. Update sensors.conf.
2. Fixed IO expander init.
3. Fixed PSU EEPROM.
4. Fixed MB EEPROM.
5. Add fancontrol and fan init.
6. Add SYS LED control (sys, fan, fan tray).
7. 2.5V compute and setup max and min.
8. Fixed typo MB eeprom delete address.
9. Remove coretemp to BMC.
10. Add active CPLD.
11. Modify SFP+ GPIO slave address.
12. Modify tmp75 Near Port 32 slave address.

Signed-off-by: Wade He <chihen.he@gmail.com>

* Add bfn script in /etc/init.d/

Signed-off-by: Wade He <chihen.he@gmail.com>

* Add bfn service in debian

Signed-off-by: Wade He <chihen.he@gmail.com>

* Fixed CPLD switch LED behavior.

Signed-off-by: Wade He <chihen.he@gmail.com>

* [Barefoot Beta] Fixed sensors and hwmon order.
1. Fixed ignore sensors Vbat.
2. Reorg hwmon order.

Signed-off-by: Wade He <chihen.he@gmail.com>

* Fixed PSU1 and PSU2 EEPROM order.

Signed-off-by: Wade He <chihen.he@gmail.com>

* initial barefoot checkin october 2017

* update refpoint

* update refpoints

* update refpoints to bf-master

* update refpoint

* update refpoint to tested version

* change to platform from asic

* update refpoint for swss

* revert core creation setting

* update refpoints

* add telnet for debug shell

* update refpoints 11/17/17

* missed change in file on previous merge

* [CPLD] Fixed blink LED issue.

* Fixed blink LED mask set error.

Signed-off-by: Wade He <chihen.he@gmail.com>

* Update bf_kdrv.c for 6.0.2.39

* Update bf kernel driver

* Add bf_fun kernel module.

* Update bf_tun for fixed build error

* merge with Azure master (12/12/17)

* update swss refpoint

* update refpoint of swss

* library dependency for stack unroll

* update refpoint to bf-master

* [DHCP relay]: Fix circuit ID and remote ID bugs (#1248)

* [DHCP relay]: Fix circuit ID and remote ID bugs

* Set circuit_id_len after setting circuit_id_len to ip->name

* [Platform] Add Psuutil and update sensors.conf for S9100-32X, S8810-32Q and S9200-64X (#1272)

* Add I2C CPLD kernel module for psuutil.
 * Support psuutil script.
 * Add voltage min and max threshold.
 * Update sensors.conf for tmp75.

Signed-off-by: Wade He <chihen.he@gmail.com>

* Allow multi platform support - infra (more changes to follow)

* update relative path to include platform for clarity

* [Platform] Add Ingrasys S9130-32X and S9230-64X with Nephos Switch ASIC for "branch 201712" (#1274)

- What I did
Add switch ASIC vendor: Nephos
Add Nephos platforms: Ingrasys S9130-32X, Ingrasys S9230-64X

- How I did it
Add platform/nephos files
Add platform/nephos/sonic-platform-modules-ingrasys submodule
Add device/ingrasys/x86_64-ingrasys_s9130_32x-r0 files
Add device/ingrasys/x86_64-ingrasys_s9230_64x-r0 files
Add SONiC to support Nephos platform
Update Head of submodule src/sonic-sairedis to "3b817bb"

- How to verify it
To build SONiC installer image and docker images, run the following commands:

make configure PLATFORM=nephos
make target/sonic-nephos.bin
Check system and network feature is worked as well

- Description for the changelog
Add switch ASIC vendor and platforms for Nephos

- A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Sam Yang <yang.kaiyu@gmail.com>

* change source of files to github (from dropbox), update sairedis refpoint

* update refpoint of sairedis

* [centec] support CENTEC SAI 1.0 on 201712 branch and update e582-48x6q board   (#1269)

* [marvel]: Marvell's updates for SONiC.201712 & SAI v1.0 (#1287)

* update sairedis (fast-boot refpoint)

* fix syncd rpc make files

* update refpoint to handle Makefile change (no functional change)

* [Marvell]: Add support for SLM5401-54x device (#1307)

* Marvell's updates for SONiC.201712 & SAI v1.0
* [Platform] Add Marvell's SLM5401-54x for branch 201712

* [Broadcom]: Update Boradcom SAI package to 3.0.3.3-3 (#1312) (#1321)

- update Arista 7050-QX32S config.bcm file
- update Accton th-as771*-32x100G.config.bcm files

* update refpoint for Makefile chnage in sairedis

* update refpoint - sairedis

* update sairedis to older refpoint till we debug clean build

* export asic platform for build

* update refpoint for makefiles

* [PLATFORM] Centec update E582 driver fan/epprom/sensor (#1332)

* Upload wnc-osw1800

* Modify for Barefoot suggest

* Revert bfn-platform.mk

* Update bfn-platform-wnc.mk

Update parameter name

* Update parameter name

* initial support for WNC platform

* change switch name to "switch"

* Delete bf modules for rel_7_0

* Add Ingrasys S9180 platform

Signed-off-by: Wade He <chihen.he@gmail.com>

* Modify bfnsdk for Ingrasys S9180 platform

Signed-off-by: Wade He <chihen.he@gmail.com>

* Resolved the conflict.

* Resolved the conflict.

* Update submodule path and url.

* Delete unused file.

* Update PSU GPIO and EEPROM for psuutil.

* Add psuutil in S9180-32X

Signed-off-by: Wade He <chihen.he@gmail.com>

* update refpoint

* update refpoint

* change contact email, update refpoint

* cleanup and update kernel modules

* updates based on review

* update refpoint

* update refpoint

* fix typo in config script to check for platforms

* remove stale file

* resolve conflicts

* cleanup diffs with Azure repo and update SDK debs

* update refpoints to Azure

* address review comments

* revert refpoint of swss-common

* porting the build fix from master

* porting build fix from master

* Minor Fix

* Minor fix

* Temp to sde deb packages url

* Update sonic - sairedis,swss & swss-common refpoints

* Update git modules url path to bfn repo

* updated paths for swss, swss-common & sairedis

* Update refpoint for sonic-swss to local bfn repo

* Update URL for downloading sde debian packages

* porting fix links of debian git server from master

* porting fix links of debian git server from master

* [Ingrasys] Add platform support for S9280-64X with Barefoot ASIC

* Update ref points for swss, swss-common and sairedis repos

* Add sonic platform scripts for bfn montara/maverick

* Call sh scripts instead of calling py scripts

* Address upstream PR Comments (#10)

* Update bf-master with azure/master
* Undo changes to some files

* Revert "Address upstream PR Comments (#10)"

This reverts commit a7fddb83ca.

* Address upstream comments (#11)

* Remove all non bfn specific changes from upstream PR

* Revert "Address upstream comments (#11)"

This reverts commit 559132103e.

* Undo non bfn changes

* Little more cleanup

* Add back code removed in merge

* export CONFIGURED_PLATFORM

* Update sairedis and swss refpoints

* Address Upstream PR comment

* change deb pkg dependency from 3.16.0-4-amd64 to 3.16.0-5-amd64

* Set default tx queue len for usb0 interface to 64

* Update sairedis refpoint

* Update swss ref point

* Add bfn buffer cfg files for montara/maverick as per new design

* Update buffer cfg templates for bfn montara

* add non zero size to buffer profile
* add macro to generate port lists

* Update buffer cfg templates for bfn mavericks

* add non zero size for buffer profiles
* add port generation macro

* Add missing psmisc package

* BGP docker seems to be missing killall utility being used by fast-reboot script. This is causing non graceful termination of BGP sessions.
Adding psmisc to resolve this issue.

* Update swss ref point

* Update swss ref point

* Update sairedis refpoint

* Update sairedis refpoint

* Update sairedis refpoint

* Update sairedis refpoint

* Update refpoint for sairedis and swss

* sairedis to azure master
* swss to latest bfn bf-master

* Update gitmodules

Update url for sairedis to azure master

* Correct typo in bfn platform script

* Update swss and sairedis ref points

* Update swss ref point

* Address Review comments

* Update swws path in gitmodules to azure master
* update swss refpoint
* update base docker j2 file -remove psmisc package (could be a concern, would cause fast reboot to not work correctly will fix in another PR)

* Fix sairedis refpoint broken in by previous merge

* Remove psmisc from docker base image

* This will break fast reboot as killall is required for killing bgp process and initiating graceful termination of BGP session.
Will fix this in a seperate PR. Need this for SONIC upstreaming

* Address upstream comments

* Remove bmc interface from interface jinja template and sample output interfaces file
* Add bmc interface at boot time to network interfaces for bfn bmc based platforms

* Remove autogen ingrasys debian files

* Revert "Remove autogen ingrasys debian files"

* Buffer and qos config template fix for bfn platforms (#21)

SWI-1509 Buffer and qos config template fix for bfn platforms

* Fix qos config files for montara & mavericks (#22)

* Reference only ppg 3,4 in qos files as no profiles are attached to 0,1 in buffer configs

* Fix vs test (#23)
2018-07-24 10:23:12 -07:00
Andriy Moroz
dadc17d9e6 [Mellanox] Use MAC from EEPROM for PortChannels and VLAN Interfaces (#1793)
* Use MAC from EEPROM for PortChannels

Signed-off-by: Andriy Moroz <c_andriym@mellanox.com>

* Use MAC from EEPROM in DEVICE_METADATA

Will affect MAC for VLAN interfaces

Signed-off-by: Andriy Moroz <c_andriym@mellanox.com>

* Get MAC via decode-syseeprom

Signed-off-by: Andriy Moroz <c_andriym@mellanox.com>

* hw-management is now a service

Signed-off-by: Andriy Moroz <c_andriym@mellanox.com>

* Add error handling for MAC fetch process

Signed-off-by: Andriy Moroz <c_andriym@mellanox.com>
2018-07-23 15:51:03 -07:00
Joe LeVeque
f04f0704f7
Build python-click Debian package from version 6.7-4 source to fix CLI autocomplete/suggest (#1824) 2018-06-29 09:59:46 -07:00
Qi Luo
7ba08e5bf6
Prefix docker container name to syslog syslogtag (program name) (#1810) 2018-06-25 10:48:42 -07:00
Rodny Molina
b5f153b3ae Introducing 'debugging' and 'profiling' options in sonic build-infra (#1782) 2018-06-19 15:59:12 -07:00
Serhey Popovych
8d88455509 [baseimage]: Improve password hashing for default user account (#1748)
* [slave.mk]: Fix displaying username and password in build summary

We display contents of DEFAULT_USERNAME and DEFAULT_PASSWORD, while
image can be build with USERNAME and/or PASSWORD given on make(1)
command line. For example:

  $ make USERNAME=adm PASSWORD=mypass target/sonic-broadcom.bin

Fix by displaying USERNAME and PASSWORD variables in build summary.

Signed-off-by: Sergey Popovich <sergey.popovich@ordnance.co>

* [baseimage]: Improve default user account handling

There are couple of issues with current implementation of default
user account management in baseimage:

  1) It uses DES to encrypt accounts password. Furthermore this
     effectively limits password length to 8 symbols, even if more
     provided with PASSWORD or DEFAULT_PASSWORD from rules/config.

  2) Salt value for password is same on all builds even with different
     password increasing attack surface.

  3) During the build process password passed as command line parameter
     either as plain text (if given to make(1) as "make PASSWORD=...")
     or DES encrypted (if given to build_debian.sh) can be seen by
     non-build users using /proc/<pid>/cmdline file that has group and
     world readable permissions.

Both 1) and 2) come from:

  perl -e 'print crypt("$(PASSWORD)", "salt"),"\n"')"

that by defalt uses DES if salt does not have format $<id>$<salt>$,
where <id> is hashing function id. See crypt(3) for more details on
valid <id> values.

To address issues above we propose following changes:

  1) Do not create password by hands (e.g. using perl snippet above):
     put this job to chpasswd(8) which is aware about system wide
     password hashing policy specified in /etc/login.defs with
     ENCRYPT_METHOD (by default it is SHA512 for Debian 8).

  2) Now chpasswd(8) will take care about proper salt value.

  3) This has two steps:

    3.1) For compatibility reasons accept USERNAME and PASSWORD as
         make(1) parameters, but warn user that this is unsafe.

    3.2) Use process environment to pass USERNAME and PASSWORD variables
         from Makefile to build_debian.sh as more secure alternative to
         passing via command line parameters: /proc/<pid>/environ
         readable only by user running process or privileged users like
         root.

Before change:
--------------

  hash1
  -----
  # u='admin'
  # p="$(LANG=C perl -e 'print crypt("YourPaSs", "salt"),"\n"')"
                                      ^^^^^^^^
                                      8 symbols
  # echo "$u:$p" | chpasswd -e

  # getent shadow admin
  admin:sazQDkwgZPfSk:17680:0:99999:7:::
        ^^^^^^^^^^^^^
        Note the hash (DES encrypted password)

  hash2
  -----
  # u='admin'
  # p="$(LANG=C perl -e 'print crypt("YourPaSsWoRd", "salt"),"\n"')"
                                      ^^^^^^^^^^^^
                                      12 symbols
  # echo "$u:$p" | chpasswd -e

  # getent shadow admin
  admin:sazQDkwgZPfSk:17680:0:99999:7:::
        ^^^^^^^^^^^^^
        Hash is the same as for "YourPaSs"

After change:
-------------

  hash1
  -----
  # echo "admin:YourPaSs" | chpasswd
  # getent shadow admin
  admin:$6$1Nho1jHC$T8YwK58FYToXMFuetQta7/XouAAN2q1IzWC3bdIg86woAs6WuTg\
           ^^^^^^^^
           Note salt here
  ksLO3oyQInax/wNVq.N4de6dyWZDsCAvsZ1:17681:0:99999:7:::

  hash2
  -----
  # echo "admin:YourPaSs" | chpasswd
  # getent shadow admin
  admin:$6$yKU5g7BO$kdT02Z1wHXhr1VCniKkZbLaMPZXK0WSSVGhSLGrNhsrsVxCJ.D9\
           ^^^^^^^^
           Here salt completely different from case above
  plFpd8ksGNpw/Vb92hvgYyCL2i5cfI8QEY/:17681:0:99999:7:::

Since salt is different hashes for same password different too.

  hash1
  -----
  # LANG=C perl -e 'print crypt("YourPaSs", "\$6\$salt\$"),"\n"'
                                             ^^^^^
                                             We want SHA512 hash
  $6$salt$qkwPvXqUeGpexO1vatnIQFAreOTXs6rnDX.OI.Sz2rcy51JrO8dFc9aGv82bB\
  yd2ELrIMJ.FQLNjgSD0nNha7/

  hash2
  -----
  # LANG=C perl -e 'print crypt("YourPaSsWoRd", "\$6\$salt\$"),"\n"'
  $6$salt$1JVndGzyy/dj7PaXo6hNcttlQoZe23ob8GWYWxVGEiGOlh6sofbaIvwl6Ho7N\
  kYDI8zwRumRwga/A29nHm4mZ1

Now with same "salt" and $<id>$, and same 8 symbol prefix in password, but
different password length we have different hashes.

Signed-off-by: Sergey Popovich <sergey.popovich@ordnance.co>
2018-06-09 11:29:16 -07:00
Joe LeVeque
e900369fa3
[slave.mk] Define SPACE variable differently such that the line doesn't end in whitespace (#1676) 2018-05-10 18:52:38 -07:00
Jipan Yang
f0690b246b [build]: Fix the problem of ENABLE_SYSTEM_TELEMETRY not taking effect in rules/config (#1615)
Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>
2018-04-18 00:31:12 -07:00
Jipan Yang
f74de8914b [telemetry]: SONiC system telemetry Support (#1526)
* SONiC system telemetry Support

Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>

* Update package name from telemetry to sonic-telemetry

Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>
2018-03-27 13:39:04 -07:00
sihuihan88
7639cc1b53 [pfcwd]: add flag to enable pfc wd on start (#1429) 2018-03-05 23:55:37 -08:00
Joe LeVeque
dc7abaf569 [slave.mk] Apply patches to dockers before building, if any exist (#1395) 2018-02-15 17:48:49 -08:00
byu343
ecf5c8d311 ssh and snmp allow list (#1363)
- Service ACL framework for Arista platforms
2018-02-08 17:43:52 -08:00
Marian Pritsak
43554c8b1a [slave.mk]: Clear deb_dist directory (#1344)
Python packages, having their version changed, will fail to build
because deb_dist directory contains both build directories for old and
new version, and (for some uncleaer reason) debian utilities don't know
which one to choose.

Signed-off-by: marian-pritsak <marianp@mellanox.com>
2018-01-29 08:14:01 -08:00
Joe LeVeque
134707f822
Move platform-specific hardware plugin base packages to sonic-platform-common submodule (#1301) 2018-01-17 17:11:31 -08:00
pavel-shirshov
47aba6f76d
[libsaithrift-dev]: Enable building libsaithrift-dev and pythonthrift libraries (#1296)
* Fix build of libsaithrift for broadcom

* Restore libsaithrift-dev building

* Comment out libsaithrift library on cavium and marvell. Both of them have old SAI drivers

* Revert back unintentional acton changes

* Don't run the package checks when building sairedis package
2018-01-11 20:32:14 -08:00
lguohan
15d433d975
[build]: allow to use http(s) proxy in the build (#1265)
* allow to use http(s) proxy in the build

To enable this, use following command
http_proxy=[your_proxy] https_proxy=[your_proxy] make
2017-12-23 23:34:15 -08:00
kaiyu22
63de341dd6 [Platform] Add Ingrasys S9130-32X and S9230-64X with Nephos Switch ASIC (#1245)
* Add switch ASIC vendor and platforms for Nephos

- What I did
Add switch ASIC vendor: Nephos
Add Nephos platforms: Ingrasys S9130-32X, Ingrasys S9230-64X

- How I did it
Add platform/nephos files
Add platform/nephos/sonic-platform-modules-ingrasys submodule
Add device/ingrasys/x86_64-ingrasys_s9130_32x-r0 files
Add device/ingrasys/x86_64-ingrasys_s9230_64x-r0 files
Add SONiC to support Nephos platform

- How to verify it
To build SONiC installer image and docker images, run the following commands:
make configure PLATFORM=nephos
make target/sonic-nephos.bin
Check system and network feature is worked as well

- Description for the changelog
Add switch ASIC vendor and platforms for Nephos

- A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Sam Yang <yang.kaiyu@gmail.com>

* Advance sonic-sairedis submodule to include #271 (Add Nephos ASIC)
2017-12-22 10:04:29 -08:00
Ying Xie
eadd74fb84
[build] allow user to override the default number of build jobs (#1255)
User could issue SONIC_BUILD_JOBS=<num> to set number of concurrent
build job(s) to run.

With this commit, the default setting is still left at 1.
2017-12-20 15:25:30 -08:00
Qi Luo
4b2225bca5 [docker-snmpd]: asyncsnmp depends on sonic-utilities so it is possible to import sonic_psu (#1212)
* asyncsnmp depends on sonic-utilities so it is possible to import sonic_psu

Signed-off-by: Qi Luo <qiluo-msft@users.noreply.github.com>

* Ignore sonic_utilities test during build
2017-12-07 13:08:23 -08:00
Liuqu
8383b1f256 [TACACS+]: Add support for TACACS+ Authentication (#1019)
* [TACACS+]: Add support for TACACS+ Authentication

* pam_tacplus - A TACACS+ protocol client library and PAM module to
  supports core TACACS+ functions for AAA.
* nss_tacplus - A NSS plugin for TACACS+ to extend function getpwnam,
  make the TACACS+ authenticated user which is not found in local
  could login successfully.

* Add make rules for pam_tacplus and install script
* Add a patch for pam_tacplus to disable pam-auth-update pam-tacplus
  by default
* Add a patch for pam_tacplus to inlucde and build nss_tacplus

  Signed-off-by: chenchen.qcc@alibaba-inc.com

* [TACACS+]: Add nss-tacplus as a separate src repo

* Separate nss-tacplus from pam-tacplus, modify tacacs.mk and
  makefile, add a patch to adapt to the new user map profile.
* Use the lastest stable version for pam-tacplus, add a dependent
  package in sonic-salve, add two patches to fix build error.
* Add scripts to disable tacplus by default.
* Remove hostcfgd service file

  Signed-off-by: Chenchen Qi <chenchen.qcc@alibaba-inc.com>

* [TACACS+]: Fix nss-tacplus filter some valid TACACS+ username

* The NAME_REGEX for username check in plugin nss-tacplus is
  the ANSI version "^[0-9a-zA-Z_-\ ]*$", but the regular expression
  in /etc/adduser.conf is not defined as ANSI version. To avoid
  nss-tacplus filter some valid TACACS+ username, remove username
  check.

  Signed-off-by: Chenchen Qi <chenchen.qcc@alibaba-inc.com>
2017-12-07 03:36:17 -08:00
lguohan
414e43bb74
[build]: start docker service if not already started (#1203)
docker start fails when docker service is already started
2017-12-01 19:24:55 -08:00
lguohan
f4e37a66f9 add INSTALLS, LAZY_INSTALLS, DEPENDS targets to image build (#1159)
1. _INSTALLS defines the packages to be installed in the image
at the build time.
2. _LAZY_INSTALLS defines the packages to be installed in the image
at the first run time
3. _DEPENDS defines the packages to at the build dependency of
the image
2017-11-16 12:27:03 -08:00
lguohan
a31c08047b [build]: add sonic-config-engine to sonic-utilitie build dependency (#1075) 2017-10-25 05:06:30 -07:00
lguohan
f073d8c654 [virtualswitch]: build docker-sonic-vs docker based on SAI virtual switch (#1065) 2017-10-23 22:01:42 -07:00
Shuotian Cheng
27cac94da7 [Makefile]: Fix unmatched variable name ENABLE_SYNCD_RPC (#1032)
Signed-off-by: Shu0T1an ChenG <shuche@microsoft.com>
2017-10-11 17:46:26 -07:00
Joe LeVeque
8a305ce9c9 [sonic-utilities]: Build as Debian package once again (revert change to wheel) (#1018) 2017-10-06 12:46:47 -07:00
Joe LeVeque
1cd9818fe9 [build]: sonic-utilities package depends on swsssdk; build as wheel and add build dependency (#1011)
* [build]: sonic-utilities package now depends on swsssdk; add build dependency

* Now building sonic-utilities Python package in wheel format

* Update sonic-utilities submodule

* Change output wheel name to match proper format
2017-10-04 19:50:39 -07:00
Andriy Moroz
9b249ceb59 Add time stamp suffix to the dirty images version string (#958)
* Add timestamp suffix to the dirty images version string

This will allow to build different debug images without
having to update BUILD_NUMBER variable before the build

* Changed timestamp format to YYYYMMDD.HHMMSS
2017-09-27 00:04:34 -07:00
ravijo2
458093fee5 Framework to plugin Organization specific scripts during ONIE Image build (#951)
* Framework to plugin Organization specific scripts

* Framework to plugin Organization specific scripts

* Framework to plugin Organization specific scripts

* add getopt option to organization script
2017-09-19 16:23:31 -07:00
Samuel Angebault
bc73082922 [devices]: Add led plugin for Arista 7060CX-32S and 7260CX3-64 (#945)
* Bump sonic-platform-modules-arista submodule

* Allow image specific mount option for containers

* Add led plugin for DCS-7060CX-32S

* Add led plugin for DCS-7260CX3-64
2017-09-11 15:29:42 -07:00
Joe LeVeque
211689ad29 [slave.mk]: Apply series of patches to SONIC_PYTHON_STDEB_DEBS targets if they exist (#941) 2017-09-07 14:02:17 -07:00
Marian Pritsak
175b952508 [slave.mk]: Add list target (#856)
Add new target to list all available packages, files, docker images and
installers.

Signed-off-by: marian-pritsak <marianp@mellanox.com>
2017-07-30 01:34:27 +03:00
Oleksandr Ivantsiv
382d52843f [mellanox]: Move FW upgrade script to base image. (#849)
- Download files to target/files directory (instead of target/debs)
- Clean python-weels log files
2017-07-28 10:57:51 -07:00
Marian Pritsak
7d95fd7e8c [rules/functions][slave.mk]: Refine build output (#838)
Print current build configuration before run
Update screen with currently running targets (only available if TERM is
available)
Change format of printed targets

Signed-off-by: marian-pritsak <marianp@mellanox.com>
2017-07-25 09:49:39 +03:00