Why I did it
This is sonic-mgmt-docker Python3 migration.
How I did it
Remove Python3 venv from sonic-mgmt-docker
How to verify it
Compile docker and verify the Python3 venv is NOT there.
Recently, netaddr 1.0.0 was released, which dropped Python 2 support
entirely. Because of this, an empty netaddr package is installed for
Python 2. Ansible still uses Python 2, so this needs to remain
functional. The Python 3 part of sonic-mgmt is fine.
Pin the version of netaddr installed in Python 2 to 0.10.1, the last
supported version.
Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
How I did it
Remove Python3 venv in Python3-only sonic-mgmt-docker
How to verify it
There is no impact to sonic-mgmt-docker:latest tag.
Build sonic-mgmt-docker with LEGACY_SONIC_MGMT_DOCKER=y, see python3 venv is there.
Build sonic-mgmt-docker with LEGACY_SONIC_MGMT_DOCKER=n, see python3 venv is NOT included.
Why I did it
This is part of Python3 migration project. This PR will add a new makefile flag: LEGACY_SONIC_MGMT_DOCKER
Now by default: LEGACY_SONIC_MGMT_DOCKER = y will build sonic-mgmt-docker with Python2 and Python3
If LEGACY_SONIC_MGMT_DOCKER = n will will sonic-mgmt-docker with Python3 only
Work item tracking
Microsoft ADO (number only): 25254349
How I did it
Add makefile flag: LEGACY_SONIC_MGMT_DOCKER
How to verify it
By default will build sonic-mgmt-docker with Python2 and Python3. No change compared to before.
Set LEGACY_SONIC_MGMT_DOCKER=n will build sonic-mgmt-docker with Python3 only
Why I did it
This PR is part of sonic-mgmt-docker Python3 migration project.
Work item tracking
Microsoft ADO (number only): 24397943
How I did it
Upgrade Ansible to 6.7.0
Make Python3 as the default interpreter. python is a soft link to python3. If you want to use python2, use the command python2 explicitly.
Upgrade some pip packages to higher version in order to meet security requirement.
How to verify it
Build a private sonic-mgmt-docker successfully.
Verify python is python3.
Verify python2 is working with 202012 and 202205 branch.
Verify python3 is working with master branch.
Verify with github PR test.
With Debian Bookworm, Paramiko 2.9 or newer will need to be used to be
able to connect to devices running that version of Debian
(specifically, to those running OpenSSH 9.2).
Paramiko is currently on 3.3.1. For now, upgrade to 2.9.5.
Signed-off-by: Saikrishna Arcot <sarcot@microsoft.com>
This reverts commit 51fb6d7d9f.
The new sonic-mgmt docker image has ansible upgraded. Encountered some issues that are hard to debug to have a quick fix. Let's revert the change for now. The new sonic-mgmt docker image was kept for further debugging and fixing. After all the issues are fixed, we'll need to apply this change again.
Why I did it
This PR is part of sonic-mgmt-docker Python3 migration project.
Work item tracking
Microsoft ADO (number only): 24397943
How I did it
Upgrade Ansible to 6.7.0
Make Python3 as the default interpreter. python is a soft link to python3. If you want to use python2, use the command python2 explicitly.
Upgrade some pip packages to higher version in order to meet security requirement.
How to verify it
Build a private sonic-mgmt-docker successfully.
Verify python is python3.
Verify python2 is working with 202012 and 202205 branch.
Verify python3 is working with master branch.
Why I did it
This change is to help us running SSH config generation for our testbed in mgmt container.
Original PR in sonic-mgmt repo can be found here: sonic-net/sonic-mgmt#9773.
Work item tracking
Microsoft ADO (number only): 25007799
How I did it
Updating sonic-mgmt docker file to add sshconf 0.2.5 into pip install under venv.
1. Remove useless temporary protobuf deb packages
2. Import dash_api to python3 env
### Why I did it
1. There are some temporary Debian packages,protobuf packages, needs to be deleted
2. The dash-api was installed in the system folder that cannot be imported by the virtual python3 environment. But the testcases of DASH in sonic-mgmt are executed in virtual python3 environment.
##### Work item tracking
- Microsoft ADO **(number only)**: 17417902
#### How I did it
1. Add missed `&&` so that all protobuf debian packaged can be downloaded to the /tmp folder
2. Add ` --system-site-packages ` to env-python so that the system library can be accessed by virtual environment
#### How to verify it
Check the dash_api can be imported in env-python3
```
AzDevOps@46a900cf8477:~$ source env-python3/bin/activate
(env-python3) zegan@46a900cf8477:~$ ls
bin env-python3
(env-python3) zegan@46a900cf8477:~$ python3
Python 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dash_api
>>>
```
Upgrade celery in the python3 to 5.2.7,
Upgrade ipython to 8.12.2 since 5.4.1 requires prompt-toolkit<2.0.0,>=1.0.4,
But celery 5.2.7 relies click-repl>=0.2.0 , click-repl>=0.2.0 relies prompt-toolkit>=3.0.36.
So upgrade ipython to resolve the prompt-toolkit version incompatible issue.
#### Why I did it
The testcases in sonic-mgmt need the packages of protobuf and dashapi
##### Work item tracking
- Microsoft ADO **(number only)**:
#### How I did it
Because the docker of sonic-mgmt is based on ubuntu20.04, it cannot directly install the packages compiled by slave due to dependency issues. Download related packaged directly from Azp.
#### How to verify it
Check azp stats.
Why I did it
This is part of sonic-mgmt-docker Python3 migration project.
Currently Python3 packages are in the Python3 virtual environment. This PR will add Python3 packages to real file system.
After we migrate all script to use Python3 in real file system, the Python3 venv will be deleted.
After this PR, in sonic-mgmt-docker,
Directly run cmd - pytest will use Python2's version.
python3 -m pytest will use Python3's version.
How I did it
Modify sonic-mgmt-docker j2 script.
How to verify it
Build a private sonic-mgmt-docker and run basic test case with Python3.
Why I did it
Current docker-sonic-mgmt build is broken. So below are two fixes which can help in mitigating the same.
PYAML - Download a specific version in python2 as after https://pypi.org/project/pyaml/23.5.5/ there was support only for python3. This update happened on May 5th. And consequently all daily builds after this changes https://dev.azure.com/mssonic/build/_build/results?buildId=266733&view=results (starting build to break) kept failing
Azure-CLI - this can be downloaded by apt-get repository. So modify as an improvement.
Work item tracking
Microsoft ADO (number only): [Build] fix docker-sonic-mgmt build #15567
How I did it
By manually checking the release notes of pyaml and install azure-cli in newly installed docker container using apt-get
How to verify it
You can run below commands to validate:
make configure PLATFORM=generic
make target/docker-sonic-mgmt.gz
Second line would fail without the commit.
Why I did it
Add AZP agent necessary packages to sonic-mgmt-docker
Remove Python 201811 venv
Update some packages in order to meet internal security requirements
How I did it
Update sonic-mgmt-docker file
How to verify it
sonic-mgmt-docker can run: bash, apt update, apt install and ping.
start.sh is under /azp with exec permission.
env-201811 venv is removed.
jinja2 is upgrade to 2.10.1
Why I did it
The Dockerfile of docker-sonic-mgmt became a little bit messy over time. Some packages are also a little bit too old. It would be better to do some cleanup and upgrade some important packages.
How I did it
Updated the dockerfile template for building docker-sonic-mgmt.
How to verify it
Locally built the docker-sonic-mgmt image and used it to run some test scripts.
Description for the changelog:
The build-essential package contains gcc and make. It's unnecessary to install them again.
The python-is-python2 package is included in the python package for Ubuntu 20.04. It's unnecessary to install it again.
Sort the apt and pip packages by alphabetic order.
Cleanup get-pip.py after installation.
Cleanup the python-scapy deb package after installation.
Ensure that the python pip, setuptools and wheel packages are up to date.
Install pytest-ansible from pip instead of from source code.
While installing docker-ce-cli, it's unnecessary to install curl and software-properties-common again.
Merged some pip install steps into one step.
Upgrade ansible from 2.8.12 to 2.9.27 for env-python3.
Upgrade pytest to 7.1.3 for env-python3.
Add ncclient package to evn-python3.
Why I did it
The python packages azure-kusto-data and azure-kusto-ingest packages for python2 are too old and not really used. The python3 environment has newer version of these packages installed. This change is to deprecate these two packages for python2 in docker-sonic-mgmt image.
How I did it
Removed the lines for installing old version of packages azure-kusto-data and azure-kusto-ingest in python2 in the Dockerfile template.
Signed-off-by: Xin Wang <xiwang5@microsoft.com>
Why I did it
Keysight provide a new version with some snappi API source code related fix: snappi[ixnetwork,convergence]==0.7.44
How I did it
Upgrade snappi version to 0.7.44
How to verify it
Whether it's installed in sonic-mgmt docker container
Why I did it
Migration of sonic-mgmt codebase from Python 2 to Python 3
How I did it
Added scapy dependencies to the env-python3 virtual environment.
How to verify it
Run test case:
py.test --testbed=testbed-t0 --inventory=../ansible/lab --testbed_file=../ansible/testbed.csv --host-pattern=testbed-t0 -- module-path=../ansible/library lldp
Signed-off-by: Oleksandr Kozodoi <oleksandrx.kozodoi@intel.com>
Why I did it
This fix is to address issue: Azure/sonic-mgmt#5280
In the sonic-mgmt Dockerfile, python package allure-pytest is installed after ENV USER $user.
Consequently the package is installed to path /home/$user/.local and is only available to the $user
account. If we use root account in sonic-mgmt docker container to run tests, any script importing
the allure package will fail with ImportError. We need to install the allure-pytest package to global
directory instead of user local directory.
How I did it
Update the sonic-mgmt Dockerfile to ensure that the allure-pytest package is installed to global directory
How to verify it
Build a new sonic-mgmt docker image based on the changes.
Use sonic-mgmt docker container of the newly built image to run test scripts that depend on the
allure-pytest package. No ImportError is raised.
This PR includes necessary changes for the setup of the Python3 virtual environment in the sonic-mgmt docker container.
How to activate Python3 virtual environment?
Connect to the sonic-mgmt container
$ docker exec -ti sonic-mgmt bash
Activate the virtual environment
$ source /var/user/env-python3/bin/activate
Why I did it
Migration of sonic-mgmt codebase from Python 2 to Python 3
How I did it
Added all necessary dependencies to the env-python3 virtual environment.
Signed-off-by: Oleksandr Kozodoi <oleksandrx.kozodoi@intel.com>
Why I did it
the motivation for this PR is to add retry_call to several test cases in the community, for example, the following cases:
test_show_platform_fanstatus_mocked
test_show_platform_temperature_mocked
are executing a command once and comparing the output to the expected mock data,
sometimes differences between the mock and the actual are causing the tests to fail.
retry will make these tests more stable.
retry will also be more efficient than sleep which will cause the tests to run longer because sometimes it is not necessary to sleep all that time, retry will only run a function only until it passed.
How I did it
added retry to the docker file
How to verify it
I run the tests with retry on the docker after installing the retry package
Signed-off-by: Sharon Lutati <slutati@nvidia.com>
A recent version of contextlib2 (https://pypi.org/project/contextlib2/21.6.0/#history) has broken Python2 compatibility, so the version picked up by netaddr when using Python2 must be specified, or else builds fail
Co-authored-by: Tom Zhu <tom.zhu@metaswitch.com>
The aiohttp package is required by azure.kusto.data which is used by sonic-mgmt/test_reporting.
This change is to ensure that the dependent package is installed in the sonic-mgmt docker.
Signed-off-by: Xin Wang <xiwang5@microsoft.com>
* Why I did it
Upgrade to the latest ixnetwork-restpy and ixnetwork-open-traffic-generator pypi packages
* How I did it
Updated the pip install entries for the packages in the Dockerfile.j2
* How to verify it
pip show ixnetwork-restpy
pip show ixnetwork-open-traffic-generator
Co-authored-by: Neetha John <nejo@microsoft.com>
