matthew/sonic-buildimage
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Upgrade Ansible to 6.7.0 and make Python3 as the default interpreter in sonic-mgmt-docker (#17021)

Browse Source 
Why I did it
This PR is part of sonic-mgmt-docker Python3 migration project.

Work item tracking
Microsoft ADO (number only): 24397943

How I did it
Upgrade Ansible to 6.7.0
Make Python3 as the default interpreter. python is a soft link to python3. If you want to use python2, use the command python2 explicitly.
Upgrade some pip packages to higher version in order to meet security requirement.

How to verify it
Build a private sonic-mgmt-docker successfully.
Verify python is python3.
Verify python2 is working with 202012 and 202205 branch.
Verify python3 is working with master branch.
Verify with github PR test.
This commit is contained in:
ShiyanWangMS 2023-10-31 09:44:55 +08:00 committed by GitHub
parent a5ee9867da
commit fe735e35c6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 78 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
dockers/docker-sonic-mgmt/0001-Fix-getattr-AttributeError-in-multi-thread-scenario.patch Normal file
View File

@ -0,0 +1,46 @@
From e323263795a0af4c9c61992bd7a3347d8928db39 Mon Sep 17 00:00:00 2001
From: Xin Wang <xiwang5@microsoft.com>
Date: Thu, 14 Sep 2023 16:52:54 +0800
Subject: [PATCH] Fix getattr AttributeError in multi-thread scenario
When multi-thread is used, the plugin loader may raise AttributeError
while getting "ActionModule" from an action plugin.
The reason is that cache is used while loading module. Before a module
is fully loaded, it is already put into the sys.modules cache. When another
thread tries to load the same module, it would load an incompletely loaded
module from the sys.modules cache. Then while getting "ActionModule"
attribute from this module, exception AttributeError could be raised.
Signed-off-by: Xin Wang <xiwang5@microsoft.com>
---
lib/ansible/plugins/loader.py | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/lib/ansible/plugins/loader.py b/lib/ansible/plugins/loader.py
index 74bdeb5719..251531b62e 100644
--- a/lib/ansible/plugins/loader.py
+++ b/lib/ansible/plugins/loader.py
@@ -801,15 +801,14 @@ class PluginLoader:
warnings.simplefilter("ignore", RuntimeWarning)
spec = importlib.util.spec_from_file_location(to_native(full_name), to_native(path))
module = importlib.util.module_from_spec(spec)
-
- # mimic import machinery; make the module-being-loaded available in sys.modules during import
- # and remove if there's a failure...
- sys.modules[full_name] = module
-
try:
spec.loader.exec_module(module)
+ # mimic import machinery; make the module-being-loaded available in sys.modules during import
+ # and remove if there's a failure...
+ sys.modules[full_name] = module
except Exception:
- del sys.modules[full_name]
+ if full_name in sys.modules:
+ del sys.modules[full_name]
raise
return module
--
2.25.1

74
dockers/docker-sonic-mgmt/Dockerfile.j2
View File

@ -38,10 +38,10 @@ RUN apt-get update && apt-get install -y apt-transport-https \
telnet \
vim
RUN pip3 install --upgrade pip setuptools wheel
RUN pip3 install aiohttp \
RUN python3 -m pip install --upgrade pip setuptools wheel
RUN python3 -m pip install aiohttp \
allure-pytest==2.8.22 \
ansible==2.9.27 \
ansible==6.7.0 \
azure-storage-blob==12.9.0 \
azure-kusto-data \
azure-kusto-ingest \
@ -49,7 +49,7 @@ RUN pip3 install aiohttp \
celery[redis]==5.2.7 \
cffi \
contextlib2==0.6.0.post1 \
cryptography==3.3.2 \
cryptography==41.0.2 \
ctypesgen \
dpkt \
dpugen==0.1.1 \
@ -60,7 +60,7 @@ RUN pip3 install aiohttp \
ixload \
ixnetwork-restpy==1.0.64 \
ixnetwork-open-traffic-generator==0.0.79 \
jinja2==2.10.1 \
jinja2==3.1.2 \
jsonpatch \
lxml \
markupsafe==2.0.1 \
@ -84,12 +84,12 @@ RUN pip3 install aiohttp \
pyro4 \
pysnmp==4.4.12 \
pysubnettree \
pytest-ansible \
pytest-ansible==4.0.0 \
pytest-html \
pytest-repeat \
pytest-xdist==1.28.0 \
python-dateutil \
pytest==7.1.3 \
pytest==7.4.0 \
PyYAML \
redis \
requests \
@ -99,7 +99,7 @@ RUN pip3 install aiohttp \
scapy==2.4.5 \
setuptools-rust \
six \
snappi[ixnetwork,convergence]==0.7.44 \
snappi[ixnetwork,convergence]==0.11.16 \
tabulate \
textfsm==1.1.2 \
thrift==0.11.0 \
@ -115,21 +115,15 @@ RUN pip3 install aiohttp \
&& cd ../.. \
&& rm -fr nanomsg-1.2 \
&& rm -f 1.2.tar.gz \
&& pip3 install nnpy
RUN curl -fsSL http://archive.ubuntu.com/ubuntu/pool/universe/s/scapy/python-scapy_2.3.3-3_all.deb \
--output python-scapy_2.3.3-3_all.deb \
&& dpkg -i python-scapy_2.3.3-3_all.deb \
&& rm -f python-scapy_2.3.3-3_all.deb
&& python3 -m pip install nnpy
RUN curl -fsSL https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py \
&& python2 get-pip.py \
&& rm -f get-pip.py \
&& ln -sf `which pip2` /usr/bin/pip
&& rm -f get-pip.py
RUN pip install --upgrade pip setuptools wheel
RUN pip install allure-pytest==2.8.22 \
ansible==2.8.12 \
RUN python2 -m pip install --upgrade pip setuptools wheel
RUN python2 -m pip install allure-pytest==2.8.22 \
ansible==2.8.20 \
azure-storage-blob==12.9.0 \
celery[redis]==4.4.7 \
cffi==1.12.0 \
@ -156,11 +150,13 @@ RUN pip install allure-pytest==2.8.22 \
netmiko==2.4.2 \
nnpy \
pandas \
paramiko==2.7.1 \
paramiko==2.7.2 \
passlib \
pexpect \
prettytable \
protobuf==3.15.0 \
psutil \
ptf \
pyaml==21.10.1 \
pyasn1==0.1.9 \
pycryptodome==3.9.8 \
@ -181,19 +177,14 @@ RUN pip install allure-pytest==2.8.22 \
retry \
rpyc \
scandir \
scapy==2.4.5 \
six \
snappi[ixnetwork,convergence]==0.7.44 \
snappi[ixnetwork,convergence]==0.11.16 \
statistics \
tabulate \
textfsm==1.1.3 \
thrift==0.11.0 \
virtualenv \
&& git clone https://github.com/p4lang/scapy-vxlan.git \
&& cd scapy-vxlan \
&& python setup.py install \
&& cd .. \
&& rm -fr scapy-vxlan \
&& pip install scapy==2.4.5 --upgrade --ignore-installed
virtualenv
# Install docker-ce-cli
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - \
@ -209,16 +200,9 @@ RUN mkdir -p /etc/apt/keyrings \
&& apt-get update && apt-get install -y azure-cli
## Copy and install sonic-mgmt docker dependencies
COPY \
{% for deb in docker_sonic_mgmt_debs.split(' ') -%}
debs/{{ deb }}{{' '}}
{%- endfor -%}
debs/
COPY debs/sonic-device-data_*.deb debs/
RUN dpkg -i \
{% for deb in docker_sonic_mgmt_debs.split(' ') -%}
debs/{{ deb }}{{' '}}
{%- endfor %}
RUN dpkg -i debs/sonic-device-data_*.deb
# Install protobuf 3.21.12 which is from https://deb.debian.org/debian/pool/main/p/protobuf/protobuf_3.21.12-3.dsc
RUN mkdir -p /tmp/protobuf \
@ -293,7 +277,7 @@ RUN python3 -m pip install aiohttp \
celery[redis]==5.2.7 \
cffi \
contextlib2==0.6.0.post1 \
cryptography==3.3.2 \
cryptography==41.0.2 \
ctypesgen \
dpkt \
dpugen==0.1.1 \
@ -316,7 +300,7 @@ RUN python3 -m pip install aiohttp \
netmiko==2.4.2 \
nnpy \
pandas \
paramiko==2.7.1 \
paramiko==2.7.2 \
passlib \
pexpect \
prettytable \
@ -344,7 +328,7 @@ RUN python3 -m pip install aiohttp \
scapy==2.4.5 \
setuptools-rust \
six \
snappi[ixnetwork,convergence]==0.7.44 \
snappi[ixnetwork,convergence]==0.11.16 \
sshconf==0.2.5 \
tabulate \
textfsm==1.1.2 \
@ -356,8 +340,14 @@ ENV PATH="$BACKUP_OF_PATH"
USER root
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh
COPY start.sh \
0001-Fix-getattr-AttributeError-in-multi-thread-scenario.patch \
./
RUN chmod +x start.sh \
&& ln -sf /usr/bin/python3 /usr/bin/python \
&& ln -sf `which pip3` /usr/bin/pip \
&& ln -sf `which pip3` /usr/local/sbin/pip \
&& patch -u -b /usr/local/lib/python3.8/dist-packages/ansible/plugins/loader.py -i /azp/0001-Fix-getattr-AttributeError-in-multi-thread-scenario.patch
USER $user
WORKDIR /var/$user