sonic-buildimage/dockers/docker-database/flush_unused_database

43 lines
1.6 KiB
Plaintext
Raw Normal View History

#!/usr/bin/python3
from swsscommon import swsscommon
import subprocess
import time
[chassis] Fix issues regarding database service failure handling and mid-plane connectivity for namespace. (#10500) What/Why I did: Issue1: By setting up of ipvlan interface in interface-config.sh we are not tolerant to failures. Reason being interface-config.service is one-shot and do not have restart capability. Scenario: For example if let's say database service goes in fail state then interface-services also gets failed because of dependency check but later database service gets restart but interface service will remain in stuck state and the ipvlan interface nevers get created. Solution: Moved all the logic in database service from interface-config service which looks more align logically also since the namespace is created here and all the network setting (sysctl) are happening here.With this if database starts we recreate the interface. Issue 2: Use of IPVLAN vs MACVLAN Currently we are using ipvlan mode. However above failure scenario is not handle correctly by ipvlan mode. Once the ipvlan interface is created and ip address assign to it and if we restart interface-config or database (new PR) service Linux Kernel gives error "Error: Address already assigned to an ipvlan device." based on this:https://github.com/torvalds/linux/blob/master/drivers/net/ipvlan/ipvlan_main.c#L978Reason being if we do not do cleanup of ip address assignment (need to be unique for IPVLAN) it remains in Kernel Database and never goes to free pool even though namespace is deleted. Solution: Considering this hard dependency of unique ip macvlan mode is better for us and since everything is managed by Linux Kernel and no dependency for on user configured IP address. Issue3: Namespace database Service do not check reachability to Supervisor Redis Chassis Server. Currently there is no explicit check as we never do Redis PING from namespace to Supervisor Redis Chassis Server. With this check it's possible we will start database and all other docker even though there is no connectivity and will hit the error/failure late in cycle Solution: Added explicit PING from namespace that will check this reachability. Issue 4:flushdb give exception when trying to accces Chassis Server DB over Unix Sokcet. Solution: Handle gracefully via try..except and log the message.
2022-05-24 18:54:12 -05:00
import syslog
while(True):
output = subprocess.Popen(['sonic-db-cli', 'PING'], stdout=subprocess.PIPE, text=True).communicate()[0]
if 'PONG' in output:
break
time.sleep(1)
Fix docker database flush_unused_database failed issue (#11600) #### Why I did it Fix docker-database flush_unused_database failed issue: https://github.com/Azure/sonic-buildimage/issues/11597 When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update. #### How I did it Change flush_unused_database code to use swsscommon API: Change get_instancelist to getInstanceList. Change get_dblist to getDbList. #### How to verify it Pass all E2E test. Manually check syslog make sure error log not exist and swss, syncd, bgp service started. Search code in Azure make sure there all similer case are fixed in this PR. #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [ ] 202205 #### Description for the changelog Fix docker-database flush_unused_database failed issue: https://github.com/Azure/sonic-buildimage/issues/11597 When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update. #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> #### A picture of a cute animal (not mandatory but encouraged) Co-authored-by: liuh-80 <azureuser@liuh-dev-vm-02.5fg3zjdzj2xezlx1yazx5oxkzd.hx.internal.cloudapp.net>
2022-08-02 21:18:00 -05:00
instlists = swsscommon.SonicDBConfig.getInstanceList()
for instname, v in instlists.items():
Fix docker database flush_unused_database failed issue (#11600) #### Why I did it Fix docker-database flush_unused_database failed issue: https://github.com/Azure/sonic-buildimage/issues/11597 When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update. #### How I did it Change flush_unused_database code to use swsscommon API: Change get_instancelist to getInstanceList. Change get_dblist to getDbList. #### How to verify it Pass all E2E test. Manually check syslog make sure error log not exist and swss, syncd, bgp service started. Search code in Azure make sure there all similer case are fixed in this PR. #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [ ] 202205 #### Description for the changelog Fix docker-database flush_unused_database failed issue: https://github.com/Azure/sonic-buildimage/issues/11597 When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update. #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> #### A picture of a cute animal (not mandatory but encouraged) Co-authored-by: liuh-80 <azureuser@liuh-dev-vm-02.5fg3zjdzj2xezlx1yazx5oxkzd.hx.internal.cloudapp.net>
2022-08-02 21:18:00 -05:00
insthost = v.hostname
instsocket = v.unixSocketPath
Fix docker database flush_unused_database failed issue (#11600) #### Why I did it Fix docker-database flush_unused_database failed issue: https://github.com/Azure/sonic-buildimage/issues/11597 When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update. #### How I did it Change flush_unused_database code to use swsscommon API: Change get_instancelist to getInstanceList. Change get_dblist to getDbList. #### How to verify it Pass all E2E test. Manually check syslog make sure error log not exist and swss, syncd, bgp service started. Search code in Azure make sure there all similer case are fixed in this PR. #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [ ] 202205 #### Description for the changelog Fix docker-database flush_unused_database failed issue: https://github.com/Azure/sonic-buildimage/issues/11597 When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update. #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> #### A picture of a cute animal (not mandatory but encouraged) Co-authored-by: liuh-80 <azureuser@liuh-dev-vm-02.5fg3zjdzj2xezlx1yazx5oxkzd.hx.internal.cloudapp.net>
2022-08-02 21:18:00 -05:00
dblists = swsscommon.SonicDBConfig.getDbList()
for dbname in dblists:
Fix docker database flush_unused_database failed issue (#11600) #### Why I did it Fix docker-database flush_unused_database failed issue: https://github.com/Azure/sonic-buildimage/issues/11597 When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update. #### How I did it Change flush_unused_database code to use swsscommon API: Change get_instancelist to getInstanceList. Change get_dblist to getDbList. #### How to verify it Pass all E2E test. Manually check syslog make sure error log not exist and swss, syncd, bgp service started. Search code in Azure make sure there all similer case are fixed in this PR. #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [ ] 202205 #### Description for the changelog Fix docker-database flush_unused_database failed issue: https://github.com/Azure/sonic-buildimage/issues/11597 When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update. #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> #### A picture of a cute animal (not mandatory but encouraged) Co-authored-by: liuh-80 <azureuser@liuh-dev-vm-02.5fg3zjdzj2xezlx1yazx5oxkzd.hx.internal.cloudapp.net>
2022-08-02 21:18:00 -05:00
dbid = swsscommon.SonicDBConfig.getDbId(dbname)
dbinst = swsscommon.SonicDBConfig.getDbInst(dbname)
# this DB is on current instance, skip flush
if dbinst == instname:
continue
[chassis] Fix issues regarding database service failure handling and mid-plane connectivity for namespace. (#10500) What/Why I did: Issue1: By setting up of ipvlan interface in interface-config.sh we are not tolerant to failures. Reason being interface-config.service is one-shot and do not have restart capability. Scenario: For example if let's say database service goes in fail state then interface-services also gets failed because of dependency check but later database service gets restart but interface service will remain in stuck state and the ipvlan interface nevers get created. Solution: Moved all the logic in database service from interface-config service which looks more align logically also since the namespace is created here and all the network setting (sysctl) are happening here.With this if database starts we recreate the interface. Issue 2: Use of IPVLAN vs MACVLAN Currently we are using ipvlan mode. However above failure scenario is not handle correctly by ipvlan mode. Once the ipvlan interface is created and ip address assign to it and if we restart interface-config or database (new PR) service Linux Kernel gives error "Error: Address already assigned to an ipvlan device." based on this:https://github.com/torvalds/linux/blob/master/drivers/net/ipvlan/ipvlan_main.c#L978Reason being if we do not do cleanup of ip address assignment (need to be unique for IPVLAN) it remains in Kernel Database and never goes to free pool even though namespace is deleted. Solution: Considering this hard dependency of unique ip macvlan mode is better for us and since everything is managed by Linux Kernel and no dependency for on user configured IP address. Issue3: Namespace database Service do not check reachability to Supervisor Redis Chassis Server. Currently there is no explicit check as we never do Redis PING from namespace to Supervisor Redis Chassis Server. With this check it's possible we will start database and all other docker even though there is no connectivity and will hit the error/failure late in cycle Solution: Added explicit PING from namespace that will check this reachability. Issue 4:flushdb give exception when trying to accces Chassis Server DB over Unix Sokcet. Solution: Handle gracefully via try..except and log the message.
2022-05-24 18:54:12 -05:00
try:
# Migrate code from py-redis to swsscommon, original code:
# r = redis.Redis(host=insthost, unix_socket_path=instsocket, db=dbid)
# py-redis will use TCP connection when unix_socket_path is None
# https://github.com/redis/redis-py/blob/d95d8a24ed2af3eae80b7b0f14cbccc9dbe86e96/redis/client.py#L1006
if instsocket is not None:
# connect with Unix socket
connector = swsscommon.DBConnector(dbid, instsocket, 0)
else:
# connect with TCP socket
port = swsscommon.SonicDBConfig.getDbPort(dbname);
connector = swsscommon.DBConnector(dbid, insthost, port, 0)
connector.flushdb()
except RuntimeError:
[chassis] Fix issues regarding database service failure handling and mid-plane connectivity for namespace. (#10500) What/Why I did: Issue1: By setting up of ipvlan interface in interface-config.sh we are not tolerant to failures. Reason being interface-config.service is one-shot and do not have restart capability. Scenario: For example if let's say database service goes in fail state then interface-services also gets failed because of dependency check but later database service gets restart but interface service will remain in stuck state and the ipvlan interface nevers get created. Solution: Moved all the logic in database service from interface-config service which looks more align logically also since the namespace is created here and all the network setting (sysctl) are happening here.With this if database starts we recreate the interface. Issue 2: Use of IPVLAN vs MACVLAN Currently we are using ipvlan mode. However above failure scenario is not handle correctly by ipvlan mode. Once the ipvlan interface is created and ip address assign to it and if we restart interface-config or database (new PR) service Linux Kernel gives error "Error: Address already assigned to an ipvlan device." based on this:https://github.com/torvalds/linux/blob/master/drivers/net/ipvlan/ipvlan_main.c#L978Reason being if we do not do cleanup of ip address assignment (need to be unique for IPVLAN) it remains in Kernel Database and never goes to free pool even though namespace is deleted. Solution: Considering this hard dependency of unique ip macvlan mode is better for us and since everything is managed by Linux Kernel and no dependency for on user configured IP address. Issue3: Namespace database Service do not check reachability to Supervisor Redis Chassis Server. Currently there is no explicit check as we never do Redis PING from namespace to Supervisor Redis Chassis Server. With this check it's possible we will start database and all other docker even though there is no connectivity and will hit the error/failure late in cycle Solution: Added explicit PING from namespace that will check this reachability. Issue 4:flushdb give exception when trying to accces Chassis Server DB over Unix Sokcet. Solution: Handle gracefully via try..except and log the message.
2022-05-24 18:54:12 -05:00
syslog.syslog(syslog.LOG_INFO,"flushdb:Redis Unix Socket connection error for path {} and dbaname {}".format(instsocket, dbname))