matthew/sonic-buildimage
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
b9d21a5779
sonic-buildimage/platform/mellanox/docker-syncd-mlnx/supervisord.conf

40 lines
857 B
Plaintext
Raw Normal View History

[Dockers]: Manage all Docker containers with Supervisord (#573) - Consolidate config.sh and start.sh scripts into one script (start.sh) - Solve issue #435 - All dockers now run supervisord as their ENTRYPOINT - All stdout/stderr output from processes managed by supervisord is now sent to syslog instead of their own files - Supervisord log messages are now also sent to syslog - Removed unused smartmontools package from docker-platform-monitor
2017-05-08 17:43:31 -05:00
[supervisord]
logfile_maxbytes=1MB
logfile_backups=2
nodaemon=true
[docker-syncd-mlnx]: use service dependency in supervisord to start services
2020-05-16 10:46:18 -05:00
[eventlistener:dependent-startup]
[supervisor] Install vanilla package once again, install Python 3 version in Buster container (#5546) **- Why I did it** We were building a custom version of Supervisor because I had added patches to prevent hangs and crashes if the system clock ever rolled backward. Those changes were merged into the upstream Supervisor repo as of version 3.4.0 (http://supervisord.org/changes.html#id9), therefore, we should be able to simply install the vanilla package via pip. This will also allow us to easily move to Python 3, as Python 3 support was added in version 4.0.0. **- How I did it** - Remove Makefiles and patches for building supervisor package from source - Install Python 3 supervisor package version 4.2.1 in Buster base container - Also install Python 3 version of supervisord-dependent-startup in Buster base container - Debian package installed binary in `/usr/bin/`, but pip package installs in `/usr/local/bin/`, so rather than update all absolute paths, I changed all references to simply call `supervisord` and let the system PATH find the executable to prevent future need for changes just in case we ever need to switch back to build a Debian package, then we won't need to modify these again. - Install Python 2 supervisor package >= 3.4.0 in Stretch and Jessie base containers
2020-11-20 01:41:32 -06:00
command=python3 -m supervisord_dependent_startup
[docker-syncd-mlnx]: use service dependency in supervisord to start services
2020-05-16 10:46:18 -05:00
autostart=true
autorestart=unexpected
startretries=0
exitcodes=0,3
events=PROCESS_STATE
[202012][dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 (#7203) #### Why I did it Backport of https://github.com/Azure/sonic-buildimage/pull/7083 to the 202012 branch. To prevent error [messages](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802) like the following from being logged: ``` Mar 17 02:33:48.523153 vlab-01 INFO swss#supervisord 2021-03-17 02:33:48,518 ERRO pool supervisor-proc-exit-listener event buffer overflowed, discarding event 46 ``` This is basically an addendum to https://github.com/Azure/sonic-buildimage/pull/5247, which increased the event buffer size for dependent-startup. While supervisor-proc-exit-listener doesn't subscribe to as many events as dependent-startup, there is still a chance some containers (like swss, as in the example above) have enough processes running to cause an overflow of the default buffer size of 10. This is especially important for preventing erroneous log_analyzer failures in the sonic-mgmt repo regression tests, which have started occasionally causing PR check builds to fail. Example [here](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802). I set all supervisor-proc-exit-listener event buffer sizes to 1024, and also updated all dependent-startup event buffer sizes to 1024, as well, to keep things simple, unified, and allow headroom so that we will not need to adjust these values frequently, if at all.
2021-04-01 14:52:19 -05:00
buffer_size=1024
[docker-syncd-mlnx]: use service dependency in supervisord to start services
2020-05-16 10:46:18 -05:00
[docker-syncd]: Restart SwSS, syncd and dependent services if a critical process in syncd container exits unexpectedly (#3534) Add the same mechanism I developed for the SwSS service in #2845 to the syncd service. However, in order to cause the SwSS service to also exit and restart in this situation, I developed a docker-wait-any program which the SwSS service uses to wait for either the swss or syncd containers to exit.
2019-11-09 12:26:39 -06:00
[eventlistener:supervisor-proc-exit-listener]
[Service] Enable/disable container auto-restart based on configuration. (#4073)
2020-02-07 14:34:07 -06:00
command=/usr/bin/supervisor-proc-exit-listener --container-name syncd
[supervisord] Monitoring the critical processes with supervisord. (#6242) - Why I did it Initially, we used Monit to monitor critical processes in each container. If one of critical processes was not running or crashed due to some reasons, then Monit will write an alerting message into syslog periodically. If we add a new process in a container, the corresponding Monti configuration file will also need to update. It is a little hard for maintenance. Currently we employed event listener of Supervisod to do this monitoring. Since processes in each container are managed by Supervisord, we can only focus on the logic of monitoring. - How I did it We borrowed the event listener of Supervisord to monitor critical processes in containers. The event listener will take following steps if it was notified one of critical processes exited unexpectedly: The event listener will first check whether the auto-restart mechanism was enabled for this container or not. If auto-restart mechanism was enabled, event listener will kill the Supervisord process, which should cause the container to exit and subsequently get restarted. If auto-restart mechanism was not enabled for this contianer, the event listener will enter a loop which will first sleep 1 minute and then check whether the process is running. If yes, the event listener exits. If no, an alerting message will be written into syslog. - How to verify it First, we need checked whether the auto-restart mechanism of a container was enabled or not by running the command show feature status. If enabled, one critical process should be selected and killed manually, then we need check whether the container will be restarted or not. Second, we can disable the auto-restart mechanism if it was enabled at step 1 by running the commnad sudo config feature autorestart <container_name> disabled. Then one critical process should be selected and killed. After that, we will see the alerting message which will appear in the syslog every 1 minute. - Which release branch to backport (provide reason below if selected) 201811 201911 [x ] 202006
2021-01-21 14:57:49 -06:00
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
[docker-syncd]: Restart SwSS, syncd and dependent services if a critical process in syncd container exits unexpectedly (#3534) Add the same mechanism I developed for the SwSS service in #2845 to the syncd service. However, in order to cause the SwSS service to also exit and restart in this situation, I developed a docker-wait-any program which the SwSS service uses to wait for either the swss or syncd containers to exit.
2019-11-09 12:26:39 -06:00
autostart=true
autorestart=unexpected
[202012][dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 (#7203) #### Why I did it Backport of https://github.com/Azure/sonic-buildimage/pull/7083 to the 202012 branch. To prevent error [messages](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802) like the following from being logged: ``` Mar 17 02:33:48.523153 vlab-01 INFO swss#supervisord 2021-03-17 02:33:48,518 ERRO pool supervisor-proc-exit-listener event buffer overflowed, discarding event 46 ``` This is basically an addendum to https://github.com/Azure/sonic-buildimage/pull/5247, which increased the event buffer size for dependent-startup. While supervisor-proc-exit-listener doesn't subscribe to as many events as dependent-startup, there is still a chance some containers (like swss, as in the example above) have enough processes running to cause an overflow of the default buffer size of 10. This is especially important for preventing erroneous log_analyzer failures in the sonic-mgmt repo regression tests, which have started occasionally causing PR check builds to fail. Example [here](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802). I set all supervisor-proc-exit-listener event buffer sizes to 1024, and also updated all dependent-startup event buffer sizes to 1024, as well, to keep things simple, unified, and allow headroom so that we will not need to adjust these values frequently, if at all.
2021-04-01 14:52:19 -05:00
buffer_size=1024
[docker-syncd]: Restart SwSS, syncd and dependent services if a critical process in syncd container exits unexpectedly (#3534) Add the same mechanism I developed for the SwSS service in #2845 to the syncd service. However, in order to cause the SwSS service to also exit and restart in this situation, I developed a docker-wait-any program which the SwSS service uses to wait for either the swss or syncd containers to exit.
2019-11-09 12:26:39 -06:00
[Dockers]: Manage all Docker containers with Supervisord (#573) - Consolidate config.sh and start.sh scripts into one script (start.sh) - Solve issue #435 - All dockers now run supervisord as their ENTRYPOINT - All stdout/stderr output from processes managed by supervisord is now sent to syslog instead of their own files - Supervisord log messages are now also sent to syslog - Removed unused smartmontools package from docker-platform-monitor
2017-05-08 17:43:31 -05:00
[program:rsyslogd]
[docker-syncd-mlnx]: use service dependency in supervisord to start services
2020-05-16 10:46:18 -05:00
command=/usr/sbin/rsyslogd -n -iNONE
priority=1
[Dockers]: Manage all Docker containers with Supervisord (#573) - Consolidate config.sh and start.sh scripts into one script (start.sh) - Solve issue #435 - All dockers now run supervisord as their ENTRYPOINT - All stdout/stderr output from processes managed by supervisord is now sent to syslog instead of their own files - Supervisord log messages are now also sent to syslog - Removed unused smartmontools package from docker-platform-monitor
2017-05-08 17:43:31 -05:00
autostart=false
[docker-syncd]: Restart SwSS, syncd and dependent services if a critical process in syncd container exits unexpectedly (#3534) Add the same mechanism I developed for the SwSS service in #2845 to the syncd service. However, in order to cause the SwSS service to also exit and restart in this situation, I developed a docker-wait-any program which the SwSS service uses to wait for either the swss or syncd containers to exit.
2019-11-09 12:26:39 -06:00
autorestart=unexpected
[Dockers]: Manage all Docker containers with Supervisord (#573) - Consolidate config.sh and start.sh scripts into one script (start.sh) - Solve issue #435 - All dockers now run supervisord as their ENTRYPOINT - All stdout/stderr output from processes managed by supervisord is now sent to syslog instead of their own files - Supervisord log messages are now also sent to syslog - Removed unused smartmontools package from docker-platform-monitor
2017-05-08 17:43:31 -05:00
stdout_logfile=syslog
stderr_logfile=syslog
[docker-syncd-mlnx]: use service dependency in supervisord to start services
2020-05-16 10:46:18 -05:00
dependent_startup=true
[Dockers]: Manage all Docker containers with Supervisord (#573) - Consolidate config.sh and start.sh scripts into one script (start.sh) - Solve issue #435 - All dockers now run supervisord as their ENTRYPOINT - All stdout/stderr output from processes managed by supervisord is now sent to syslog instead of their own files - Supervisord log messages are now also sent to syslog - Removed unused smartmontools package from docker-platform-monitor
2017-05-08 17:43:31 -05:00
[program:syncd]
[docker-syncd-*]: Properly manage syncd with supervisord (#617) - This PR allows supervisord to log syncd exit events to syslog - Syncd dockers now are built from docker-config-engine instead of docker-base - Supervisord in all syncd dockers now call syncd_start.s which is installed by sonic-sairedis repo
2017-05-24 13:53:38 -05:00
command=/usr/bin/syncd_start.sh
[docker-syncd-mlnx]: use service dependency in supervisord to start services
2020-05-16 10:46:18 -05:00
priority=2
[Dockers]: Manage all Docker containers with Supervisord (#573) - Consolidate config.sh and start.sh scripts into one script (start.sh) - Solve issue #435 - All dockers now run supervisord as their ENTRYPOINT - All stdout/stderr output from processes managed by supervisord is now sent to syslog instead of their own files - Supervisord log messages are now also sent to syslog - Removed unused smartmontools package from docker-platform-monitor
2017-05-08 17:43:31 -05:00
autostart=false
[dockers]: Disable autorestart on all supervisor processes inside containers (#580)
2017-05-09 19:37:08 -05:00
autorestart=false
[Dockers]: Manage all Docker containers with Supervisord (#573) - Consolidate config.sh and start.sh scripts into one script (start.sh) - Solve issue #435 - All dockers now run supervisord as their ENTRYPOINT - All stdout/stderr output from processes managed by supervisord is now sent to syslog instead of their own files - Supervisord log messages are now also sent to syslog - Removed unused smartmontools package from docker-platform-monitor
2017-05-08 17:43:31 -05:00
stdout_logfile=syslog
stderr_logfile=syslog
[docker-syncd-mlnx]: use service dependency in supervisord to start services
2020-05-16 10:46:18 -05:00
dependent_startup=true
dependent_startup_wait_for=rsyslogd:running
Reference in New Issue Copy Permalink