sonic-buildimage/dockers/docker-orchagent/supervisord.conf.j2

302 lines
7.0 KiB
Plaintext
Raw Normal View History

[supervisord]
logfile_maxbytes=1MB
logfile_backups=2
nodaemon=true
[eventlistener:dependent-startup]
command=python3 -m supervisord_dependent_startup
autostart=true
autorestart=unexpected
startretries=0
exitcodes=0,3
events=PROCESS_STATE
[dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 (#7083) To prevent error [messages](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802) like the following from being logged: ``` Mar 17 02:33:48.523153 vlab-01 INFO swss#supervisord 2021-03-17 02:33:48,518 ERRO pool supervisor-proc-exit-listener event buffer overflowed, discarding event 46 ``` This is basically an addendum to https://github.com/Azure/sonic-buildimage/pull/5247, which increased the event buffer size for dependent-startup. While supervisor-proc-exit-listener doesn't subscribe to as many events as dependent-startup, there is still a chance some containers (like swss, as in the example above) have enough processes running to cause an overflow of the default buffer size of 10. This is especially important for preventing erroneous log_analyzer failures in the sonic-mgmt repo regression tests, which have started occasionally causing PR check builds to fail. Example [here](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802). I set all supervisor-proc-exit-listener event buffer sizes to 1024, and also updated all dependent-startup event buffer sizes to 1024, as well, to keep things simple, unified, and allow headroom so that we will not need to adjust these values frequently, if at all.
2021-03-27 23:14:24 -05:00
buffer_size=1024
[eventlistener:supervisor-proc-exit-listener]
command=/usr/bin/supervisor-proc-exit-listener --container-name swss
[supervisord] Monitoring the critical processes with supervisord. (#6242) - Why I did it Initially, we used Monit to monitor critical processes in each container. If one of critical processes was not running or crashed due to some reasons, then Monit will write an alerting message into syslog periodically. If we add a new process in a container, the corresponding Monti configuration file will also need to update. It is a little hard for maintenance. Currently we employed event listener of Supervisod to do this monitoring. Since processes in each container are managed by Supervisord, we can only focus on the logic of monitoring. - How I did it We borrowed the event listener of Supervisord to monitor critical processes in containers. The event listener will take following steps if it was notified one of critical processes exited unexpectedly: The event listener will first check whether the auto-restart mechanism was enabled for this container or not. If auto-restart mechanism was enabled, event listener will kill the Supervisord process, which should cause the container to exit and subsequently get restarted. If auto-restart mechanism was not enabled for this contianer, the event listener will enter a loop which will first sleep 1 minute and then check whether the process is running. If yes, the event listener exits. If no, an alerting message will be written into syslog. - How to verify it First, we need checked whether the auto-restart mechanism of a container was enabled or not by running the command show feature status. If enabled, one critical process should be selected and killed manually, then we need check whether the container will be restarted or not. Second, we can disable the auto-restart mechanism if it was enabled at step 1 by running the commnad sudo config feature autorestart <container_name> disabled. Then one critical process should be selected and killed. After that, we will see the alerting message which will appear in the syslog every 1 minute. - Which release branch to backport (provide reason below if selected) 201811 201911 [x ] 202006
2021-01-21 14:57:49 -06:00
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
autostart=true
autorestart=unexpected
[dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 (#7083) To prevent error [messages](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802) like the following from being logged: ``` Mar 17 02:33:48.523153 vlab-01 INFO swss#supervisord 2021-03-17 02:33:48,518 ERRO pool supervisor-proc-exit-listener event buffer overflowed, discarding event 46 ``` This is basically an addendum to https://github.com/Azure/sonic-buildimage/pull/5247, which increased the event buffer size for dependent-startup. While supervisor-proc-exit-listener doesn't subscribe to as many events as dependent-startup, there is still a chance some containers (like swss, as in the example above) have enough processes running to cause an overflow of the default buffer size of 10. This is especially important for preventing erroneous log_analyzer failures in the sonic-mgmt repo regression tests, which have started occasionally causing PR check builds to fail. Example [here](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802). I set all supervisor-proc-exit-listener event buffer sizes to 1024, and also updated all dependent-startup event buffer sizes to 1024, as well, to keep things simple, unified, and allow headroom so that we will not need to adjust these values frequently, if at all.
2021-03-27 23:14:24 -05:00
buffer_size=1024
[program:rsyslogd]
command=/usr/sbin/rsyslogd -n -iNONE
priority=1
autostart=false
autorestart=unexpected
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
{% set is_fabric_asic = 0 %}
{% set orchagent_dependent_startup_wait_for = "portsyncd:running" %}
{% if DEVICE_METADATA.localhost.switch_type %}
{% if DEVICE_METADATA.localhost.switch_type == "fabric" %}
{% set is_fabric_asic = 1 %}
{% set orchagent_dependent_startup_wait_for = "rsyslogd:running" %}
{%- endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
Add gearbox phy device files and a new physyncd docker to support VS gearbox phy feature (#4851) * buildimage: Add gearbox phy device files and a new physyncd docker to support VS gearbox phy feature * scripts and configuration needed to support a second syncd docker (physyncd) * physyncd supports gearbox device and phy SAI APIs and runs multiple instances of syncd, one per phy in the device * support for VS target (sonic-sairedis vslib has been extended to support a virtual BCM81724 gearbox PHY). HLD is located at https://github.com/Azure/SONiC/blob/b817a12fd89520d3fd26bbc5897487928e7f6de7/doc/gearbox/gearbox_mgr_design.md **- Why I did it** This work is part of the gearbox phy joint effort between Microsoft and Broadcom, and is based on multi-switch support in sonic-sairedis. **- How I did it** Overall feature was implemented across several projects. The collective pull requests (some in late stages of review at this point): https://github.com/Azure/sonic-utilities/pull/931 - CLI (merged) https://github.com/Azure/sonic-swss-common/pull/347 - Minor changes (merged) https://github.com/Azure/sonic-swss/pull/1321 - gearsyncd, config parsers, changes to orchargent to create gearbox phy on supported systems https://github.com/Azure/sonic-sairedis/pull/624 - physyncd, virtual BCM81724 gearbox phy added to vslib **- How to verify it** In a vslib build: root@sonic:/home/admin# show gearbox interfaces status PHY Id Interface MAC Lanes MAC Lane Speed PHY Lanes PHY Lane Speed Line Lanes Line Lane Speed Oper Admin -------- ----------- --------------- ---------------- --------------- ---------------- ------------ ----------------- ------ ------- 1 Ethernet48 121,122,123,124 25G 200,201,202,203 25G 204,205 50G down down 1 Ethernet49 125,126,127,128 25G 206,207,208,209 25G 210,211 50G down down 1 Ethernet50 69,70,71,72 25G 212,213,214,215 25G 216 100G down down In addition, docker ps | grep phy should show a physyncd docker running. Signed-off-by: syd.logan@broadcom.com
2020-09-25 10:32:44 -05:00
[program:gearsyncd]
command=/usr/bin/gearsyncd -p /usr/share/sonic/hwsku/gearbox_config.json
priority=3
autostart=false
autorestart=false
startsecs=0
startretries=0
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=rsyslogd:running
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/gearsyncd-asan.log"
{% endif %}
{%- endif %}
Add gearbox phy device files and a new physyncd docker to support VS gearbox phy feature (#4851) * buildimage: Add gearbox phy device files and a new physyncd docker to support VS gearbox phy feature * scripts and configuration needed to support a second syncd docker (physyncd) * physyncd supports gearbox device and phy SAI APIs and runs multiple instances of syncd, one per phy in the device * support for VS target (sonic-sairedis vslib has been extended to support a virtual BCM81724 gearbox PHY). HLD is located at https://github.com/Azure/SONiC/blob/b817a12fd89520d3fd26bbc5897487928e7f6de7/doc/gearbox/gearbox_mgr_design.md **- Why I did it** This work is part of the gearbox phy joint effort between Microsoft and Broadcom, and is based on multi-switch support in sonic-sairedis. **- How I did it** Overall feature was implemented across several projects. The collective pull requests (some in late stages of review at this point): https://github.com/Azure/sonic-utilities/pull/931 - CLI (merged) https://github.com/Azure/sonic-swss-common/pull/347 - Minor changes (merged) https://github.com/Azure/sonic-swss/pull/1321 - gearsyncd, config parsers, changes to orchargent to create gearbox phy on supported systems https://github.com/Azure/sonic-sairedis/pull/624 - physyncd, virtual BCM81724 gearbox phy added to vslib **- How to verify it** In a vslib build: root@sonic:/home/admin# show gearbox interfaces status PHY Id Interface MAC Lanes MAC Lane Speed PHY Lanes PHY Lane Speed Line Lanes Line Lane Speed Oper Admin -------- ----------- --------------- ---------------- --------------- ---------------- ------------ ----------------- ------ ------- 1 Ethernet48 121,122,123,124 25G 200,201,202,203 25G 204,205 50G down down 1 Ethernet49 125,126,127,128 25G 206,207,208,209 25G 210,211 50G down down 1 Ethernet50 69,70,71,72 25G 212,213,214,215 25G 216 100G down down In addition, docker ps | grep phy should show a physyncd docker running. Signed-off-by: syd.logan@broadcom.com
2020-09-25 10:32:44 -05:00
{% if is_fabric_asic == 0 %}
[program:portsyncd]
command=/usr/bin/portsyncd
priority=3
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=rsyslogd:running
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/portsyncd-asan.log"
{% endif %}
{%- endif %}
[program:orchagent]
command=/usr/bin/orchagent.sh
priority=4
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for={{ orchagent_dependent_startup_wait_for }}
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/orchagent-asan.log"
{% endif %}
[program:swssconfig]
command=/usr/bin/swssconfig.sh
priority=5
autostart=false
autorestart=unexpected
startretries=0
startsecs=0
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=orchagent:running
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/swssconfig-asan.log"
{% endif %}
{% if is_fabric_asic == 0 %}
[program:restore_neighbors]
command=/usr/bin/restore_neighbors.py
priority=6
autostart=false
autorestart=false
startsecs=0
startretries=0
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:coppmgrd]
command=/usr/bin/coppmgrd
priority=6
autostart=false
autorestart=false
startretries=0
startsecs=0
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=orchagent:running
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/coppmgrd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:neighsyncd]
command=/usr/bin/neighsyncd
priority=7
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/neighsyncd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:vlanmgrd]
command=/usr/bin/vlanmgrd
priority=8
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/vlanmgrd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:intfmgrd]
command=/usr/bin/intfmgrd
priority=9
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/intfmgrd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:portmgrd]
command=/usr/bin/portmgrd
priority=10
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/portmgrd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:buffermgrd]
[Dynamic buffer calc] Support dynamic buffer calculation (#6194) **- Why I did it** To support dynamic buffer calculation. This PR also depends on the following PRs for sub modules - [sonic-swss: [buffermgr/bufferorch] Support dynamic buffer calculation #1338](https://github.com/Azure/sonic-swss/pull/1338) - [sonic-swss-common: Dynamic buffer calculation #361](https://github.com/Azure/sonic-swss-common/pull/361) - [sonic-utilities: Support dynamic buffer calculation #973](https://github.com/Azure/sonic-utilities/pull/973) **- How I did it** 1. Introduce field `buffer_model` in `DEVICE_METADATA|localhost` to represent which buffer model is running in the system currently: - `dynamic` for the dynamic buffer calculation model - `traditional` for the traditional model in which the `pg_profile_lookup.ini` is used 2. Add the tables required for the feature: - ASIC_TABLE in platform/\<vendor\>/asic_table.j2 - PERIPHERAL_TABLE in platform/\<vendor\>/peripheral_table.j2 - PORT_PERIPHERAL_TABLE on a per-platform basis in device/\<vendor\>/\<platform\>/port_peripheral_config.j2 for each platform with gearbox installed. - DEFAULT_LOSSLESS_BUFFER_PARAMETER and LOSSLESS_TRAFFIC_PATTERN in files/build_templates/buffers_config.j2 - Add lossless PGs (3-4) for each port in files/build_templates/buffers_config.j2 3. Copy the newly introduced j2 files into the image and rendering them when the system starts 4. Update the CLI options for buffermgrd so that it can start with dynamic mode 5. Fetches the ASIC vendor name in orchagent: - fetch the vendor name when creates the docker and pass it as a docker environment variable - `buffermgrd` can use this passed-in variable 6. Clear buffer related tables from STATE_DB when swss docker starts 7. Update the src/sonic-config-engine/tests/sample_output/buffers-dell6100.json according to the buffer_config.j2 8. Remove buffer pool sizes for ingress pools and egress_lossy_pool Update the buffer settings for dynamic buffer calculation
2020-12-13 13:35:39 -06:00
command=/usr/bin/buffermgrd.sh
priority=11
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/buffermgrd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:vrfmgrd]
command=/usr/bin/vrfmgrd
priority=13
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/vrfmgrd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:nbrmgrd]
command=/usr/bin/nbrmgrd
priority=15
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/nbrmgrd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:vxlanmgrd]
command=/usr/bin/vxlanmgrd
priority=16
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/vxlanmgrd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:tunnelmgrd]
command=/usr/bin/tunnelmgrd
priority=17
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/tunnelmgrd-asan.log"
{% endif %}
{%- endif %}
[program:enable_counters]
command=/usr/bin/enable_counters.py
priority=12
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if is_fabric_asic == 0 %}
[program:fdbsyncd]
command=/usr/bin/fdbsyncd
priority=17
autostart=false
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{% if ENABLE_ASAN == "y" %}
environment=ASAN_OPTIONS="log_path=/var/log/asan/fdbsyncd-asan.log"
{% endif %}
{%- endif %}
{% if is_fabric_asic == 0 %}
[program:tunnel_packet_handler]
command=/usr/bin/tunnel_packet_handler.py
priority=12
autostart=false
autorestart=unexpected
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
dependent_startup_wait_for=swssconfig:exited
{%- endif %}