matthew/sonic-buildimage
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Add monit_snmp file to monitor memory usage (#14464)

Browse Source 
#### Why I did it

When CPU is busy, the sonic_ax_impl may not have sufficient speed to handle the notification message sent from REDIS.
Thus, the message will keep stacking in the memory space of sonic_ax_impl.

If the condition continues, the memory usage will keep increasing.

#### How I did it

Add a monit file to check if the SNMP container where sonic_ax_impl resides in use more than 4GB memory.
If yes, restart the sonic_ax_impl process.

#### How to verify it

Run a lot of this command: `while true; do ret=$(redis-cli -n 0 set LLDP_ENTRY_TABLE:test1 test1); sleep 0.1; done;`
And check the memory used by sonic_ax_impl keeps increasing.

After a period, make sure the sonic_ax_impl is restarted when the memory usage reaches the 4GB threshold.
And verify the memory usage of sonic_ax_impl drops down from 4GB.
This commit is contained in:
Kuanyu Chen 2023-04-07 03:19:11 +08:00 committed by GitHub
parent dd6659ae07
commit cffd87a627
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 6 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
dockers/docker-snmp/base_image_files/monit_snmp Normal file
View File

@ -0,0 +1,5 @@
###############################################################################
## Monit configuration for snmp container
###############################################################################
check program container_memory_snmp with path "/usr/bin/memory_checker snmp 4294967296"
if status == 3 for 10 times within 20 cycles then exec "/usr/bin/docker exec snmp supervisorctl restart snmp-subagent"

1
rules/docker-snmp.mk
View File

@ -32,6 +32,7 @@ $(DOCKER_SNMP)_RUN_OPT += --privileged -t
$(DOCKER_SNMP)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro $(DOCKER_SNMP)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro
$(DOCKER_SNMP)_RUN_OPT += -v /usr/share/sonic/scripts:/usr/share/sonic/scripts:ro $(DOCKER_SNMP)_RUN_OPT += -v /usr/share/sonic/scripts:/usr/share/sonic/scripts:ro
$(DOCKER_SNMP)_FILES += $(SUPERVISOR_PROC_EXIT_LISTENER_SCRIPT) $(DOCKER_SNMP)_FILES += $(SUPERVISOR_PROC_EXIT_LISTENER_SCRIPT)
$(DOCKER_SNMP)_BASE_IMAGE_FILES += monit_snmp:/etc/monit/conf.d
SONIC_BULLSEYE_DOCKERS += $(DOCKER_SNMP) SONIC_BULLSEYE_DOCKERS += $(DOCKER_SNMP)
SONIC_BULLSEYE_DBG_DOCKERS += $(DOCKER_SNMP_DBG) SONIC_BULLSEYE_DBG_DOCKERS += $(DOCKER_SNMP_DBG)