sonic-buildimage/dockers/docker-snmp/snmpd.conf.j2

226 lines
7.9 KiB
Plaintext
Raw Normal View History

###############################################################################
# Managed by sonic-config-engine
###############################################################################
#
# EXAMPLE.conf:
# An example configuration file for configuring the Net-SNMP agent ('snmpd')
# See the 'snmpd.conf(5)' man page for details
#
# Some entries are deliberately commented out, and will need to be explicitly activated
#
###############################################################################
#
# AGENT BEHAVIOUR
#
[SNMP][IPv6]: Fix SNMP IPv6 reachability issue in certain scenarios (#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: https://github.com/sonic-net/sonic-buildimage/pull/5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
2023-07-12 11:52:06 -05:00
# Listen for connections on all ip addresses, including eth0, ipv4 lo for multi-asic platform
# Listen on managment and loopback0 ips for single asic platform
#
{% macro protocol(ip_addr) %}
{%- if ip_addr|ipv6 -%}
{{ 'udp6' }}
{%- else -%}
{{ 'udp' }}
{%- endif -%}
{% endmacro %}
{% if SNMP_AGENT_ADDRESS_CONFIG %}
{% for (agentip, port, vrf) in SNMP_AGENT_ADDRESS_CONFIG %}
[SNMP][IPv6]: Fix SNMP IPv6 reachability issue in certain scenarios (#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: https://github.com/sonic-net/sonic-buildimage/pull/5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
2023-07-12 11:52:06 -05:00
agentAddress {{ protocol(agentip) }}:[{{ agentip }}]{% if port %}:{{ port }}{% endif %}{% if vrf %}%{{ vrf }}{% endif %}{{ "" }}
{% endfor %}
[SNMP][IPv6]: Fix SNMP IPv6 reachability issue in certain scenarios (#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: https://github.com/sonic-net/sonic-buildimage/pull/5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
2023-07-12 11:52:06 -05:00
{% elif NAMESPACE_COUNT is not defined or NAMESPACE_COUNT|int <= 1 %}
{% if MGMT_INTERFACE is defined %}
{% for if, ip in MGMT_INTERFACE %}
{% set agentip = ip.split('/')[0] %}
agentAddress {{ protocol(agentip) }}:[{{ agentip }}]:161
{% endfor %}
{% endif %}
{% if LOOPBACK_INTERFACE is defined %}
{% for lo in LOOPBACK_INTERFACE %}
{% if lo | length == 2 %}
{% set agentip = lo[1].split('/')[0] %}
agentAddress {{ protocol(agentip) }}:[{{ agentip }}]:161
{% endif %}
{% endfor %}
{% endif %}
{% else %}
agentAddress udp:161
agentAddress udp6:161
{% endif %}
###############################################################################
#
# ACCESS CONTROL
#
# system + hrSystem groups only
view systemonly included .1.3.6.1.2.1.1
view systemonly included .1.3.6.1.2.1.25.1
# Default access to basic system info
{% if SNMP_COMMUNITY is defined %}
{% for community in SNMP_COMMUNITY %}
{% if SNMP_COMMUNITY[community]['TYPE'] == 'RO' %}
rocommunity {{ community }}
rocommunity6 {{ community }}
{% endif %}
{% endfor %}
{% endif %}
{% if SNMP_COMMUNITY is defined %}
{% for community in SNMP_COMMUNITY %}
{% if SNMP_COMMUNITY[community]['TYPE'] == 'RW' %}
rwcommunity {{ community }}
rwcommunity6 {{ community }}
{% endif %}
{% endfor %}
{% endif %}
{% if SNMP_USER is defined %}
{% for user in SNMP_USER %}
{% if SNMP_USER[user]['SNMP_USER_PERMISSION'] == 'RO' %}
rouser {{ user }} {{ SNMP_USER[user]['SNMP_USER_TYPE'] }}
CreateUser {{ user }} {{ SNMP_USER[user]['SNMP_USER_AUTH_TYPE'] }} {{ SNMP_USER[user]['SNMP_USER_AUTH_PASSWORD'] }} {{ SNMP_USER[user]['SNMP_USER_ENCRYPTION_TYPE'] }} {{ SNMP_USER[user]['SNMP_USER_ENCRYPTION_PASSWORD'] }}
{% elif SNMP_USER[user]['SNMP_USER_PERMISSION'] == 'RW' %}
rwuser {{ user }} {{ SNMP_USER[user]['SNMP_USER_TYPE'] }}
CreateUser {{ user }} {{ SNMP_USER[user]['SNMP_USER_AUTH_TYPE'] }} {{ SNMP_USER[user]['SNMP_USER_AUTH_PASSWORD'] }} {{ SNMP_USER[user]['SNMP_USER_ENCRYPTION_TYPE'] }} {{ SNMP_USER[user]['SNMP_USER_ENCRYPTION_PASSWORD'] }}
{% endif %}
{% endfor %}
{% else %}
{% endif %}
###############################################################################
#
# SYSTEM INFORMATION
#
# Note that setting these values here, results in the corresponding MIB objects being 'read-only'
# See snmpd.conf(5) for more details
{% if SNMP is defined and SNMP.LOCATION is defined %}
sysLocation {{ SNMP.LOCATION.Location }}
{% else %}
sysLocation public
{% endif %}
{% if SNMP is defined and SNMP.CONTACT is defined %}
sysContact {{ SNMP.CONTACT.keys() | first }} {{ SNMP.CONTACT.values() | first }}
{% else %}
sysContact Azure Cloud Switch vteam <linuxnetdev@microsoft.com>
{% endif %}
# Application + End-to-End layers
sysServices 72
#
# Process Monitoring
#
# todo: should we enable snmp based monitoring of sswsyncd and other processes?
# At least one 'sendmail' process, but no more than 10
#proc sendmail 10 1
# Walk the UCD-SNMP-MIB::prTable to see the resulting output
# Note that this table will be empty if there are no "proc" entries in the snmpd.conf file
#
# Disk Monitoring
#
# 10MBs required on root disk, 5% free on /var, 10% free on all other disks
disk / 10000
disk /var 5%
includeAllDisks 10%
# Walk the UCD-SNMP-MIB::dskTable to see the resulting output
# Note that this table will be empty if there are no "disk" entries in the snmpd.conf file
#
# System Load
#
# Unacceptable 1-, 5-, and 15-minute load averages
load 12 10 5
# Walk the UCD-SNMP-MIB::laTable to see the resulting output
# Note that this table *will* be populated, even without a "load" entry in the snmpd.conf file
###############################################################################
#
# ACTIVE MONITORING
#
# Note: disabled snmp traps due to side effect of causing snmpd to listen on all ports (0.0.0.0)
#
# send SNMPv1 traps
{% if SNMP_TRAP_CONFIG and SNMP_TRAP_CONFIG['v1TrapDest'] %}
{% set v1SnmpTrapIp = SNMP_TRAP_CONFIG['v1TrapDest']['DestIp'] %}
{% set v1SnmpTrapPort = SNMP_TRAP_CONFIG['v1TrapDest']['DestPort'] %}
{% set v1SnmpTrapVrf = SNMP_TRAP_CONFIG['v1TrapDest']['vrf'] %}
{% set v1SnmpTrapComm = SNMP_TRAP_CONFIG['v1TrapDest']['Community'] %}
trapsink {{ v1SnmpTrapIp }}:{{ v1SnmpTrapPort }}{% if v1SnmpTrapVrf != 'None' %}%{{ v1SnmpTrapVrf }}{% endif %} {{ v1SnmpTrapComm }}{{ "" }}
{% else %}
#trapsink localhost public
{% endif %}
# send SNMPv2c traps
{% if SNMP_TRAP_CONFIG and SNMP_TRAP_CONFIG['v2TrapDest'] %}
{% set v2SnmpTrapIp = SNMP_TRAP_CONFIG['v2TrapDest']['DestIp'] %}
{% set v2SnmpTrapPort = SNMP_TRAP_CONFIG['v2TrapDest']['DestPort'] %}
{% set v2SnmpTrapVrf = SNMP_TRAP_CONFIG['v2TrapDest']['vrf'] %}
{% set v2SnmpTrapComm = SNMP_TRAP_CONFIG['v2TrapDest']['Community'] %}
trap2sink {{ v2SnmpTrapIp }}:{{ v2SnmpTrapPort }}{% if v2SnmpTrapVrf != 'None' %}%{{ v2SnmpTrapVrf }}{% endif %} {{ v2SnmpTrapComm }}{{ "" }}
{% else %}
#trap2sink localhost public
{% endif %}
# send SNMPv2c INFORMs
{% if SNMP_TRAP_CONFIG and SNMP_TRAP_CONFIG['v3TrapDest'] %}
{% set v3SnmpTrapIp = SNMP_TRAP_CONFIG['v3TrapDest']['DestIp'] %}
{% set v3SnmpTrapPort = SNMP_TRAP_CONFIG['v3TrapDest']['DestPort'] %}
{% set v3SnmpTrapVrf = SNMP_TRAP_CONFIG['v3TrapDest']['vrf'] %}
{% set v3SnmpTrapComm = SNMP_TRAP_CONFIG['v3TrapDest']['Community'] %}
trapsink {{ v3SnmpTrapIp }}:{{ v3SnmpTrapPort }}{% if v3SnmpTrapVrf != 'None' %}%{{ v3SnmpTrapVrf }}{% endif %} {{ v3SnmpTrapComm }}{{ "" }}
{% else %}
#informsink localhost public
{% endif %}
# Note that you typically only want *one* of these three lines
# Uncommenting two (or all three) will result in multiple copies of each notification.
#
# Event MIB - automatically generate alerts
#
# Remember to activate the 'createUser' lines above
2017-05-16 18:46:30 -05:00
#iquerySecName internalUser
#rouser internalUser
# generate traps on UCD error conditions
#defaultMonitors yes
#note, this release of snmpd does not support linkUpDownNotifications
# generate traps on linkUp/Down
#linkUpDownNotifications yes
#
# AgentX Sub-agents
#
# Run as an AgentX master agent
master agentx
# internal socket to allow extension to other docker containers
# Currently the other container using this is docker-fpm-frr
# make sure this line matches bgp:/etc/snmp/frr.conf
# please see testing procedure in the same file to verify this works
# to verify the SNMP docker side look for the following string in the log file:
# INFO snmp-subagent [ax_interface] INFO: Using agentx socket type tcp with path tcp:localhost:3161
# INFO supervisord snmp-subagent INFO:ax_interface:Using agentx socket type tcp with path tcp:localhost:3161
agentxsocket tcp:localhost:3161
#
# SysDescription pass-through
#
pass -p 10 .1.3.6.1.2.1.1.1 /usr/share/snmp/sysDescr_pass.py