[Mellanox][sai_failure_dump]Added platform specific script to be invoked during SAI failure dump (#13533) (#13749)

- Why I did it
Added platform specific script to be invoked during SAI failure dump. Added some generic changes to mount /var/log/sai_failure_dump as read write in the syncd docker

- How I did it
Added script in docker-syncd of mellanox and copied it to /usr/bin

- How to verify it
Manual UT and new sonic-mgmt tests

Co-authored-by: Sudharsan Dhamal Gopalarathnam <dgsudharsan@users.noreply.github.com>

files/build_templates/docker_image_ctl.j2

@@ -495,6 +495,7 @@ start() {
         -v mlnx_sdk_socket:/var/run/sx_sdk \
         -v mlnx_sdk_ready:/tmp \
         -v /dev/shm:/dev/shm:rw \
+        -v /var/log/sai_failure_dump:/var/log/sai_failure_dump:rw \
         -e SX_API_SOCKET_FILE=/var/run/sx_sdk/sx_api.sock \
 {%- elif docker_container_name == "pmon" %}
         -v /var/run/hw-management:/var/run/hw-management:rw \

platform/mellanox/docker-syncd-mlnx/Dockerfile.j2

@@ -65,6 +65,7 @@ RUN apt-get clean -y && \
 COPY ["supervisord.conf.j2", "/usr/share/sonic/templates/"]
 COPY ["files/supervisor-proc-exit-listener", "/usr/bin"]
 COPY ["critical_processes", "/etc/supervisor/"]
+COPY ["platform_syncd_dump.sh", "/usr/bin/"]
 
 RUN mkdir -p /etc/supervisor/conf.d/
 RUN sonic-cfggen -a "{\"ENABLE_ASAN\":\"{{ENABLE_ASAN}}\"}" -t /usr/share/sonic/templates/supervisord.conf.j2 > /etc/supervisor/conf.d/supervisord.conf

platform/mellanox/docker-syncd-mlnx/platform_syncd_dump.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+#
+# Script for sai failure dump
+#
+
+# Source the platform specific dump file
+
+sai_dump_name="sai_sdk_dump_$(date +"%m_%d_%Y_%I_%M_%p")"
+sai_dump_path="${DUMPDIR}/$sai_dump_name"
+mkdir -p $sai_dump_path
+sai_dump_file="${sai_dump_path}/$sai_dump_name"
+saisdkdump -f $sai_dump_file
+cd "${DUMPDIR}"
+tar -cvf "$sai_dump_name".tar $sai_dump_name
+gzip "$sai_dump_name".tar
+rm -rf $sai_dump_name
+
+# Update max failure dumps
+if grep -q SAI_DUMP_STORE_AMOUNT /usr/share/sonic/hwsku/sai.profile; then
+    SAI_MAX_FAILURE_DUMPS=$(grep SAI_DUMP_STORE_AMOUNT /usr/share/sonic/hwsku/sai.profile | cut -d '=' -f2)
+fi
+