[mlnx-fw-upgrade] Add FW reactivation in case 2 FW upgrades were done without reboot (#17092)

- Why I did it
In order to activate FW after it was upgraded need to perform reboot.
If reboot wasn't performed and user need to upgrade to another SONiC image then it will fail.
The reason for that is that during SONiC upgrade new FW should be installed but it will fail because previously installed FW wasn't activated.
In order to allow 2nd FW upgrade without reboot in-between need to reactivate FW image.
This change handles such flow.

Example of issue scenario:

User installed SONiC image on the switch
Then for some reason FW was upgraded by user or script but reboot was not performed to activate it.
After that upgrade to new SONiC image will fail because new image need to install FW but it fails due to previous one wasn't activated.

- How I did it
In "mlnx-fw-upgrade" script check if FW upgrade failed with the error that FW was already installed but reboot was not performed.
If so then perform FW image reactivation and try to upgrade FW again.

- How to verify it
Install SONiC image on the switch
Then upgrade FW but don't perform reboot.
After that upgrade to new SONiC image and check that upgrade was successfull.

Signed-off-by: Volodymyr Samotiy <volodymyrs@nvidia.com>

platform/mellanox/mlnx-fw-upgrade.j2

@@ -30,6 +30,7 @@ declare -r VERBOSE_MIN="${VERBOSE_ERROR}"
 
 declare -r EXIT_SUCCESS="0"
 declare -r EXIT_FAILURE="1"
+declare -r FW_ALREADY_UPDATED_FAILURE="2"
 
 declare -r QUERY_CMD="mlxfwmanager --query"
 declare -r LIST_CONTENT_CMD="mlxfwmanager --list-content"
@@ -243,6 +244,22 @@ function RunFwUpdateCmd() {
     fi
 
     ERROR_CODE="$?"
+
+    if [[ "${ERROR_CODE}" == "${FW_ALREADY_UPDATED_FAILURE}" ]]; then
+        LogInfo "FW reactivation is required. Reactivating and updating FW ..."
+        local -r _MST_DEVICE="$(GetMstDevice)"
+        local -r _CMD="flint -d ${_MST_DEVICE} ir"
+        output=$(eval "${_CMD}")
+
+        if [[ "${VERBOSE_LEVEL}" -eq "${VERBOSE_MAX}" ]]; then
+            output=$(eval "${COMMAND}")
+        else
+            output=$(eval "${COMMAND}") >/dev/null 2>&1
+        fi
+    fi
+
+    ERROR_CODE="$?"
+
     if [[ "${ERROR_CODE}" != "${EXIT_SUCCESS}" ]]; then
         echo "${output}"
         failure_msg="${output#*Fail : }"