staging/rdma/hfi1: Fix fabric serdes reset by re-downloading firmware

A host fabric serdes reset is required to go back to polling.
However, access to the fabric serdes may have been invalidated
by the sibling HFI when it downloads its fabric serdes firmware.
Work around this by re-downloading and re-validating the serdes
firmware at reset time on Bx hardware.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: Dean Luick <dean.luick@intel.com>
Signed-off-by: Jubin John <jubin.john@intel.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>