staging/rdma/hfi1: Fix downgrade race

A link downgrade can race with link up. Avoid the race
in two ways. First, by having the downgrade application logic
take the link state mutex for all of its checking. Second, by
waiting for the link to move out of the going up state.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: Dean Luick <dean.luick@intel.com>
Signed-off-by: Jubin John <jubin.john@intel.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>