staging: rtl8712: _r8712_init_sta_priv(): Change return values

Add check for the return value of function _r8712_init_sta_priv at call
site.
Change return values of the function from _SUCCESS/_FAIL to 0/-ENOMEM
respectively.
Change return type of the function from u32 to int to enable return of
-ENOMEM.

Signed-off-by: Nishka Dasgupta <nishkadg.linux@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>