rtlwifi: btcoexist: Replace RT_TRACE with rtl_dbg

Change the misleading macro name to one that is more descriptive for
btcoexist.

This change exposed some conditions reported by checkpatch.pl.
These are also fixed.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
Signed-off-by: Kalle Valo <kvalo@codeaurora.org>
Link: https://lore.kernel.org/r/20200723204244.24457-4-Larry.Finger@lwfinger.net