[media] drx-j: Fix qam/256 mode

QAM/256 currently doesn't work, as the code is only called if
channel->mirror is DRX_MIRROR_AUTO, but a prevous if prevents
this condition to happen.

While here, returns -EINVAL to not supported QAM modes and
simplify the code, reducing the number of indents.

Acked-by: Devin Heitmueller <dheitmueller@kernellabs.com>
Signed-off-by: Mauro Carvalho Chehab <m.chehab@samsung.com>