media: staging/imx: of: allow for recursing downstream

Calling of_parse_subdev() recursively to a downstream path that has
already been followed is ok, it just means that call will return
immediately since the subdevice was already added to the async list.

With that there is no need to determine whether a subdevice's port
is a sink or source, so 'num_{sink|src}_pads' is no longer used and
is removed.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>