greybus: es2: clean up es2 destructor

Now that the host-device is deregistered separately from deallocation,
we can simplify the cport_to_ep-array deallocation.

Signed-off-by: Johan Hovold <johan@hovoldconsulting.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>