btrfs: tests: return errors from extent map tests

The individual testcases for extent maps do not return an error on
allocation failures. This is not a big problem as the allocation don't
fail in general but there are functional tests handled with ASSERTS.
This makes tests dependent on them and it's not reliable.

This patch adds the allocation failure handling and allows for the
conversion of the asserts to proper error handling and reporting.

Signed-off-by: David Sterba <dsterba@suse.com>