drm/amdgpu: fix ocl test performance drop

partial revert commit <6971d3d> - drm/amdgpu: cleanup logic in
amdgpu_vm_flush

Signed-off-by: Flora Cui <Flora.Cui@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <David1.Zhou@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>