Apparently this was broken by the "ctx->hwdec" check in the if condition
guarding the destroy call, and "ctx->hwdec = NULL;" was moved up
earlier, making this always dead code.
This should probably be refcounted or so, although that could make it
worse as well. For now, add a flag whether the device should be
destroyed.
Fixes#4735.