This might not always be correct depending on filtering done in VS. But
unless VS send us all the metadata we have to get it from somewhere.
Fixes dynamic frame metadata, which was copied from fmt_in, initialized
only upon reinitialization.
Fixes crashes caused by the use of stale pointers.
Fixes: #13956