As a time player I could be everyone's sprite by going back in time to prototype themself repeatedly. Would there be any major downsides to this?

Lack of variety. But technically, Doomed Versions are sorta apparently un-doomed when Prototyped so… Maybe?