I thought only some of them heard it as a voice, like Terezi and Rose.

No, everyone heard the voices.  Rose and Terezi just heard it more clearly, probably due to them both being Seers.