We know that The Distortion has a yellow door that can lead anywhere, including to it’s center.

And in “Killing Floor” we hear about a green door that’s supposed to lead outside but actually lead’s it’s victim deeper into a flesh dominion-eques slaughterhouse.

So. I’m choosing to headcanon that all the fears have a different colored door attached to them that can lead into their dominion. The Distortion uses theirs the most since “weird door that doesn’t lead where it should” is right up madness alley, but the other fears can utilize theirs as well in certain situations.