I'm a bit confused about the difference between the dancestors and the ancestor trolls. are the ancestors just adult versions of them in a different timeline. or is it their adult versions if they were to live on alternia?

The ancestors are the post-scratch version of the dancestors.

Just like Nanna is the pre-scratch version of Jane.