And if they were then you couldn't have existed to come back in time. You did so they didn't. Or you would return to the future to learn that you have different ancestors than you did when you left.