Ok, tin foil hats for this one, our universe isn't exactly infinite in the way people traditionally think like numbers. The edges of the universe bend and form a large shape, say a sphere for simplicity. That cover speeds through and circles back eventually, but do to it's speed and travelling along the edges of everything and relativity, when it returns it's not at the same point or even at the same speed. It arrives before it initially left, quite a bit before it left... So much so that it kills off the dinosaurs.