The consequence of always choosing the middle ground is that the middle ground will be progressively pulled toward the right.
It's like finding the average of an array of integers. If someone keeps adding negative numbers to the array, eventually the average will be negative. The people who instinctively go "but compromise!!!" without question or criticism are no smarter than a computer blindly computing the mean of a list.