First, a stop "in the middle of nowhere" does not really add that much time to a train's running time. Both by observation and by calculation on a 79 mph railroad a stop adds just about exactly 3 minutes 00 seconds plus dwell time to a passenger train's running time. The actual time does vary somewhat on power to weight ratio, but if you use 3 minutes in your analysis the difference between calculation and reality will be measured in seconds, usually single digits of those.
Second, financial and political reality says were are not going to spend megabucks to permit faster run times. As part of this, for much of the eastern US permitting faster run times requires improved alignment, and anybody who deals with consturction of ANYTHING on a new location knows how difficult it is to even be allowed to do so even if you have the money in hand.
There are more points, but at this time I must start working on the things I am paid to do.