In general, though, I think the answer is that in general you need an intact (or mostly-intact) ROW that you can access to make things viable for a private investor. That could be an existing railroad (Brightline), an interstate (Brightline and Brightline West), or conceivably a shortline (probably a Class II given the scale/scope needed). In theory one of the Class Is could decide to play ball, but that seems unlikely given the current management mindset - they're so short-term focused that I don't think most of them would take a contract for 99 years of paid access with guaranteed base payments for access.
[I jest, slightly - UP was willing to entertain some of the LA-Vegas folks, but I can't speak to what priority those folks would have had.]