Basically, it comes down to this:
From the
Amtrak Service Line Plans FY 21-25:
"Millennials, the largest population cohort, seek travel experiences that are inexpensive yet Instagram-worthy, with seamless Wi-Fi capability for any work or leisure/social activity. In contrast, Baby Boomers gravitate toward luxury experiences with differentiated amenities, yet also value seamless connectivity. As a responsible steward of federal dollars, Amtrak must ensure we are making investments that maximize public benefit"
The Long Distance Trains provide Instagram-worthy sights and relative comfort even if food is not included. The market that cares deeply about the food is not the market Amtrak wants on the trains. Most of the Instagram set I've seen on the LD trains don't really care one way or another about the food, it wouldn't prevent them from making the trip.
The other thing is that the LD trains are a hybrid product. The Auto Train IS an all-sleeper luxury product designed to get people down to their vacation homes in Florida. It will continue to grow as a service line since it was designed from the beginning to serve one purpose and one purpose only.
All of the other LD routes serve a variety of customers going to a variety of different destinations. These LD routes are there because they serve rural populations that like having a train around, but make most of their money from those traveling longer distances.