I do both directions on a pretty regular basis (North 3-4x/month, South 1x/month).
The Southbound does have some pretty scenery as you get to the last hour or so of the journey, around say Oceanside (sorry, I can't recall more specifically).
I'm a bigger fan of the Northbound, though. Once you pass Ventura, roughly, you've got lots and lots of pretty coastal viewing til you reach Santa Barbara.
While there's certainly a lot walking distance in San Diego, Santa Barbara is beautiful as well. The Pier is a 5-7 minute walk from the train station, during which you'll pass a couple of nice restaurants, an ice cream parlor, a wine bar, and a Tapas lounge.
If the SWC gets in on time, grab the 765 Northbound to SB, then come back in the evening on the 794, after a slightly early dinner. If the SWC runs late, you can explore LA for the afternoon, then take the 777 North. You'll only have about 90 minutes in Santa Barbara - but that's enough to grab dinner on the pier, then board the train home just in time for the sunset.
Edit: If you want to make the day trip entirely about the train, you can indeed push all the way to San Luis Obisbo. The concern there is that, if schedules do go sideways, it's a lot bigger hassle to make it back to LA. From Santa Barbara, if things go bad, you can catch an Uber to Moorpark for $75 or so, then the metrolink rest of the way back to Union Station. Can't do that from San Luis Obisbo. If you look at this option, look at the Coast Starlight schedules as well.