In my opinion, there are 4 significant problems that result in extended station dwell times.
#1 is 'street car-ing'. I want to scream every time I ride the Lakeshore Ltd when I see the conductor scanning every boarding passenger's tickets on the ground! It happens on other trains, too, but on the LSL, it's EVERY STOP. I fully understand their need to tell the passenger which car to find a seat as the 'shorts' are usually limited to 1 car and the longer-distance passengers grouped by destination, especially the endpoint cities. Instead, the conductor should simply ASK where they are going as passengers too often don't have their cell phone ticket displayed, or printed ticket in their hand, etc, when they reach the conductor.
#2 is short platforms. Consider the Lakeshore Ltd. The smaller city intermediate stations frequently have a platform long enough for 4 or 5 cars. If I recall correctly, the Waterloo IN platform is only 2-3 cars long. The pre-Covid LSL usually ran with 11 or 12 not counting the now-gone 448/449 baggage car to/from BOS. And to make things more complicated, passengers going to points in MA must be in the 'front' of train #448/48. When there's a need to board a sleeper passenger for BOS, either they spot the sleeper at the front of the platform or the boarding sleeper passenger(s) have to traipse through coach and the cafe/BC car to get to their sleeper. Fortunately, all the coaches are together...2 from BOS and 2-4 from NYP, followed by the diner/aka sleeper lounge, 2 sleepers and the baggage car. Worst case scenario, a sleeper passenger for BOS and sleeper passengers for NYP. The only workable solution is to do a 'triple stop' at the station. I've witnessed numerous times. That's 15-20 minutes for what the timetable shows as a departure time only, ie, 2-5 minute stop. Double stops are far more common.
Fortunately, at least on the LSL, all stations that handle baggage have 10-12 car platforms so that everyone can deboard/board simultaneously and baggage can be worked. And for passengers going to/from stations without baggage handling, simply lugging their oversized suitcases up and down the steps takes 30-60 seconds per passenger to get on or off the train! I've seen 'monster' rolling suitcases that likely weigh 70 pounds or more and some petite woman trying to maneuver them down the aisle. Fortunately, the (assistant) conductor is alert to take the suitcase off or on the train for them.
#3 is the lack of conductors to be able to open and staff every door. I never counted them, but the combined LSL has, I think, 1 assistant conductor and 1 conductor. That means that at most, only 2 doors can be open at a stop, plus any sleeping car doors for passengers on/off there. The exact number is determined by contract. So, there is no choice but to single file passengers on or off.
#4 is really a non-issue if the train is running late. But if it arrives early enough to complete all the station work prior to the scheduled departure time, there is no choice but to 'wait time' until the clock says it's time to go. Don't forget padding in the schedule so if the train is running 'on time', it's not unusual to be done at a station early due to padding. I don't recall waiting time on the LSL except on #49/449 when everything gets wrapped up in CLE or TOL and we have to 'wait time' to leave. On other LD trains, I've noticed it maybe several times per year. Having a scanner tuned in keeps me abreast of what's happening.