PTC "Server disruption" service issue (March 2023)

[AmtrakBlue]

Could this affect the northeast corridor? I have a trip from NJ to Mystic, CT on a regional on Monday.

If there are PTC issues, I don’t mind finding hotel space for a few more days there, but I’d definitely like to get there!

I think the NEC uses a different type of PTC (ACSES) than what the rest of the country uses. @jis mentioned I-ETMS at one point.
Watching the tracker the NEC trains are running fine.

 

[AmtrakBlue]

 

[AmtrakBlue]

Amtrak San Joaquins

As of 1:54pm PT: ALERT: Due to ongoing nationwide intermittent server issues, customers should expect some delays. #SanJoaquins

 

[jis]

I think the NEC uses a different type of PTC (ACSES) than what the rest of the country uses. @jis mentioned I-ETMS at one point.
Watching the tracker the NEC trains are running fine.

NEC primarily uses ACSES. There are sections of NEC South that have I-ETMS in addition to ACSES for the use of CSX/NS/CSA and allegedly MARC too. I don’t know how those are doing.

 

[PerRock]

I think the NEC uses a different type of PTC (ACSES) than what the rest of the country uses. @jis mentioned I-ETMS at one point.
Watching the tracker the NEC trains are running fine.

Michigan is unique too, with our ITCS system. Although I think the issue is getting out of Chicago... once the train gets to MI trackage it's on our unique PTC.

peter

 

[allanorn]

The Pere Marquete (370) just posted as departed CUS so that might be good news?

 

[Just-Thinking-51]

The Pere Marquete (370) just posted as departed CUS so that might be good news?

Last night a few trains were able to departure after 7pm. So maybe it’s a good thing. Will see if the system crashes in the morning.

 

[joelkfla]

"You may change your reservation via the Amtrak app or Amtrak.com. If your trip has been canceled, cancellation fees will be waived."

Sounds like Amtrak is charging cancellation fees to people who proactively cancel their trip to avoid getting stuck in the mess, like not wanting to get stuck halfway in CHI.

 

[allanorn]

Last night a few trains were able to departure after 7pm. So maybe it’s a good thing. Will see if the system crashes in the morning.

Guess not. Just posted on Amtrak Alerts: tonight's 30 and 48/448 cancelled. No word on 59 yet.

Adding: Tonight's 59 canceled.

 

[west point]

As far as I can tell nothing has left CHI as of 1900 EDT today..
EDIT 370 toward Michigan has made it out of CHI 46 minutes late.. Now if no problems on NS ??

 

[Burns651]

Michigan is unique too, with our ITCS system. Although I think the issue is getting out of Chicago... once the train gets to MI trackage it's on our unique PTC.

Also could be a problem on the opposite end getting out of Pontiac, as they're on CN and then CRSA until Dearborn. Then there's the 2 miles on CN through Battle Creek.

 

[Steve4031]

Crew just timed out on 92 which had a very slow trip into Tampa. 92 is sitting there. This is Penny’s train. She just provided an update because she’s waiting in Orlando.

 

[pennyk]

92 arrived in Tampa over 4 hours late. According to the Orlando Station agent, there are “signal” problems and the train is being held in Tampa waiting for new crew.
The wait in ORL has been long.

 

[user 14106]

What a mess. I’m sure Monday’s RPA meeting will be interesting to say the least!

 

[pennyk]

92 arrived in Tampa over 4 hours late. According to the Orlando Station agent, there are “signal” problems and the train is being held in Tampa waiting for new crew.
The wait in ORL has been long.

I think it left Tampa.

 

[allanorn]

I think it left Tampa.

And supposedly scheduled to depart Orlando in five minutes.

 

[pennyk]

And supposedly scheduled to depart Orlando in five minutes.

In fact, the board in Orlando was never electronically updated and still said 9:53 departure. At 8:43, there was an automated boarding announcement. Some passengers got up and started walking outside. The board is now blank.

 

[jis]

Amtrak isn't affected on CSX around Washington DC either.

Spoke too soon. Everything seems to be restricted to 59mph except for a few exceptions. Go figure.

And there is something strange going on between Lakeland and Tampa beyond just PTC failure.

 

[Old MM]

This is sad. Something needs to change.

 

[allanorn]

This is sad. Something needs to change.

I'd love to read the incident report from IT, if it ever is made public.

Thinking about what I've seen in twenty years of financial IT: it could be a single-point-of-failure, or a (hot) backup failed with a bricked production server, or a combo issue (like multiple servers failed and need each other online to boot properly) that just cascades out of control. Kind of like Southwest where their systems worked great if there are minimal exceptions throughout the system, but you can't have 50% of your capacity frozen in place for two days with terrible/manual recovery procedures.

(edited for a better example)

 

[Old MM]

I don’t know, I don’t remember that Amtrak ever had these large scale systemic issues before.

As far as Southwest, we don’t even know the full story there. Reportedly their flight crews were in contract negotiations.

There are answers, hopefully.

 

[west point]

Why is it FRA's anything? FRA laid down some pretty reasonable requirements (as required of them by the Congress and the President) that are incidentally also met by things like ACSES in the US and various European standard implementations (e.g. ETCS/ERTMS L2) and others in the world (e.g. Kavach in India). There are perfectly good implementations meeting the PTC requirements in the US that seem to not have the problems that I-ETMS seems to be having. It really is Wabtec's and freight railroad's issue more than anyone else's

Sorry but Where is the FRA and Sec of Transportation in this mess?? Fra certainly could have paused the initial departure initiation sequence. Evidently that protocol needs rewriting. Our local freight operation is certainly all screwed up here.

 

[Eric in East County]

This is just another example of how dependent we have become upon computers and how easily things can come to a screeching halt when they cease to function. Mischance can and will occur in all human undertakings, so we must assume that this current glitch was not caused deliberately. Let’s hope that computers are never allowed to become our Archilles Heel during a major world conflict.

 

[jis]

Sorry but Where is the FRA and Sec of Transportation in this mess?? Fra certainly could have paused the initial departure initiation sequence. Evidently that protocol needs rewriting. Our local freight operation is certainly all screwed up here.

FRA is the regulator, not the protocol designer. The latter is on the likes of Wabtec. FRA specifications does not say that the system shall require a specific initialization. That is why ACSES can be compliant with FRA spec without having a complex initialization protocol. That is why FRA does not much to do directly with this fiasco. If everyone used something like ACSES there would be no initialization protocol to break and still be PTC spec compliant, like ACSES is.

 

[Thirdrail7]

FRA is the regulator, not the protocol designer.

Indeed, Jis. They make the regulations that govern the (non)movement of trains under these circumstances.

That is why FRA does not much to do directly with this fiasco.

This is EXACTLY why the FRA has EVERYTHING to do with this mess and has been directly involved in why some train have actually been allowed to depart. What you're leaving out is not too long ago, regulations stated a train COULD depart an initial terminal with initialization or partial initialization failure, as long as the host railroad allowed it. It could also depart an initial or turnaround location that isn't designated a PTC repair point with initialization failure.

It was the FRA's ruling that stated a train can no longer depart with initialization failure. Indeed, they even changed the definition of "initial terminal" for PTC, which flies in the face of other applications of initial terminal (e.g. cab signals, brakes, non-running gear defects.) This is likely due to the fact that despite being told otherwise, the FRA probably assumed that if a piece of equipment fails to initialize, there must be something wrong with said piece of equipment.

Anyone that is actually out in the field and not sitting in an office knows this isn't the truth, and they were warned.

The best part of this is a train can depart with PTC COMPLETELY cut out/failed state in certain locations, but if it doesn't initialize in the same area, it can't move.

Example:

A MARC train operates from Washington DC to Perryville, Maryland and turns for another train. Since Perryville has no facilities, no mechanical forces and no other equipment to swap with, Perryville is considered a "turnaround location." Considering the circumstances, the trains were previously allowed to depart. Now, the current rules state that same train cannot depart Perryville with initialization failure even though there are no forces to correct the problem and no other equipment to swap with. Meanwhile, if the same train has outright PTC failure and is in cut out/failed state, the same train CAN depart Perryville under PTC failure rules....as long as it can be initialized.

This episode will certainly help the battle that has been raging between the FRA and various railroads that want this rule amended.