PTC "Server disruption" service issue (March 2023)

[fillyjonk]

This just came across my Twitter dash, anyone knows what's up and who the "host railroad" is? This seems ungood:

 

[AmtrakBlue]

This just came across my Twitter dash, anyone knows what's up and who the "host railroad" is? This seems ungood:
View attachment 31794

Looking at the tracker, I'd say UP. Looks like all trains are going "slow on the west coast. It's not CSX on the east coast as those trains are going nearly full speed.

 

[Acela150]

A guess says that the "server issues" are related to PTC.

 

[joelkfla]

Current alert on Amtrak home page:

 

[jis]

A guess says that the "server issues" are related to PTC.

I think your guess is correct. PTC failure would automatically cause trains to run at reduced speed across the board.

However, the folks who put together these notifications would do better to characterize that as PTC issue rather than Server Issues IMHO. Would be less obfuscatory and more meaningfully informative.

Apparently BNSF had a two hour outage yesterday. Apparently CP had some issue too leading to cancellation of at least one Hiawatha. Additionally UP had some issues. This indicates that it was the backbone telecom provider who had the root cause issues, and suggests that the designers forgot to design the communication network to be sufficiently failsafe, and with all railroads depending on the same thing.

This is also not exactly the first time that this has happened.

 

[AmtrakBlue]

It appears all trains out of CUS are "on hold" due to the server issue. Someone posted an FB that they announced "no trains are leaving". But then she shared a message she got saying 421 would be DELAYED (not cancelled) 4 hours.

 

[AmtrakBlue]

Looking at the alerts, they are cancelling the short haul trains out of CUS.

 

[AmtrakBlue]

From an Amtrak employee on FB

Due to ongoing Positive Train Control (PTC) issues systemwide, Train 5 departing Chicago, Illinois, today, March 24, 2023, is schedule to depart 4 hours late (6:00PM CT). Amtrak and other parties are continuing to try and resolve the issues. Trains 3, 7, 21, and some Midwest Services out of Chicago are also affected.

 

[Just-Thinking-51]

This is going to be fun for me. Will be in Chicago tomorrow to connect with the Texas Eagle.

 

[AmtrakBlue]

This is going to be fun for me. Will be in Chicago tomorrow to connect with the Texas Eagle.

Good Luck!

 

[RalphCT]

I received this message concerned departure of Train 7 on Friday 3/24/2023 at 1:30 pm Mountain time from Amtrak:
"AMTRAK: Hello Customer, Your train will depart from its originating station Chicago 4hr late due to an ongoing server issue. We apologize for any inconvenience this may cause. -Amtrak CCT"

I'm scheduled to board this train in Montana tomorrow for a trip to Seattle.

 

[Just-Thinking-51]

I received this message concerned departure of Train 7 on Friday 3/24/2023 at 1:30 pm Mountain time from Amtrak:
"AMTRAK: Hello Customer, Your train will depart from its originating station Chicago 4hr late due to an ongoing server issue. We apologize for any inconvenience this may cause. -Amtrak CCT"

I'm scheduled to board this train in Montana tomorrow for a trip to Seattle.

Strong recommendation your are available at the station at your schedule time. Then play it by ear if your train is actually running late.

You will miss your train, the train does not miss you.

 

[moselman66]

Looking at the alerts, they are cancelling the short haul trains out of CUS.

Apparently the Hiawatha is seeing bustitution. Being a spring break weekend most of the trains are relatively full.

 

[west point]

You gets what you paid for. Making PTC overly complicated will cause it to bite you in the behind not only now but in the future. FRA Its your bag of manure.

 

[Just-Thinking-51]

Yesterday Texas Eagle was canceled it seems.
Today is showing sold out. But schedule for a on time departure.

Still on the LSL so I am make sure my phone is full charge. Crossing my fingers and hope for the best.

 

[jis]

You gets what you paid for. Making PTC overly complicated will cause it to bite you in the behind not only now but in the future. FRA Its your bag of manure.

Why is it FRA's anything? FRA laid down some pretty reasonable requirements (as required of them by the Congress and the President) that are incidentally also met by things like ACSES in the US and various European standard implementations (e.g. ETCS/ERTMS L2) and others in the world (e.g. Kavach in India). There are perfectly good implementations meeting the PTC requirements in the US that seem to not have the problems that I-ETMS seems to be having. It really is Wabtec's and freight railroad's issue more than anyone else's

 

[John Bobinyec]

It really is Wabtec's and freight railroad's issue more than anyone else's

Do we know what actually failed and whose it is and why (evidently) there is no redundancy built in?

jb

 

[RalphCT]

Strong recommendation your are available at the station at your schedule time. Then play it by ear if your train is actually running late.

You will miss your train, the train does not miss you.

In response to the message I received about the Empire Builder departing 4 hours late: I fully know about train tracking and how to be at my station on time. Doesn't matter this time because Amtrak canceled Train 7 from Chicao "in its entirety" on Friday evening. So it's a flight to Seattle then a flight to Maui. Fortunately I never schedule an Amtrak adventure with a tight schedule.

 

[jis]

Do we know what actually failed and whose it is and why (evidently) there is no redundancy built in?

jb

We know it is observed in Wabtec's implementation of I-ETMS as deployed. Loco Engineers are unable to initialize the system at the origin, or connection with the central server is lost en route followed by inability to re-initialize to reconnect. The problem actually may be Wabtec's or underlying communication network provider.

As for the architectural issues regarding having single points of failure, I guess the service providers are probably asking that question as we speak.

I wonder if this may be another of those unfortunate cases where a triple redundant communication system is built with all three cables running through the same conduit, so if the conduit goes caput everything fails. Remember the DC-10 redundant hydraulic control lines and control cable routing issues?

 

[John Bobinyec]

So the failure in this case, whatever it is, is particular to one freight railroad only? Which one?

jb

 

[dadonatrain]

I just noticed #1 has been canceled. So it’s clearly more than just Chicago

 

[jis]

So the failure in this case, whatever it is, is particular to one freight railroad only? Which one?

jb

Multiple freight railroads, all using Wabtec's I-ETMS apparently. It affected UP, BNSF, CSX at various times. I am not sure about NS. Apparently it did not affect the I-ETMS installed along the southern portion of the NEC spine either, since I have not heard of any issues faced by MARC.

Oddly, Amtrak put yesterday's Chicago departure of 30 and 48 into Service Disruption possibly prematurely, but they are actually running within an hour or so of their schedule. This suggest NS did not have the problem at least yesterday afternoon, but Amtrak for a while thought NS did.

Today CSX and NS don;t seem to have any problem as trains appear to be running at full speed on them.

 

[Just-Thinking-51]

In the Chicago lounge right now. These morning trains were canceled.

Wolverine
Lincoln
Carl Sandburg
Hiawatha

Not sure what running later in the day.
The information available is lacking here.

These four routes are on multiple railroads so maybe it’s just a single section of track here in Chicago. Or Amtrak is just throwing in the towel again.

 

[jis]

In the Chicago lounge right now. These morning trains were canceled.

Wolverine
Lincoln
Carl Sandburg
Hiawatha

Not sure what running later in the day.
The information available is lacking here.

These four routes are on multiple railroads so maybe it’s just a single section of track here in Chicago. Or Amtrak is just throwing in the towel again.

What I find interesting is that the LSL, Cap and Card are running fine. LSL and Cap share route with the Wolverines. I guess something further down the line for Wolverines is not working? A portion that has not been converted to ITCS yet perhaps?

Hiawatha and EB would indicate that CP has problems.

Things are generally running fine in the East, so NS and CSX appear to be OK today.

 

[Just-Thinking-51]

The Texas Eagle, the California Zephyrs, and the 2:15p Wolverine is now showing Great Haul Hall. But at least not canceled.