NS PTC failure on 8/28/23 affecting all NS routes

[Railroad Bill]

All trains out of Chicago eastbound now cancelled We are sitting in Metro Lounge and just announced 30, 48 all Michigan trains cancelled with no alt transport. They will meet with us SOON. Just spending day in Chicago. Meds only fir tomorrow..Natives in lounge are restless

 

[AmtrakBlue]

All trains out of Chicago eastbound now cancelled We are sitting in Metro Lounge and just announced 30, 48 all Michigan trains cancelled with no alt transport. They will meet with us SOON. Just spending day in Chicago. Meds only fir tomorrow..Natives in lounge are restless

NS’s PTC went kaput

 

[Old MM]

Going east out of Chicago anyway.
Why? Wish I knew.
Maybe someone here does.

 

[John Bredin]

Amtrak website says the suspensions of 30, 48, and Michigan service are due to signal problems on the host railroad, not the coming storm. Link.

(Edit: ninja'd by AmtrakBlue)

 

[Thirdrail7]

PTC issues on NS

 

[John Bobinyec]

Generally delays due to PTC failures are called "signal failures". However, signal failures are generally local in nature. Today (8/28/2023) it looks like NS is having server failures which is probably shutting down the entire railroad. Amtrak trains affected today are:

Piedmonts canceled: 72, 74, 75, 77, 78
Piedmonts holding in Charlotte: 76 (two sets of equipment)
Carolinian holding at Raleigh: 79
Capitol Limited canceled: 30
Lake Shore Limited canceled: 48
Wolverines canceled: 352, 354, 355
Blue Water canceled: 364

I've had discussions with train crew friends about this. They place the blame on the FRA. Evidently the rule states that at the initial terminal (where there is a crew change, not at the train's origin) PTC must be reinitialized and working. Period. There's no contingency for what to do when it is NOT working.

So, with that kind of situation, the PTC central servers absolutely must work, or else there is the potential to shut the railroad down.

There must be server/communication redundancy, backup systems, alternative methods, etc.

jb

 

[Trogdor]

I was on 352. We departed 30 minutes late after what the crew called “mechanical issues” on the PA (though a friend of mine said NS was having server problems). We made it to Lumber Street and sat for about 30 minutes before they backed us up to Roosevelt to clear the way for other trains trying to make it around the wye. After another 30 minutes, they backed us into the station and said the train would be cancelled and they were trying to arrange alternate transportation.

I didn’t stick around to see what the results of those arrangements were, as I just opted for a ticket refund and made separate arrangements to get where I was going.

 

[jis]

This problem is more prevalent in certain PTC implementations of I-ETMS more than certain other more distributed systems with fewer single point of failure.

 

[Thirdrail7]

FYI-This is not just the Piedmonts and the Chicago area. Northeastern trains like Pennsylvanian, the Crescent, the Lynchburger and whatever you call the Norfolk services are impacted as well.

 

[City of Miami]

Amtrak says 66 canceled tonight due to host rr server issues.
Another alert says 171 held @ WAS 2 hours due to host rr signal issues. What's the diffo?
Much more of this and I will cancel all trips and stay in CVS.

 

[John Bobinyec]

News Release Issued: Aug 28, 2023 (10:22pm EDT)

System outage update​

ATLANTA, Aug. 28, 2023 /PRNewswire/ -- Norfolk Southern Corporation (NYSE: NSC) provided an update Monday on a technology outage that impacted rail operations:

This morning, Norfolk Southern experienced a hardware-related technology outage that impacted rail operations. At this time, we have no indication that this was a cybersecurity incident. Our teams worked throughout the day and successfully restored all systems at 7:00 p.m. ET. We are safely bringing our rail network back online. Throughout this, we have been in contact with our customers and will work with them on updated timing for their shipments. We expect the impact to our operations to last at least a couple of weeks.

SOURCE Norfolk Southern Corporation

 

[Just-Thinking-51]

Well this is the third time this has happened. Maybe one of these days it will get addressed.
Blaming it on the FRA for a safety policy is a bit disingenuous, although they could update their policy. Or maybe enforce a requirement for a more sturdy system. I personally got clipped in Chicago and had my connecting train cancelled back in March 25th. Interesting they were cancelled train shortly before departure hoping the system would be restored.

 

[Steve4031]

I hope they figure this out. PTC will be an area of vulnerability until this is resolved.

 

[jis]

Well this is the third time this has happened. Maybe one of these days it will get addressed.
Blaming it on the FRA for a safety policy is a bit disingenuous, although they could update their policy. Or maybe enforce a requirement for a more sturdy system. I personally got clipped in Chicago and had my connecting train cancelled back in March 25th. Interesting they were cancelled train shortly before departure hoping the system would be restored.

It has always seemed that what is being asked of the FRA is to say that as long as the system is installed on the train you can operate it whether it works or not. Which has sounded a bit bizarre to me, but I don't work in the railroad industry. They do operate to their own tune which I sometimes do not understand. But I suppose it is what it is.

I have a suspicion that partly they are unwilling to ease up and allow operation at reduced speed from a terminal is that it would open up the door for simply operating everything reduced speed and save the money on keeping the safety system properly operational. The ingenuity of management is well known to all I suppose.

 

[John Bobinyec]

Just ruminating here about what a completely robust system would look like.

If I had my druthers, I would design a twin system. Each central system would be located apart from the other - say Atlanta for one and Cleveland for the other, in the case of NS. That would mitigate problems with weather. Both systems would be operational, in order to share the workload, but be capable of taking over from the other system, if it went down. Communications between the systems would be such that there would be no single point of failure.

Ah, but I can dream, can't I?

jb

 

[jis]

Just ruminating here about what a completely robust system would look like.

If I had my druthers, I would design a twin system. Each central system would be located apart from the other - say Atlanta for one and Cleveland for the other, in the case of NS. That would mitigate problems with weather. Both systems would be operational, in order to share the workload, but be capable of taking over from the other system, if it went down. Communications between the systems would be such that there would be no single point of failure.

Ah, but I can dream, can't I?

jb

None of this is rocket science. The technology is known. But when you try to get something on the cheap you get what you pay for.

 

[GDRRiley]

Just ruminating here about what a completely robust system would look like.

If I had my druthers, I would design a twin system. Each central system would be located apart from the other - say Atlanta for one and Cleveland for the other, in the case of NS. That would mitigate problems with weather. Both systems would be operational, in order to share the workload, but be capable of taking over from the other system, if it went down. Communications between the systems would be such that there would be no single point of failure.

Ah, but I can dream, can't I?

jb

I'd personally go for a 3 headed system here so you can take 1 offline for upgrades and then not be at risk of failing. It really isn't that hard to do now with the ability to lease large amounts of data center space from companies. It just requires not being cheap.
Atlanta, Cincinnati and Altoona or Harrisburg would be roughly where you'd want stuff to have it well distributed

the other option would have been to pick a system thats not so tied to the back office.

I have a suspicion that partly they are unwilling to ease up and allow operation at reduced speed from a terminal is that it would open up the door for simply operating everything reduced speed and save the money on keeping the safety system properly operational. The ingenuity of management is well known to all I suppose.

Yep railroads have killed safety systems before to run slower, IIATS and Cab signaling were pretty wide spread before they got pulled up as train speeds decreased.

 

[west point]

What does the hardware problem mean? What ever the failure is will lessons be learned? 2 weeks to fix appears a major equipment failure but who knows for sure. Now the POLS & FRA need to examine this problem a fix it. Suggest that all CTC equipment have a default to ABS if it does not detect a command for some length of time.

Too much dependence on centralized control can have a system wide shut down as this event . What happens if a CME wipes out central control?

 

[AmtrakBlue]

What does the hardware problem mean? What ever the failure is will lessons be learned? 2 weeks to fix appears a major equipment failure but who knows for sure. Now the POLS & FRA need to examine this problem a fix it. Suggest that all CTC equipment have a default to ABS if it does not detect a command for some length of time.

Too much dependence on centralized control can have a system wide shut down as this event . What happens if a CME wipes out central control?

I read it as 2 weeks to get the trains back on schedule due to all the trains that were stopped during the outage. Not 2 weeks to fix the hardware problem.

 

[Railroad Bill]

As a follow up to our Chicago dilemma, after a few hours of Amtrak organizing, those of us on 30, 48, 448 and Michigan service were put on a bus to downtown hotel. Nice place. Next morning bus arrived at 11am for ride back to CUS. I luckily got a call from Amtrak that evening and arranged for a sleeper on 448 back to CLE. Talked to several others who ended up in coach with refund of part of their fare. We all received food vouchers $40 each, although many people did not know they had to go to Amtrak ticket counter to get them. Not passed out in lounge. Just about all the places in the food court would take them except for Starbucks. . Waited until 9:30 and 48.448 headed out about half hour late due to another electronic issue but made up some time and back to CLE this morning about 20 minutes down. Thought Amtrak handled the situation well since it wasnt their fault. Will NS get a bill from Amtrak for the hotel costs, food etc??

 

[trimetbusfan]

As a follow up to our Chicago dilemma, after a few hours of Amtrak organizing, those of us on 30, 48, 448 and Michigan service were put on a bus to downtown hotel. Nice place. Next morning bus arrived at 11am for ride back to CUS. I luckily got a call from Amtrak that evening and arranged for a sleeper on 448 back to CLE. Talked to several others who ended up in coach with refund of part of their fare. We all received food vouchers $40 each, although many people did not know they had to go to Amtrak ticket counter to get them. Not passed out in lounge. Just about all the places in the food court would take them except for Starbucks. . Waited until 9:30 and 48.448 headed out about half hour late due to another electronic issue but made up some time and back to CLE this morning about 20 minutes down. Thought Amtrak handled the situation well since it wasnt their fault. Will NS get a bill from Amtrak for the hotel costs, food etc??

Wow! I’m surprised Amtrak did so much to accommodate you, even with a cancellation. Glad to hear that.

 

[danasgoodstuff]

Wow! I’m surprised Amtrak did so much to accommodate you, even with a cancellation. Glad to hear that.

I've had them put me up on a missed connection, nice place too (across from DC Union Station). But they didn't buy me any food.

 

[Old MM]

I think they did both a good, and not so good, job handling the situation. I recognize the station employees were doing the best they could with the tools they had however there are clear training issues that I will describe more in a trip report soon.