NEC Meltdown 12/31/23

[user 14106]

Major communications/server issue affecting a the entire Northeast - basically sounds like anything Amtrak dispatched and is affecting NJ transit, MARC, and SEPTA too. Sounds like Metro North is not affected.

 

[Just-Thinking-51]

Well this is the third time communication - server have taken down train service. This is the first time I have heard it impacting the NEC. Last two were restricted to the National trains, both freight and passenger ones.

Someday we will fix the problem. Just not this year.

 

[Rasputin]

Just in time for New Year's maybe we go back to the timetable and train order system? Time of dust off those Form 19s and Form 31s and get out the copy of Josserand's Rights of Trains to make sure things are done right.

 

[Maverickstation]

From Amtrak's Website.

Server Issues Causing Delays on Northeast Corridor
December 31, 2023 08:15 AM
We're currently experiencing server/ signal issues in the Northeast Corridor. All Trains operating in the area are being delayed. As of 8:00 am, Amtrak is anticipating up to a 2-hour delay.

We will continue to keep you updated and will let you know when this situation is resolved and normal service has resumed.

Here is what you can do to stay informed:
• Self-modify your reservation via the Amtrak app or Amtrak.com (or call 800-USA-RAIL) if your trip has been impacted and change and cancelation fees will be waived.
• Visit Amtrak.com or download the Amtrak app for the latest information on train status
• Subscribe to delay alerts for any train via email or text message at Amtrak.com/DelayAlerts
• Follow @AmtrakAlerts on Twitter for real-time service information on Amtrak trains across the country

Amtrak regrets any inconvenience. This information is correct as of the above time and date. Information is subject to change as conditions warrant.

 

[City of Miami]

What exactly are server-signal issues?

 

[user 14106]

Looks like things are moving now on the Empire corridor. 250 and 260 have finally departed Albany - Rensellear and 280 which was holding near Hoffmans just outside of Amtrak territory is moving again.

 

[jis]

What exactly are server-signal issues?

Most likely the electronic interlocking software has issues causing them to be unable to get route clearance for dispatching trains. So net net they are unable to turn any Red signal to Green. It is possible that a patch was applied last night which either triggered a latent bug or introduced a new one. I have lived through a few of those in the world of telephony in early part of my career when I worked for Bell Labs.

 

[user 14106]

99 and 2263 have departed Boston and 143 is now boarding and preparing to depart Springfield.

 

[user 14106]

Interestingly CTrail was somehow operating their trains this morning on the Springfield line. So I don’t know what that indicates as far as what the problem is.

 

[jis]

Interestingly CTrail was somehow operating their trains this morning on the Springfield line. So I don’t know what that indicates as far as what the problem is.

Corruption of only part of the database?

 

[user 14106]

157 just followed 143 out of Springfield station so it looks like at least the northern end is up and running. Still haven’t seen any movement out of Washington.

 

[Amtrak25]

When I was an application programmer for a major Fortune 100, we had software install freezes for various lengths of time at quarter end and year end, the latter being from around Dec 10 to early January to avoid hell to pay at financial close periods with junk code being installed with "releases" - mass software installs in one night, adequately tested or not to make a target date, rather than less at a time incrementally when it was ready, which was less risky. That even applied to mainframes. "Change Management" has brought with it some undesirable results.

 

[AmtrakBlue]

Trains are moving in both directions now

 

[jis]

https://www.trains.com/trn/news-rev...er-issues-disrupt-northeast-corridor-service/

 

[joelkfla]

Most likely the electronic interlocking software has issues causing them to be unable to get route clearance for dispatching trains. So net net they are unable to turn any Red signal to Green. It is possible that a patch was applied last night which either triggered a latent bug or introduced a new one. I have lived through a few of those in the world of telephony in early part of my career when I worked for Bell Labs.

There's also the possibility of malware, as in a foreign actor probing for vulnerabilities.

 

[west point]

There probably will be a lot of cancellations later today as the T&E personnel will run up against HOS?

 

[jis]

There's also the possibility of malware, as in a foreign actor probing for vulnerabilities.

In my experience plain old goofupware is way more common than malware, and people try to shove them under the carpet as fast as they can. Sometimes their effect becomes too pervasive to hide and that is when they show up outside the inner circle's consciousness.

 

[user 14106]

It also could be some sort of communications infrastructure failure as opposed to software. Unfortunately not enough information has been released to know for sure.

 

[west point]

Would not be surprised if Amtrak never really explains the problem.

 

[BlackDiamond]

Apparently Metro North New Haven Line was not affected. I have a friend who was trying to leave NYC to get to his car in New Haven and he was able to transfer to GCT and catch a MNR train to NHV.

 

[jis]

Apparently Metro North New Haven Line was not affected. I have a friend who was trying to leave NYC to get to his car in New Haven and he was able to transfer to GCT and catch a MNR train to NHV.

Of course not. MNRR uses a different dispatching system from Amtrak. AFAICT the problem was with Amtrak’s CTEC only.

 

[user 14106]

Yes it also affected Penn Station Central Control for a period as Long Island Rail Road reported service into both Penn Station and Grand Central Madison was disrupted for a period Saturday evening.

 

[Amtrak25]

Yes it also affected Penn Station Central Control for a period as Long Island Rail Road reported service into both Penn Station and Grand Central Madison was disrupted for a period Saturday evening.

I wonder if that was the same thing or a coincidence. LIRR only said there were switch problems west of Woodside, which is a repeated occurance. They didn't blame it on Amtrak.

 

[jis]

Yes it also affected Penn Station Central Control for a period as Long Island Rail Road reported service into both Penn Station and Grand Central Madison was disrupted for a period Saturday evening.

Harold is under PSCC, so if PSCC was affected then trains to both Penn Station and Grand Central Madison would be affected. Actually I believe the underlying dispatching platform is shared between the network of CTECs and PSCC.

 

[JWM]

Will they ever be honest about exactly what caused the issue?