Massive delays out of Chicago Union Station today (2/28)

[nti1094]

 

[MikefromCrete]

WLS-TV reports signal problems at both ends of the station. Manual control of switches limit movements to one train at a time. Metra suggests finding another way home. 

 

[Railroad Bill]

As of 7:30 Eastern, delays continue. Most commuter people were told to take buses or other forms of transportation to leave the city.  Amtrak trains will be in major delay. Last news conference said it may continue into this evening.  

 

[keelhauled]

WLS-TV reports signal problems at both ends of the station. Manual control of switches limit movements to one train at a time.

Have they tried turning it off and on again?

 

[west point]

All eggs in one basket?  Why hasn't Amtrak set up the NEC locations to take over?  Many of the Class 1s  have ability to switch to other centers if there is a computer crash. Are there any problems with NOL and  Michigan dispatched trains?

 

[SarahZ]

 

[west point]

All eggs in one basket?  Why hasn't Amtrak set up the NEC locations to take over?  Many of the Class 1s  have ability to switch to other centers if there is a computer crash. Are there any problems with NOL and  Michigan dispatched trains?

Michigan Amtrak tracks having big delays

 

[cpotisch]

View attachment 12696

That bit really is eternally funny.  "Just kidding!"  :lol:

 

[NorthShore]

Yay....maybe this means we'll get another free weekend to ride!

 

[NorthShore]

I'd actually love to learn more about any alternative routes people took home.

For instance, the Ravenswood station is in the city and one of the highest for ridership on the entire system.   People could have easily taken the L, which stops two blocks away.    Or, even to get to the suburbs on the same line, a Pace bus connects and serves many of the same stations as far as Highland Park.  Given it would take a good deal longer.  Of course, that's out of Ogilvie, so I imagine it ought to have been unaffected, unless the switching problem was a little further out.

Still, closer in suburbs served by the lines using Union station do have available services via Pace, CTA, and other Metra lines.

The larger problem would be with some suburbs which really have no alternate services.

It would be interesting to see what the actual arrival/departure schedules were like.

As far as effective manual switch operations, isn't that just a matter of adding extra crews to throw more switches and flag trains?  

 

[John Bredin]

I'd actually love to learn more about any alternative routes people took home.

Still, closer in suburbs served by the lines using Union station do have available services via Pace, CTA, and other Metra lines.

The larger problem would be with some suburbs which really have no alternate services.

I ride the North Central line from Wheeling. I took the UP Northwest line and my girlfriend drove me from the station.

I imagine people who live farther out still take an alternate Metra route as close as they can get* and then either (1) call someone to pick them up, or (2) Uber/Lyft/taxi.

I know some people going to the same station/town will split an Uber/Lyft/taxi all the way from downtown, but that's simultaneously expensive and slow, the "express"ways being anything but.  :giggle:

* Rock Island for Southwest and Heritage Corridor. UP-West or possibly UP-Northwest (closer to Elgin) for Milwaukee-West. UP-Northwest for North Central, and UP-North for Milwaukee-North. None of those is ideal, but a whole lot closer to home than Union Station with no Metra service. 

 

[StanJazz]

Human Error

http://www.chicagotribune.com/business/ct-biz-amtrak-metra-union-station-switching-problem-20190301-story.html

 

[NorthShore]

I wonder whether there would have been any possibility of switching a limited number of trains into other downtown terminals further out, where they intersect those rail lines.

 

[NorthShore]

Someone fell on a circuit board?   As in fell asleep?   Tripped and fell?   Had a heart attack?   That's just terribly unclear.

And, indeed, why was a software update taking place at rush hour?

 

[NorthShore]

I ride the North Central line from Wheeling. I took the UP Northwest line and my girlfriend drove me from the station.

I imagine people who live farther out still take an alternate Metra route as close as they can get* and then either (1) call someone to pick them up, or (2) Uber/Lyft/taxi.

I know some people going to the same station/town will split an Uber/Lyft/taxi all the way from downtown, but that's simultaneously expensive and slow, the "express"ways being anything but.  :giggle:

* Rock Island for Southwest and Heritage Corridor. UP-West or possibly UP-Northwest (closer to Elgin) for Milwaukee-West. UP-Northwest for North Central, and UP-North for Milwaukee-North. None of those is ideal, but a whole lot closer to home than Union Station with no Metra service. 

See....if Meigs was still open, you could have flown to Palwaukee!

 

[John Santos]

Someone fell on a circuit board?   As in fell asleep?   Tripped and fell?   Had a heart attack?   That's just terribly unclear.

And, indeed, why was a software update taking place at rush hour?

The article says they fell off a ladder and landed on the circuit board.

It also says Anderson blamed the problem on multiple mistakes, the main one being they decided to upgrade (or update, it isn't clear) the servers in the middle of a busy working day.  Duh!  The local management was probably trying to make its numbers look good by reducing overtime.

He also blamed a previous decision (trying to deflect blame to previous management?) to decentralize Amtrak's communications system and wants to recentralize it.  That strikes me (as someone with 45 years experience in computer, networking and communications systems in the banking and telecom industries) as completely wrong.  A properly designed and implemented decentralized system would have no single point of failure and would have automatic resiliency to cover single and many multiple outages without skipping a beat.  A centralized system would be vulnerable to the entire rail network going down, instead of just Chicago.  It would have been hundreds or thousands of times worse.  This problem smacks of inept design or incompetent implementation or both.

Also, it should have taken only minutes (no more than half an hour, including sanity checking after a rarely performed operation) to get the system back up after it crashed, not all day.

This is assuming it was a computer problem (they were allegedly "ugprading the servers") and not a problem with the circuitry that controls and drives the track switches and signals.

 

[Thirdrail7]

This is assuming it was a computer problem (they were allegedly "ugprading the servers") and not a problem with the circuitry that controls and drives the track switches and signals.

The latter is my impression.

All eggs in one basket?  Why hasn't Amtrak set up the NEC locations to take over?  Many of the Class 1s  have ability to switch to other centers if there is a computer crash.

Amtrak currently requires their dispatchers to be qualified on the physical characteristics of the territory they dispatch.  This is not conducive to centralized dispatching.  Most centers have local back up.

This is why I'm curious about this "all things on or off" circuit board.

 

[west point]

It is curious how so many business go thru this merry go round over and over.  RRs over and over as well.  Centralize then whole company goes down. Decentralize and some Harvard graduate wants to centralize to save money and the wheel goes round and round!

 

[cpotisch]

Someone fell on a circuit board?   As in fell asleep?   Tripped and fell?   Had a heart attack?   That's just terribly unclear.

And, indeed, why was a software update taking place at rush hour?

It's bizarre, but here:

“They tried to do a server upgrade, sadly, in the middle of a busy day, exactly the wrong time to do it,” said Durbin said in a phone interview Friday. A workman fell off a ladder and bumped against a panel, which caused the shutdown, said Durbin, who praised Anderson’s honesty.

 

[cpotisch]

The article says they fell off a ladder and landed on the circuit board.

It also says Anderson blamed the problem on multiple mistakes, the main one being they decided to upgrade (or update, it isn't clear) the servers in the middle of a busy working day.  Duh!  The local management was probably trying to make its numbers look good by reducing overtime.

He also blamed a previous decision (trying to deflect blame to previous management?) to decentralize Amtrak's communications system and wants to recentralize it.  That strikes me (as someone with 45 years experience in computer, networking and communications systems in the banking and telecom industries) as completely wrong.  A properly designed and implemented decentralized system would have no single point of failure and would have automatic resiliency to cover single and many multiple outages without skipping a beat.  A centralized system would be vulnerable to the entire rail network going down, instead of just Chicago.  It would have been hundreds or thousands of times worse.  This problem smacks of inept design or incompetent implementation or both.

Also, it should have taken only minutes (no more than half an hour, including sanity checking after a rarely performed operation) to get the system back up after it crashed, not all day.

This is assuming it was a computer problem (they were allegedly "ugprading the servers") and not a problem with the circuitry that controls and drives the track switches and signals.

Well said.

I've gotta say, something about this story just doesn't seem right. I mean, a worker falls off a ladder and onto a circuit board and it devastates Union Station for a day? I'm not totally sure I trust the accuracy of an article in the Chicago Tribune discussing train signal issues, that got its information from some senator that apparently talked to Anderson. That's pretty much the equivalent of a game of "telephone" where everybody is hard of hearing; there's just too many ways for info to get screwed up.

 

[Dakota 400]

A properly designed and implemented decentralized system would have no single point of failure and would have automatic resiliency to cover single and many multiple outages without skipping a beat.  

Perhaps this situation is the result of years of under funding Amtrak.

 

[Rover]

Perhaps this situation is the result of years of under funding Amtrak.

..the result of years of planning by Amtrak.

 

[neroden]

No replacement circuit board ready, seriously?

 

[NorthShore]

Welcome to Chicago: where we set our train tracks on fire to keep the switches from freezing, then find others ways so we can't switch them, anyway!

 

[tim49424]

What happened in Chicago today? I’m noticing a bunch of really late departures again.