How does Amtrak Arrow system work?

[LinPhil]

I'm just curious on how the Arrow system works? Anyone have any experience or pictures?

 

[me_little_me]

Watch the old westerns. You will see:

Arrows often miss and mislead
Arrows seem to be a poor way of positive communication
Arrows can be poorly made.

Applies to the old arrows and the Amtrak one.

 

[Qapla]

I'm just curious on how the Arrow system works?

The general consensus on how the Amtrak Arrow works is - Not Very Good

 

[railiner]

I believe its original programming was purchased from American Airlines then owned SABRE system. At first it was called "Spike", and then changed to Arrow.
I still have in my "archives", an Amtrak Spike Owner's Manual...
When first implemented, we had CRT integrated terminals and archaic modems to connect with the main frame in Philadelphia. A couple of years later, we upgraded into using PC's.

 

[jis]

I believe its original programming was purchased from American Airlines then owned SABRE system. At first it was called "Spike", and then changed to Arrow.
I still have in my "archives", an Amtrak Spike Owner's Manual...
When first implemented, we had CRT integrated terminals and archaic modems to connect with the main frame in Philadelphia. A couple of years later, we upgraded into using PC's.

But the main Arrow system still run on a backend large server. It now has myriads of front ends connecting to it using myriads of protocols with myriads of caches which have no consistent consistency maintenance mechanisms,. So a view of the situation from one type of front end may not match what you see from another type, and then all the problems that follow from such. Typical of a system that has grown like an Onion with multiple layers over time.

 

[bratkinson]

Many moons ago, I recall reading that the system Amtrak purchased/licensed was written in IBM 360/370 Assembler language using 'macro level' CICS, the only 'interface' to CICS at that time. Having maintained a couple of similarly written programs at a client back in the 70s, I couldn't think of a more difficult way to do online communications other than building and sending/receiving 3270 control strings and coming up with a means to keep 100s of terminals simultaneously working without getting into each others' ways.

Unfortunately, it is now 100% impossible to find a talented IBM mainframe assembler person that not only knows CICS macro level but also worth his/her pay. Us old fogies have certainly forgotten more about both than some whipper-snapper will ever know! We can't forget the incredible learning curve for a new hire just to understand 'how' it fits together. There's countless ways of passing data from one task to another, 'timing' issues, and many more 'deep wells' to fall into. The biggest fear of anyone new maintaining an existing system is 'breaking' something unexpectedly...as in 'been there done that' a couple of times as a contractor!

It would be several years later that IBM developed the 'command level' that could be easily used by Assembler and COBOL programmers. I probably wrote well over 100 CICS command level COBOL programs, and later wrote a translator that would input Intercomm COBOL programs (predecessor of CICS, but not by IBM) and convert them to CICS Command Level COBOL. Over 2200 were converted for a Fortune 500 company that had about 1500 of their own Intercomm programs, but they bought out a competitor that had 700 or more of their own!

As Jis noted above, there is now a number of front-end servers that interface with the mainframe designed and written by folks long gone from Amtraks' payroll, and modified, perhaps clumsily, by newbies that didn't stay too long either as less stressful, higher paying jobs are always available in the industry.

Also, the replacement of bulky, clunky-looking 3270s with PCs was nothing more than an optional interface card in the PC and appropriate 3270 'emulator' program to make it work. The mainframe and the controllers between it and every terminal didn't 'know' it wasn't a real 3270 at the end of the wire.

And by the way...Ever since they made that switch, my home PCs have ALWAYS had keyboards with the function keys to the left of the keyboard, exactly the same pattern as 3270s and 3278s! Northgate keyboards are almost indestructable!

 

[Devil's Advocate]

I'm just curious on how the Arrow system works? Anyone have any experience or pictures?

If people still knew how it worked it probably would have been replaced by now.

 

[Willbridge]

Just a side note. CN's reservation system in the mid-1960's was based on Air Canada's. And in Spain in 1971 I found that RENFE's reservation system was based on Iberia's. In hindsight it might have been better to have started over but it seems that in the beginning an airline system was the easy way to go.

 

[Steve4031]

How much would it cost Amtrak to start over? And how much money could they save with a more efficient newer system. Additionally, there would be ways to increase revenue with an entirely new system.

 

[Devil's Advocate]

How much would it cost Amtrak to start over?

Enterprise software that could replace ARROW from the ground up typically starts at a few million dollars, but the total outlay for a highly available mission critical implementation could stretch into tens-of-millions when all factors are considered. A long term hosted contract could defray some of the up-front cost and spread it out over time.

And how much money could they save with a more efficient newer system. Additionally, there would be ways to increase revenue with an entirely new system.

That would be hard to quantify from outside the organization.

 

[jis]

One of the more recent replacement of an older airline reservation system with a newer one was at United when they merged with Continental. They chose to use Continental's Shares system and ditched their existing system. So it involved transition a bit more than half its customers. They also transitioned their flight dispatching etc. system to a Shares module.

What they got as a result is being able to attach all front ends data access devices in real time to a single source of "truth" and the ability to update customer facing information simultaneously with staff at gate and staff at service center facing information. This actually opens up a world of possibilities. Other airlines have also retrofitted such ability now but it required some very significant rearchitecting the access layers in their systems.

Amtrak's problem is exacerbated by the haphazard manner in which various access modules have been added to the original system many with their own caches that are not synchronized in near real time. And I am told there is no one around from the time they were bolted on so the expertise for pealing the thing apart is difficult to come by.

This reminds me of a problem that AT&T/Bell Labs had with one of their very large main office digital switches, the ESS-5. Its software had grown so big and so complex that everyone was afraid to touch the core of it, since they could not predict what would break if the change one line of code there. I pictured the situation as hundreds of software guys with various missions to add new features standing around a tar pit poking around with sticks trying to figure out which part of the pit they could dive into without having the pit blow up. Sad but hilarious. Finally they had to redo a whole lot of it to get a handle on it.

 

[NSC1109]

When Delta and Northwest merged, Worldspan was merged into the DLTerm system and was left online. I can still access everything that Worldspan could do back when NWA was still around but I mainly just use it for weather.

These systems have been around for ages and still work (mostly) well. I don't have any experience with Arrow to know how "dated" the system may be but if it's anything like PARS then it should have a decent degree of functionality to it.

 

[Willbridge]

Even before computers there were risks in changes to telephone or telegraph networks. When I was in college the Bell System guys who provided the links for our campus radio station broadcasts of away games invited our engineering staff to salvage some cables, jacks, wire harnesses, etc. from the CApitol exchange (downtown Portland). Our guys did as they were instructed. The next day there were news articles about a mysterious service outage in part of the CApitol exchange. It was never publicly resolved but we concluded that something in the equipment to be salvaged was still in service.

 

[MilwaukeeRoadLover]

Many moons ago, I recall reading that the system Amtrak purchased/licensed was written in IBM 360/370 Assembler language using 'macro level' CICS, the only 'interface' to CICS at that time. Having maintained a couple of similarly written programs at a client back in the 70s, I couldn't think of a more difficult way to do online communications other than building and sending/receiving 3270 control strings and coming up with a means to keep 100s of terminals simultaneously working without getting into each others' ways.

Unfortunately, it is now 100% impossible to find a talented IBM mainframe assembler person that not only knows CICS macro level but also worth his/her pay. Us old fogies have certainly forgotten more about both than some whipper-snapper will ever know! We can't forget the incredible learning curve for a new hire just to understand 'how' it fits together. There's countless ways of passing data from one task to another, 'timing' issues, and many more 'deep wells' to fall into. The biggest fear of anyone new maintaining an existing system is 'breaking' something unexpectedly...as in 'been there done that' a couple of times as a contractor!

It would be several years later that IBM developed the 'command level' that could be easily used by Assembler and COBOL programmers. I probably wrote well over 100 CICS command level COBOL programs, and later wrote a translator that would input Intercomm COBOL programs (predecessor of CICS, but not by IBM) and convert them to CICS Command Level COBOL. Over 2200 were converted for a Fortune 500 company that had about 1500 of their own Intercomm programs, but they bought out a competitor that had 700 or more of their own!

As Jis noted above, there is now a number of front-end servers that interface with the mainframe designed and written by folks long gone from Amtraks' payroll, and modified, perhaps clumsily, by newbies that didn't stay too long either as less stressful, higher paying jobs are always available in the industry.

Also, the replacement of bulky, clunky-looking 3270s with PCs was nothing more than an optional interface card in the PC and appropriate 3270 'emulator' program to make it work. The mainframe and the controllers between it and every terminal didn't 'know' it wasn't a real 3270 at the end of the wire.

And by the way...Ever since they made that switch, my home PCs have ALWAYS had keyboards with the function keys to the left of the keyboard, exactly the same pattern as 3270s and 3278s! Northgate keyboards are almost indestructable!

I still have a 360 assembler reference card. I'm not sure Newbies would know assembler, as it hasn't been taught in awhile. Even newer versions of Conol and CICS are built on the old.

In the other hand, a mainframe rarely returned an OS exception and was amazingly CPU efficient. However, it lacked multi stream processing.

 

[me_little_me]

In the other hand, a mainframe rarely returned an OS exception and was amazingly CPU efficient. However, it lacked multi stream processing.

That's because gerbils can only run so fast!

 

[bratkinson]

My 'green card', 'yellow card', and even various disk drive 'cards' are still floating around here somewhere, along with a number of flowcharting templates and a couple of printer-rulers.

I DID, however, dump my private library of old IBM manuals ranging from multi-volume messages and codes, to COBOL, Fortran, PL/I and Assembler manuals, DOS JCL, OS JCL, JES2, JES3, and even IMS DB/DC, DB/2, and multiple CICS manuals. I might have had a Panvalet and Librarian manuals in that pile as well. At 73, I don't think I'll ever need them again. Maybe I should have sold them on ebay instead.

 

[railiner]

Thats an idea...wonder if my “Spike” manual would sell?
Naaah, I’ll probably just dump it, too...

 

[niemi24s]

Have an extra Fluke 8502A manual if anybody needs one.

 

[neroden]

How much would it cost Amtrak to start over? And how much money could they save with a more efficient newer system. Additionally, there would be ways to increase revenue with an entirely new system.

Amtrak started by ripping out a lot of "auxilliary" functions from Arrow; it used to do all kinds of things like fuel calculations, now it *just* does reservations.

What's left has been moved over to a z390 emulator to remove hardware fragility. But it's still running on 370 machine code. Amtrak's IT department has been attempting to pull out pieces of it one at a time and replace them with sections written in C. But it wasn't properly modular to start with, so each such "extraction" is a huge pain in the neck.

It didn't even have code-data separation. One of the things they're working on is modifying it to have code-data separation, so that the list of "currently active reservations" can be exported from ARROW and imported into a new system. Last I checked, a couple of years ago, they were making... some... progress?

The thing's a mess. Lacking code-data separation and lacking full structured programming and lacking full modularization make it a very bad system from a maintenance perspective -- you can do code-data separation and structured programming and modularization in machine language, but apparently they didn't.

railiner, your Spike manual WOULD sell, BTW

 

[Steve4031]

Thanks Nerodrn. I don’t understand all the computer lingo but you did show me the big issue is transferring all the information from current reservations into the new system.

 

[railiner]

Not being into computer programming, I have no idea of what it all takes to modernize a reservation system, but I’m sure it is very substantial.
Seems like it might be easier, and perhaps even more cost effective to start up an entirely new system, run it for a period concurrently with the old system, and have a dedicated team move the data from one into the other as quickly as they could, and then when caught up, cutover to the new system.

Didn’t merging airlines, such as AA and US Air, manage something similar, a few years ago?
IIRC, they learned from some mistakes from the DL-NW, and UA-CO mergers,
and did a better job of it. Not sure if it was the reservations systems, or the flight operations in those cases, or maybe both...

 

[Devil's Advocate]

Not being into computer programming, I have no idea of what it all takes to modernize a reservation system, but I’m sure it is very substantial. Seems like it might be easier, and perhaps even more cost effective to start up an entirely new system, run it for a period concurrently with the old system, and have a dedicated team move the data from one into the other as quickly as they could, and then when caught up, cutover to the new system.

The cheapest and easiest solution is to simply emulate an older environment on newer hardware, which Amtrak has already done. I'd be surprised if they were starting over from actual scratch. They'd most likely buy an enterprise framework for the nearest industry and build on that foundation. Analyze and document the old system, stand up the foundation for the new system, plan and implement the specialized aspects, test the results, and train the users. Concurrent processing for a couple hours or days is sometimes unavoidable but the longer it goes the worse it gets. Kind of like one man with two watches trying to decide what time it is as they move further and further apart.

 

[Brian Battuello]

The cheapest and easiest solution is to simply emulate an older environment on newer hardware, which Amtrak has already done. I'd be surprised if they were starting over from actual scratch. They'd most likely buy an enterprise framework for the nearest industry and build on that foundation. Analyze and document the old system, stand up the foundation for the new system, plan and implement the specialized aspects, test the results, and train the users. Concurrent processing for a couple hours or days is sometimes unavoidable but the longer it goes the worse it gets.

FWIW, that's exactly what I did for 35 years. I did banks, brokerages, car dealership systems, and of course Y2K. I left the industry to become a commercial tour pilot and never looked back. My spouse says if I'm so smart, why don't I go back and make some money. I have no intention of ever sitting in a cubicle again.

 

[railiner]

FWIW, that's exactly what I did for 35 years. I did banks, brokerages, car dealership systems, and of course Y2K. I left the industry to become a commercial tour pilot and never looked back. My spouse says if I'm so smart, why don't I go back and make some money. I have no intention of ever sitting in a cubicle again.

That’s quite a change in careers. Takes real nerve to accomplish. Good for you!
Just curious on how the touring business is going nowadays?

 

[neroden]

The cheapest and easiest solution is to simply emulate an older environment on newer hardware, which Amtrak has already done. I'd be surprised if they were starting over from actual scratch. They'd most likely buy an enterprise framework for the nearest industry and build on that foundation. Analyze and document the old system, stand up the foundation for the new system, plan and implement the specialized aspects, test the results, and train the users. Concurrent processing for a couple hours or days is sometimes unavoidable but the longer it goes the worse it gets.

I can't remember where I read the last published report from Amtrak's IT on their migration process, but my memories of that are where I'm getting all this information.

So, in a modern system, code is over here, and data is over there, so it is easy to export the data, and import it to a new program. By "modern system", I mean, uh, anything made after roughly 1975.

Unfortunately, Amtrak's system is, at its core, older. The data was stored intermixed with the code. A saved copy of the system is a mix of data and code. This is really, genuinely bad design. There has been a lot of work ongoing to unpick this mess, so that they *can* export the reservations data from ARROW and import it into a new system, but last I checked they were still in the middle of the process.

They want to have this working reliably before they get a new system because they can't afford to have a week's downtime for the reservations system, nor can they afford to lose previously-made reservations -- and most of all, they really do not want to have reservations on two systems at once. (I've seen that sort of transition -- "Oh, is your reservation in the old system or the new system? Let me check both" -- it works, but it is pure yuck for the customers and the frontline agents.)

In the meantime, they have been removing all the "auxiliary" features from ARROW -- features where it can *tolerate* a week's downtime -- and moving them to modern systems. They have modern front-ends with interfaces that can be "replumbed" to interface with a new system fairly easily.

It sounded like the core problem of getting the current reservations data out of the system where they could be injected into a new system was quite difficult and required highly skilled machine language programmers. I believe last I checked, a few years ago, 90% of the reservations were stored in a reasonable place and format but there were always a set of "in process" reservations which were still intermixed with the code. They don't want to lose those during a system transition. Since I'm remembering a document I read in the Obama administration, I could be wrong about some of this.