Fail by design: Banking's legacy of dark code

High street banking used to be a simple business. If you were lucky enough to have any shrapnel to your name, you would go into your local branch, maybe chat with the manager (who you'd invariably find with a rag in his hand, dusting the counters), count out your copper, scribble on a chit, pay in a check, and walk out with a smile on your face. There was even an ad on TV, while I was growing up in the UK, telling you to "laugh all the way to the Leeds."

Those days are gone, of course. We now want, or have been told to expect, secure mobile internet banking, with all the complexity and automation that it demands.

As for laughs, you just fondle the phone in your pocket with a giggly grin.

But no one was laughing when theTSB Bank tried to upgrade its computer system one Sunday afternoon in April, while simultaneously migrating a billion customer records.

It did not go well.

TSB Bank's internet and mobile banking suffered a meltdown. Many customers were unable to access their accounts, or they saw wrong balances. And some even saw other people's accounts instead of their own.

Generational difference

IT upgrades seldom go without a glitch. But when it comes to banks, the problems were built into the system 60 years ago — albeit unwittingly — when computer languages like COBOL and FORTRAN first reared their sturdy heads.

"When these large scale financial systems were developed, they were developed on mini and mainframe [computer] systems," says Simon Moores, a former UK "IT ambassador" and managing director at Zentelligence Research.

They were big systems with inscrutable names, like HP minis, DEC VAX, Dexcom, or IBM MVS, running in big rooms, creating lots of heat.

"Those things are robust. The best analogy is that of a tank or a Kalashnikov — you can drop it, kick, fill it full of sand and it just works," says Moores. "It was created with COBOL running underneath, and it was absolutely suitable for the environment and the requirements of the time."

Read more: Australia's largest bank loses records of 20 million accounts

But over time we've added more and more requirements at increasing speed as the technology has advanced, and it's getting harder to tell how each new layer will interact with the old — especially as COBOL is now what some programmers call "dark code." All the experts have either retired or died, few universities teach it, and as a result even fewer people can understand or fix it.

When the TSB Bank tried to upgrade its system, it appears the upgrade couldn't cope with the level of transactions coming in at that same time.

"A slight incompatibility cascades into something catastrophic, and, I would suggest, that maybe nobody existed to be able to look at the code, or even understand the code, because it was compiled [Ed.: source code is compiled or "interpreted" before it is executed], to know what might possibly go wrong — other than to code it with your fingers crossed," says Moores.  

Bury the COBOL?

COBOL (common business-oriented language) was developed in 1959, and as we've seen it's still in use, worldwide. Which means this TSB debacle could happen again, anywhere — in Germany, the US, the Middle East or Asia. Just think of the data loss story out of Australia's Commonwealth Bank this Thursday (3.5.2018) … So they misplaced some magnetic tape? Sounds pretty mainframe to me.

Yet COBOL is still run for a reason. It's rock solid.

Part of that has to do with the language itself. It's written in a highly-structured, yet readable language that is very close to what you're reading now, rather than a machine code or an assembly language.

That's thanks to Grace Hopper, a rear admiral in the US Navy, who was also known as "Amazing Grace." Without Hopper, we may never have had COBOL — or banking as we knew it. She developed the first English-like computer language called FLOW-MATIC, and later had a hand in COBOL and FORTRAN, too.

"People who are conversant in terse languages, like C, do not like COBOL because it's very wordy. You say things out in plain English sentences and before you even get down to doing anything you have to describe all of your data in a very elaborate structure," says retired COBOL programmer, Jay Moseley.

That can have its advantages.

"I like COBOL in that it's self-documenting. You can pick up something that somebody wrote 40 years ago and figure out pretty much what they were doing and why they were doing it," says Moseley.

Millions of lines of code

Moseley worked as a programmer in banking, and in city administration in Fort Worth, Texas, among other places and industries. And his experience seems to jar with the "dark code" theory put forward by people like Simon Moores and others.

"The first bank I worked at," says Moseley, "people were literally using hand-written notes scratched in machine code to maintain systems. So I have seen the criticism of COBOL, but they haven't seen anything. COBOL is very self-documenting."

 

Even so, you have to understand the structure of a language — and in COBOL's case, its many "language divisions" — to find the bugs in a huge system, or change the way it behaves.

"The reason there are still lots of legacy systems is a simple matter of size," says Neel Krishnaswami, a programming languages researcher at Cambridge University in the UK. "A typical big bank, financial service or institution might have legacy systems ranging from tens to hundreds of millions of lines of code. That's like hundreds of thousands to millions of pages of legal text."

Read more: A bitcoin how-to for hapless hackers

All those lines of code represent "decades of institutional knowledge that's been encoded into the computer code," says Krishnaswami. "And as programmers either retire or leave for new jobs that living, human understanding of why the software was written the way it was gets gradually forgotten."

As a result banks and other financial institutions are reluctant to rewrite their software "because changing a condition when you don't understand why it's there is a recipe for disaster."

"Computers follow the instructions they are given without any knowledge or awareness of the goals or the context of the process," says Krishnaswami. "A few years ago, the Knight Capital Group lost US$440 million (€370 million) and nearly went bankrupt over a period of 30 minutes just due to a programming error."

Gnu — as in new

It's too early to say how big TSB's losses will be, although its chief executive Paul Pester will "forfeit" a bonus of £2 million (€2.2 million, $2.7 million).

Perhaps some of that money could go into training young programmers with COBOL, FORTRAN and other forgotten computer skills, because these issues are likely to keep on arising.

"As I understand it, COBOL courses are becoming popular again," says Moores. "There is a demand for COBOL programmers for this very reason — to assist in porting the old systems into the new systems."

And it's not just COBOL. There are all kinds of lingering legacy code. Do you remember the Y2K scare in 1999?

"That was another example of when assumptions that were made suddenly became invalid and a whole bunch of software had to be rewritten," says Krishnaswami. 

"Back in the 1970s people thought, 'Okay, we can encode the year with two digits because there's no way that software I wrote in 1975 will still be running 25 years from now.' But indeed it still was," he says.

And now we have this similar situation with COBOL. Only COBOL isn't the problem, so it doesn't have to die. The problem is how the newer code, like Java, interacts with COBOL.

I've wondered whether it would be possible to design new code that takes these "communication" issues into consideration, but without the COBOL expertise there is no way of knowing in advance what problems will arise. I've also wondered whether an artificial intelligence could spot and fix communication errors in the code better than human programmers (Krishnaswami: "It'd make things worse. You wouldn't only have a system that you used to understand but don't understand anymore, you'd then have a system that no one ever understood.")

Perhaps instead COBOL has to evolve. There is a community working on a new kind of COBOL called GnuCOBOL (formerly OpenCOBOL). One of the leading figures in the community, says Moseley, is a man in Canada called Brian Tiffin.

"He's done lots of experiments in this area," says Moseley. "He's interested in using COBOL code to do things like block chain, and security, like 256 bit secure key encryption. That's probably the future of COBOL — if people want to put the effort into doing it."

And why not? There's every chance that a new, updated version of COBOL, like GnuCOBOL, could meet the demands of today's digital banking landscape, and at the same time do a better job of bridging the gap between the old and the new code. 

DW tried to contact GnuCOBOL programmer Brian Tiffin, but he did not respond in time for publication.