As the Canadian government embarked on its payroll transformation project, IBM was tasked with customizing software for a complex public service pay system. Pressure to stay on budget led the government to defer or eliminate important functions and cancel a planned pilot test. In the months before the launch in 2016, there were many warnings that the Phoenix pay system wasn’t ready — but the government pressed on anyway.
Episode Three - Timestamps
(00.00) Intro
(03:28) IBM/PeopleSoft
(06:24) The sole bidder
(07:37) Senate hearing
(09:14) Justin Trudeau
(13:56) Failure rates
(18:34) Phoenix goes live
(20:36) Complaints pour in
Phoenixed: Inside Canada’s payroll disaster is produced by the Global Payroll Association and Storythings.
[Soundbite of speech in Miramichi]
Stephen Harper: Today is a very proud day for New Brunswickers and particularly for people in this region, because …
Glen McGregor, host: In his final year in power, in 2015, Canadian Prime Minister Stephen Harper travelled to Miramichi to give an update on the transformation of his government’s payroll system.
[Soundbite of speech in Miramichi]
Harper: The modernization of the federal government's pay system and services will save taxpayers millions of dollars, something like $70 million a year once it's fully up. And of course, it has created over 500 public service jobs right here in this community. I don't have to tell you that this is a major step forward for Miramichi and for the region, one that will benefit generations to come.
McGregor: At the time of the announcement, the official launch of the Phoenix pay system was only about a year away. But behind the scenes, things were not going well.
I’m Glen McGregor. I’ve been an investigative reporter in Ottawa for more than 25 years. This is Phoenixed: Inside Canada’s payroll disaster, produced by the Global Payroll Association and Storythings.
Kim Coles: They didn't want to admit the errors in the first place, and then they couldn't hide it anymore. It was way too big.
Beth Bell [in a Senate committee meeting]: There was a culture at the time that made it difficult to speak truth to power, and that bad news didn’t flow uphill.
James Bagnall: “Good IT costs a lot, bad IT costs a fortune.”
McGregor: Episode 3, “Going Live.” In 2011, Harper's government awarded a contract to handle the technical overhaul of the existing 40-year-old regional pay system. The automation would accompany the physical centralization of a huge number of payroll workers in the province of New Brunswick. But senior managers in government were wary of big IT projects and their potential to run out of control. Here’s journalist Jim Bagnall, who covered Phoenix for the Ottawa Citizen.
Bagnall: I remember getting a, a quote from one of these guys, he said that, “Look, I hate anything to do with information technology, and I'd be very happy never to have anything to do with this for the rest of my career.” Like, these guys were not engaged in this, they didn't really understand it.
McGregor: Part of the problem with large IT projects is that government is spending public money, and it’s highly sensitive to the political risks if costs go out of control — costs that ultimately get passed on to taxpayers.
Bagnall: Government generally spends about 2% of its revenues on information technology, and for large private-sector organizations like banks, the figure’s more like 10. So, so …
McGregor: You think, they’re cheaping out a little bit.
Bagnall: Oh yeah, you know, big time.
McGregor: Generally, federal governments are not in the business of making software, so they contract the work out. So PSPC, the government department responsible for procurement, would oversee the payroll project but put the automation component of the job out to tender, setting an initial budget of $155 million for the tech part of the upgrade.
Now, there’s an old maxim in the business world that says, “No one ever got fired for choosing IBM.” Such was the reputation of Big Blue that it was the safe choice, with a proven track record over decades of automating business tasks like payroll. IBM got the contract.
Bagnall: They were doing a specific role and they were being guided by the public servants. So IBM only had limited say. I mean, they could — they pushed back here and there, but they didn't, as far as I know, in the end, say, “Look, you know, this isn't going to work.”
McGregor: IBM is not designing the system …
Bagnall: No.
McGregor: They’re saying — they’re talking to the public servants, saying, “This is what we want done” …
Bagnall: Yeah.
McGregor: “You figure out how — the technology behind it.”
Bagnall: Yeah. And even though one of the ministers involved later on acknowledged that, you know, “What we should have done was tell IBM, here's what we want the system to do, you figure out how to do it.” Instead, the civil servants were trying to tell them or figure it out and tell them, “Look, do this, do this, do this.”
McGregor: The government told IBM to use software called PeopleSoft, a dominant brand in the HR world owned by Oracle, and one that the Canadian government had lots of familiarity with.
Bagnall: PeopleSoft was a California company, a pioneer in the whole idea of automating all the processes related with hiring and compensation and fitting that all into one unit. So the federal government bought PeopleSoft software decades earlier. And, you know, they have different variants, you know, they upgraded. And so it was IBM's job, basically, to take the PeopleSoft platform and adapt it for this crazy, crazily complicated system of federal government pay. I mean, they had to work in, I think, something like 80,000 rules and regulations, one kind or another.
McGregor: The problem with Phoenix wasn't that the software didn't work. It was just being asked to do things it couldn't do.
Bagnall: Or that they didn't have enough time to work through all the issues and, and to make it work. It — I mean, the amount of speed that was required for a, a massive job like this, you're talking 300,000 government workers and pay systems stretched end to end.
McGregor: You had a line in one of your stories about cheap technology versus expensive technology. Do you remember that?
Bagnall: Oh, yes, yes. I shouldn't take credit for that. That was an IT guy who told me, “Good IT costs a lot, bad IT costs a fortune.”
McGregor: That’s great. Fair to say that's what happened with Phoenix?
Bagnall: Yeah.
McGregor: Okay.
Bagnall: Yeah.
McGregor: As it turned out, IBM was the only bidder for the contract to build the new payroll system. That could have been a warning sign that the government’s expectations were unrealistic or too expensive. A consultant named Sandy Moir who later studied Phoenix’s implementation suggested that the narrow constraints on the project discouraged other potential bidders who might have wanted the work.
[Soundbite of a Senate committee meeting]
Sandy Moir: Obviously, it’s not an ideal situation when you have one bidder. I mean, I — you know, you want a competitive process. But the Government of Canada came out with a very prescribed “this is exactly what we want.” They’d already decided they wanted to go with PeopleSoft. It wasn’t IBM that suggested it. They were looking for a system integrator that could implement PeopleSoft in the Government of Canada in the following ways. And I believe that the, the bidding community considered the requirements, considered the ask and simply felt that they didn’t want to undertake the project.
McGregor: Other companies might have been nervous, too, when they realized the scope of the project. Phoenix was not just a software update. It was a whole new way of doing pay. Here’s what then-IBM executive Beth Bell later recalled when speaking to a Senate committee.
[Soundbite of a Senate committee meeting]
Bell: When things happened to people’s pay, like they were hired, they perhaps had to act for someone or they went on leave, those events were reported to both the HR and the compensation advisors.
McGregor: HR would enter the data in their systems and payroll would make sure the employee got paid.
[Soundbite of a Senate committee meeting]
Bell: There were, in fact, and still today remain, two concurrent systems: a payroll system and multiple human resource systems. So under the old process, there was no link between the two systems. Each of the departments had its own HR staff, compensation advisors, business processes, organizational structure and, their — in fact, their own HR system. Employee data in these separate HR and payroll systems were often not aligned, meaning there was no single source of data.
McGregor: The government wanted to change this with Phoenix.
[Soundbite of a Senate committee meeting]
Bell: We understood the Government of Canada’s business case was based on three things: eliminating the duplication between the HR and the payroll systems, establishing a single source of data and leveraging well-known commercial off-the-shelf software.
McGregor: A few months after he visited Miramichi, the prime minister called a general election.
[Soundbite of federal election call]
Harper: Once again, a few moments ago, I met with His Excellency the governor general and he has agreed to my request that Parliament be dissolved. In accordance with our commitment to a fixed election date, the next general election will be held as prescribed by law.
McGregor: Stephen Harper’s government had been battered by ethics scandals, and after nearly a decade of Conservative rule, Canadians were looking for a change. And many of them found it in the son of a former Prime Minister.
[Soundbite of election night speech]
Justin Trudeau: Sunny ways, my friends, sunny ways.
McGregor: Justin Trudeau led the Liberal Party to a majority government in October 2015, pledging to take on climate change and be an advocate for the middle class. He also said he’d name an equal number of women and men in his cabinet of ministers. Among them was a Liberal member of parliament from Newfoundland and Labrador. Judy Foote was named as the minister of public services and procurement, and that meant she would be accountable for the payroll transformation file.
Foote might not have known it at the time, but there were sharply divided opinions in her new department about whether Phoenix would be ready to launch just a few months after the election. The government had initially aimed to launch the new system in two stages, beginning in October 2015. It planned a smaller pilot version for July of that year, but the software wasn’t ready. The pilot was scrapped. Here’s Beth Bell from IBM again.
[Soundbite of a Senate committee meeting]
Bell: As another senator indicated, you know, the pilot was cancelled, but there was a scheduled go-live to go live in two phases in 2015, both October and December, okay? And around the July/August time frame, IBM — based on its understanding of the project and the visibility that we had to some of the other aspects of the project — started to signal to the government that we’re not sure that you’re going to make October/December of that year. And the, the executives we were working with asked us, “Well, when could we make it?” And so we did some work, and we came back and we said, “Well, we think July and August 2016,” based on the volume of work and changes they still had for us to do, “would be a good time frame.” Now, that wasn’t a time frame that was, was great for them. Because they really needed to go live in full by April 2016 because the compensation advisors had already been given their notices.
McGregor: In other words, Bell’s saying the government had to launch the new system because it was laying off payroll workers who were previously doing the work. IBM had no role in that decision, but it was affecting the implementation.
[Soundbite of a Senate committee meeting]
Bell: And so we came back to them and we said, “Okay, so there’s this much work left to do, and we think that’s going to take till July and August. What of this work could you not do right now so we can go live in April for you?” And it was a February/April staged — two-stage go-live. And so we worked together, we gave them some guidance in terms of the amount of work we could do, and they made a selection of what would go in that go-live and what functionality would be deferred till after the original go-live.
McGregor: There were complications, too, in porting the old payroll data to Phoenix. According to IBM executives, the government had to extract employee data from the old regional pay system, clean it up, then send it to IBM to be uploaded into PeopleSoft. The process was glitchy: IBM ran 13 data conversion tests and found errors in all of them. But the launch date loomed and concerns about the system weren’t always making it up the chain, according to IBM.
[Soundbite of a Senate committee meeting]
Bell: There was a culture at the time that made it difficult to speak truth to power, and that bad news didn’t flow uphill.
McGregor: Bell was questioned about the crucial decision to launch by Senator Douglas Black.
[Soundbite of a Senate committee meeting]
Douglas Black: And you would say you went live against your advice?
Bell: Prior to January 2016, we had some visibility in the governance as to what was going on outside.
Black: Right.
Bell: So we could make some assessments. In late January 2016, we didn’t get any more reports telling us what was going on outside, so we were in a position where we couldn’t provide any more advice on the whole transformation program.
Black: But why, as a sophisticated consultancy, would you accept that from your client? Why did you take that risk? Why didn’t you resign?
Bell: Why didn’t we resign?
Regan Watts: Senator.
Black: No, no. No, no. Please.
Bell: So, at the time, we were really put into a position where IBM just focused on the software implementation. And we did continue to express our concern, but the executive at the time told us, “Stick to what you’re responsible for.”
Black: And why did you accept that?
Bell: Because at the end of the day, our responsibility to our client is we knew we could deliver what they wanted within the scope of our work, and that was our responsibility and we delivered on that.
Black: Even though — and I’m, I’m not being critical of you — even though you knew at the time that the risk of failure was high?
Bell: We did not know at the time because we had — we no longer got any of the data on the broader transformation updates after the end of January 2016. So we didn’t know, senator.
Black: And did you find that acceptable, that you were cut out of the loop?
Bell: Not really.
Black: Did you express that? And it made no difference?
Bell: We were, we were asked to focus on the work …
Black: “Stick to your knitting — if I want to hear from you, I’ll ask you?”
Bell: Correct, sir.
McGregor: “Stick to your knitting,” as the senator puts it. IBM knew the system wasn’t totally ready, but in their executives’ account, the government didn’t want to hear it. The February 2016 target date remained in place. A likely factor in that decision was an analysis the government received from a consulting company called S.i. Systems. The month before the scheduled launch, the firm provided a draft report saying Phoenix was ready to roll.
The payroll transformation project, the draft said, “is very likely to achieve its goals and desired outcomes within the first year or two of full operations … All in-scope work has been completed, a code freeze (referring to the software) has been imposed on Phoenix, and the Miramichi Pay Centre is fully operational.” Sure, the consultant warned, there could be risks, and the launch of Phoenix would be challenging. But it said, “It is likely that the problems and difficulties that will be encountered will be manageable.”
Promising news, except that this wasn’t what the agency responsible for employment in the federal public service had been told. Officials at the Treasury Board of Canada Secretariat had heard concerns from other departments. There were rumblings across the public service that Phoenix was not ready for prime time. Concerns, too, that the departments that would pass on pay information to the new system hadn’t been properly trained. Initial testing results were not promising.
Bagnall: You had a number of different agencies saying, “Look, we're getting 30%, 40% failure rates. What's happening here?”
McGregor: What, what do you mean by failure rates? Like what, what kind of things are they, are they finding?
Bagnall: Well, in a department like Corrections, where they have a ton of staff that's working nights, shifts, weekends …
McGregor: These are basically jail guards.
Bagnall: Jail guards. The system, as it was constructed, wasn't making allowances for all of the different variations in the, in the pay. It was still pretty straightforward. And here, you were getting feedback from key departments, or Treasury Board was saying, “You know what, folks? This isn't ready to run yet.”
So you have a system that's on the verge of coming out, but it's clearly not ready. But the people in charge of the program, directly in charge of the program within the federal government — we're talking about the bureaucrats now, not the politicians — were reassuring the politicians, “Look, yeah, there's mistakes here, but we're fixing them. And when we go live, we'll be able to deal with these problems.”
McGregor: Treasury Board officials decided to commission their own study from another outside consulting firm called Gartner. According to the Ottawa Citizen, Gartner consultants interviewed workers across a number of departments, and what they learned was concerning. HR staff in the departments had not been properly trained to pass on the right information that would get fed into Phoenix and the PeopleSoft system. What could go wrong? Gartner correctly anticipated one likely outcome: “unanticipated consequences such as incorrect pay calculation.”
Gartner also predicted a flood of queries at the pay centre in Miramichi after the new pay system’s launch. The firm recommended that Phoenix be implemented only gradually, starting with a few departments with less complicated pay needs. And, this is crucial, the company recommended keeping the existing regional pay system in place as Phoenix was phased in, in case something went wrong.
But the new Liberal government and its minister Judy Foote pressed on.
[Soundbite of media scrum]
Judy Foote: The problem we had was that we had a payroll system that was 40 years old, that had completely broken down on occasion. So there was no doubt that the payroll system needed to be, needed to be changed. And the previous government made that decision to do that.
When I asked questions about when they were ready to launch Phoenix, because as you know, we inherited this decision, whether or not there were any problems that we could expect, whether or not we were actually, we were ready to go, I was assured that we were. But with any system — I mean, I'm not naive, I realized that there are bound to be challenges — anything of this magnitude, when you're talking about technology, there are bound to be challenges.
McGregor: 34 departments and agencies were moved over to Phoenix in February 2016. Another 67 would follow in April. When the switch was flipped and Phoenix went live, the phones in Miramichi began ringing, just as consulting company Gartner had predicted. Alain St-Arnaud, the seasoned payroll pro from Montreal I spoke to in the last episode, had to start fielding calls about what was going wrong.
Alain St-Arnaud: I was working with the call centre, helping them answering questions because most of the questions that the pay centre was getting was not really with the transaction itself, but how a file is dealt with: let's say, “If I have to send a leave without pay form, where should I send it?” Because it was pretty new with the, the new system. So the, the managers, the employees and the HR teams, they didn't get enough information before we went live to be able to have a smooth transition from the old to the new pay system.
McGregor: Naturally, this was frustrating for St-Arnaud and his colleagues.
St-Arnaud: They were mainly upset because they were not able to do as much transaction as they thought they would be able to do, and they were frustrated that they were not able to, to — on the call center side, that they were not able to do the transaction right away, even though they were, they were, they had the capacity to do so. But they were asked not to do the transaction, but just answer, answering calls as much as possible.
McGregor: The employees on the other end of the line were frustrated too.
St-Arnaud: When the, the employees were calling, they were trying to get their issue fixed and not being told that it's going to be fixed eventually when a compensation advisor will get to it.
McGregor: I asked Chris Aylward about what the launch day was like. He’s the president of the Public Service Alliance of Canada, one of the largest unions for federal public sector workers.
McGregor: Do you remember the day that it went active and when you first — maybe it's not a day, maybe it's the week, the payroll week — when you start hearing from members that there are problems?
Chris Aylward: Yes. Yes. We — and we started hearing from members right away. And right from the get-go, we, we were hearing, you know, problems and we just thought, “Okay, I mean, this, these are just glitches with a new system.” We were only probably into maybe the second or third pay period after Phoenix was introduced when we realized this is not a glitch; we have a catastrophe on our hands.
McGregor: What sort of things are your members telling you, that the problems they're having in that, in that initial rollout period?
Aylward: Initially, we, we were hearing the, the real horror stories: “I didn't get paid. I checked my bank account, my pay is not in my bank account. When am I gonna get paid?” That, to, you know, “I got paid too much. I got an extra $5,000 on my paycheck,” right? And, you know, “Don't spend it, because you're going to have to pay it back eventually” kind of thing. But yeah, it ran the whole gamut, seriously, even just within the first three or four pay periods: everything from zero pay to being overpaid to being underpaid. And like I said, it just ran the whole gamut. It wasn't very long when we realized this is extremely serious here.
McGregor: Workers who had pay problems tried to contact the pay units in their own departments, but often they found they were referred to call centres which were swamped. Many workers turned to their union representatives to untangle their pay issues. Kim Coles was working as a labour relations officer for the National Health Union. When Phoenix launched, the calls started pouring in.
Coles: It, it, it took a few weeks for people to start to see the impacts. And, of course, the impacts we were seeing were, at first and sort of continued, were anybody that was having a change in their pay. So if they were still, you know, in the same pay level and grade and classification, they didn't have any issues when it switched over. But anybody that had a, a leave without pay, so it could be, you know, maternity leave, going off on sick leave, like unpaid sick leave, and retiring, or acting, or going into another pay level — and this is where people started to see issues with, with their pay. And things started to unravel quite quickly.
McGregor: Trying to resolve members’ pay problems became a big part of the union’s job. Union reps spent a lot of time talking to government managers about what was going wrong.
Coles: Well, I mean, they were like, “Ah.” I think they were a little overwhelmed, for sure. They didn't want to admit the errors in the first place, and then they couldn't hide it anymore. It was way too big.
McGregor: Wow.
Coles: And then they had to frantically go looking to get sort of retired or people that had been in the position before, to come back and set up teams to try to deal with it.
McGregor: Word quickly spread that public servants whose job titles changed — including those given the title “acting” in their position — were at risk of pay problems. Some even turned down or deferred promotions; the pay raise wouldn’t mean much if they didn’t actually get paid.
Coles: Like, you know, if, if a promotion came up, like, you know, they would be offered a position, you know, at a higher pay level or a different, a different classification. They wouldn't take it because they didn't want it to have an impact on their pay.
McGregor: When people came to you, I mean, did they have a sort of sense that this was sort of a violation of the basic thing about working for government? The hours can be tough, sometimes, and the job may not be as lucrative as the private sector, but it was dependable, and you would get paid, you're supposed to get paid. And this, they never thought this would be an issue.
Coles: Government jobs are supposed to be known for life, right? “Become a lifer.” But even when you're, when you're working for a business in the private sector, the business could go out of business, right? The company you're working for could go out of business. So you know, you — could be a chance that you don't have a paycheck. But for the government, everybody's going, “Well, jeez, you know, there's a system in place, there shouldn't be an issue.” You know, you do your work, you get paid.
McGregor: But with Phoenix, that was no longer guaranteed. About a year after the launch of Phoenix, a report from the auditor general found there were nearly 495,000 outstanding pay requests from departments and agencies that used the Miramichi pay centre.
Pay requests are sort of like IT tickets. They include requests to change an employee’s personal information, to enter overtime, and to fix problems with pay.
51% of all government employees reported errors in their pay. Workers in Miramichi scrambled to resolve the problems, but they simply didn’t have the numbers — or necessarily the experience — to handle the deluge of complaints. And behind each outstanding pay request was a worker: a real person trying to pay their bills, their mortgage, their rent or their daycare fees. Over time, the Phoenix problems would have profound effects on employees, on their finances, on their marriages, even on their kids.
Eight years after the launch of Phoenix, people are still dealing with the effects of these pay problems: lost homes, lost savings and worse. In the next episode, you’ll hear from the government workers who were affected.
