We were upgrading our ERP system. We did it every year, so it became routine for us. The last upgrade events ran like a charm, so we didn’t expect problems this time. But this year, we were going to be faced with making a risky decision…
If you’ll read the post until the end and like it, you may want to follow Netstero.com on LinkedIn.
The trouble leading to risky decision
Unfortunately, as soon as the vendor puts the new version into production, the system starts to work very slowly. Some of the features don’t work at all. On the first day, we check the network to eliminate the hardware issue. The next day, we give the vendor a chance to fix the system. The third day starts with growing nervousness. The system is still sluggish. HR department reports that there is a threat of paycheck delay because of the broken ERP. We give the vendor time to solve the issues until the end of the day.
Day four. The ERP system shows no improvement. As the risk of paycheck delay rises, the HR team requests to roll back the ERP system upgrade. We call all key users to a meeting. We ask them if they agree to roll back the upgrade. If they approve, they’ll have to input all data from the last 3 days into the system again. Faced with the paycheck delay, they agree. We’re clear to roll back the upgrade.
The risky decision doing its job
We call the vendor to coordinate the rollback. Unfortunately, there is a problem with it. The primary database is integrated with another one responsible for the employees’ portal. Within 3 days, new records were created in the portal’s database. Rolling back the primary database and inputting the data again would corrupt the data in the portal’s database. The vendor refuses to perform the rollback despite our hard push to do it.
A hard path to a quick victory
Within several minutes the vendor representatives call a meeting with their technical staff. We join together with our HR representatives. We keep pushing for rollback. Within half an hour, the vendor finds solutions and workarounds. We agree to disable some of the latest minor features for some time. The system speeds back up. It generates the paycheck data smoothly.
For the next 2 days, the vendor resolves the rest of the severe and medium issues. Everyone feels relief and starts to celebrate the success. The IT department is recognized as heroes who saved the paycheck.
We made the right decision and resolved the crisis. Or did we?
The great result caused by the risky decision
Hindsight bias is a phenomenon of judging past events based on the outcomes. If the consequences are good, the decision must have been right. But have it? If the decision was reckless but led to good results by pure luck, is it the merit of the decision maker?
Pushing for rollback turned out to be a good negotiation strategy. It forced the vendor to prevent troublesome data corruption on his database. But it wasn’t only the negotiation tactic. We were ready to roll back the ERP database and cause more significant problems for ourselves.
We had several other databases integrated with the ERP database. The ERP vendor wasn’t responsible for their maintenance. We were. How much trouble with corrupted data we cause ourselves if we roll the upgrade back?…
Thank you for taking the time to read the post. If you liked it, don’t forget to follow Netstero.com on LinkedIn.
