As an IT team, we detected malware on several endpoint machines in our network. The situation started to escalate. More and more computers were getting infected. But the worst was still to come. And it was to be caused by my IT mistake…
If you read the post until the end and like it, you may want to follow Netstero.com on LinkedIn or Facebook.
A bad day
I was a local desk side helpdesk specialist. My boss asked me to take each infected computer, copy data, and scan it against malware. Next, he asked me to remove the malware and reinstall a fresh operating system. Finally, I had to connect the machine to the network to join it to an AD domain. Not a complex task…
Unfortunately, we ran out of IP numbers in our DHCP pool. I had to assign IP manually to every machine. After cleaning the computer’s disk I had to map a network drive from NAS to download the necessary applications. I did it with every cleaned computer.
Connect to network -> map network drive -> do all installations -> next machine -> connect to network and so on…
As there were no IPs left in the dynamic pool, I’ve had several static IP addresses selected for the job. I rotated the assignments around the computers I had at hand. It was late, I was after many hours of work already. But I was racing with the virus spreading around the network. So I decided to do another computer, or perhaps two… I cleaned another machine and picked one of the static IP addresses for it. I put it into the config interface along with other connection details. I’ve connected the computer to a domain and left it for some time for updates to download and install. I switched to take care of another machine…
Small mistake, but massive consequences
But, what’s wrong? It seems like I cannot connect to a network drive. I’ve checked the network – it was fine. I thought to myself: I’m busy fighting with the virus, let admins do their job to repair the NAS… But, wait for a second, hadn’t I just?… I’ve checked the network config of the last machine. Oops! I’ve assigned an IP address of the NAS storage to the computer. It made all employees lose access to their files on network shares. But not only… In our company applications were run directly from the network drives. So… I’ve turned all these applications off! For hundreds of users! Big oops! I’ve caused a quite significant amount of trouble…
What were my largest sins?
- I was already tired and in hurry. Both of these things don’t help when you put a series of numbers into the interface to connect to a network… Especially if you do it repeatedly for a whole day.
- I was cleaning computers and doing my other helpdesk duties in the meantime. Multitasking doesn’t help to focus…
- I wasn’t quite aware that this thing may happen. I had technical knowledge of what may be the consequences of assigning the same IP address to two devices. I just haven’t imagined a scenario of what could happen if I assign the address of something important to another device. If I had this scenario in my head I would probably be more careful…
But above all, there were several major issues right from the beginning:
- There was a lack of free dynamic addresses. It may seem silly how it could happen. Unfortunately in that organization, it was not that easy to assign more address pools. It required a sequence of acceptances and planned work of global network teams.
- The users’ network was not separated from the servers’ network. Both users and the NAS were in the same subnet. Lack of network separation may cause several concerns, mostly related to security. One of them is assigning an address of some important resource to an endpoint device.
Conclusion – how to prevent this kind of thing to happen?
From the story, it seems quite obvious how to secure your environment to prevent bad consequences of IP address misassigning.
- Be fresh and rested at work. Even if it seems that you do simple, repetitive actions. Or especially then. Of course, it’s easier said than done. You not always can follow this advice. What you can do is mind your condition and double-check what you do when you’re tired and in hurry.
- Don’t multitask. Yes, this is another “brilliant” piece of advice, which you at least sometimes have to break. Again though – mind what you do. If you multitask, remember that you don’t put all your focus on each of your actions. Double-check what you do and how you do it.
- Think about what consequences you have to trigger by doing a relatively simple task. Don’t underestimate the importance of the work you perform.
- Make sure that there are enough free addresses in the dynamic pool in your network. Don’t let them run out.
- Don’t make it too complicated to get more pools in your organization. Especially if your company grows, expands, or, for some reason, increases the number of endpoint devices.
- Do separate the servers’ network from the users’ network. You should do it for many reasons. Costly consequences of misassignment of server’s IP address are just one of them.
Do you see some other conclusions from this story? If yes, please share them in the comments 🙂
Thank you for taking the time to read the post. If you liked it, don’t forget to follow Netstero.com on LinkedIn or Facebook.
