Grid Attack And Restore – Fire And Ice Grid

Grid Attack And Restore – Fire And Ice Grid. Our last blog post detailed the grids backup procedures. At the time of writing, I would never have imagined those systems being tested so quickly. We have suffered a series of DDOS attacks for a couple of months. Mistakenly I assumed they were random attacks and not something explicitly focused on our servers. Every attack was quickly detected by the hosting companies DDOS protection system.

Grid Attack And Restore -What Happened?

The DDOS attacks seemed to end; however, they had reduced the speed of the attack to avoid detection. Unfortunately, we were unaware of a sustained brute force attack. All three of our servers were attacked almost a million times each. The server containing ROBUST (assets) was breached on the 18th of May at 07:42 GMT. The attacks on all three servers were split between the SSH port and PhpMyAdmin. It was the SSH password which was finally cracked.

No Financial details or personally identifiable information was compromised

All of our billing details are held on a different system entirely. The only information anyone could potentially use is avatar names and UUID’s. Our Paypal account uses an altogether different password with no details of this stored on the compromised server what so ever. Our billing system has always been kept behind a much higher level of security. Naively I believed a small grid server holding no financial or personal details didn’t need high levels of security.

Grid Attack And Restore – What was lost?

The attack was very clearly targeted at our ability to run an Opensimulator grid. Everything related to the grid was erased. Robust service folders, bash script files (used to launch the service), duplicates of the settings files and MySQL databases. Sadly it didn’t end there, they also mounted our primary backup and erased everything it contained. Finally, they attempted to lock me out of our own server. The passwords on 3 out of 4 sudo users (administrator) accounts were changed. Once inside the first server, they were able to connect to the other two (simulator servers). The files removed were identical, opensimulator folders, launch files and MySQL databases.

Grid Attack And Restore – Why did it take 30 hours to restore?

Restoring the grid took much longer than we initially expected. Our secondary backup store is connected to a substantially slower service than the primary system. The primary backup was erased entirely, leaving us restoring a huge amount of data over a slow connection. Moving forwards, we will be upgrading our secondary backup store. Everything had been compromised, which is different from a hardware failure. Every single password needed to be changed. All the settings files required adjustment. Essentially it is a big job to restore three servers. The slow secondary backup still took the vast majority of the time.

Improvements

Some of our improvements can not be discussed without undermining the actual gains. All of our servers are now using key pair SSH access with no password authentication possible. To prevent access to other servers in the event of a breach, each server also has its own passcode. The SSH port is different, and the server is configured to block the IP address of anyone who fails the passcode 3 times. Further measures to protect other areas which were attacked have been implemented. The servers are now far more secure than they were previously.

Details of the security improvements can now be found on securing-an-opensim-ubuntu-server/