Tag Archives: networking

Hacked!

Laptop in chains

 

With our school’s graduation ceremony last night, the school year is now officially finished. This year will definitely go down in my memory as the year that the students got the best of me… twice!

IP-gate
To give some background on the first “hack,” our current network uses a flat IP network with IP subnets used for each different set of machines (for organizational purposes). We don’t use IP-based security for obvious reasons, but we do use the subnets for deciding internet speed. IP addresses (fixed except for the guest subnet) are given out using DHCP, and each of the subnets except the guest subnet gets decent speed.

When I set up this system ten years ago, I was well aware of the obvious drawback: any person could set a static IP address on any subnet they chose, and, given our lack of managed switches (at the time we had none, though things are changing), there wasn’t much of anything I could do about it. On the flip side, the worst that could happen is that these users would get faster internet, hardly the end of the world.

It took ten years, but, finally, someone figured it out. One of our more intelligent students decided that his IP address of 10.10.10.113 didn’t make a whole lot of sense, given that the gateway is 10.10.1.1. He set his IP to 10.10.1.113, and, voilĂ , his internet speed shot through the roof!

Naturally, he shared his findings with his friends, who managed to keep it under the radar until one of the friends decided to see how well BitTorrent would work with the school internet. What none of these students realized is that the 10.10.1.* subnet was for servers, and, oddly enough, none of our servers uses BitTorrent. The traffic stuck out like a sore thumb, and I finally caught on.

My first step was to blacklist all unrecognized MAC addresses using the server subnet. The next step was more difficult. Now that the cat was out of the bag and everyone knew how to get faster internet, I needed a way to block anybody not using the IP they’d been assigned through our DHCP server. Obviously, there is a correct way of doing this, but that seems to be using 802.1x, and we’re just not there yet. My quick and dirty solution was to copy the dhcp configuration file containing all the host and IP information to our firewall, and then generate a list of iptables rules that only allow traffic through if the IP address matches the expected MAC address.

The problem with this solution is that it doesn’t account for the fact that spoofing MAC addresses is actually relatively simple, so it looks like one of my summer projects is going to be a complete revamp of our network. I’m hoping I can configure our FreeIPA server to also operate as the backend for a RADIUS server so we can implement 802.1x security.

In this case, the consequences of the “hack” for us were pretty insignificant. Students got some extra bandwidth for a while. The students who changed their IP addresses also didn’t suffer any major consequences. Their devices were blacklisted from the internet until they came to speak with me, and then were put on the guest subnet. All of the students were in their final year, so they were only stuck on the guest subnet for the last month or so.

The most obvious lesson I learned from “IP-gate” is that security through obscurity works great…until someone turns on the light. And when that happens, you’d better have a plan.

The Grade-changing Scandal
This was a far messier situation. One of our teachers allowed a student to access their computer to set up a video for class. On the computer, the teacher had saved their login credentials for LESSON, our web-based marking system. While the teacher was distracted, the student used this trick to find the teacher’s password, and then shared the password with different members of the class. Throughout the next few days, the class average for that teacher’s subjects rose at a remarkable rate.

Three days later, one of the students finally told the principal what had happened, and the principal called me. What followed was a day of tying together evidence from multiple sources to work out who changed what and when.

What the students weren’t aware of was that LESSON logs everything at the assignment level, so I could see which IP addresses changed which assignments. If the IP was an internal school address, I could also see which user changed the assignment. One of the students used their laptop (registered on the network, so I knew who it was) to change some marks, then logged in from a lab computer (so once again, I knew who it was), and then finally logged in from home.

The students who logged in from home were harder to track, at least until they did something foolish, like logging in as themselves to verify that the marks had actually changed ten seconds after logging out as the teacher.

We also do daily backups of the LESSON database that we keep for a full year, so it was a piece of cake to restore all of the marks back to their original scores.

Obviously though, this went much further than the IP-spoofing going in in “IP-gate.” This wasn’t just some kids wanting faster internet, this was a case of flagrant academic dishonesty.

In the end, we came up with the following consequences:

  • The students who masterminded the break-in received a zero for the subject for the term
  • The students who we caught changing the marks received zeros for any assignment of theirs that had a changed mark
  • The students we knew that they knew their marks were changed received three Saturday detentions (they have to sit in complete silence for four hours on a Saturday)
  • The students we suspected that they knew their marks were changed received one Saturday detention, though these students were allowed to appeal, and most who did had their Saturday detention reversed

One of the things I’ve learned from this is that there’s never too much audit information. LESSON is going to be changed to record not just who changes each assignment, but who changes each mark, and there will be a history of every changed mark so that teachers can see when marks are changed.

Apart from this, I would be curious as to what others think about the consequences for these two “hacks.” Were we too lenient on the first? Too harsh on the second? What should we have done differently? And what should we do differently going forward?

Laptop computer locked with chain and padlock by Santeri Viinamäki. Used under a CC BY-SA 4.0 license.

Managing the unmanageable

Sisyphus pushing a boulder up the mountain

Uphill battle

When I first started working at LES, sometime in the last century, the computers were networked together using some high-tech gizmos called “hubs“. These hubs would reach a maximum speed of 10Mbps on a good day, if there were only two devices connected and the solar flares were at a minimum.

Time marched on and we upgraded to 10/100Mbps hubs, then 10/100Mbps switches, and then, finally, in the last few years to unmanaged gigabit switches. One of the biggest problems with using unmanaged switches is that the network can be brought to a standstill using a simple patch cable, plugged into two network sockets. I’ve become pretty adept at recognizing the signs of a network switching loop (the lights on the switches are flickering like the last few seconds on the timer in Mission Impossible, the servers are inaccessible, the teachers are waiting outside my office with baseball bats). One of our network loop disasters hiccups even managed to anonymously make it to a site dedicated to technology-related problems.

Over the last month, though, I had lots of small problems that never quite reached the level of crashing the network. Our Fedora systems, connected to the server via NFS, would occasionally freeze for a few seconds, and then start working again. Our accountants, who are running Windows, complained that their connection to the server was being broken a couple of times each day, causing their accounting software to crash. And pinging any server would result in a loss of ten-fifteen packets every ten minutes or so.

I checked our switches for the flicker of death and came up dry. I tried dumping packets from a server on one side of the school to a server on the other side of the school and consistently reached 1Gbps. In desperation, I retipped the Cat6 cable connecting the switches that form the backbone of our network. All to no avail. I decided to wait until evening and then unplug the switches one at a time until I found the problem. The problem disappeared.

The next morning it was back. I had two options. Disconnect the switches one port at a time in the middle of the school day, while teachers, students and accountants are all trying to use the system. Or put in a request for some managed switches and see if they could help us figure out what the heck was going on. Hundreds of irritated users outside my door… or new kit. It was a hard call, but I went for the new kit.

We started with an eight-port MikroTik switch/router, and, after I tested it for a day, we quickly grabbed a couple more 24-port MikroTik switches (most of our backbone locations have nine or ten ports that need to be connected and MikroTik either does 8 or 24 ports).

After we got the three core locations outfitted with switches, I quickly got messages on the switches pointing to a potential network loop on a link to one of our unmanaged leaf switches in the computer room, which was connected to another unmanaged five-port switch that had apparently had a bad day and decided it would start forwarding packets back through itself.

I replaced the five port switch with a TP-Link five-port router running OpenWRT and, just like that, everything was back to normal.

I am never going back to unmanaged switches again. Having managed switches as our network’s backbone reduced the time to find the problem by a factor of 10 to 20, and, if we’d had managed switches all the way through the network from the beginning, we could have zeroed in directly on the bad switch rather than spending weeks trying to work out what the problem was.

So now we’re back to a nice quiet network where packet storms are but a distant nightmare. Knock on wood.

First Work: Myth of Sisyphus detail #1 by AbominableDante, used under a CC BY-NC-ND license