Category Archives: Computers

Hacked!

Laptop in chains

 

With our school’s graduation ceremony last night, the school year is now officially finished. This year will definitely go down in my memory as the year that the students got the best of me… twice!

IP-gate
To give some background on the first “hack,” our current network uses a flat IP network with IP subnets used for each different set of machines (for organizational purposes). We don’t use IP-based security for obvious reasons, but we do use the subnets for deciding internet speed. IP addresses (fixed except for the guest subnet) are given out using DHCP, and each of the subnets except the guest subnet gets decent speed.

When I set up this system ten years ago, I was well aware of the obvious drawback: any person could set a static IP address on any subnet they chose, and, given our lack of managed switches (at the time we had none, though things are changing), there wasn’t much of anything I could do about it. On the flip side, the worst that could happen is that these users would get faster internet, hardly the end of the world.

It took ten years, but, finally, someone figured it out. One of our more intelligent students decided that his IP address of 10.10.10.113 didn’t make a whole lot of sense, given that the gateway is 10.10.1.1. He set his IP to 10.10.1.113, and, voilà, his internet speed shot through the roof!

Naturally, he shared his findings with his friends, who managed to keep it under the radar until one of the friends decided to see how well BitTorrent would work with the school internet. What none of these students realized is that the 10.10.1.* subnet was for servers, and, oddly enough, none of our servers uses BitTorrent. The traffic stuck out like a sore thumb, and I finally caught on.

My first step was to blacklist all unrecognized MAC addresses using the server subnet. The next step was more difficult. Now that the cat was out of the bag and everyone knew how to get faster internet, I needed a way to block anybody not using the IP they’d been assigned through our DHCP server. Obviously, there is a correct way of doing this, but that seems to be using 802.1x, and we’re just not there yet. My quick and dirty solution was to copy the dhcp configuration file containing all the host and IP information to our firewall, and then generate a list of iptables rules that only allow traffic through if the IP address matches the expected MAC address.

The problem with this solution is that it doesn’t account for the fact that spoofing MAC addresses is actually relatively simple, so it looks like one of my summer projects is going to be a complete revamp of our network. I’m hoping I can configure our FreeIPA server to also operate as the backend for a RADIUS server so we can implement 802.1x security.

In this case, the consequences of the “hack” for us were pretty insignificant. Students got some extra bandwidth for a while. The students who changed their IP addresses also didn’t suffer any major consequences. Their devices were blacklisted from the internet until they came to speak with me, and then were put on the guest subnet. All of the students were in their final year, so they were only stuck on the guest subnet for the last month or so.

The most obvious lesson I learned from “IP-gate” is that security through obscurity works great…until someone turns on the light. And when that happens, you’d better have a plan.

The Grade-changing Scandal
This was a far messier situation. One of our teachers allowed a student to access their computer to set up a video for class. On the computer, the teacher had saved their login credentials for LESSON, our web-based marking system. While the teacher was distracted, the student used this trick to find the teacher’s password, and then shared the password with different members of the class. Throughout the next few days, the class average for that teacher’s subjects rose at a remarkable rate.

Three days later, one of the students finally told the principal what had happened, and the principal called me. What followed was a day of tying together evidence from multiple sources to work out who changed what and when.

What the students weren’t aware of was that LESSON logs everything at the assignment level, so I could see which IP addresses changed which assignments. If the IP was an internal school address, I could also see which user changed the assignment. One of the students used their laptop (registered on the network, so I knew who it was) to change some marks, then logged in from a lab computer (so once again, I knew who it was), and then finally logged in from home.

The students who logged in from home were harder to track, at least until they did something foolish, like logging in as themselves to verify that the marks had actually changed ten seconds after logging out as the teacher.

We also do daily backups of the LESSON database that we keep for a full year, so it was a piece of cake to restore all of the marks back to their original scores.

Obviously though, this went much further than the IP-spoofing going in in “IP-gate.” This wasn’t just some kids wanting faster internet, this was a case of flagrant academic dishonesty.

In the end, we came up with the following consequences:

  • The students who masterminded the break-in received a zero for the subject for the term
  • The students who we caught changing the marks received zeros for any assignment of theirs that had a changed mark
  • The students we knew that they knew their marks were changed received three Saturday detentions (they have to sit in complete silence for four hours on a Saturday)
  • The students we suspected that they knew their marks were changed received one Saturday detention, though these students were allowed to appeal, and most who did had their Saturday detention reversed

One of the things I’ve learned from this is that there’s never too much audit information. LESSON is going to be changed to record not just who changes each assignment, but who changes each mark, and there will be a history of every changed mark so that teachers can see when marks are changed.

Apart from this, I would be curious as to what others think about the consequences for these two “hacks.” Were we too lenient on the first? Too harsh on the second? What should we have done differently? And what should we do differently going forward?

Laptop computer locked with chain and padlock by Santeri Viinamäki. Used under a CC BY-SA 4.0 license.

Finishing the LESSON

LESSON

At our school, we’ve been using a web-based marking system called LESSON for the last twelve years or so. I originally wrote LESSON because I was annoyed with the complexity of using a spreadsheet to deal with both assignments of different weights and absences. It started out as a personal web-based gradebook, written in PHP with all the compromises that implies. I didn’t use classes, created only a small number of core functions, and made liberal use of copy and paste. My code was a mess of spaghetti that only got worse as time went on. In other words, I wrote a typical PHP website.

During the final half of 2011, I went home to the States on sabbatical, and, when I returned in early 2012, my teaching load was less than normal for the remainder of the school year. One of my goals for the year was a rewrite of LESSON. I had a vision of LESSON 2.0 having a backend written in Python, a well-written web frontend in PHP or Python and an Android client. Using sqlalchemy and web.py, I came up with a core backend that automatically generated pages based on the database tables, allowed filtering using URLs, and allowed modules to override the automatically generated pages when necessary. The code was elegant (compared to the old LESSON 1.0 code), resilient and much easier to extend.

Unfortunately, other projects started to take my attention off of the backend, and though I was happy with the core, I never reached the point of even starting to port our current system to the backend. The backend has languished since mid-2012 (coincidentally, when I started teaching my usual load again). There was a small resurgence of commits last October, when I had great intentions of resurrecting the project in preparation for some major work integrating family information, but that rapidly tapered off as other projects took precedence.

Compare that to LESSON 1.0 which has seen more or less continuous development over the last twelve years. This year, I’ve mainly focused on moving our school registration process over to LESSON, and, more recently, Telegram integration, but it’s telling that all that work has been done on LESSON 1.0.

So why does the “old” LESSON get all the updates? I think it’s for the following reasons:

  • LESSON 1.0 works and works well. Its code might be a nightmare, but it’s fast, low-bandwidth and easy to use, even for teachers who have never touched a computer before (and we had a few of those when we first started using LESSON). LESSON has a grand total of two images. Last month, we had 1,009,000 external hits on LESSON with a total bandwidth of less than 2GB. That’s under 2KB per hit. And, despite multiple student attempts to convince me otherwise, there hasn’t been a bug in LESSON’s grading code in years.
  • New features trump better code. I am a system administrator, teacher and programmer. Our day-to-day sysadmin emergencies always need to be dealt with before long-term projects and my teaching is definitely next on the list of priorities. When I do have time to work on long-term projects (maybe 10-15 hours a week), new features always have the highest priority. The principal wants to know when he’ll be able to create a Telegram channel, populate it with all of his teachers, and send them a message. He’s never asked me when I’ll be done with LESSON 2.0.
  • I fell victim to the second system effect. When developing LESSON 2.0, I spent a lot of time trying to create a system rather than a program. LESSON 1.0 wasn’t designed to be extensible, but LESSON 2.0 was, complete with the concept of modules and all sorts of other goodies. Unfortunately, these added a level of complexity that made the whole thing more and more difficult to work with.

So where does this leave us? The code for LESSON 1.0 may be ugly, but it works well enough. If I ever get caught up with everything else, I’ll probably continue working on LESSON 2.0, but for now it’s on the back burner, and, like anything left on the back burner for too long, it may eventually get thrown out.

Treasure Chest by JHarake

Locks in the classroom – 2016

For the fourth year now, our grade nine students have been doing 3D modeling using Blender. A couple of weeks ago, our students finished up their first assignments, and I gave the top modelers the option of showing off their work. This year one of my students came in already an expert at 3D modeling (far, far better than I am), so he’s been inspiring other students to go above what I’ve seen in previous years. Without further ado, here are the top models from each of the three grade nine classes (click on the pictures for Full HD renders).

Lock by Barbour - CC0

Lock by Barbour – CC0Source

Nice use of materials, with a colorful choice for a background image.

Lock by Tala - CC BY

Lock by Tala – CC BYSource

Very creative and very different than most of the others. I love the key.


Lock by Yubin – CC0Source
Textures by Laitr Keiows – CC BY-SA 3.0

I normally don’t teach animation until the end of the year, but Yubin didn’t want to wait and came up with this. Very nice!

Lock by Carmouna

Lock by Carmouna – CC BYSource

The artistic eye used to compose this scene makes this project really stand out.

Lock by Nahuda

Lock by Nahuda – CC BYSource

I love the double lock thing going on here.

Lock by RayRay

Lock by RayRay – CC BYSource

The chain just makes this look cool.

Lock by Annan

Lock by Annan – CC BYSource

And wow! Annan used the game physics in Blender to get this chain to rest on the table. This looks great!

Lock by Mohammad

Lock by Mohammad – CC0Source

The final render looks good, but you need to download and open the source to really appreciate this scene. We’re seeing the front of a classroom from inside a locker, with the lock hanging on the door. Mohammad designed the entire classroom and it looks great (and incidentally, looks like most of the classes in the school).

Treasure Chest by JHarake

Treasure Chest by JHarake – CC0Source

When I first looked at this, I thought JHarake had found a great background image. Then I realized that he didn’t use a background image. This is a complete 3D scene, created by a student who makes his teacher’s most intricate Blender projects look like the scribblings of a small child. Well done JHarake!

Notes on a mass upgrade to Fedora 23

Picture of Fedora 23 desktop

Fedora 23

One of the hardest parts of running Fedora in a school setting is keeping on top of the upgrades, and I ended up falling a few months behind. Fedora 23 was released back in November, and it took me until February to start the upgrade process.

For our provisioning process, we’ve switched from a custom koji instance to ansible (with our plays on github), and this release was the first time I was really able to take advantage it. I changed our default kickstart to point to the Fedora 23 repositories, installed it on a test system, ran ansible on it, and voilà, I had a working Fedora 23 setup, running perfectly with all our school’s customizations. It was the easiest upgrade experience I’ve ever had!

Well, mostly.

As usual, the moment you think everything is perfect is the moment everything goes wrong. On our multiseat systems, we have three external AMD graphics cards along with the internal Intel graphics. The first bug I noticed was that the Intel card wasn’t doing any graphics acceleration. It turns out that VGA arbitration is automatically turned on if you have more than one video card, and Intel cards don’t support it in DRI2. DRI3 does handle arbitration just fine, but it was (and still is) disabled in the latest xorg-x11-drv-intel in the updates repository. Luckily for me, there’s a build in koji that re-enables DRI3. Problem solved.

The second bug was…odd. While we use gnome-shell as the default desktop environment in the school, we use lightdm for logging in, mainly because of it’s flexibility. We run xscreensaver in the login screen (and only in the login screen) to make it clear which computers are off, which are on, and which are logged in. GDM doesn’t support xscreensaver, but lightdm does. And this brings us back to the bug. On the Intel seat, moving the mouse or pressing a key would stop the screensaver as expected, but the screen would remain black except for the username control. It seems that the “VisibilityNotify” event isn’t being honored by the driver (though don’t ask me why it should be passed down to the driver). I filed a bug, and then finally figured out that fading xscreensaver back in works around the problem.

The third bug is even stranger. On the teacher’s machine, we have a small script that starts x11vnc (giving no control to anyone connecting to it) so the teacher can give a demonstration to the students. But after install Fedora 23 on the teacher’s machine, the demo kept showing the same three frames over and over. The teacher’s system isn’t multiseat and is using the builtin Intel graphics, so, oddly enough, disabling DRI3 fixed the problem. I filed another bug.

When upgrading the staff room systems, I ran into a bug in which cups runs screaming into the night (ok, slight exaggeration) if you have a server announcing printers over both the old cups and new dnssd protocols. Since we don’t have any pre-F21 systems any more, I’ve just disabled the old cups protocol on the server.

And, finally, my principal, who teachers computers to grades 11 and 12, came in to ask me why LibreOffice was crashing for a couple (and only a couple) of his students when they were formatting cells on a spreadsheet that he gave them. After some fancy footwork involving rm’d .config/libreoffice directories and files saved into random odd formats and then back into ods, we finally managed to format the cells without a crash. Lovely.

All this brings me back to ansible. In each of the bugs that required changes to the workstations, all I had to do was update the ansible scripts and push the changes out. Talk about painless! Ansible has made this job so much easier!

And I do want to finish by saying that these bugs are part of the reason that I love Fedora. With Fedora, I have the freedom to fix these problems myself. For both the cups bug and the xscreensaver bug, I was able to dig into the source code to start tracking down where the problem lay and come up with a workaround. And if I can just get the LibreOffice bug to reproduce, I could get a crash dump off of it and possibly figure it out too. Hurrah for source code!

Scratch group projects – 2016

Scratch

Scratch

As I mentioned last year, programming is on the syllabus for our grade 10 students, and they have just finished this year’s group projects in Scratch. We’re moving on to python, but I’ve posted their projects at http://scratch.lesbg.com Feel free to play them and rate them. This is a first attempt for students, so do please be gentle on the ratings.

If you want to check out previous years’ projects, they’re also available at the links at the top left. If you have any comments or suggestions for the site itself, please leave them below.

Virtualizing Windows (and simplifying my life)

Picture of fireworks

Freedom

At our school, we’ve been running Fedora on most of the desktops since Fedora 8, but the one department that’s stuck with Windows is the accounting department, mainly because their software is Windows-only.  This has long been a problem because most of our infrastructure is built around Linux and we haven’t put nearly as much energy into making sure Windows systems are maintained properly.

Obviously, this led to problems that started out small, but grew until the systems were bordering on unusable.  When it reached the point that we were considering yet another reinstall of Windows, I suggested switching the accountants over to Fedora and having them use a virtual machine for the software that required the other OS.

It took a few days to get something that worked, and another week (including one very late night) to tie down the little glitches and get the virtual machine beyond just-usable to easy-to-use.

I started with VirtualBox, but there were a number of issues with stability, so I decided to take another look at QEMU.  I thought about using libvirt, but one of my requirements was that everything needed to run under the user’s permissions, so it turned out to be easier to run qemu-kvm directly.  I used SPICE and installed the guest agent, which gave us a far better experience with QEMU than the last time I used it for a desktop OS (which, granted, was over five years ago).

Most of my time was spent fixing problems inherent to Windows 7 itself, rather than the virtualization process.  It turns out that there are bugs in how it handles network printers, causing delays every time you want to print.  Oddly enough, the fix was pretty simple, but it took a while to figure it out.  There was also the bug where network drives aren’t mapped properly if the system boots so quickly that the network isn’t up in time, which was only fixable by using a batch file for mapping the network drives.

One change I made was to insist that we use throw-away snapshots for day-to-day work (the data is stored on a network drive) and only keep changes when we’re updating the accounting software.  This should help protect us from viruses and malware that can’t be easily removed.

The best part of all this is that the new accounting VM and the scripts necessary to start it are sitting in a network folder only accessible by the accountants.  This means that they can now do their work from any computer in the school, if necessary, while still protecting them.

And I’m no longer stuck keeping unmanaged Windows systems running.  What a way to close out the year!

Colorful Fireworks by 久留米市民(Kurume-Shimin) used under a CC BY-SA 3.0 unported license

Here's your VOIP

Freeswitch vs. Asterisk?

canned-phone-568056_1920

VOIP of the highest quality

We’ve been experimenting with VOIP in our school, primarily for internal communication.  I’ve set up both asterisk and freeswitch servers, and have been quite frustrated with the limitations of both.

Asterisk only allows one registration to be connected to each extension.  Yes, there are ways to work around this restriction (for extension 101, set up multiple extensions – 980101, 981101, 982101, and then set up a ring group 101 that rings those extensions simultaneously), but it’s an incredibly irritating workaround.

Freeswitch does allow multiple registrations on a single extension, but it has other problems.  Some of our softphones are running over WiFi and we need SRTP for these systems.  Other hardware phones don’t support SRTP, which, while not ideal, is less of an issue because they’re connected via a physical link that we have complete control over.  Unfortunately, even with Freeswitch in bridging mode, it refuses to use SRTP on the softphone link, while using no encryption on the hardware phone link.  It’s either all or nothing.  Which means, during our testing phase, we’re stuck at nothing.  Lovely.

So should I bail on Freeswitch and switch over to Asterisk?  Stick with Freeswitch and hope that I can work out some way of fixing the SRTP problem?  Or should I just give our staff tin cans attached to Cat-6 cable and tell them that’s the new VOIP system?