Finishing the LESSON


At our school, we’ve been using a web-based marking system called LESSON for the last twelve years or so. I originally wrote LESSON because I was annoyed with the complexity of using a spreadsheet to deal with both assignments of different weights and absences. It started out as a personal web-based gradebook, written in PHP with all the compromises that implies. I didn’t use classes, created only a small number of core functions, and made liberal use of copy and paste. My code was a mess of spaghetti that only got worse as time went on. In other words, I wrote a typical PHP website.

During the final half of 2011, I went home to the States on sabbatical, and, when I returned in early 2012, my teaching load was less than normal for the remainder of the school year. One of my goals for the year was a rewrite of LESSON. I had a vision of LESSON 2.0 having a backend written in Python, a well-written web frontend in PHP or Python and an Android client. Using sqlalchemy and, I came up with a core backend that automatically generated pages based on the database tables, allowed filtering using URLs, and allowed modules to override the automatically generated pages when necessary. The code was elegant (compared to the old LESSON 1.0 code), resilient and much easier to extend.

Unfortunately, other projects started to take my attention off of the backend, and though I was happy with the core, I never reached the point of even starting to port our current system to the backend. The backend has languished since mid-2012 (coincidentally, when I started teaching my usual load again). There was a small resurgence of commits last October, when I had great intentions of resurrecting the project in preparation for some major work integrating family information, but that rapidly tapered off as other projects took precedence.

Compare that to LESSON 1.0 which has seen more or less continuous development over the last twelve years. This year, I’ve mainly focused on moving our school registration process over to LESSON, and, more recently, Telegram integration, but it’s telling that all that work has been done on LESSON 1.0.

So why does the “old” LESSON get all the updates? I think it’s for the following reasons:

  • LESSON 1.0 works and works well. Its code might be a nightmare, but it’s fast, low-bandwidth and easy to use, even for teachers who have never touched a computer before (and we had a few of those when we first started using LESSON). LESSON has a grand total of two images. Last month, we had 1,009,000 external hits on LESSON with a total bandwidth of less than 2GB. That’s under 2KB per hit. And, despite multiple student attempts to convince me otherwise, there hasn’t been a bug in LESSON’s grading code in years.
  • New features trump better code. I am a system administrator, teacher and programmer. Our day-to-day sysadmin emergencies always need to be dealt with before long-term projects and my teaching is definitely next on the list of priorities. When I do have time to work on long-term projects (maybe 10-15 hours a week), new features always have the highest priority. The principal wants to know when he’ll be able to create a Telegram channel, populate it with all of his teachers, and send them a message. He’s never asked me when I’ll be done with LESSON 2.0.
  • I fell victim to the second system effect. When developing LESSON 2.0, I spent a lot of time trying to create a system rather than a program. LESSON 1.0 wasn’t designed to be extensible, but LESSON 2.0 was, complete with the concept of modules and all sorts of other goodies. Unfortunately, these added a level of complexity that made the whole thing more and more difficult to work with.

So where does this leave us? The code for LESSON 1.0 may be ugly, but it works well enough. If I ever get caught up with everything else, I’ll probably continue working on LESSON 2.0, but for now it’s on the back burner, and, like anything left on the back burner for too long, it may eventually get thrown out.

Locks in the classroom – 2016

For the fourth year now, our grade nine students have been doing 3D modeling using Blender. A couple of weeks ago, our students finished up their first assignments, and I gave the top modelers the option of showing off their work. This year one of my students came in already an expert at 3D modeling (far, far better than I am), so he’s been inspiring other students to go above what I’ve seen in previous years. Without further ado, here are the top models from each of the three grade nine classes (click on the pictures for Full HD renders).

Lock by Barbour - CC0

Lock by Barbour – CC0Source

Nice use of materials, with a colorful choice for a background image.

Lock by Tala - CC BY

Lock by Tala – CC BYSource

Very creative and very different than most of the others. I love the key.

Lock by Yubin – CC0Source
Textures by Laitr Keiows – CC BY-SA 3.0

I normally don’t teach animation until the end of the year, but Yubin didn’t want to wait and came up with this. Very nice!

Lock by Carmouna

Lock by Carmouna – CC BYSource

The artistic eye used to compose this scene makes this project really stand out.

Lock by Nahuda

Lock by Nahuda – CC BYSource

I love the double lock thing going on here.

Lock by RayRay

Lock by RayRay – CC BYSource

The chain just makes this look cool.

Lock by Annan

Lock by Annan – CC BYSource

And wow! Annan used the game physics in Blender to get this chain to rest on the table. This looks great!

Lock by Mohammad

Lock by Mohammad – CC0Source

The final render looks good, but you need to download and open the source to really appreciate this scene. We’re seeing the front of a classroom from inside a locker, with the lock hanging on the door. Mohammad designed the entire classroom and it looks great (and incidentally, looks like most of the classes in the school).

Treasure Chest by JHarake

Treasure Chest by JHarake – CC0Source

When I first looked at this, I thought JHarake had found a great background image. Then I realized that he didn’t use a background image. This is a complete 3D scene, created by a student who makes his teacher’s most intricate Blender projects look like the scribblings of a small child. Well done JHarake!

Notes on a mass upgrade to Fedora 23

Picture of Fedora 23 desktop

Fedora 23

One of the hardest parts of running Fedora in a school setting is keeping on top of the upgrades, and I ended up falling a few months behind. Fedora 23 was released back in November, and it took me until February to start the upgrade process.

For our provisioning process, we’ve switched from a custom koji instance to ansible (with our plays on github), and this release was the first time I was really able to take advantage it. I changed our default kickstart to point to the Fedora 23 repositories, installed it on a test system, ran ansible on it, and voilà, I had a working Fedora 23 setup, running perfectly with all our school’s customizations. It was the easiest upgrade experience I’ve ever had!

Well, mostly.

As usual, the moment you think everything is perfect is the moment everything goes wrong. On our multiseat systems, we have three external AMD graphics cards along with the internal Intel graphics. The first bug I noticed was that the Intel card wasn’t doing any graphics acceleration. It turns out that VGA arbitration is automatically turned on if you have more than one video card, and Intel cards don’t support it in DRI2. DRI3 does handle arbitration just fine, but it was (and still is) disabled in the latest xorg-x11-drv-intel in the updates repository. Luckily for me, there’s a build in koji that re-enables DRI3. Problem solved.

The second bug was…odd. While we use gnome-shell as the default desktop environment in the school, we use lightdm for logging in, mainly because of it’s flexibility. We run xscreensaver in the login screen (and only in the login screen) to make it clear which computers are off, which are on, and which are logged in. GDM doesn’t support xscreensaver, but lightdm does. And this brings us back to the bug. On the Intel seat, moving the mouse or pressing a key would stop the screensaver as expected, but the screen would remain black except for the username control. It seems that the “VisibilityNotify” event isn’t being honored by the driver (though don’t ask me why it should be passed down to the driver). I filed a bug, and then finally figured out that fading xscreensaver back in works around the problem.

The third bug is even stranger. On the teacher’s machine, we have a small script that starts x11vnc (giving no control to anyone connecting to it) so the teacher can give a demonstration to the students. But after install Fedora 23 on the teacher’s machine, the demo kept showing the same three frames over and over. The teacher’s system isn’t multiseat and is using the builtin Intel graphics, so, oddly enough, disabling DRI3 fixed the problem. I filed another bug.

When upgrading the staff room systems, I ran into a bug in which cups runs screaming into the night (ok, slight exaggeration) if you have a server announcing printers over both the old cups and new dnssd protocols. Since we don’t have any pre-F21 systems any more, I’ve just disabled the old cups protocol on the server.

And, finally, my principal, who teachers computers to grades 11 and 12, came in to ask me why LibreOffice was crashing for a couple (and only a couple) of his students when they were formatting cells on a spreadsheet that he gave them. After some fancy footwork involving rm’d .config/libreoffice directories and files saved into random odd formats and then back into ods, we finally managed to format the cells without a crash. Lovely.

All this brings me back to ansible. In each of the bugs that required changes to the workstations, all I had to do was update the ansible scripts and push the changes out. Talk about painless! Ansible has made this job so much easier!

And I do want to finish by saying that these bugs are part of the reason that I love Fedora. With Fedora, I have the freedom to fix these problems myself. For both the cups bug and the xscreensaver bug, I was able to dig into the source code to start tracking down where the problem lay and come up with a workaround. And if I can just get the LibreOffice bug to reproduce, I could get a crash dump off of it and possibly figure it out too. Hurrah for source code!

Scratch group projects – 2016



As I mentioned last year, programming is on the syllabus for our grade 10 students, and they have just finished this year’s group projects in Scratch. We’re moving on to python, but I’ve posted their projects at Feel free to play them and rate them. This is a first attempt for students, so do please be gentle on the ratings.

If you want to check out previous years’ projects, they’re also available at the links at the top left. If you have any comments or suggestions for the site itself, please leave them below.

Virtualizing Windows (and simplifying my life)

Picture of fireworks


At our school, we’ve been running Fedora on most of the desktops since Fedora 8, but the one department that’s stuck with Windows is the accounting department, mainly because their software is Windows-only.  This has long been a problem because most of our infrastructure is built around Linux and we haven’t put nearly as much energy into making sure Windows systems are maintained properly.

Obviously, this led to problems that started out small, but grew until the systems were bordering on unusable.  When it reached the point that we were considering yet another reinstall of Windows, I suggested switching the accountants over to Fedora and having them use a virtual machine for the software that required the other OS.

It took a few days to get something that worked, and another week (including one very late night) to tie down the little glitches and get the virtual machine beyond just-usable to easy-to-use.

I started with VirtualBox, but there were a number of issues with stability, so I decided to take another look at QEMU.  I thought about using libvirt, but one of my requirements was that everything needed to run under the user’s permissions, so it turned out to be easier to run qemu-kvm directly.  I used SPICE and installed the guest agent, which gave us a far better experience with QEMU than the last time I used it for a desktop OS (which, granted, was over five years ago).

Most of my time was spent fixing problems inherent to Windows 7 itself, rather than the virtualization process.  It turns out that there are bugs in how it handles network printers, causing delays every time you want to print.  Oddly enough, the fix was pretty simple, but it took a while to figure it out.  There was also the bug where network drives aren’t mapped properly if the system boots so quickly that the network isn’t up in time, which was only fixable by using a batch file for mapping the network drives.

One change I made was to insist that we use throw-away snapshots for day-to-day work (the data is stored on a network drive) and only keep changes when we’re updating the accounting software.  This should help protect us from viruses and malware that can’t be easily removed.

The best part of all this is that the new accounting VM and the scripts necessary to start it are sitting in a network folder only accessible by the accountants.  This means that they can now do their work from any computer in the school, if necessary, while still protecting them.

And I’m no longer stuck keeping unmanaged Windows systems running.  What a way to close out the year!

Colorful Fireworks by 久留米市民(Kurume-Shimin) used under a CC BY-SA 3.0 unported license

Freeswitch vs. Asterisk?


VOIP of the highest quality

We’ve been experimenting with VOIP in our school, primarily for internal communication.  I’ve set up both asterisk and freeswitch servers, and have been quite frustrated with the limitations of both.

Asterisk only allows one registration to be connected to each extension.  Yes, there are ways to work around this restriction (for extension 101, set up multiple extensions – 980101, 981101, 982101, and then set up a ring group 101 that rings those extensions simultaneously), but it’s an incredibly irritating workaround.

Freeswitch does allow multiple registrations on a single extension, but it has other problems.  Some of our softphones are running over WiFi and we need SRTP for these systems.  Other hardware phones don’t support SRTP, which, while not ideal, is less of an issue because they’re connected via a physical link that we have complete control over.  Unfortunately, even with Freeswitch in bridging mode, it refuses to use SRTP on the softphone link, while using no encryption on the hardware phone link.  It’s either all or nothing.  Which means, during our testing phase, we’re stuck at nothing.  Lovely.

So should I bail on Freeswitch and switch over to Asterisk?  Stick with Freeswitch and hope that I can work out some way of fixing the SRTP problem?  Or should I just give our staff tin cans attached to Cat-6 cable and tell them that’s the new VOIP system?

Where’s my blankey? (aka IPv6 on a flat network)

Boy on beach holding small soft toy while sucking thumb

Familiarity is comforting

At the Lebanon Evangelical School – Loueizeh, we try to push the envelope when it comes to testing new technology.  Our website makes efficient use of some of the latest CSS technology from 1996, and we have just started experimenting with this new-fangled IPv6 technology, also from the mid 90’s.  Cue Macarena single…

If you’re on a flat network and want to match MAC addresses to IPv6 addresses, ditch ISC’s DHCPv6 server, and instead use dhcpy6d

After years of asking, our ISP has finally given us an IPv6 /48 prefix, so I finally have the privilege of setting up an IPv6 network to coexist with our IPv4 network.  Yay!  As an aside, I think we’re one of the first organizations in Lebanon to be working with IPv6.  A couple of years ago, I asked our old ISP about getting an IPv6 address and was told that they didn’t have any.  There were plenty of IPv4 addresses to go around, so why bother?  But I digress…

Our current IPv4 topology is… flat.  Very flat.  We use a single 10.x.y.z internal IP range, given out by two DHCP servers, where x is a fixed number, y is the category of the device and z is the individual address.  The categories allow us to divide devices into guest, staff and student groups.  We are very aware that MAC addresses can be easily spoofed, so these categories are primarily used for traffic shaping and nothing security related.  I am aware of (and would like to start working with) VLANs, but not all of our networking equipment supports them.

So our primary requirement for IPv6 is that we are able to set up corresponding categories to match our IPv4 categories (y in 10.x.y.z).  Another requirement is that we use some form of static IP addresses for logging purposes and that these static addresses be somehow tied to a device’s corresponding IPv4 address, mainly for convenience.

My first plan was to use SLAAC, which allows us to use 16 bits to specify categories (double what we have with our IPv4 addresses), while still allowing the client 64 bits to use in coming up with its own address.  Unfortunately, SLAAC falls down for us on both categories and static addresses.  To implement categories properly, we’d need to divide our network into VLANs, and we still have a few unmanaged switches that don’t know how to handle them.  And, while some devices just use their MAC addresses to generate their unique address, others (especially the more modern mobile OS’s which automatically implement RFC4941) will generate random unique addresses, which causes problems if we want to track who’s doing what.

Plan B was to use DHCPv6, which isn’t that much of a stretch, seeing as we’re already using DHCP on our IPv4 network.  Granted, since Android doesn’t work with DHCPv6 (really, Google?), that means all of the Android devices on our network will be stuck on IPv4, but, since we’re still in the experimental stage, I’m ok with that.  With DHCPv6 I can choose which address each device gets, allowing me to specify both categories and static (randomly generated) IP addresses.  So I started putting together an ISC DHCPv6 server, looked at the examples, and… “Where the heck am I supposed to put the MAC address?”

You see, in their infinite wisdom, the IETF decided that MAC addresses were no longer sufficient to tell the DHCPv6 server what IP address we should get.  No, now we get to use these new things called DUIDs which last forever like MAC addresses, but stay the same for all interfaces on a system, unlike MAC addresses.  Woo-hoo!  There are some small exceptions, especially with a computer or phone as opposed to an embedded device.  The DUID will not be automatically be the same if you dual boot.  Or if you wipe the OS.  Or, if like us, you make extensive use of netbooting for system maintenance.  But other than that, DUIDs will last forever.

The best part is that there are multiple ways of generating a DUID (including straight from the MAC address), but the server doesn’t have any say in telling the clients how it wants its DUIDs generated.  That means that clients like NetworkManager will quite happily generate a DUID based on /etc/machine-id, which is itself randomly generated the first time a system is booted.  Not very useful if you want to pass the same IP address whether a system is being netbooted into maintenance mode or booting normally off of the hard drive.

Now, all this wouldn’t be insurmountable, except for the fact that I currently have a database of MAC addresses matched with IPv4 addresses (complete with categories), but, because DUIDs don’t necessarily have anything to do with the MAC address, I have no idea how to match that same device to the IPv6 address I want to give it.

Apparently ISC’s DHCPv6 server has implemented a relatively new specification that allows link-local addresses to be sent to the DHCPv6 server through a relay, but our network is already flat, so I shouldn’t need to screw around with a relay.

There’s also some ambiguity about the Confirm message in DHCPv6.  The specific section says:

When the server receives a Confirm message, the server determines whether the addresses in the Confirm message are appropriate for the link to which the client is attached. If all of the addresses in the Confirm message pass this test, the server returns a status of Success. If any of the addresses do not pass this test, the server returns a status of NotOnLink.

The question is, if I’ve changed the static IP address for a client, is the old address still appropriate for the link?  As the sysadmin, my answer is, “No, please discontinue use of the old address immediately.”  Unfortunately, the ISC DHCPv6 server disagrees with me and will happily confirm the old addresses until the cows come home.

After a bit of searching, I’ve found a solution that I’m actually quite happy with.  The Leibniz Institute for Solid State and Materials Research in Dresden has released a DHCPv6 server called dhcpy6d written in Python that allows you to match based on MAC address, DUID or a combination of both.  There have been a few bugs in it, and it’s not yet handling Confirm messages the way I’d like it to, but upstream has been very responsive, and I’m looking forward to having an IPv6 system that works for us.

We’re not quite up and running 100% yet, but I hope to be there by the second week of November.  Of course, I had hoped to get some work done during this long weekend, but God had other plans (at least, judging by the lightning that destroyed quite a bit of my networking equipment at home just before school let out for the weekend).