Category Archives: Computers

Multiseat systems and the NVIDIA binary driver

Building mesa

Building mesa

Ever since our school switched to Fedora on the desktop, I’ve either used the onboard Intel graphics or AMD Radeon cards, since both are supported out of the box in Fedora. With our multiseat systems, we now need three external video cards on top of the onboard graphics on each system, so we’ve bought a large number of Radeon cards over the last few years.

Unfortunately, our local supplier has greatly reduced the number of AMD cards that they stock. In their latest price lists, they have a grand total of two Radeon cards in our price range, and one of them is almost seven years old!

This has led me to take a second look at NVIDIA cards, and I’m slowly coming back around to the concept of buying them and maybe even using their binary drivers. Our needs have changed since we first started using Linux, and NVIDIA’s binary driver does offer some unique benefits.

As we’ve started teaching 3D modeling using Blender, render time has become a real bottleneck for some of our students. We allow students to use the computers before and after school, but some of them don’t have much flexibility in their transportation and need to get their rendering done during the school breaks. Having two or three students all trying to render at the same time on a single multiseat system can lead to a sluggish system and very slow rendering. The easiest way to fix this is to do the rendering in the GPU, which Blender does support, but only using NVIDIA’s binary driver.

So about a month ago, I ordered a cheap NVIDIA card for testing purposes. I swapped it with an AMD card on one of our multiseat systems and powered it up. Fedora recognized the card using the open-source nouveau driver and everything just worked. Beautiful!

Then, a few hours later, I noticed the system had frozen. I rebooted it, and, after a few hours, it had frozen again. I moved the NVIDIA card into a different system, and, after a few hours, it froze while the original system just kept running.

Some research showed that the nouveau driver sometimes has issues with multiple video cards on the same system. There was some talk about extracting the binary driver’s firmware and using it in nouveau, but I decided to see if I could get the binary driver working without breaking our other Intel and AMD seats.

The first thing I did was upgrade the test system to Fedora 25 in hopes of taking advantage of the work done to make mesa and the NVIDIA binary driver coexist. I then installed the binary NVIDIA drivers from this repository (mainly because his version of blender already has the CUDA kernels compiled in). The NVIDIA seat came up just fine, but I quickly found that mesa in Fedora 25 isn’t built with libglvnd (a shim between either the mesa or NVIDIA OpenGL implementation, depending on which card you’re using and your applications) enabled, so all of the seats based on open drivers didn’t come up. But, even when it was enabled, I ran into this bug, so I ended up extending this patch so it would also work with Gallium drivers and applying it.

This took me several steps closer, but apparently the X11 GLX module is not part of libglvnd and NVIDIA sets the Files section in xorg.conf to use it’s own GLX module (which, oddly enough, doesn’t work with the open drivers). I finally worked around this via the ugly hack of creating two different xorg.conf.d directories and telling lightdm to use the NVIDIA one when loading the NVIDIA seat.

Voilà! We now have a multiseat system with one Intel built-in card using the mesa driver, two AMD cards using the mesa Gallium driver, and one NVIDIA card using the NVIDIA binary driver. And it only cost me eight hours and my sanity.

So what needs to happen to make this Just Work™? Either libglvnd needs to also include the X11 GLX module or we need a different shim to accomplish the same thing. And Fedora needs to build mesa with libglvnd enabled (but not until this bug is fixed!)

My mesa build is here and the source rpm is here. There is a manual “Provides: libGL.so.1()(64bit)” in there that isn’t technically correct, but I really didn’t want to recompile negativo17’s libglvnd to add it in and my mesa build requires that libglvnd implementation.

My xorg configs are here and my lightdm configuration is here. Please note that the xorg configs have my specific PCI paths; yours may differ.

And I do plan to write a script to automate the xorg and lightdm configs. I’ll update this post when I’ve done so.

Sidenote: As I was looking through my old posts to see if I had anything on NVIDIA, I came across a comment by Seth Vidal. He was an excellent example of what the Fedora community is all about, and I really miss him.

From NFS to LizardFS

If you’ve been following me for a while, you’ll know that we started our data servers out using NFS on ext4 mirrored over DRBD, hit some load problems, switched to btrfs, hit load problems again, tried a hacky workaround, ran into problems, dropped DRBD for glusterfs, had a major disaster, switched back to NFS on ext4 mirrored over DRBD, hit more load problems, and finally dropped DRBD for ZFS.

As of March 2016, our network looked something like this:

Old server layout

Old server layout

Our NFS over ZFS system worked great for three years, especially after we added SSD cache and log devices to our ZFS pools, but we were starting to overload our ZFS servers and I realized that we didn’t really have any way of scaling up.

This pushed me to investigate distributed filesystems yet again. As I mentioned here, distributed filesystems have been a holy grail for me, but I never found one that would work for us. Our problem is that our home directories (including config directories) are stored on our data servers, and there might be over one hundred users logged in simultaneously. Linux desktops tend to do a lot of small reads and writes to the config directories, and any latency bottlenecks tend to cascade. This leads to an unresponsive network, which then leads to students acting out the Old Testament practice of stoning the computer. GlusterFS was too slow (and almost lost all our data), CephFS still seems too experimental (especially for the features I want), and there didn’t seem to be any other reasonable alternatives… until I looked at LizardFS.

LizardFS (a completely open source fork of MooseFS) is a distributed filesystem that has one fascinating twist: All the metadata is stored in RAM. It gets written out to the hard drive regularly, but all of the metadata must fit into the RAM. The main result is that metadata lookups are rocket-fast. Add to that the ability to direct different paths (say, perhaps, config directories) to different storage types (say, perhaps, SSDs), and you have a filesystem that is scalable and fast.

LizardFS does have its drawbacks. You can run hot backups of your metadata servers, but only one will ever be the active master at any one time. If it goes down, you have to manually switch one of the replicas into master mode. LizardFS also has a very complicated upgrade procedure. First the metadata replicas must be upgraded, then the master and finally the clients. And finally, there are some corner cases where replication is not as robust as I would like it to be, but they seem to be well understood and really only seem to affect very new blocks.

So, given the potential benefits and drawbacks, we decided to run some tests. The results were instant… and impressive. A single user’s login time on a server with no load… doubled. Instead of five seconds, it took ten for them to log in. Not good. But when a whole class logged in simultaneously, it took only 15 seconds for them to all log in, down from three to five minutes. We decided that a massive speed gain in the multiple user scenario was well worth the speed sacrifice in the single-user scenario.

Another bonus is that we’ve gone from two separate data servers with two completely different filesystems (only one which ever had high load) to five data servers sharing the load while serving out one massive filesystem, giving us a system that now looks like this:

New server setup

New server layout

So, six months on, LizardFS has served us well, and will hopefully continue to serve us for the next (few? many?) years. The main downside is that Fedora doesn’t have LizardFS in its repositories, but I’m thinking about cleaning up my spec and putting in a review request.

Updated to add graphics of old and new server layouts, info about Fedora packaging status, LizardFS bug links, and remove some grammatical errors

Flock 2016

Man and woman in driving horse-drawn buggy

Downtown Kraków

I have just returned from a vacation in beautiful Kraków, where, entirely coincidentally, there just happened to be a Fedora conference! My family and I enjoyed the amazing sights around Kraków (if you haven’t visited the salt mine, you really should), but my personal highlight was getting to attend my first Flock, where I got to meet people face to face who I’d only previously talked with via IRC and email.

I got to chance to speak about how we use Fedora in the classroom in our school (slides here). There were some excellent questions from the audience at the end, and I realized (again!) that my biggest problem is creating decent documentation about what we’re doing so others can follow.

One of my goals over the next year is to make sure that our work is easily reproducible in other schools, both from the sysadmin side and from the educational side.

My biggest take-away from the conference is that Fedora is moving into some very interesting times as it starts to expand from rpms being the only system delivery mechanism. I’m very interested in ostree with its concept of a read-only system partition and the work they’re doing on layered trees so you can have multiple system images branching off of one base image.

I’d really like to thank the event organizers for all the work they did putting Flock together, the design team for the beautiful t-shirts, and the Fedora community for just being great. And, while I’m at it, I’d like to extend personal thanks to Rafał Luzynski and his wife, Maja, for their hospitality.

Talk – Using Fedora in the classroom

Spreadsheet assignment

Spreadsheet assignment

So I’m sitting here in Kraków, doing some last-minute preparation for my talk (Fedora in the Classroom) at the upcoming Flock conference next week.

I’ll be looking at why we use Fedora in our school, what tools we use to setup and maintain our workstations, and the actual subjects that we teach our students, complete with actual projects1 that our students have done.

If you’re a teacher looking for ways to use open source software in the classroom, an administrator looking for a computer curriculum that emphasizes creativity and comprehension over memorization and rote learning, or you’re just interested in seeing how Fedora is effectively used in a school environment, please do come check it out.

 
[1] Projects have been anonymized to protect student privacy

Hacked!

Laptop in chains

 

With our school’s graduation ceremony last night, the school year is now officially finished. This year will definitely go down in my memory as the year that the students got the best of me… twice!

IP-gate
To give some background on the first “hack,” our current network uses a flat IP network with IP subnets used for each different set of machines (for organizational purposes). We don’t use IP-based security for obvious reasons, but we do use the subnets for deciding internet speed. IP addresses (fixed except for the guest subnet) are given out using DHCP, and each of the subnets except the guest subnet gets decent speed.

When I set up this system ten years ago, I was well aware of the obvious drawback: any person could set a static IP address on any subnet they chose, and, given our lack of managed switches (at the time we had none, though things are changing), there wasn’t much of anything I could do about it. On the flip side, the worst that could happen is that these users would get faster internet, hardly the end of the world.

It took ten years, but, finally, someone figured it out. One of our more intelligent students decided that his IP address of 10.10.10.113 didn’t make a whole lot of sense, given that the gateway is 10.10.1.1. He set his IP to 10.10.1.113, and, voilà, his internet speed shot through the roof!

Naturally, he shared his findings with his friends, who managed to keep it under the radar until one of the friends decided to see how well BitTorrent would work with the school internet. What none of these students realized is that the 10.10.1.* subnet was for servers, and, oddly enough, none of our servers uses BitTorrent. The traffic stuck out like a sore thumb, and I finally caught on.

My first step was to blacklist all unrecognized MAC addresses using the server subnet. The next step was more difficult. Now that the cat was out of the bag and everyone knew how to get faster internet, I needed a way to block anybody not using the IP they’d been assigned through our DHCP server. Obviously, there is a correct way of doing this, but that seems to be using 802.1x, and we’re just not there yet. My quick and dirty solution was to copy the dhcp configuration file containing all the host and IP information to our firewall, and then generate a list of iptables rules that only allow traffic through if the IP address matches the expected MAC address.

The problem with this solution is that it doesn’t account for the fact that spoofing MAC addresses is actually relatively simple, so it looks like one of my summer projects is going to be a complete revamp of our network. I’m hoping I can configure our FreeIPA server to also operate as the backend for a RADIUS server so we can implement 802.1x security.

In this case, the consequences of the “hack” for us were pretty insignificant. Students got some extra bandwidth for a while. The students who changed their IP addresses also didn’t suffer any major consequences. Their devices were blacklisted from the internet until they came to speak with me, and then were put on the guest subnet. All of the students were in their final year, so they were only stuck on the guest subnet for the last month or so.

The most obvious lesson I learned from “IP-gate” is that security through obscurity works great…until someone turns on the light. And when that happens, you’d better have a plan.

The Grade-changing Scandal
This was a far messier situation. One of our teachers allowed a student to access their computer to set up a video for class. On the computer, the teacher had saved their login credentials for LESSON, our web-based marking system. While the teacher was distracted, the student used this trick to find the teacher’s password, and then shared the password with different members of the class. Throughout the next few days, the class average for that teacher’s subjects rose at a remarkable rate.

Three days later, one of the students finally told the principal what had happened, and the principal called me. What followed was a day of tying together evidence from multiple sources to work out who changed what and when.

What the students weren’t aware of was that LESSON logs everything at the assignment level, so I could see which IP addresses changed which assignments. If the IP was an internal school address, I could also see which user changed the assignment. One of the students used their laptop (registered on the network, so I knew who it was) to change some marks, then logged in from a lab computer (so once again, I knew who it was), and then finally logged in from home.

The students who logged in from home were harder to track, at least until they did something foolish, like logging in as themselves to verify that the marks had actually changed ten seconds after logging out as the teacher.

We also do daily backups of the LESSON database that we keep for a full year, so it was a piece of cake to restore all of the marks back to their original scores.

Obviously though, this went much further than the IP-spoofing going in in “IP-gate.” This wasn’t just some kids wanting faster internet, this was a case of flagrant academic dishonesty.

In the end, we came up with the following consequences:

  • The students who masterminded the break-in received a zero for the subject for the term
  • The students who we caught changing the marks received zeros for any assignment of theirs that had a changed mark
  • The students we knew that they knew their marks were changed received three Saturday detentions (they have to sit in complete silence for four hours on a Saturday)
  • The students we suspected that they knew their marks were changed received one Saturday detention, though these students were allowed to appeal, and most who did had their Saturday detention reversed

One of the things I’ve learned from this is that there’s never too much audit information. LESSON is going to be changed to record not just who changes each assignment, but who changes each mark, and there will be a history of every changed mark so that teachers can see when marks are changed.

Apart from this, I would be curious as to what others think about the consequences for these two “hacks.” Were we too lenient on the first? Too harsh on the second? What should we have done differently? And what should we do differently going forward?

Laptop computer locked with chain and padlock by Santeri Viinamäki. Used under a CC BY-SA 4.0 license.

Finishing the LESSON

LESSON

At our school, we’ve been using a web-based marking system called LESSON for the last twelve years or so. I originally wrote LESSON because I was annoyed with the complexity of using a spreadsheet to deal with both assignments of different weights and absences. It started out as a personal web-based gradebook, written in PHP with all the compromises that implies. I didn’t use classes, created only a small number of core functions, and made liberal use of copy and paste. My code was a mess of spaghetti that only got worse as time went on. In other words, I wrote a typical PHP website.

During the final half of 2011, I went home to the States on sabbatical, and, when I returned in early 2012, my teaching load was less than normal for the remainder of the school year. One of my goals for the year was a rewrite of LESSON. I had a vision of LESSON 2.0 having a backend written in Python, a well-written web frontend in PHP or Python and an Android client. Using sqlalchemy and web.py, I came up with a core backend that automatically generated pages based on the database tables, allowed filtering using URLs, and allowed modules to override the automatically generated pages when necessary. The code was elegant (compared to the old LESSON 1.0 code), resilient and much easier to extend.

Unfortunately, other projects started to take my attention off of the backend, and though I was happy with the core, I never reached the point of even starting to port our current system to the backend. The backend has languished since mid-2012 (coincidentally, when I started teaching my usual load again). There was a small resurgence of commits last October, when I had great intentions of resurrecting the project in preparation for some major work integrating family information, but that rapidly tapered off as other projects took precedence.

Compare that to LESSON 1.0 which has seen more or less continuous development over the last twelve years. This year, I’ve mainly focused on moving our school registration process over to LESSON, and, more recently, Telegram integration, but it’s telling that all that work has been done on LESSON 1.0.

So why does the “old” LESSON get all the updates? I think it’s for the following reasons:

  • LESSON 1.0 works and works well. Its code might be a nightmare, but it’s fast, low-bandwidth and easy to use, even for teachers who have never touched a computer before (and we had a few of those when we first started using LESSON). LESSON has a grand total of two images. Last month, we had 1,009,000 external hits on LESSON with a total bandwidth of less than 2GB. That’s under 2KB per hit. And, despite multiple student attempts to convince me otherwise, there hasn’t been a bug in LESSON’s grading code in years.
  • New features trump better code. I am a system administrator, teacher and programmer. Our day-to-day sysadmin emergencies always need to be dealt with before long-term projects and my teaching is definitely next on the list of priorities. When I do have time to work on long-term projects (maybe 10-15 hours a week), new features always have the highest priority. The principal wants to know when he’ll be able to create a Telegram channel, populate it with all of his teachers, and send them a message. He’s never asked me when I’ll be done with LESSON 2.0.
  • I fell victim to the second system effect. When developing LESSON 2.0, I spent a lot of time trying to create a system rather than a program. LESSON 1.0 wasn’t designed to be extensible, but LESSON 2.0 was, complete with the concept of modules and all sorts of other goodies. Unfortunately, these added a level of complexity that made the whole thing more and more difficult to work with.

So where does this leave us? The code for LESSON 1.0 may be ugly, but it works well enough. If I ever get caught up with everything else, I’ll probably continue working on LESSON 2.0, but for now it’s on the back burner, and, like anything left on the back burner for too long, it may eventually get thrown out.

Treasure Chest by JHarake

Locks in the classroom – 2016

For the fourth year now, our grade nine students have been doing 3D modeling using Blender. A couple of weeks ago, our students finished up their first assignments, and I gave the top modelers the option of showing off their work. This year one of my students came in already an expert at 3D modeling (far, far better than I am), so he’s been inspiring other students to go above what I’ve seen in previous years. Without further ado, here are the top models from each of the three grade nine classes (click on the pictures for Full HD renders).

Lock by Barbour - CC0

Lock by Barbour – CC0Source

Nice use of materials, with a colorful choice for a background image.

Lock by Tala - CC BY

Lock by Tala – CC BYSource

Very creative and very different than most of the others. I love the key.


Lock by Yubin – CC0Source
Textures by Laitr Keiows – CC BY-SA 3.0

I normally don’t teach animation until the end of the year, but Yubin didn’t want to wait and came up with this. Very nice!

Lock by Carmouna

Lock by Carmouna – CC BYSource

The artistic eye used to compose this scene makes this project really stand out.

Lock by Nahuda

Lock by Nahuda – CC BYSource

I love the double lock thing going on here.

Lock by RayRay

Lock by RayRay – CC BYSource

The chain just makes this look cool.

Lock by Annan

Lock by Annan – CC BYSource

And wow! Annan used the game physics in Blender to get this chain to rest on the table. This looks great!

Lock by Mohammad

Lock by Mohammad – CC0Source

The final render looks good, but you need to download and open the source to really appreciate this scene. We’re seeing the front of a classroom from inside a locker, with the lock hanging on the door. Mohammad designed the entire classroom and it looks great (and incidentally, looks like most of the classes in the school).

Treasure Chest by JHarake

Treasure Chest by JHarake – CC0Source

When I first looked at this, I thought JHarake had found a great background image. Then I realized that he didn’t use a background image. This is a complete 3D scene, created by a student who makes his teacher’s most intricate Blender projects look like the scribblings of a small child. Well done JHarake!