Tag Archives: lesbg

Changes ahead

Lebanon Evangelical School

I’ve been working at the Lebanon Evangelical School for over fifteen years, and it has been an amazing experience. During that time we’ve switched to Linux on both servers and desktops, setup an open-source infrastructure that’s both powerful and user-friendly, and created an IT curriculum that pushes our students well beyond the local standards.

More importantly, I’ve had the opportunity to invest in students’ lives and watch them grow. One of the students I taught in the early 2000s is now teaching my daughter! Our school really is “lighting the fires that burn for eternity.”

Having said all that, the time has come for a change, and this school year will be our family’s last in Lebanon. I will be spending this school year simplifying and documenting our system to make sure that the school is in good shape for my departure, while also working to train up those under me.

Next summer, we will be leaving Lebanon, but our next step is still up in the air. Ireland and the US are the obvious choices, but we’d definitely be open to opportunities elsewhere. For now, our focus is on making sure we finish the school year strong.

Advertisements

Locks in the classroom – 2017

For the fifth year now, our grade nine students have been doing 3D modeling using Blender. Our students finished up their first assignments over a month ago, but it’s taken this long for me to get the top models together. So, with no further delay, here are the top models from each of the three grade nine classes (click on the pictures for Full HD renders).

We start with this nice mix of color and reflection.

Lock by A. Badr – CC BYSource

Next is a double-lock combination.

Lock by Amanda S. – CC BYSource

And I love the addition of a key in this lock.

Lock by christian07 – CC BY-SASource

Next is the missing link. I really like the mix of pink and black.

Lock by fromdawn02 – CC BY-SASource

Here we have safari-themed locks. Nice choice of textures!

Lock by A. Ayvazian – CC BY-SASource

And here’s a lock in a cave. The cave itself is a proper model, not just a background image.

Lock by A. Zamroud – CC BY-SASource

Here we have the lock on the Pearly Gates. I wish the lock texture was of better quality, but I really appreciate how @david190 has shaped the lock to match the texture, especially on the top.

Lock by @david190 – CC BY-SASource

And here we have a beautiful lock in a velvety box. Very nice!

Lock by Waad H – CC BY-SASource

We can’t have a lock assignment without the obligatory pirate’s treasure chest. I love the attention to detail!

Lock by Sarah B. – CC BY-SASource

And, our final submission includes a monkey. Because… monkeys rock? What a wonderful level of detail in this scene.

Lock by JP Kazzi – CC BY-SASource

Bare-metal Kubernetes

A few years ago, I attended my first Linux conference, DevConf 2014. Many of the speakers talked about containers and how wonderful they were, and my interest was piqued, but I’ve never really had an opportunity to use them.

As the sysadmin for a school, there just isn’t much need for the scalability provided for by containers. Our internal web site runs on a single VM, and the short downtimes required for system updates and upgrades are not a problem, especially if I plan them for the weekends. On the flip side, having something that we can use to spin up web services quickly isn’t a bad idea, so, over the last few months, I’ve been experimenting with Kubernetes.

My main goal was to get something running that was easy to set up and Just Works™. Well, my experience setting up Kubernetes was anything but easy, but, now that it’s up, it does seem to just work. My main problem was that I wanted to use my three oVirt nodes (running CentOS 7) as both Kubernetes nodes and masters, which meant the tutorials took a bit of finessing to get working.

I mostly followed this guide for the initial setup, and then this guide, but I did run into a few problems that I’d like to document. The first was that my containers were inaccessible on their cluster IP range, the primary symptom being that the kube-dashboard service couldn’t connect to the kubernetes service. It turned out that I, rather stupidly, forgot to start kube-proxy, which does all the iptables magic to direct traffic to the correct destination.

The second problem I ran into was that I couldn’t get pretty graphs in kube-dashboard because the heapster service wouldn’t start because I hadn’t set up the cluster DNS service, kube-dns. To be fair, the instructions for doing so are pretty unclear. In the end, I downloaded skydns-rc.yaml.sed and skydns-svc.yaml.sed, and replaced $DNS_DOMAIN and $DNS_SERVER_IP with the values I wanted to use.

The final problem I ran into is that I’m using our school’s local Certificate Authority for all the certificates we use, and I’ve had to keep on adding new subject alternative names to the server cert and then regenerate it. At the moment, it’s got the following:
DNS:kubernetes.example.com
DNS:node01.example.com
DNS:node02.example.com
DNS:node03.example.com
DNS:localhost
DNS:localhost.localdomain
DNS:kubernetes.default.svc.local
# Where I replaced $DNS_DOMAIN with “local”
DNS:kubernetes.local
DNS:kubernetes.default
IP Address:127.0.0.1
IP Address:172.30.0.1
# Where our cluster IP range is 172.30.0.0/16

I suspect I could now get rid of some of those hostnames/addresses, and I’m not even sure if this method is best practice, but at least it’s all working.

So I’m at the point now where I need to see if I can setup our MikroTik router as a load balancer and then see if I can get our web based marking system, LESSON, moved over to a container with multiple replicas. Hurrah for redundancy!

Broken Wagon Wheel by Kevin Casper is in the public domain / used under a CC0 license

Scratch group projects – 2017

Scratch

Scratch

It’s January, so it must be time for this year’s Scratch projects from my grade 10 students. We’re moving on to python, but I’ve posted their projects at http://scratch.lesbg.com Feel free to play them and rate them. This is a first attempt for students, so do please be gentle on the ratings.

One of my personal favorites is Gravity Clash, which is strangely addicting, given how simple it is.

If you want to check out previous years’ projects, they’re also available at the links at the top left. If you have any comments or suggestions for the site itself, please leave them below.

From NFS to LizardFS

If you’ve been following me for a while, you’ll know that we started our data servers out using NFS on ext4 mirrored over DRBD, hit some load problems, switched to btrfs, hit load problems again, tried a hacky workaround, ran into problems, dropped DRBD for glusterfs, had a major disaster, switched back to NFS on ext4 mirrored over DRBD, hit more load problems, and finally dropped DRBD for ZFS.

As of March 2016, our network looked something like this:

Old server layout

Old server layout

Our NFS over ZFS system worked great for three years, especially after we added SSD cache and log devices to our ZFS pools, but we were starting to overload our ZFS servers and I realized that we didn’t really have any way of scaling up.

This pushed me to investigate distributed filesystems yet again. As I mentioned here, distributed filesystems have been a holy grail for me, but I never found one that would work for us. Our problem is that our home directories (including config directories) are stored on our data servers, and there might be over one hundred users logged in simultaneously. Linux desktops tend to do a lot of small reads and writes to the config directories, and any latency bottlenecks tend to cascade. This leads to an unresponsive network, which then leads to students acting out the Old Testament practice of stoning the computer. GlusterFS was too slow (and almost lost all our data), CephFS still seems too experimental (especially for the features I want), and there didn’t seem to be any other reasonable alternatives… until I looked at LizardFS.

LizardFS (a completely open source fork of MooseFS) is a distributed filesystem that has one fascinating twist: All the metadata is stored in RAM. It gets written out to the hard drive regularly, but all of the metadata must fit into the RAM. The main result is that metadata lookups are rocket-fast. Add to that the ability to direct different paths (say, perhaps, config directories) to different storage types (say, perhaps, SSDs), and you have a filesystem that is scalable and fast.

LizardFS does have its drawbacks. You can run hot backups of your metadata servers, but only one will ever be the active master at any one time. If it goes down, you have to manually switch one of the replicas into master mode. LizardFS also has a very complicated upgrade procedure. First the metadata replicas must be upgraded, then the master and finally the clients. And finally, there are some corner cases where replication is not as robust as I would like it to be, but they seem to be well understood and really only seem to affect very new blocks.

So, given the potential benefits and drawbacks, we decided to run some tests. The results were instant… and impressive. A single user’s login time on a server with no load… doubled. Instead of five seconds, it took ten for them to log in. Not good. But when a whole class logged in simultaneously, it took only 15 seconds for them to all log in, down from three to five minutes. We decided that a massive speed gain in the multiple user scenario was well worth the speed sacrifice in the single-user scenario.

Another bonus is that we’ve gone from two separate data servers with two completely different filesystems (only one which ever had high load) to five data servers sharing the load while serving out one massive filesystem, giving us a system that now looks like this:

New server setup

New server layout

So, six months on, LizardFS has served us well, and will hopefully continue to serve us for the next (few? many?) years. The main downside is that Fedora doesn’t have LizardFS in its repositories, but I’m thinking about cleaning up my spec and putting in a review request.

Updated to add graphics of old and new server layouts, info about Fedora packaging status, LizardFS bug links, and remove some grammatical errors

Updated 12 April 2017 I’ve just packaged up LizardFS to follow Fedora’s guidelines and the review request is here.

Flock 2016

Man and woman in driving horse-drawn buggy

Downtown Kraków

I have just returned from a vacation in beautiful Kraków, where, entirely coincidentally, there just happened to be a Fedora conference! My family and I enjoyed the amazing sights around Kraków (if you haven’t visited the salt mine, you really should), but my personal highlight was getting to attend my first Flock, where I got to meet people face to face who I’d only previously talked with via IRC and email.

I got to chance to speak about how we use Fedora in the classroom in our school (slides here). There were some excellent questions from the audience at the end, and I realized (again!) that my biggest problem is creating decent documentation about what we’re doing so others can follow.

One of my goals over the next year is to make sure that our work is easily reproducible in other schools, both from the sysadmin side and from the educational side.

My biggest take-away from the conference is that Fedora is moving into some very interesting times as it starts to expand from rpms being the only system delivery mechanism. I’m very interested in ostree with its concept of a read-only system partition and the work they’re doing on layered trees so you can have multiple system images branching off of one base image.

I’d really like to thank the event organizers for all the work they did putting Flock together, the design team for the beautiful t-shirts, and the Fedora community for just being great. And, while I’m at it, I’d like to extend personal thanks to Rafał Luzynski and his wife, Maja, for their hospitality.

Talk – Using Fedora in the classroom

Spreadsheet assignment

Spreadsheet assignment

So I’m sitting here in Kraków, doing some last-minute preparation for my talk (Fedora in the Classroom) at the upcoming Flock conference next week.

I’ll be looking at why we use Fedora in our school, what tools we use to setup and maintain our workstations, and the actual subjects that we teach our students, complete with actual projects1 that our students have done.

If you’re a teacher looking for ways to use open source software in the classroom, an administrator looking for a computer curriculum that emphasizes creativity and comprehension over memorization and rote learning, or you’re just interested in seeing how Fedora is effectively used in a school environment, please do come check it out.

 
[1] Projects have been anonymized to protect student privacy