Posted by: jdieter | October 22, 2014

Using FreeIPA as a backend for DHCP


Yeah, this…

Disclaimer: This is not an official guide and in no way represents best practices for FreeIPA. It is ugly and involves the digital equivalent of bashing on screws with a hammer. Having said that, when nobody has invented the right screwdriver yet, sometimes you just have to hammer away.

First, some history. We’ve been running separate DHCP, DNS and LDAP servers since we switched from static IP addresses and a Windows NT domain somewhere around ten years ago. The DHCP server was loosely connected with the DNS server, and I had written this beautifully complex (read: messily unreadable) script that would allow you to quickly add a system to both DHCP and DNS. A few months ago, we migrated all of our users over to FreeIPA, and I started the process of migrating our DNS database over. Unfortunately, this meant that our DHCP fixed addresses were being configured separately from our DNS entries.

Last week I investigated what it would take to integrate our DHCP leases into FreeIPA. First I checked on the web to see if something like this had already been written, but the closest thing I could find was a link to a design page for a feature that’s due to appear in FreeIPA 4.x.

So here’s my (admittedly hacky) contribution:

  1. sync_dhcp – A bash script (put in /srv, chmod +x)that constantly checks whether the DNS zone’s serial number has changed, and, if it has, runs…
  2. – A python script (put in /srv, chmod +x) that regenerates a list of fixed-addresses in /etc/dhcp/hosts.conf
  3. dhcpd.conf – A sample dhcpd.conf (put in /etc/dhcp) that uses the list generated by
  4. sync-dhcp.service – A systemd service (put in /etc/systemd/system) to run sync_dhcp on bootup
  5. make_dns – A script (chmod +x) that allows the sysadmin to easily add new dns entries with a mac address

sync_dhcp does need to know your domain so it knows which DNS zone serial to check, but other than that, the first four files should work with little or no modification. You will need to create a dnsserver user in FreeIPA, give the user read access to DNS entries, and put its password in /etc/dhcp/dnspasswd (readable only by root).

make_dns makes a number of assumptions that are true of our network, but may not be true of yours. It first assumes that you’re using a network (yes, I know that’s not right; it’s long story) and that 10.10.9.x and 10.10.10.x IPs are for unrecognized systems. It also requires that you’ve installed freeipa-admintools and run kinit for a user with permissions to change DNS entries, as it’s just basically a fancy wrapper around the IPA cli tools.

Bent Screw Hole Backyard Metal Macros by Steven Depolo used under a CC BY 2.0 license

Posted by: jdieter | July 1, 2014

On Vacation

Longview, Washington

The rain in Washington

On Thursday, my family and I departed beautiful Lebanon and started the long trek (at least as far as sitting in an airplane can be considered trekking) back home to Washington State. We were greeted with some rain when we arrived, which was definitely proof that we were home.

We’ll be here until the beginning of September, and then it’s back to sunny Beirut. I’m looking forward to the kids getting to celebrate the 4th of July for the first time.

I’m also hoping to get some time to look into making applydeltarpm more efficient. If you’ve been following the conversation on the fedora-devel list, you’ll have noticed that, oddly enough, some people don’t like deltarpms, and the reasons given are definitely valid.

At the moment, recreating an rpm from a deltarpm includes recompressing it so that signatures match, and that recompression is *very* expensive in terms of CPU time. If you’re on a slow computer with decent storage, it might make more sense to rebuild uncompressed rpms, but if we did this, then signatures would no longer match. I’d like to see if there’s some way that we can reasonably store the signature of the uncompressed payload as well as the compressed payload in the rpm. Ideally, this will be done in such a way so as to require minimal (if any) changes to the buildsystem.

If I can manage a proof-of-concept that works without too much trouble for the infrastructure guys, then we might just be able to pull off much faster deltarpm rebuilds.

Posted by: jdieter | May 31, 2014

Canon copier/printer on Fedora

Canon Copy Machine


<tl;dr>There is a decent cups print driver for Canon copiers if you don’t mind using proprietary software and making some manual changes</tl;dr>

Recently, our school got a couple of Canon copy machines that can be configured as network printers, but up until a month ago we only used them as copy machines. Last month, I started the process of getting them configured to print using CUPS, and, in the process, learned a bit about the printers and a lot about how CUPS works.

The first problem I ran into is that Canon’s printer drivers aren’t open source, which led to some crazy problems finding the correct drivers. It turns out that Canon produces two cups print drivers, the first which prints using Canon’s proprietary UFR-II, and the second which prints using PCL or PXL. Both drivers are a pain to find, but once found, install in a halfway-reasonable way.

I was interested to find that the UFR-II driver left some odd shading any time I printed a graphic. The cups test page had a weird gradient in the middle where I’ve never seen a gradient before, and PDFs would print with the same strange gradient. The PCL/PXL driver also had the gradient, but, after mixing a few options (Image Refinement – On, Line Refinement – On, Halftones – High HighResolution), it almost completely disappeared.

The other nice thing about the PCL/PXL driver is that it’s actually mostly using the built-in (open source) tools already available in cups, and the only proprietary parts (at least as far as I can see) are the PPD itself and a small program that adds the extra print options (like double-sided printing, stapling, etc) to the PCL print job. Given all that, I figured there wasn’t much point in sticking with the UFR-II driver, and started working with the PCL/PXL driver.

However, on using the driver, I ran into some other strange problems. The first was that the cups page log didn’t actually show any information on some of the print jobs. After poking at the PPD, I discovered that if an incoming job is PDF, the print driver can’t count the pages, while if it’s PostScript the driver can. That was an easy fix. To force cups to convert incoming PDF jobs into PostScript before passing them to the driver, in the PPD delete the following lines:
*cupsFilter: "application/vnd.cups-pdf 0 foomatic-rip"
*cupsFilter: "application/ 25 foomatic-rip"

The second problem was a bit more subtle. Let’s imagine that I want to print a four-page test… 30 times, because I have 30 students in my class. I go to the print dialog, select the staple option, ask for 30 copies, and send it to print. Out of the copier come 120 pages… and one staple. The Canon driver will only staple it once because it’s one job. Because, obviously, if you’re printing 30 copies of the same job, you must want them to be stapled together.

Fortunately, the Canon driver does support an extra “Repeat job” count that you can use in place of the copy count. If you set the “Repeat job” count to 30 and leave the copy count at 1, it will print 30 four-page tests, with each test stapled separately. Unfortunately, this feature is in the advanced settings, while the copy count is sitting right there in the print dialog.

So I wrote a wrapper script for the Canon driver that automatically sets the “Repeat job” count to the copy count, and then sets the copy count to 1. Now the teachers can turn on stapling and set the copy count to whatever they want, and it will print as expected. You do have to change the *FoomaticRIPCommandLine line to say:
*FoomaticRIPCommandLine: "sicgsfilter-autonumpages &user; &quot;&title;&quot; &quot;%A&quot; &quot;%B&quot; &quot;%C&quot; &quot;%D&quot; &quot;%E&quot; &quot;%F&quot; &quot;%G&quot; &quot;%H&quot; &quot;%I&quot;"

I also went to the trouble of stripping out a bunch of unused options from the PPD, to make sure that they don’t appear when the teachers are going through the print options.

So now we have Canon copiers that are functioning great as printers, and our teachers love it!

Posted by: jdieter | April 27, 2014

Cillian James Dieter

Cillian James Dieter

Cillian James Dieter

At 7:45AM on April 14, 2014, Cillian (KILL-ee-an) James Dieter was born. He was 9 lbs, 4 oz (4.2 kg).

This was a very quick delivery, and I was still able to be at a class trip that started two hours after he was born. Easter break started two days after he was born, and Naomi’s parents are here now to spend some time with us all, so life has been pretty relaxed for the last week and a half.

Saoirse, Ailíse and Eoin are very excited that they have a little brother, but I think it will take Eoin a little while to grasp that he’s no longer the baby in the family.

The Dieter Family

Saoirse, Ailíse, Eoin, Jonathan, Naomi and Cillian

Posted by: jdieter | March 31, 2014

Locks in the classroom – 2014

For the second consecutive year, our grade nine students have been doing 3D modeling using Blender. A couple of weeks ago, our students finished up their first assignments, and I gave the top modelers the option of showing off their work. So, without further ado, here are the top three models in each of the three grade nine classes.

Lock by Ali Ab

Lock by Ali Ab – CC BY-SA 4.0Source

Lock by Abi Baadarani

Lock by Abi Baadarani – CC BY 4.0Source

Lock by J Mona

Lock by J Mona – CC BY-SA 4.0Source

Lock by Wael

Lock by Wael – CC BY-SA 4.0Source

Lock by Majd

Lock by Majd – CC BY-SA 4.0Source

Lock by Abo Ror

Lock by Abo Ror – CC BY-SA 4.0Source

Lock by Anonymous

Lock by Anonymous – CC BY 4.0Source

Lock by CN

Lock by CN – CC BY-SA 4.0Source

Lock by KR

Lock by KR – CC BY-SA 4.0Source

Posted by: jdieter | January 28, 2014

DevConf 2014

Brno cityscape


Thanks to some help from the school, Tro Chakerian (one of my assistants) and I will be going to Brno, Czech Republic for DevConf 2014. I’m really looking forward to getting to meet some other Fedora developers, and I’m hoping to learn a few things that will advance our school’s system over the next year or so.

I’m particularly looking forward to Why use a SAT solver for package management?, DNF API and Future of Fedora Big Picture. The last one is probably the most important for us as a school as we plan to continue to use Fedora on the desktop for the foreseeable future.

View of Brno from Spilberk Castle by Norbert Aepli used under a CC BY 2.5 license

Posted by: jdieter | December 2, 2013

Setting up a multiseat system

Panorama view of our multiseat Computer Center

Multiseat Computer Center

On Saturday, I described the new multiseat systems that we’re using at the school here. A number of people asked for some more details, so here they are.

First, the hardware for a multiseat system (and the price at time of order from our local supplier):

  • 1 x Intel G2020 – 2.90 GHz – $65
  • 1 x Kingston DDR3-1600 8G – $65
  • 1 x MSI Z77A-G45 motherboard – $155
    1 x Asus P8Z77-V LK motherboard – $160
  • 1 x Kingston SSDNow V300 60GB – $70
  • 3 x Sapphire Radeon HD6450 – $50
  • 1 x Generic case – $20
  • 4 x 4 Port USB hub – $5
  • Tax – 10%

The final price is somewhere between $600 and $610, depending on the motherboard.

Once you have the hardware built, make sure the onboard video is enabled in the BIOS and is set to be the primary display. Plug the USB hubs into the computer. Make sure you don’t swap ports after they’ve been plugged in. Then, install the standard Fedora 19 GNOME desktop and install the latest version of the lesbg-multiseat package from the school’s repositories. Enable the multiseat service (systemctl enable prepare-multiseat).

Make sure GDM is installed and that you’re using it as your display manager. You can use any desktop environment you’d like but you must use GDM (or LightDM with some patches) as other display managers don’t recognize systemd’s seat management. Reboot the computer.

When the computer comes up, there should be a login screen on each monitor. Each USB hub should automatically match a monitor, but you may have to swap ports so the hubs match the right monitor. lesbg-multiseat will always try to match the USB hubs to the video cards in order, so the first usb port will match the first video card, and so on.

Congratulations, you now have a multiseat system. Note that the configuration is designed to be minimal. We use the same OS image for single-seat or multiseat systems.

Posted by: jdieter | November 30, 2013

Multiseat in Fedora 19

This year in our main computer room, we switched from single-seat systems to multiseat systems. Our old single-seat systems cost us roughly $300 a system, and we would generally buy 20 a year. The goal with our multiseat systems was to see if we could do better than $300/seat. I also had a number of requirements, some of which would raise the cost, while others couldn’t be met the last time I looked into multiseat systems.

My first requirement was 3D acceleration on all seats. I know someone’s been working on separating OpenGL processing from the display server, which would theoretically allow us to use Plugable devices, but until that’s done, we need a separate video card for each seat. We also need motherboards that can support more than one PCIE video card (as well as preferably supporting the built-in GPU). This is the main extra expense for our multiseat systems.

My second requirement was plug-and-play USB. The last time I looked into multiseat, that wasn’t supported under Linux; USB devices would only be detected if they were plugged in when the X server started. But, thanks to some relatively new code in systemd which is now controlling logins using logind, USB ports can be directed to specific seats, with the devices plugged into them appearing in the correct seat when they’re plugged in.

In June, we bought a test system that came to just under $600. To our normal order we added a gaming motherboard, three of the cheapest PCIE AMD Radeon 5xxx/6xxx series cards we could find, extra RAM, and four USB hubs. The idea with the USB hubs was to place one next to each monitor and create our own wannabe-Plugable devices. I then wrote a small program that would deterministically assign each USB hub to a different monitor on bootup. An extra bonus to this program is that we can daisy chain the USB hubs. Once the program was working, I let the students play with the test system… and it worked!

So, during the summer, we bought ten more systems and put them in our main computer room. At four seats per system, we are saving 50%, so we were able to replace all forty computers in the main room in one year (and add four more seats as a bonus).

The main annoyance we’re still dealing with is that the USB hubs we got aren’t that great, and we’ve had a few fail on us. But they’re easy (and cheap) to replace. I also had to make some changes to X, like re-enabling Ctrl+Alt+Backspace as a solution for a stuck seat, which is better than rebooting the whole computer. And we do have the occasional hang where all four seats stop working, which I think is tied to the number of open files, but I haven’t tracked it down yet.

I’ve been very happy with our multiseat systems and would like to extend a huge thank you to the systemd developers for their work on logind.

Edit: More details are available in this post.

Posted by: jdieter | September 25, 2013

How do you rank a sysadmin?

Sysadmin at work

Sysadmin at work

When I heard about the 4th Linux Showdown, sponsored by TrueAbility, I was pretty excited. I’m a pretty competitive guy, so the idea of competing in a sysadmin’s challenge sounded like fun.

In the Linux Showdown, you get 30 minutes to complete a certain number of sysadmin tasks. Some of the tasks are pretty simple, while some of the others become more difficult. I entered the first day and managed to get 9th place with a score of 100% and a time of just under 17 minutes.

The second day I ran into trouble. One of the tasks was to reset the mysql root password, and, though I followed the directions here, twice, I was never able to log into mysql as root. The commands seemed to be running correctly, but I was locked out.

In my day-job as the system administrator for a school, I would keep bashing away at the problem until I figured out what I was doing wrong. In the competition, I ran out of time after fifteen minutes of debugging and ended up with a lousy 40%. Ouch!

I was frustrated, but figured the third day’s competition should fit a bit better. The hint said that it was a scripting competition, and my python foo is pretty decent. Sure enough, day three involved finding files with modification times between two dates, adding them to a database, and then tarring them up.

I came up with a python script that found the necessary files and added them to the database. Except my clever ‘INSERT’ statement didn’t actually work. If I manually copied and pasted it into mysql, it worked perfectly, but it didn’t run from the script. Grrr. I spent ten minutes debugging… and my time was up!

Well, that sucked. This time I got an impressive 20%. Double ouch!

After finishing the test, I went to bed and spent fifteen minutes ranting to my poor wife. The next day, after cooling off, I decided I was done. The hint for the last competition said that it had something to do with security, and I wouldn’t call myself an expert on that. If I’m getting 20% in the areas that I’m relatively good at, then what should I expect in areas that I’m less comfortable with.

Then it hit me. If I’m not comfortable with it, why not just do it for fun? If I know I’m probably going to get a zero, who cares? I checked the leaderboard, and the highest score at the time was 67%, so my zero wouldn’t be so bad. I went ahead and started the last competition.

Step one, secure the mail server. We don’t run our own mail servers here at the school and I know nothing about postfix, so I spent ten minutes or so Googling for some kind of solution, typed in what I thought was a partial fix, and then decided to give up.

Step two, secure a page on the webserver. This is something I have to do quite often, so I was able to get it done in five minutes or so.

Finally, step three, secure an FTP server. Who still uses FTP? We don’t! I wasn’t even sure what the ftp daemon’s name was, so I ran a ‘ps aux | grep ftp’. This was the only reason that I noticed that the ftp daemon wasn’t using the config file in /etc, but rather some config file in someone’s home directory. I did what I thought would secure the ftp server in both config files, and saw that I had a little over two minutes left.

Ok, I could have spent some more time on postfix, but I knew nothing about it, so I decided that I was finished. Worst case, I’d get 33% for the webserver (which was the only fix I’d actually tested). Best case, 67% for the ftp server, which I was pretty sure I’d fixed. If so, I might actually get in the top twenty. So, I logged in to the leaderboard, checked my ranking… First!??!? With 100%? What?

Apparently the random lines from Google that I put into my postfix config had secured it. Pure luck. As I followed the leaderboard for the rest of the day, it became obvious that many people with a lot of experience with apache, postfix and ftp were whipping right through the contest, missing the ftp config file in the home directory, and getting 67%, while I kept sitting on top with the lone 100%. I felt like such a fraud.

Finally, in the last hour before the contest ending, someone else found the solution five minutes faster than I did and got first place. Praise God! I still felt like a fraud, but at least first place was going to someone who knew what they were doing.

So, in four days of competitions, I got the highest score in the areas I was weakest in and the lowest score in the areas I was strongest in. That seems to indicate either that I don’t know what my strengths and weaknesses are, or that the competition needs some tweaking. Well, I think I’m at least reasonably aware of my strengths and weaknesses, and I’m very aware of how much of a role chance played in all four days of competition. So how can this competition be tweaked?

The strengths of the competition are pretty obvious. The whole point of TrueAbility is to winnow out people who talk the talk, but can’t walk the walk. When you get a résumé, you don’t know whether the applicant can actually do all the things they claim to be able to do, so, with TrueAbility, you give someone a VM and a list of tasks, and see whether or not they can do them. TrueAbility doesn’t care how they do the tasks, they just check that the tasks are completed. Brilliant!

The biggest weakness in the competition is the time limit. A vast majority of the problems we face as sysadmins need to be fixed quickly, but rarely does a complex problem need to be solved within 30 minutes. This time limit in the competition introduces a bias against those who work methodically. While hiring fast workers is always nice, basing hiring decisions based on how fast someone can code rather than how well they code is not wise.

In addition, the marking (especially for the last few days) was extremely coarse, so ranking was heavily dependent on how quickly you finished. This was especially noticeable in the first day, where the only difference between 1st place and 28th place was whether you took 10 minutes to finish the job or 30 minutes. As was obvious in the last day’s competition, this emphasis on time caused people to rush so much that they made mistakes. Time makes a lousy basis for ranking.

So what’s the solution? I see two complementary things that could be done to improve the competition. The first is to break down the grading even more, and assign different values to the different tasks. I’d even add in some standard tasks (with a total score of a maximum of 20%) along the lines of “Make sure that you close any ports not needed for your task”, “Disallow password logins over ssh and set up the server to trust your ssh key”, and “Replace your Ubuntu install with the real sysadmin’s OS: Fedora”. Ok, I’m half joking on that last one, but you get the idea. The key thing is that it should be almost impossible to get 100%, but a mediocre sysadmin should be able to hit 70% with only minor difficulty, and a talented sysadmin shouldn’t have much trouble reaching 90%.

The other thing that would help would be a removal of the hard deadline. Instead, allow candidates to continue working beyond the time limit, with a deduction of 1-2% for every minute. This introduces a cost to breaking the deadline without causing the candidate to completely fail because they needed ten more minutes.

With these two adjustments, time should become secondary to doing the job right. If I spend 10 minutes getting 90%, I’ll still get a lower score than someone who takes their time to do it right in 30 minutes. And, if I spend 40 minutes reaching 90%, I’ll only lose 20% for going over and end with a score of 70%, rather than sitting at zero because I just couldn’t finish my script within the deadline.

TrueAbility, thank you for the time and effort you’ve put into developing the problems for this competition, and thank you for the creative idea of a sysadmin’s competition in the first place.

And I really want to congratulate those who were able to consistently get high scores under the tough time limits.

Now I’m off to get some sleep before our first day of school.

Messy wires credit – Cisco Spaghetti by CHRISTOPHER MACSURAK. Used under the CC-BY 2.0 license.

Posted by: jdieter | June 30, 2013

Running cars through a blender

For their final Blender assignment, my grade nine students were asked to make an object (in most cases a car) go around some kind of track that followed the contour of the ground. These are some of the more creative projects that I received. Please note that in some cases the source doesn’t perfectly match the video. For some there were extenuating circumstances, and the others had it reflected in their scores.

I like the solid feel that this scene has, despite the glitches in the terrain texture mapping. I did keep expecting the truck to transform.
Car on track by Khodr – CC BY-SA 3.0Source

I’m not completely sure, but I think I would really enjoy driving this car.
Car on track by Shamas – CC BY-SA 3.0Source

I’m not sure if the camera angle really does it justice, but this scene has a really nice switchback. The car, the road and the terrain are all nicely modelled and textured, a rare combination.
Car on track by Fayez – CC BY 3.0Source

This was the only project where the car had headlights that actually project light. Very nice.
Car on track by C.H.W. – CC BY-SA 3.0Source

While not technically brilliant, the creativity and sheer strangeness of this scene made it impossible to exclude.
Car on track by Lynn – CC BY 3.0Source

I love the creative use of white balls for clouds and the covered wagon was icing on the cake.
Car on track by Su A – CC BY 3.0Source

The car bouncing around the track is pure vanilla, but the end is pretty impressive. It brought to mind the end of a movie that I saw as a kid.
Car on track by Oliver – CC BY 3.0Source

Older Posts »



Get every new post delivered to your Inbox.