Tag Archives: ati

Multiseat systems and the NVIDIA binary driver

Building mesa

Building mesa

Ever since our school switched to Fedora on the desktop, I’ve either used the onboard Intel graphics or AMD Radeon cards, since both are supported out of the box in Fedora. With our multiseat systems, we now need three external video cards on top of the onboard graphics on each system, so we’ve bought a large number of Radeon cards over the last few years.

Unfortunately, our local supplier has greatly reduced the number of AMD cards that they stock. In their latest price lists, they have a grand total of two Radeon cards in our price range, and one of them is almost seven years old!

This has led me to take a second look at NVIDIA cards, and I’m slowly coming back around to the concept of buying them and maybe even using their binary drivers. Our needs have changed since we first started using Linux, and NVIDIA’s binary driver does offer some unique benefits.

As we’ve started teaching 3D modeling using Blender, render time has become a real bottleneck for some of our students. We allow students to use the computers before and after school, but some of them don’t have much flexibility in their transportation and need to get their rendering done during the school breaks. Having two or three students all trying to render at the same time on a single multiseat system can lead to a sluggish system and very slow rendering. The easiest way to fix this is to do the rendering in the GPU, which Blender does support, but only using NVIDIA’s binary driver.

So about a month ago, I ordered a cheap NVIDIA card for testing purposes. I swapped it with an AMD card on one of our multiseat systems and powered it up. Fedora recognized the card using the open-source nouveau driver and everything just worked. Beautiful!

Then, a few hours later, I noticed the system had frozen. I rebooted it, and, after a few hours, it had frozen again. I moved the NVIDIA card into a different system, and, after a few hours, it froze while the original system just kept running.

Some research showed that the nouveau driver sometimes has issues with multiple video cards on the same system. There was some talk about extracting the binary driver’s firmware and using it in nouveau, but I decided to see if I could get the binary driver working without breaking our other Intel and AMD seats.

The first thing I did was upgrade the test system to Fedora 25 in hopes of taking advantage of the work done to make mesa and the NVIDIA binary driver coexist. I then installed the binary NVIDIA drivers from this repository (mainly because his version of blender already has the CUDA kernels compiled in). The NVIDIA seat came up just fine, but I quickly found that mesa in Fedora 25 isn’t built with libglvnd (a shim between either the mesa or NVIDIA OpenGL implementation, depending on which card you’re using and your applications) enabled, so all of the seats based on open drivers didn’t come up. But, even when it was enabled, I ran into this bug, so I ended up extending this patch so it would also work with Gallium drivers and applying it.

This took me several steps closer, but apparently the X11 GLX module is not part of libglvnd and NVIDIA sets the Files section in xorg.conf to use it’s own GLX module (which, oddly enough, doesn’t work with the open drivers). I finally worked around this via the ugly hack of creating two different xorg.conf.d directories and telling lightdm to use the NVIDIA one when loading the NVIDIA seat.

Voilà! We now have a multiseat system with one Intel built-in card using the mesa driver, two AMD cards using the mesa Gallium driver, and one NVIDIA card using the NVIDIA binary driver. And it only cost me eight hours and my sanity.

So what needs to happen to make this Just Work™? Either libglvnd needs to also include the X11 GLX module or we need a different shim to accomplish the same thing. And Fedora needs to build mesa with libglvnd enabled (but not until this bug is fixed!)

My mesa build is here and the source rpm is here. There is a manual “Provides: libGL.so.1()(64bit)” in there that isn’t technically correct, but I really didn’t want to recompile negativo17’s libglvnd to add it in and my mesa build requires that libglvnd implementation.

My xorg configs are here and my lightdm configuration is here. Please note that the xorg configs have my specific PCI paths; yours may differ.

And I do plan to write a script to automate the xorg and lightdm configs. I’ll update this post when I’ve done so.

Sidenote: As I was looking through my old posts to see if I had anything on NVIDIA, I came across a comment by Seth Vidal. He was an excellent example of what the Fedora community is all about, and I really miss him.

Update: Configuration has become much simpler. An updated post is here.

Setting up a multiseat system

Panorama view of our multiseat Computer Center

Multiseat Computer Center

On Saturday, I described the new multiseat systems that we’re using at the school here. A number of people asked for some more details, so here they are.

First, the hardware for a multiseat system (and the price at time of order from our local supplier):

  • 1 x Intel G2020 – 2.90 GHz – $65
  • 1 x Kingston DDR3-1600 8G – $65
  • 1 x MSI Z77A-G45 motherboard – $155
    OR
    1 x Asus P8Z77-V LK motherboard – $160
  • 1 x Kingston SSDNow V300 60GB – $70
  • 3 x Sapphire Radeon HD6450 – $50
  • 1 x Generic case – $20
  • 4 x 4 Port USB hub – $5
  • Tax – 10%

The final price is somewhere between $600 and $610, depending on the motherboard.

Once you have the hardware built, make sure the onboard video is enabled in the BIOS and is set to be the primary display. Plug the USB hubs into the computer. Make sure you don’t swap ports after they’ve been plugged in. Then, install the standard Fedora 19 GNOME desktop and install the latest version of the lesbg-multiseat package from the school’s repositories. Enable the multiseat service (systemctl enable prepare-multiseat).

Make sure GDM is installed and that you’re using it as your display manager. You can use any desktop environment you’d like but you must use GDM (or LightDM with some patches) as other display managers don’t recognize systemd’s seat management. Reboot the computer.

When the computer comes up, there should be a login screen on each monitor. Each USB hub should automatically match a monitor, but you may have to swap ports so the hubs match the right monitor. lesbg-multiseat will always try to match the USB hubs to the video cards in order, so the first usb port will match the first video card, and so on.

Congratulations, you now have a multiseat system. Note that the configuration is designed to be minimal. We use the same OS image for single-seat or multiseat systems.

Multiseat in Fedora 19

This year in our main computer room, we switched from single-seat systems to multiseat systems. Our old single-seat systems cost us roughly $300 a system, and we would generally buy 20 a year. The goal with our multiseat systems was to see if we could do better than $300/seat. I also had a number of requirements, some of which would raise the cost, while others couldn’t be met the last time I looked into multiseat systems.

My first requirement was 3D acceleration on all seats. I know someone’s been working on separating OpenGL processing from the display server, which would theoretically allow us to use Plugable devices, but until that’s done, we need a separate video card for each seat. We also need motherboards that can support more than one PCIE video card (as well as preferably supporting the built-in GPU). This is the main extra expense for our multiseat systems.

My second requirement was plug-and-play USB. The last time I looked into multiseat, that wasn’t supported under Linux; USB devices would only be detected if they were plugged in when the X server started. But, thanks to some relatively new code in systemd which is now controlling logins using logind, USB ports can be directed to specific seats, with the devices plugged into them appearing in the correct seat when they’re plugged in.

In June, we bought a test system that came to just under $600. To our normal order we added a gaming motherboard, three of the cheapest PCIE AMD Radeon 5xxx/6xxx series cards we could find, extra RAM, and four USB hubs. The idea with the USB hubs was to place one next to each monitor and create our own wannabe-Plugable devices. I then wrote a small program that would deterministically assign each USB hub to a different monitor on bootup. An extra bonus to this program is that we can daisy chain the USB hubs. Once the program was working, I let the students play with the test system… and it worked!

So, during the summer, we bought ten more systems and put them in our main computer room. At four seats per system, we are saving 50%, so we were able to replace all forty computers in the main room in one year (and add four more seats as a bonus).

The main annoyance we’re still dealing with is that the USB hubs we got aren’t that great, and we’ve had a few fail on us. But they’re easy (and cheap) to replace. I also had to make some changes to X, like re-enabling Ctrl+Alt+Backspace as a solution for a stuck seat, which is better than rebooting the whole computer. And we do have the occasional hang where all four seats stop working, which I think is tied to the number of open files, but I haven’t tracked it down yet.

I’ve been very happy with our multiseat systems and would like to extend a huge thank you to the systemd developers for their work on logind.

Edit: More details are available in this post.

Catalyst vs. Mesa (round 2)

Two Boxers

In February, I wrote about my frustration with AMD’s binary Catalyst drivers and my switch to the free Mesa drivers for my media/gaming system. During the summer, I updated to Fedora 13 and continued to enjoy the reliability of free drivers.

Then, a problem. Sometime in September, some of the rendering in XBMC started to be corrupted. Movies played fine and the picture slideshow continued to work correctly, but any controls rendered on top of the Movie or slideshow seemed to be missing the background texture and instead rendered as a very light grey. With white text rendered on top of it, it made the controls pretty unreadable.

Upgrading mesa didn’t work. Neither did upgrading the kernel or XBMC. And a full upgrade to Fedora 14 didn’t help either. Given the insanity of getting everything else up and running at the beginning of the school year, this was the point that I stopped. After all, our main use of the system is to watch movies or the pictures slideshow, and, though annoying, the bug wasn’t a show stopper.

With some of the free time we had for Christmas, I decided to try to tackle the bug. I figured the easiest way would be to go back to the Catalyst drivers and see if the rendering was still screwed up. Sure enough, the Catalyst drivers fixed the rendering. I then tried to open Nexuiz full-screen over XBMC. The display froze. One reboot later I tried again. And the display froze again.

After several hours of trying different kernel and xorg.conf options, I was ready to put the computer in the middle of the Damascus highway during rush-hour traffic.

Then I had an epiphany. In Fedora 15, the r600 driver will switch from the standard Mesa driver to the new Gallium3D driver. So I installed fedora-release-rawhide and then did a:
yum update mesa-* libdrm-* xorg-x11-* gdm-*

One reboot later and XBMC is rendered correctly in all of its glory, all of my games run correctly over it, and I still don’t have to worry about keeping Catalyst up to date.

Closed drivers: 0
Open drivers: 2

Note: Some may wonder why I updated gdm. For some reason, the old version interacts with X in such a way that X crashes and I’m left with the boot screen and can only ssh into the system. It seems to be somewhat related to this bug, and updating gdm fixes it.

F11 Catalyst vs. F12 mesa-drivers-experimental

Last May I bought an ATI Radeon HD4830 video card for my music/gaming system. I would have gone with nVidia, but my last laptop had an nVidia graphics driver and I got quite frustrated with trying to keep the binary drivers up-to-date on it (even using RPM Fusion, I would run into odd problems every now and then).

I wanted a card that would have open drivers. Now, I knew when I bought the 4830 that it wasn’t supported by the open radeon driver, but I also knew it would be ready soon, and I figured I could use the closed drivers until then.

And that’s when I found out that ATI’s closed-source binary Catalyst driver is so poorly maintained, it makes Windows 95 look up-to-date. It took several months for AMD/ATI to put out a Catalyst driver that would support Fedora 11, and they still haven’t put out a Catalyst driver that supports Fedora 12.

The driver also is extremely buggy. Any release after 9.8 wouldn’t work with my original motherboard (see this bug), and after upgrading to a new Intel motherboard and processor a couple of weeks ago, I had random crashes that ranged from once every couple of days to once every fifteen minutes.

If I could ssh into the computer (which was only about half of the time), I’d see some message about an “ASIC hang”. Googling it didn’t give much information. I originally thought it had something to do with the power supply, but even a brand name power supply didn’t fix the problem.

Yesterday, I finally had enough. I wiped the hard drive and did a clean install of Fedora 12. Yeah, Catalyst won’t work, but I’d been hearing good things about mesa-drivers-experimental, so I decided to give it a go.

So, first the down side to switching:

  • Nexuiz runs much slower. With the binary drivers, I was able to run the game at 1920×1080 ultimate quality at 50-60 fps. With the experimental driver, it’s down to 3-4 fps (though medium quality works great at 50-60 fps).
  • XBMC ProjectM visualizations are slow. Again, with the binary drivers, the ProjectM visualizations ran at full speed, while with the experimental drivers, they run at 5-6 fps.

But, the good news is:

  • I can play my 3D games just fine. Even though I can’t run them at full quality, I can still play my games.
  • XBMC movies run at full speed. Even with the cool effects that XBMC uses for its controls, movies work perfectly.
  • XBMC slideshows work fine, with panning and zooming. The slideshow work the way they are supposed to with no delays at all. In fact, I may be imagining it, but I think there was the occasional slight delay in catalyst that isn’t there now
  • THE SYSTEM DOESN’T HANG/CRASH. I haven’t had a single system hang since switching. And that is worth far more to me than the best Nexuiz framerate ever.

Using the open experimental drivers brings me back to the reason I bought ATI in the first place, and I can finally say that I’m glad I bought ATI. Binary drivers are a pain to keep up with, and I’m so glad I don’t have to deal with that any more. I’m hoping to see things running even better in Fedora 13.

Update: The story continues here.