AIME on Technology of Controls for Accelerators and Detectors

Last week, Javier Muñoz and myself attended the conference HEPTech Academia – Industry Matching Event on Technology of Controls for Accelerators and Detectors, which was hosted at DEMOKRITOS, Athens.

The target of this event was to have Academia and Industry together in the same place to share ideas and potential applications around Control Systems. Igalia was invited to participate as member of the Industry side to talk about our collaboration with CERN on Linux device drivers, KiCAD and HW virtualization areas.

Open Hardware slideThe conference’s agenda covered very exciting control system’s technologies used by High Energy Physics facilities. For example, CERN staff explained ATLAS and CMS detectors control systems, what they are planning to do for next years and how the next-gen accelerator, CLIC, is being designed.

However, there were also very good talks from other accelerators. One of the talks explained how the proton therapy cures eye cancer with a huge percentage of success. I recommend you to check out all the slides from the agenda because you will learn how the accelerator’s control systems are made and how the companies are collaborating on this area. Very interesting for both scientists and engineers!

Our talk was part of the “Open Hardware vs Conventional Development approach” track, where concepts like Open Hardware or projects like WhiteRabbit were explained to the audience. We were explaining our work developing the FMC TDC driver and how QEMU helped us a lot to debug the driver and improve its robustness by using SW techniques such as continuous integration and testing. Our slides are publicly available if you want to read them.

In the same track, we represented Igalia in the round table by giving our opinion about Open Hardware and sharing some advices based on our experience in the Free Software world that can be applied on Open Hardware based companies.

I would like to finish this post by saying that the organizers did a great job in taking care of the success of the event. Everything was well explained and organized, even when this was the first event of this kind organized around control systems for accelerators and detectors.

Introduction to Linux Graphics drivers: DRM

Linux support for graphics cards are very important for desktop and mobile users: they want to run games, composite their applications and have a nice and modern user experience.

AGP Video Card photo

So it’s usual that all the eyes are on this area when you want to optimize your embedded device’s user experience… but how the graphics drivers communicate with user-space applications?

Víctor Jáquez, from Igalia, wrote a very nice introduction to this topic. If you are interest on Linux graphics stack in general, you have this post wrote by Jasper St. Pierre.

Direct Rendering Infraestructure schema

Direct Rendering Infraestructure schema (by Víctor Jáquez)

The Direct Rendering Infrastructure (DRI) in Linux is like is shown at the above picture. Using Xorg as an X server example, we see that it is in charge of X11 drawing commands along with Mesa or Gallium3D as the OpenGL software implementation for rendering three-dimensional graphics.

How they communicate with the actual HW? One possibility is using libdrm as a backend to talk with the Direct Rendering Manager at Linux kernel. DRM is divided in two in-kernel drivers: a generic drm driver and another which has specific support for the video hardware.

There is a Xorg driver running in user-space who receives the data to be passed to the HW. Using Nouveau as an example, the xf86-video-nouveau is the DDX (Device Dependent X) component inside of X stack.

It communicates with the in-kernel driver using ioctl. Some of them are generic for all the drivers and they are defined in the generic drm driver. Inside of drivers/gpu/drm/drm_drv.c file you have the relationship between the ioctl number, its corresponding function and its capabilities.

However, the specific driver for the video hardware can define its own ioctls in a similar way. For example, here you have the corresponding ones for the Nouveau driver (drivers/gpu/drm/nouveau/nouveau_drm.c file).

As you can see, most of these ioctls can be grouped by functionality:

  • Get information from the drm generic driver: stats, version number, capabilities, etc.
  • Get/set parameters: gamma values, planes, color pattern, etc.
  • Buffer’s management: ask for new buffer, destroy, push buffer, etc.
  • Memory management: GEM, setup MMIO, etc.
  • Framebuffer management.
  • In case of Nouveau: channel allocation (for context switching).

Basically, the xorg user-space driver may prepare a buffer of commands to be executed on the GPU and pass it through a ioctl call. Sometimes, it just want to use the framebuffer to draw on the screen directly. Others, it uses KMS capabilities to change parameters of the screen: video mode, resolution, gamma correction, etc.

There are more things that DRM can do inside of the kernel. It can setup the color pattern used to draw pixels on the screen, it can select which encoder is going to be used and on which connector (LVDS, D-Sub, HDMI, etc), pick EDID information from the monitor, manage vblank IRQ…

At Igalia we have a long background working in Linux multimedia stack (GStreamer, WebGL, OpenCL, drivers, etc). If you need help to develop, optimize or just you want advice, please don’t hesitate to contact us.

A great year has passed

Exactly one year ago, it was my first day as Igalia employee. I had arrived from Geneva some days before, I had rented an apartment here in Coruña the day before. I was very nervous and willing to start.

Today, I can say I am more than happy to be working for this company: I started contributing drivers upstream, I became one of the official maintainers of them, I went to LinuxCon Europe, gave talks at OSHWCon Madrid 2012 and 7th White Rabbit workshop, I learned a lot of things… but the most important thing is that I made real friends here.

Regarding my personal life, everything has been getting better last year. Living so close to my parent’s home gives me the possibility to see my relatives and friends more often. I just need to drive 3 hours and I am there!

I don’t know what is going to happen next year, but I am sure it’s going to be awesome :-)

7th White Rabbit workshop

This week I traveled to Madrid where the 7th White Rabbit workshop was held. It was a two day event where the White Rabbit community presented all the latest development efforts, installations, the White Rabbit standardization process and much more. It was amazing to find a broad range of experiments using or considering to use this technology, the applications, the new PCB designs to come, etc.

Apart from Research institutions like CERN, GSI, DESY, NIKHEF among others, there were a lot of HW companies that design, manufacture and support all the required boards needed by the experiments. They were showing their products like the White Rabbit switch and Sevensols‘s White Rabbit starter kit.

I went there to present, together with Javier Muñoz, the latest work Igalia‘s OS team did. We spoke about testing and how virtual HW helps on this task. We showed our experience on the FMC TDC driver development, the FMC TDC’s virtual model and its integration in a testing suite with continuous integration.

Also, some software companies were there, like Gnudd represented by Alessandro Rubini, Integrasys and others. They were describing their work on the software part: FMC bus drivers, sdbfs filesystem, White Rabbit switch software, etc.

I found this event very interesting to share ideas, talk about technologies and plan new designs to improve all the White Rabbit stack.

I wouldn’t like to finish this post without saying anything about the organization. It was a perfect event: CDTI made a great work hosting the event at their offices and BE/CO/HT section at CERN organizing the rest.

FMC TDC driver

During last few months we have been involved in the development of FMC TDC software support for Linux, i.e., writing a driver for it along with an user-space library and a bunch of test programs.

But first of all, let me show you what is a Time-to-Digital converter (TDC)

In electronic instrumentation and signal processing, a time to digital converter (abbreviated TDC) is a device for recognizing events and providing a digital representation of the time they occurred. For example, a TDC might output the time of arrival for each incoming pulse. Some applications wish to measure the time interval between two events rather than some notion of an absolute time.

Source: Wikipedia

In summary, it measures the time of arrival of each incoming pulse and saves the timestamp into the memory, ready to be read later on by the user. This time, this is a mezzanine board plugged to a carrier board across the FMC bus, hence its name.

FMC TDC board image

The board was designed by CERN to fulfill their needs in the control system. They published the schematics and all the needed design files under CERN Open Hardware license. You can check them out from its project page at OHWR website. Remember that it is still in the prototyping phase.

The FMC TDC driver depends on ZIO framework and FMC bus driver to work. Also, as we were developing it using the SPEC board as a carrier board, the SPEC driver is needed to perform the I/O operations from/to the FMC TDC board.

There is still work to be done like adapting the code to last ZIO changes and improve the documentation, but it is quite close to what we can call “1.0″ version :-)

All the code is hosted in OHWR, under its own project. If you are interested on following its development, I recommend you to subscribe to the mailing list.

New GPG key

Hello lazy web,

For a number of reasons, I have set up a new OpenPGP key, and I will be transitioning away from my old one. I created a new 4096-bits key following Debian’s instructions.

At the time of writing GnuPG unfortunately defaults to a 1024 bit DSA key as the primary with SHA1 as the preferred hash. Due to weaknesses found with the SHA1 hashing algorithm Debian prefers to use keys that are at least 2048 bits and preferring SHA2.

keyring.debian.org

The old key will continue to be valid for some time, but i prefer all future correspondence to come to the new one. I would also like this new key to be re-integrated into the web of trust.

The old key was:

pub 1024D/F99DF5E2 2006-01-04
Key fingerprint = DC45 A767 9618 ACA8 FDD1 989A 5AB7 DBC9 F99D F5E2

And the new key is:

pub 4096R/F17DC343 2012-11-16
Key fingerprint = 40FF 9902 F697 5A47 EE29 7884 7FF4 BA32 F17D C343

To fetch my new key from a public key server, you can simply do:

gpg –keyserver pgp.mit.edu –recv-key F17DC343

If you already know my old key, you can now verify that the new key is signed by the old one:

gpg –check-sigs F17DC343 | grep F99DF5E2

And if you don’t have my old key then you can check the following link, and see the signatures done with my old key (F99DF5E2):

http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x7FF4BA32F17DC343

If you are satisfied that you have got the right key, and the UIDs match what you expect ( gpg –fingerprint F17DC343 ), then I would appreciate it if you would sign my key:

gpg –sign-key F17DC343

Lastly, if you could upload these signatures, I also would appreciate it.

gpg –keyserver pgp.mit.edu –send-key F17DC343

Please let me know if there is any trouble, and sorry for the inconvenience.

LinuxCon Europe 2012 experience

Last week, I have been in Barcelona attending to LinuxCon Europe 2012 and the Embedded Linux Conference Europe 2012. This was actually my second LinuxCon Europe (the first was in Prague, last year) but I was there with the same feelings that the first time!

It was very nice to find old friends there, chat with them or, even, drink some beers while talking about some funny anecdotes we shared together :-) Also, I enjoyed some more technical conversations we had related to real time issues, virtualization and industrial hardware.

However, the best part of these conferences is not only talk with friends but meet new ones. I had very nice talks with people from a broad range of countries, careers and interests. It’s there when you realize that you are not the only one suffering a certain problem or interested on a certain area. It definitively boosts my desire to change the world!

Thanks to Igalia for sponsoring my trip to LinuxCon Europe!

Industry Pack Linux support on Sourceforge

Some weeks ago, I have created a project in Sourceforge to host the Industry Pack Linux kernel support that keeps the source available in a public repository and provides a mailing list for public discussion.

This support started few months ago with the submission of three drivers into the Linux kernel. However, there was a lot of work to be done (improving the API, adding new features, fixing bugs, cleaning up the drivers…). Using the Sourceforge project, we are improving Industry Pack support and we plan to add drivers for new devices very soon.

I would like to thank Jens Traprogge for his contributions fixing some pending issues, my ex-colleagues at CERN who allow to test the code in their facilities and Igalia which allows me to devote some working hours to Industry Pack Linux support :-)

Going to OSHWCon Madrid 2012

A few weeks ago, I have submitted a talk at OSHWCon 2012 and it got accepted… Yay!

The talk will explain some concepts related to Industrial HW, the benefits of open development and collaborating upstream, the support of Industrial HW in the Linux kernel and how to test our drivers using virtualization software.

So, if you are a hardware engineer or your company is interested on open development or you want to see how the virtualization can boost the software support for your HW project… you cannot miss this talk!

 

Magic SysRq key

When you are developing in the Linux kernel you easily find yourself dealing with a kernel oops, kernel bugs or something alike. Sometimes you can recover from it and continue working, sometimes not. If you are developing drivers and testing them over the same machine… you are doing something wrong.

It is recommended to have a virtual machine or another computer to test your code there, if not you might have productivity problems (you might need to reboot, so it is a waste of time waiting to boot, log in and open the editor to fix the bug) or, even worse, you can lose part of your work because the computer hanged and you forgot to save the file!

In case of having a second machine, you would probably access to it using SSH or telnet. Sometimes, the machine’s location is far away from where you are, even in another continent. So you cannot go there, reboot the machine manually and come back in a reasonable time; or you need to bother people to do it for you.

You have a lot of ways to sort out this problem. You can use a remote power switch (for example, this one) but it is quite expensive. Another way is to use some hidden features in the Linux kernel, like the files present in /proc/sys/kernel that I will explain in another post.

However, the topic of this post is to learn how to use the SysRq keys. Now you will understand the presence of the SysRq key on your keyboard!

Definition

As Wikipedia says:

The magic SysRq key is a key combination understood by the Linux kernel, which allows the user to perform various low level commands regardless of the system’s state. It is often used to recover from freezes, or to reboot a computer without corrupting the filesystem.

This special key allows you to recover from a crash on your system and send some commans, like shutdown, reboot, sync data with your filesystem, etc.

Enable SysRq support

The support for SysRq commands should be enabled before use it. In the common GNU/Linux distributions, this support is usually enabled by default in the kernels they provided. If you are compiling your own kernel, be sure that you have the following in your .config file:

There is a sysctl to toggle this feature on and off. To enable it:

How to use it

On x86, you press the key combo ‘Alt’ + ‘PrintScrn/SysRq’ + key. For other architectures you can read the corresponding Linux Documentation file.

Under graphical environments (such as Gnome or KDE) ‘Alt’+'PrintScrn/SysRq’+key combination generally only leads to a screenshot being dumped. To avoid this, you should press ‘Ctrl’ before: ‘Ctrl’+'Alt’+'SysRq’+key.

Extracted from the Wikipedia page, here you have the available options:

Action QWERTY Dvorak AZERTY
Set the console log level, which controls the types of kernel messages that are output to the console 0 through 9 0 through 9 0 through 9
(without using shift)
Immediately reboot the system, without unmounting or syncing filesystems b x b
Reboot kexec and output a crashdump c j c
Display all currently held Locks d e d
Send the SIGTERM signal to all processes except init (PID 1) e . e
Call oom_kill, which kills a process to alleviate an OOM condition f u f
When using Kernel Mode Setting, provides emergency support for switching back to the kernel’s framebuffer console[4] If the in-kernel debugger ‘kdb’ is present, enter the debugger. g i g
Output a terse help document to the console
Any key which is not bound to a command should also perform this action
h d h
Send the SIGKILL signal to all processes except init i c i
Forcibly “Just thaw it” – filesystems frozen by the FIFREEZE ioctl. j h ?
Kill all processes on the current virtual console (Can be used to kill X and svgalib programs, see below)
This was originally designed to imitate a Secure Access Key
k t k
Shows a stack backtrace for all active CPUs. l n ?
Output current memory information to the console m m ,
Reset the nice level of all high-priority and real-time tasks n b n
Shut off the system o r o
Output the current registers and flags to the console p l p
Display all active high-resolution timers and clock sources. q a
Switch the keyboard from raw mode, the mode used by programs such as X11 and svgalib, to XLATE mode r p r
Sync all mounted filesystems s o s
Output a list of current tasks and their information to the console t y t
Remount all mounted filesystems in read-only mode u g u
Forcefully restores framebuffer console, except for ARM processors, where this key causes ETM buffer dump v k v
Display list of blocked (D state) tasks w , z
Used by xmon interface on PPC/PowerPC platforms. x q ?
Show global CPU Registers (SPARC-64 specific) y f ?
Dump the ftrace buffer z ; ?

 How to use SysRq remotely

However, the key combos explained before are just for a local machine with access to a keyboard plugged to it. How do we use these low-level commands in a machine that is a thousand km far away?

Easily, we setup a daemon to connect to it and it will execute this key combos for us. At this case, I will explain sysrqd because is the one I am using to debug one remote machine in Igalia.

Remember that sysrqd opens the 4094 port by default to allow the access via Telnet. You should have to provide a password (later I will explain how) but… the password won’t be encrypted, so be aware of the security-risks!

sysrqd setup

Firstly, you need to install it on the remote’s GNU/Linux distro. As I am using Debian:

After that, you need to setup the password and change the file permissions:

And then reboot or start the sysrqd daemon directly:

Access to sysrqd

Once you have enable sysrqd, you can access to your machine using telnet:

It will prompt waiting for your password, then write it (remember that it won’t be sent encrypted and it will appear on your screen too!), and finally press your desire commands as they are described in the aforementioned table.

You need to take into account that leaving the Magic SysRq enabled on a production machine can be dangerous, anyone with physical access to the machine can bring it down. It is even worse with sysrqd daemon running, anyone sniffing the network could pick the clear-text password and later access via telnet. So use it for testing and under your own risk!