Barebones Linux on Barreleye G1

Following up on the previous article, this post will be building a similar barebones Linux for the POWER8-based OpenPOWER Barreleye system.

For this round, I will be building the system on a freshly kicked x86_64 Ubuntu 16.04 VM to capture all dependencies needed to follow the same steps.

Cross Compiling Linux

To get things started I will pull in the various dependencies needed to build the system.

sudo apt install -y build-essential gcc-powerpc64le-linux-gnu libssl-dev bc

With these dependencies in place, I’ll create a new directory to work out of and pull down the kernel the same way as the last post. A minor difference is today the latest stable version is 4.13.1, so I’ll be rolling with that version.

mkdir barebones-ppc64le
cd barebones-ppc64le/
tar xf linux-4.13.1.tar.xz
cd linux-4.13.1/

Next I setup some environment variables to prepare for cross compilation.

export ARCH=powerpc
export CROSS_COMPILE=/usr/bin/powerpc64le-linux-gnu-

The next big step is to configure the kernel. For this platform, I have not yet had success using one of the default configs that comes with the kernel source tree. The good news is that the OpenPOWER firmware of this platform is Linux based and open source. The kernel configuration from the firmware build repo is a known good configuration for this exact system, so that gives me assurance it will work pretty well.

I’ll download the skiroot kernel config, run the kernels make oldconfig to get the kernel build happy with the configuration (piping yes '' to make to accept the default answers to its many questions), and build the kernel using the number of threads returned to me by nproc.

wget -O .config
yes '' | make oldconfig
make -j`nproc`
cp vmlinux ..
cd ..

Building initramfs and Testing

Building the initramfs is also nearly identical to the previous post, just using the cross compiler instead of the default. This is the program that will be my init process.

#include <stdio.h>
#include <unistd.h>
int main() {
  while (1) {
    printf("Hello from the land of Barreleye\n");

And the steps I used to compile and pack into my initramfs

powerpc64le-linux-gnu-gcc -o init -static test.c
echo init | cpio -o --format=newc > initramfs

With that all done, I will upload it to a webserver I have in our lab and boot up my Barreleye system to Petitboot, the firmware stacks primary interface. I’ll exit to a shell, wget the kernel and initramfs and kexec load and execute!

Since I’m connecting into the system from home to the BMC console the interface is a wee bit slow to respond to input, but it does the trick.

Testing my build on a Barreleye system

All looks good! Please feel free to comment or leave any questions you may have.


Building a Barebones Linux System

In this post, I will describe the process I’ve used to create a very minimal Linux build. It is minimal for the sake of learning, to get a clearer picture of how barebones a operating system could be put together. The build will consist of an operating system kernel and an initial ram disk

I will be using Ubuntu 16.04 on my laptop as my build box, and use QEMU VMs as my early test systems. If you’d like more information on building a linux kernel there is a decent tutorial on kernelnewbies as well as the kernel building section of the Gentoo handbook.

The Linux Kernel

It can be difficult to see the separation of what is Linux kernel and what is the Linux distribution. The Linux Kernel itself is there primarily to facilitate access to hardware and provide a safe haven for processes to live and breathe. The kernel’s user interface is its API. It provides a foundation for all other software to allow storing files, using networks, render graphics, handle keyboard input and so on. On top of the kernel your terminal, your login process, your window manager, your browser, etc. are usually provided by your distribution.

Obtaining and building the kernel is very simple, though configuring the kernel can be a little trickier depending on your needs. For the sake of this excersize the default configurations for an x86_64 system will do just fine.

First off, I’ll create a new directory to work out of and get started. As of today, the latest linux kernel at the top of is 4.13, I will download this latest tar file, extract it and jump into the extracted directory.

Downloading the kernel

Now that I have the kernel source files, I’ll use make defconfig to configure the defaults for a standard x86_64 machine and will do make -j4 to compile the kernel using all 4 threads my laptop has to offer.

Starting the kernel build

Even with the extra threads, it takes about 8 minutes to compile on my machine. It’s not the simplest kernel as there are many default features we could disable to trim down the kernel size and compile time but I’m not worried about that for this build. Once it’s complete I copy the bzImage file that was generated into my barebones parent directory.

Kernel build finishing

Init process and initramfs

Typically when a computer boots, the host firmware will look for a bootloader on a bootable device. The bootloader will load the kernel and pass control of the system to it. The kernel gets control, does it’s own startup shenanigans and looks for a program to run as the primary (init) userspace process (PID 1)

In most Linux distributions, this init process can be a SysV style init process, SystemD, Upstart or Upstart. In this case, it’s going to be a looping hello world program. I compile it with -static so that libraries are staticly linked

Compiling a simple hello world looping program

The next step is to build the initramfs, the initial file system that will go in ram for the kernel to interact with. To do this, I will pipe a newline separated list of files to pack to cpio. I only have one file so I’ll just echo that to it. cpio will output the archive and I’ll store that to a file.

Building an uncompressed initramfs with a single file

Testing the new build in QEMU

I’ll be using qemu to test the new build. The simplest way I could boot the VM would be to use qemu-system-x86_64 -kernel vmlinuz -initrd initramfs, but to run this within my terminal I’ll also add -append console=ttyS0 -nographic.

Testing the kernel in QEMU

It boots fast and my hello world endlessly spits out its message! My barebones OS works! I may eventually use this to build a few utility images to network boot but for now this will do as a base for more tinkering.

Using CAPI within a Docker container

In this post I will cover the process I used to get a program that uses CAPI to run within a docker container.

Base system config

The system I’m working on has been used inthe past to deploy my Hello AFU example project, which I will get to run in this Docker container. The host system already has the cxl support in the kernel, as well as my Hello AFU flashed onto my AlphaData KU3 device.

I first tested to ensure my AFU and CAPI application were still runing properly on the host system and installed Docker using Ubuntu’s docker package in apt.

Base docker image for PPC64LE

It’s been a while since I’ve used docker, so I’m a bit rusty. I decided to use the ppc64le/ubuntu from docker hub as my base image.

I started with a fairly minimal image, just to test that everything is working as expected. This simply pulls the dockerhub image, and runs a few apt commands.

FROM ppc64le/ubuntu

RUN apt update -y
RUN apt upgrade -y
RUN apt install -y git build-essential

CMD ["/bin/echo", "hellllooooooo!"]

Next I tag a new image, baseimage, with docker build, output below truncated for brevity.

root@Barreleye-15:~/docker-capi-test# docker build -t baseimage .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ppc64le/ubuntu
latest: Pulling from ppc64le/ubuntu
0847857e6401: Pull complete 
f8c18c152457: Pull complete 
8643975d001d: Pull complete 
d5802da4b3a0: Pull complete 
fe172ed92137: Pull complete 
Digest: sha256:5349f00594c719455f2c8e6f011b32758dcd326d8e225c737a55c15cf3d6948c
Status: Downloaded newer image for ppc64le/ubuntu:latest
 ---> 1967d889e07f
Step 2 : RUN apt update -y
 ---> Running in 8bb2d361d36c

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Get:1 xenial InRelease [247 kB]


Fetched 24.1 MB in 4s (4993 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
25 packages can be upgraded. Run 'apt list --upgradable' to see them.
 ---> e6e3775a3cd9
Removing intermediate container 8bb2d361d36c
Step 3 : RUN apt upgrade -y
 ---> Running in b617cbfa00f8

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
  apt base-files bsdutils gcc-5-base libapt-pkg5.0 libblkid1 libc-bin libc6


Processing triggers for libc-bin (2.23-0ubuntu5) ...
 ---> 6cc12a86896f
Removing intermediate container b617cbfa00f8
Step 4 : RUN apt install -y git build-essential
 ---> Running in c42e2e2f5e52

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  binutils bzip2 ca-certificates cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc


173 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
 ---> a3c8b863af74
Removing intermediate container c42e2e2f5e52
Step 5 : CMD /bin/bash echo hellllooooooo!
 ---> Running in e320ed315285
 ---> 7f633abdf66d
Removing intermediate container e320ed315285
Successfully built 7f633abdf66d

And I test to see if my image is working:

root@Barreleye-15:~/docker-capi-test# docker run -t baseimage


Building my CAPI application into the image

Next I will extend my Dockerfile to pull down my hello-afu code and build it. The first I’ll add an additional apt call to pull the necessary library and headers in from the libcxl-dev package. Then I use git and make commands to build my application the same as I would anywhere else. I also set my command line to automaticaly execute the test_afu application when running the container.

FROM ppc64le/ubuntu

RUN apt update -y
RUN apt upgrade -y
RUN apt install -y git build-essential 
RUN apt install -y libcxl-dev

RUN git clone
RUN cd hello-afu && make

CMD ["/hello-afu/test_afu"]

I’ll build this image and tag it as hello-afu

root@Barreleye-15:~/docker-capi-test# docker build -t hello-afu .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ppc64le/ubuntu
 ---> 1967d889e07f
Step 2 : RUN apt update -y
 ---> Using cache
 ---> daf3a9437751
Step 3 : RUN apt upgrade -y
 ---> Using cache
 ---> 3882f8f83b78
Step 4 : RUN apt install -y git build-essential
 ---> Using cache
 ---> 2cf498f74f15
Step 5 : RUN apt install -y libcxl-dev
 ---> Running in dbeb87ec38fd

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
The following NEW packages will be installed:
  libcxl-dev libcxl1
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 58.5 kB of archives.
After this operation, 178 kB of additional disk space will be used.
Get:1 xenial/universe ppc64el libcxl1 ppc64el 1.3-0ubuntu2 [12.6 kB]
Get:2 xenial/universe ppc64el libcxl-dev ppc64el 1.3-0ubuntu2 [45.9 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 58.5 kB in 0s (124 kB/s)
Selecting previously unselected package libcxl1.
(Reading database ... 16348 files and directories currently installed.)
Preparing to unpack .../libcxl1_1.3-0ubuntu2_ppc64el.deb ...
Unpacking libcxl1 (1.3-0ubuntu2) ...
Selecting previously unselected package libcxl-dev.
Preparing to unpack .../libcxl-dev_1.3-0ubuntu2_ppc64el.deb ...
Unpacking libcxl-dev (1.3-0ubuntu2) ...
Processing triggers for libc-bin (2.23-0ubuntu5) ...
Setting up libcxl1 (1.3-0ubuntu2) ...
Setting up libcxl-dev (1.3-0ubuntu2) ...
Processing triggers for libc-bin (2.23-0ubuntu5) ...
 ---> db6ef39a11d3
Removing intermediate container dbeb87ec38fd
Step 6 : RUN git clone
 ---> Running in ca75aa96477f
Cloning into 'hello-afu'...
 ---> 4471e2887fad
Removing intermediate container ca75aa96477f
Step 7 : RUN cd hello-afu && make
 ---> Running in e9cd952ef56c
gcc -Wall -o test_afu test_afu.c -I ~/workprojects/pslse/libcxl -L ~/workprojects/pslse/libcxl -lcxl -lpthread 
test_afu.c: In function 'main':
test_afu.c:56:9: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type '__u64 {aka long unsigned int}' [-Wformat=]
  printf("  example->size: %llu\n", example->size);
 ---> 75346e99c3a7
Removing intermediate container e9cd952ef56c
Step 8 : CMD /hello-afu/test_afu
 ---> Running in 53c0cb116657
 ---> dd2ea88f3a11
Removing intermediate container 53c0cb116657
Successfully built dd2ea88f3a11

And I’ll test out this new image:

root@Barreleye-15:~/docker-capi-test# docker run -t hello-afu
Failed to open AFU: No such file or directory

So far my docker file seems setup properly, but the test_afu application is not finding the capi device, which is expected as it’s not yet within the containers view of the world.

Sharing the CXL device

In my first attempt, I tried to directly mount my /dev/cxl from host to container, this resulted in a different error regarding the permissions

root@Barreleye-15:~/docker-capi-test# docker run -v /dev/cxl/:/dev/cxl -t hello-afu
Failed to open AFU: Operation not permitted

After some documentation perusing, I found the --privileged flag that allows me to use this type of device sharing.

root@Barreleye-15:~/docker-capi-test# docker run -v /dev/cxl/:/dev/cxl --privileged -t hello-afu
[example structure
  example: 0x1002c4d0200
  example->size: 128
  example->stripe1: 0x1002c4d0300
  example->stripe2: 0x1002c4d0400
  example->parity: 0x1002c4d0580
  &(example->done): 0x1002c4d0220
Attached to AFU
Waiting for completion by AFU
That is some proper parity! This is exactly what I'm expecting to see. I'd also like to see this running on some real gear soon
Releasing AFU

Success! I now have my application from my docker container that can interface with the host CAPI device. I hope this post proves helpful, please leave any feedback you may have in the comments!