Using CAPI within a Docker container

In this post I will cover the process I used to get a program that uses CAPI to run within a docker container.

Base system config

The system I’m working on has been used inthe past to deploy my Hello AFU example project, which I will get to run in this Docker container. The host system already has the cxl support in the kernel, as well as my Hello AFU flashed onto my AlphaData KU3 device.

I first tested to ensure my AFU and CAPI application were still runing properly on the host system and installed Docker using Ubuntu’s docker package in apt.



Base docker image for PPC64LE

It’s been a while since I’ve used docker, so I’m a bit rusty. I decided to use the ppc64le/ubuntu from docker hub as my base image.

I started with a fairly minimal image, just to test that everything is working as expected. This simply pulls the dockerhub image, and runs a few apt commands.

FROM ppc64le/ubuntu

RUN apt update -y
RUN apt upgrade -y
RUN apt install -y git build-essential

CMD ["/bin/echo", "hellllooooooo!"]

Next I tag a new image, baseimage, with docker build, output below truncated for brevity.

root@Barreleye-15:~/docker-capi-test# docker build -t baseimage .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ppc64le/ubuntu
latest: Pulling from ppc64le/ubuntu
0847857e6401: Pull complete 
f8c18c152457: Pull complete 
8643975d001d: Pull complete 
d5802da4b3a0: Pull complete 
fe172ed92137: Pull complete 
Digest: sha256:5349f00594c719455f2c8e6f011b32758dcd326d8e225c737a55c15cf3d6948c
Status: Downloaded newer image for ppc64le/ubuntu:latest
 ---> 1967d889e07f
Step 2 : RUN apt update -y
 ---> Running in 8bb2d361d36c

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Get:1 http://ports.ubuntu.com/ubuntu-ports xenial InRelease [247 kB]

[...]

Fetched 24.1 MB in 4s (4993 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
25 packages can be upgraded. Run 'apt list --upgradable' to see them.
 ---> e6e3775a3cd9
Removing intermediate container 8bb2d361d36c
Step 3 : RUN apt upgrade -y
 ---> Running in b617cbfa00f8

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
  apt base-files bsdutils gcc-5-base libapt-pkg5.0 libblkid1 libc-bin libc6

[...]

Processing triggers for libc-bin (2.23-0ubuntu5) ...
 ---> 6cc12a86896f
Removing intermediate container b617cbfa00f8
Step 4 : RUN apt install -y git build-essential
 ---> Running in c42e2e2f5e52

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  binutils bzip2 ca-certificates cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc

[...]

173 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
 ---> a3c8b863af74
Removing intermediate container c42e2e2f5e52
Step 5 : CMD /bin/bash echo hellllooooooo!
 ---> Running in e320ed315285
 ---> 7f633abdf66d
Removing intermediate container e320ed315285
Successfully built 7f633abdf66d

And I test to see if my image is working:

root@Barreleye-15:~/docker-capi-test# docker run -t baseimage
hellllooooooo!

Huzzah!

Building my CAPI application into the image

Next I will extend my Dockerfile to pull down my hello-afu code and build it. The first I’ll add an additional apt call to pull the necessary library and headers in from the libcxl-dev package. Then I use git and make commands to build my application the same as I would anywhere else. I also set my command line to automaticaly execute the test_afu application when running the container.

FROM ppc64le/ubuntu

RUN apt update -y
RUN apt upgrade -y
RUN apt install -y git build-essential 
RUN apt install -y libcxl-dev

RUN git clone https://github.com/KennethWilke/hello-afu
RUN cd hello-afu && make

CMD ["/hello-afu/test_afu"]

I’ll build this image and tag it as hello-afu

root@Barreleye-15:~/docker-capi-test# docker build -t hello-afu .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ppc64le/ubuntu
 ---> 1967d889e07f
Step 2 : RUN apt update -y
 ---> Using cache
 ---> daf3a9437751
Step 3 : RUN apt upgrade -y
 ---> Using cache
 ---> 3882f8f83b78
Step 4 : RUN apt install -y git build-essential
 ---> Using cache
 ---> 2cf498f74f15
Step 5 : RUN apt install -y libcxl-dev
 ---> Running in dbeb87ec38fd

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libcxl1
The following NEW packages will be installed:
  libcxl-dev libcxl1
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 58.5 kB of archives.
After this operation, 178 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial/universe ppc64el libcxl1 ppc64el 1.3-0ubuntu2 [12.6 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports xenial/universe ppc64el libcxl-dev ppc64el 1.3-0ubuntu2 [45.9 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 58.5 kB in 0s (124 kB/s)
Selecting previously unselected package libcxl1.
(Reading database ... 16348 files and directories currently installed.)
Preparing to unpack .../libcxl1_1.3-0ubuntu2_ppc64el.deb ...
Unpacking libcxl1 (1.3-0ubuntu2) ...
Selecting previously unselected package libcxl-dev.
Preparing to unpack .../libcxl-dev_1.3-0ubuntu2_ppc64el.deb ...
Unpacking libcxl-dev (1.3-0ubuntu2) ...
Processing triggers for libc-bin (2.23-0ubuntu5) ...
Setting up libcxl1 (1.3-0ubuntu2) ...
Setting up libcxl-dev (1.3-0ubuntu2) ...
Processing triggers for libc-bin (2.23-0ubuntu5) ...
 ---> db6ef39a11d3
Removing intermediate container dbeb87ec38fd
Step 6 : RUN git clone https://github.com/KennethWilke/hello-afu
 ---> Running in ca75aa96477f
Cloning into 'hello-afu'...
 ---> 4471e2887fad
Removing intermediate container ca75aa96477f
Step 7 : RUN cd hello-afu && make
 ---> Running in e9cd952ef56c
gcc -Wall -o test_afu test_afu.c -I ~/workprojects/pslse/libcxl -L ~/workprojects/pslse/libcxl -lcxl -lpthread 
test_afu.c: In function 'main':
test_afu.c:56:9: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type '__u64 {aka long unsigned int}' [-Wformat=]
  printf("  example->size: %llu\n", example->size);
         ^
 ---> 75346e99c3a7
Removing intermediate container e9cd952ef56c
Step 8 : CMD /hello-afu/test_afu
 ---> Running in 53c0cb116657
 ---> dd2ea88f3a11
Removing intermediate container 53c0cb116657
Successfully built dd2ea88f3a11

And I’ll test out this new image:

root@Barreleye-15:~/docker-capi-test# docker run -t hello-afu
Failed to open AFU: No such file or directory

So far my docker file seems setup properly, but the test_afu application is not finding the capi device, which is expected as it’s not yet within the containers view of the world.

Sharing the CXL device

In my first attempt, I tried to directly mount my /dev/cxl from host to container, this resulted in a different error regarding the permissions

root@Barreleye-15:~/docker-capi-test# docker run -v /dev/cxl/:/dev/cxl -t hello-afu
Failed to open AFU: Operation not permitted

After some documentation perusing, I found the --privileged flag that allows me to use this type of device sharing.

root@Barreleye-15:~/docker-capi-test# docker run -v /dev/cxl/:/dev/cxl --privileged -t hello-afu
[example structure
  example: 0x1002c4d0200
  example->size: 128
  example->stripe1: 0x1002c4d0300
  example->stripe2: 0x1002c4d0400
  example->parity: 0x1002c4d0580
  &(example->done): 0x1002c4d0220
Attached to AFU
Waiting for completion by AFU
PARITY:
That is some proper parity! This is exactly what I'm expecting to see. I'd also like to see this running on some real gear soon
Releasing AFU

Success! I now have my application from my docker container that can interface with the host CAPI device. I hope this post proves helpful, please leave any feedback you may have in the comments!

Simulating CAPI Designs with PSLSE and Vivado

Following up on the Hello AFU tutorial, this post covers the process to bring simulate that design in Vivavo’s xsim.

Setting up PSLSE

Assuming readers may not have setup PSLSE before, I will start by cloning that down again and building it with support for use with Vivado.

First, I’ll just clone down the repo and enter it’s directory.

kwilke@kbawx:~/projects$ git clone https://github.com/ibm-capi/pslse
Cloning into 'pslse'...
remote: Counting objects: 2789, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 2789 (delta 0), reused 0 (delta 0), pack-reused 2778
Receiving objects: 100% (2789/2789), 954.75 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1952/1952), done.
Checking connectivity... done.

Next, I’ll set the VPI_USER_H_DIR to point to the xsim include directory from my local Vivado installation and build the afu_driver, pslse and libcxl from the PSLSE repo.

kwilke@kbawx:~/projects/pslse$ export VPI_USER_H_DIR=/opt/Xilinx/Vivado/2015.4/data/xsim/include
kwilke@kbawx:~/projects/pslse$ cd afu_driver/src
kwilke@kbawx:~/projects/pslse/afu_driver/src$ make
 [CC]    afu_driver.o
...
 [CC]    libdpi.so
kwilke@kbawx:~/projects/pslse/afu_driver/src$ cd ../../pslse/
kwilke@kbawx:~/projects/pslse/pslse$ make
 [CC]    shim_host.o
 ...
 [CC]    pslse
kwilke@kbawx:~/projects/pslse/pslse$ cd ../libcxl/
kwilke@kbawx:~/projects/pslse/libcxl$ make
 [CC]    libcxl.o
...
 [AR]    libcxl.a 

Next I’ll add some symlinks to te psl_interface headers into my pslse/afu_driver/src directory to ease the library compilation via Vivado’s xsc.

kwilke@kbawx:~/projects/pslse/libcxl$ cd ../afu_driver/src/
kwilke@kbawx:~/projects/pslse/afu_driver/src$ ln -s ../../common/psl_interface.h .
kwilke@kbawx:~/projects/pslse/afu_driver/src$ ln -s ../../common/psl_interface_t.h .

Now PSLSE should be ready for us.



Compiling the DPI library and AFU

The next step is to enter the Hello AFU directory and build the AFU driver for Vivado.

kwilke@kbawx:~/projects/hello-afu$ xsc ~/projects/pslse/afu_driver/src/afu_driver.c
Multi-threading is on. Using 2 slave threads.
Running compilation flow
Done compilation
Done linking: "/home/kwilke/projects/hello-afu/xsim.dir/xsc/dpi.so"

I’ll create a symlink to the libdpi.so into my Hello AFU directory.

kwilke@kbawx:~/projects/hello-afu$ ln -s ~/projects/pslse/afu_driver/src/libdpi.so .

Next I’ll use xvlog to build the AFU code.

kwilke@kbawx:~/projects/hello-afu$ xvlog --sv *.sv *.v
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/hello-afu/afu.sv" into library work
INFO: [VRFC 10-311] analyzing module afu
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/hello-afu/capi.sv" into library work
INFO: [VRFC 10-443] port direction not specified for function/task, assuming input [/home/kwilke/projects/hello-afu/capi.sv:136]
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/hello-afu/mmio.sv" into library work
INFO: [VRFC 10-311] analyzing module mmio
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/hello-afu/parity_afu.sv" into library work
INFO: [VRFC 10-311] analyzing module parity_afu
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/hello-afu/parity_workelement.sv" into library work
INFO: [VRFC 10-443] port direction not specified for function/task, assuming input [/home/kwilke/projects/hello-afu/parity_workelement.sv:27]
INFO: [VRFC 10-311] analyzing module parity_workelement
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/hello-afu/shift_register.sv" into library work
INFO: [VRFC 10-311] analyzing module shift_register
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/hello-afu/top.v" into library work
INFO: [VRFC 10-311] analyzing module top

Finally, I’ll use xelab to elaborate the design.

kwilke@kbawx:~/projects/hello-afu$ xelab -timescale 1ns/1ps -svlog ~/projects/pslse/afu_driver/verilog/top.v -sv_root . -sv_lib libdpi -debug all
Vivado Simulator 2015.4
Copyright 1986-1999, 2001-2015 Xilinx, Inc. All Rights Reserved.
Running: /opt/Xilinx/Vivado/2015.4/bin/unwrapped/lnx64.o/xelab -timescale 1ns/1ps -svlog /home/kwilke/projects/pslse/afu_driver/verilog/top.v -sv_root . -sv_lib libdpi -debug all 
Multi-threading is on. Using 2 slave threads.
INFO: [VRFC 10-2263] Analyzing SystemVerilog file "/home/kwilke/projects/pslse/afu_driver/verilog/top.v" into library work
INFO: [VRFC 10-311] analyzing module top
Starting static elaboration
Completed static elaboration
Starting simulation data flow analysis
Completed simulation data flow analysis
Time Resolution for simulation is 1ps
Compiling module work.shift_register
Compiling module work.shift_register(width=64)
Compiling module work.mmio
Compiling module work.shift_register(width=512)
Compiling module work.parity_workelement
Compiling module work.parity_afu
Compiling module work.afu
Compiling module work.top
Compiling package work.CAPI
Compiling package work.$unit_1
Built simulation snapshot work.top

We should now be in a good state to begin simulation.

Running the simulation

To kick off the simulation, I’ll point xsim to the project.

kwilke@kbawx:~/projects/hello-afu$ xsim -g work.top

****** xsim v2015.4 (64-bit)
  **** SW Build 1412921 on Wed Nov 18 09:44:32 MST 2015
  **** IP Build 1412160 on Tue Nov 17 13:47:24 MST 2015
    ** Copyright 1986-2015 Xilinx, Inc. All Rights Reserved.

start_gui

At this point the Vivado tools will come up and enter simulation for our project.

Vivado simulation window

Hitting the Run All button from the top (looks like a Play button with a square wave) it’ll start off the simulation and wait for a connection from PSLSE.

Simulation Console Output

I can now kick off PSLSE and it’ll connect to xsim and wait for a connection from my application.

Starting PSLSE with Vivado

At this time the master branch of PSLSE isn’t building libcxl.so, so I’ve pointed my application to the library of a previous build and it’s worked just fine.

Hello simulation in Vivado

Hooray! I now have my system setup to simulate CAPI designs in Vivado! I hope this post is useful for others working to get their CAPI designs simulated in Vivado, if you have any questions or comments please pass them my way!

Hello AFU on Alpha-Data KU3

Picking up on the Hello AFU project, I’ve recently gone through the motions of building the Hello AFU project for an actual CAPI device and tested it out. This post documents the process I followed to build and deploy this on real hardware.

Requirements

To complete this process you’ll need a few things:
* A POWER8 based machine, for me I’m using a Barreleye server
* An Alpha-Data KU3 card
* The latest HDK archive from Alpha Data’s support site, at this time that file is named ADMPCIEKU3_CAPI_HDK_REL18MAR16.zip
* A licensed version of Xilinx’s Vivado



Preparing files for the build

First off, we need to extract the HDK

unzip ADMPCIEKU3_CAPI_HDK_REL18MAR16.zip

In the HDK by default, there will be some AFU source files in adku060_capi_1_1_release/Sources/afu/ we’ll jump in there and delete them, then copy over the SystemVerilog files from the hello-afu repository

cd adku060_capi_1_1_release/Sources/afu/
rm *
cp ~/projects/hello-afu/*.sv .

Next, open the project file adku060_capi_1_1_release/Sources/prj/psl_fpga.prj in a text editor to change a few lines. Remove all of the lines that start with verilog work, then add lines to reference the source files we copied into the afu directory. Some bash-fu for that:

cd ../prj
sed -i '/^verilog work/d' psl_fpga.prj
for i in `ls ../afu/*.sv | cut -d'/' -f3`; do echo "verilog work \"afu/$i\"" >> psl_fpga.prj; done

That should have us setup to build our AFU in leiu of the one that comes with the HDK!

Build and flash the binfile

With our files in the right spot and our project file modified, we just need to run a few of the tcl scripts in the HDK through vivado.

vivado -mode batch -source psl_fpga.tcl -notrace
vivado -mode batch -source write_bitstream.tcl -notrace

The first run here does the heavy lifting of synthesis, place and route, etc. The second command generates the actual binfile and bitfile that we can use to flash the device. The first command takes a significant amount of time on my i7-equipped laptop, about 40 minutes, the second command completed in about 9 seconds. Maybe someday we’ll have a CAPI-based accelerator for synthesis and place & route! Now that the building is complete I have my bitfile at capi-adku060/psl_fpga_flash.bin

To flash this to your device to a card that already has the PSL working you can use the capi-flash-script utility. If your card is factory-fresh or in a bad state, you can use a JTAG programmer and Vivado’s Hardware Manager to flash directly from your laptop, or remotely via xvcserver.

Using the AFU

After I flashed my AFU, I ensured libcxl was setup on my server. Since I’m running Ubunt 16.04 I simply installed it via apt.

apt-get install -y libcxl-dev

Next I rebooted the machine so that everything is nice and fresh, as part of the PCIe reset the bitfile from the KU3’s flash chip will be flashed onto the FPGA. I can verify the card is in a good state because I have my cxl device at /dev/cxl/afu0.0d.

I run my test_afu binary from the hello-afu project and boom! The same result as I get from simulation, woo-hoo!