[FIXED] How to unit test graphics with python3, CircleCI and Mayavi

Issue

I wrote a bunch of visualization functions in my python3 library using Mayavi. I am not very familiar with this library, nor am I with testing visualizations using python.

Ideally, I would just like the visualization code to generate some graphics on disk, I don't care too much about popping up windows (although I'm not sure to understand if Mayavi can work properly without popping such windows).

Anyway, my code works on local, but when I push it on develop, CircleCI fails at running the tests with the following error:

!/bin/bash -eo pipefail
python3 tests/test.py

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl, xcb.


Received "aborted" signal

The Docker Image I use is the following:

FROM ubuntu:focal

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update -y
RUN apt-get install -y --no-install-recommends\
                    vim \
                    git \
                    gcc-9 \
                    g++ \
                    build-essential \
                    libboost-all-dev \
                    cmake \
                    unzip \
                    tar \
                    ca-certificates \
                    doxygen \
                    graphviz
                    
RUN apt-get install -y libgdal-dev g++ --no-install-recommends && \
    apt-get clean -y

ENV CPLUS_INCLUDE_PATH=/usr/include/gdal
ENV C_INCLUDE_PATH=/usr/include/gdal

RUN git clone --recurse-submodules https://github.com/Becheler/quetzal-EGGS \
&& cd quetzal-EGGS \
&&  mkdir Release \
&&  cd Release \
&& cmake .. -DCMAKE_INSTALL_PREFIX="/usr/local/quetzal-EGGS" \
&& cmake --build . --config Release --target install

RUN set -xe \
    apt-get update && apt-get install -y \
    python3-pip \
    --no-install-recommends

RUN pip3 install --upgrade pip
RUN pip3 install build twine pipenv numpy # for crumbs publishing
RUN pip3 install rasterio && \
    pip3 install matplotlib && \
    pip3 install imageio && \
    pip3 install imageio-ffmpeg && \
    pip3 install pyproj && \
    pip3 install shapely && \
    pip3 install fiona && \
    pip3 install scikit-learn && \ 
    pip3 install pyimpute && \ 
    pip3 install geopandas && \
    pip3 install pygbif

########## MAYAVI 

# xcb plugin 
RUN apt-get install -y --no-install-recommends libxkbcommon-x11-0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 && \
    apt-get clean -y
    
RUN python3 -m pip install vtk
RUN apt-get update && apt-get install -y python3-opencv && apt-get clean -y
RUN pip3 install opencv-python
RUN pip3 install mayavi PyQt5
  
RUN pip3 install GDAL==$(gdal-config --version) pyvolve==1.0.3 quetzal-crumbs==0.0.15

# Clean to make image smaller
RUN apt-get autoclean && \
    apt-get autoremove && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

• Am I missing some dependencies?
• Should I had a -X option somewhere?
• Should I deactivate the mlab.show() and imshow() calls in my library?

Solution

I missed a dependency, qt5-default. I ended up having these lines for Mayavi running on Docker/CircleCi:

########## MAYAVI 

# xcb plugin 
RUN apt-get install -y --no-install-recommends xvfb libxkbcommon-x11-0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 && \
    apt-get clean -y
# Trying to solve the weird xcfb error
RUN apt-get install -y --no-install-recommends qt5-default && \
    apt-get clean -y
RUN python3 -m pip install vtk
RUN apt-get update && apt-get install -y python3-opencv && apt-get clean -y
RUN pip3 install opencv-python
RUN pip3 install PyVirtualDisplay
RUN pip3 install mayavi PyQt5

I am pretty sure that many of these dependencies are redundant or not required. But it works, so I will leave it this way until I have some time to make a bit of cleanup.

I also added this line in my python library:

mlab.options.offscreen = True

The rational for it comes from the mayavi offscreen rendering documentation

Answered By - Arnaud Becheler

Answer Checked By - Timothy Miller (PHPFixing Admin)