[FIXED] How to extend timeout for tests in circleci?

Issue

Im running some tests in circleci and some of the tests are taking longer then 10 min cause its ui tests that run on a headless browser that Im installing in my circle.yml

How can I extend the time of the timeout?

thanks

Solution

You need to use the timeout modifier in your config as explained in this doc: https://circleci.com/docs/configuration#modifiers

Here is an example doubling the default 600s to 1200s:

commands:
    - /bin/bash build_scripts/deploy_to_eb.sh:
        timeout: 1200

Cheers

Answered By - Joan

Answer Checked By - Mildred Charles (PHPFixing Admin)

Read More

[FIXED] How to disable builds-concurrency on master in circle-ci/azure-pipeline/any-other-ci?

Issue

It is considered a bug to run deployment scripts concurrenctly.

I failed to find a solution to it in circle-ci, azure-pipeline, code-fresh and more..

Bitbucket-pipelines has a very nice solution:

• only one deployment script is running against a specific enviroment
• new builds will automatically stop older (running) builds

As I want to move out of bitbucket, I can't do it until I will find any other CI that has this basic ability.

---

Maybe I'm missing something because to my knowlege, all the CIs themselves must have this problem as well when they deploy their new features. unless they use jenkins and just lock the project haha...

Solution

I think you can achieve this with azure release pipeline.

If you are to use Classic UI Azure release pipeline. You can achieve stopping older builds when new builds is queued by configuring Deployment queue settings. See below screenshot:

1,Set the Maximum number of parallel deployments to control the parallel deployment. Check Deploy latest and cancel others will only deploy the latest queued deployment. All the previous queued(not running yet) deployment will be cancelled. But if a previous deployment is running. The latest queued build will have to wait until the running build to be completed unless you manually cancel it.

If you want to cancel the older running builds you can add a script task to call the rest api to cancel the previous running builds. See the example in below yaml pipeline example:

2, To only deploy to one target you can configure the Deployment targets as below in a deployment group job

If you are to use Yaml pipleline. The Deploy latest and cancel others and Deployment group jobs are not supported for yaml pipeline. See this use voice here.

In Yaml pipeline, you can use deploy jobs and environments instead. You can configure the Define approvals and checks for the Environment to enable the Exclusive lock to ensure only a single run deploys to this environment at a time. And configure the deployment strategy to set the maxparallel deployment.

To cancel the older running builds you can add a script task to call the rest api. See below example: Check my answer to this thread for more information.

- task: PowerShell@2
  inputs:
        targetType: inline
        script: |
          $header = @{ Authorization = "Bearer $(system.accesstoken)" }
          $buildsUrl = "$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_apis/build/builds?api-version=5.1"
          echo $buildsUrl
          $builds = Invoke-RestMethod -Uri $buildsUrl -Method Get -Header $header
          $buildsToStop = $builds.value.Where({ ($_.status -eq 'inProgress') -and ($_.definition.name -eq "$(Build.DefinitionName)") -and ($_.id -ne $(Build.BuildId))})
          ForEach($build in $buildsToStop)
          {
            echo $build.id
            $build.status = "cancelling"
            $body = $build | ConvertTo-Json -Depth 10
            $urlToCancel = "$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_apis/build/builds/$($build.id)?api-version=5.1"
            echo $urlToCancel
            Invoke-RestMethod -Uri $urlToCancel -Method Patch -ContentType application/json -Body $body -Header $header
          }

Answered By - Levi Lu-MSFT

Answer Checked By - Mildred Charles (PHPFixing Admin)

Read More

[FIXED] How to run Rubocop only on the changed files in a pull request?

Issue

I have created spec/lint/rubocop_spec.rb which runs Rubocop style checker on the files changed between current branch and master. This works when I test locally but not when the test run on the build server Circle.ci. I suspect it is because only the branch in question is downloaded, so it does not find any differences between master. Is there a better way than git co master && git pull origin master? Can I query the Github API perhaps to get the files changed listed?

require 'spec_helper'

describe 'Check that the files we have changed have correct syntax' do
  before do
    current_sha = `git rev-parse --verify HEAD`.strip!
    files = `git diff master #{current_sha} --name-only | grep .rb`
    files.tr!("\n", ' ')
    @report = 'nada'
    if files.present?
      puts "Changed files: #{files}"

      @report = `rubocop #{files}`
      puts "Report: #{@report}"
    end
  end

  it { @report.match('Offenses').should_not be true }
end

Solution

I fixed it by querying api.github.com. This will run rubocop on all files that has been changed between current_sha and the master branch.

require 'spec_helper'

describe 'Check that the files we have changed have correct syntax' do
  before do
    current_sha = `git rev-parse --verify HEAD`.strip!
    token = 'YOUR GITHUB TOKEN'
    url = 'https://api.github.com/repos/orwapp/orwapp/compare/' \
          "master...#{current_sha}?access_token=#{token}"
    files = `curl -i #{url} | grep filename | cut -f2 -d: | grep \.rb | tr '"', '\ '`
    files.tr!("\n", ' ')
    @report = 'nada'
    if files.present?
      puts "Changed files: #{files}"

      @report = `rubocop #{files}`
      puts "Report: #{@report}"
    end
  end

  it { expect(@report.match('Offenses')).to be_falsey }
end

Answered By - martins

Answer Checked By - Cary Denson (PHPFixing Admin)

Read More

[FIXED] How to solve: yarn package not found on CircleCi?

Issue

I have been trying to learn CircleCi workflows by building simple jobs and I keep getting this error: /bin/bash: yarn: command not found.

So all the steps run but when it comes to the job is self it stops. So spin up environment, preparing env variables, checkout code, Restore Yarn Package Cache are green(successful).

My workflow below:

default-environment: &default-environment
  docker:
    - image: cimg/base:2020.01

step-restore-build-cache: &step-restore-build-cache
  restore_cache:
    name: Restore Yarn Package Cache
    keys:
      - yarn-packages-{{ checksum "yarn.lock" }}

step-save-build-cache: &step-save-build-cache
  save_cache:
    name: Save Yarn Package Cache
    key: yarn-packages-{{ checksum "yarn.lock" }}
    paths:
      - ~/.cache/yarn

step-run-cache: &step-run-cache
  run:
    name: Install Dependencies
    command: yarn install --immutable

version: 2.1
jobs:
  build:
    <<: *default-environment
    steps:
      - checkout
      - <<: *step-restore-build-cache
      - run:
          name: Build
          command: yarn run build
      - <<: *step-save-build-cache
  lint:
    <<: *default-environment
    steps:
      - checkout
      - <<: *step-restore-build-cache
      - run:
          name: Lint
          command: yarn run lint
  format:
    <<: *default-environment
    steps:
      - checkout
      - <<: *step-restore-build-cache
      - run:
          name: Format
          command: yarn run format
  type-check:
    <<: *default-environment
    steps:
      - checkout
      - <<: *step-restore-build-cache
      - run:
          name: Type-check
          command: yarn run type-check
workflows:
  version: 2
  build-and-lint:
    jobs:
      - build
      - lint
      - format
      - type-check

Not really sure how to fix this..

Thank you very much :)

Solution

You're using cimg/base:2020.01 which is a general purpose image and does not have node or the yarn binary installed. You should use the cimg/node image which has yarn pre-installed.

Answered By - Idan Elhalwani

Answer Checked By - Terry (PHPFixing Volunteer)

Read More

[FIXED] How to unit test graphics with python3, CircleCI and Mayavi

Issue

I wrote a bunch of visualization functions in my python3 library using Mayavi. I am not very familiar with this library, nor am I with testing visualizations using python.

Ideally, I would just like the visualization code to generate some graphics on disk, I don't care too much about popping up windows (although I'm not sure to understand if Mayavi can work properly without popping such windows).

Anyway, my code works on local, but when I push it on develop, CircleCI fails at running the tests with the following error:

!/bin/bash -eo pipefail
python3 tests/test.py

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl, xcb.


Received "aborted" signal

The Docker Image I use is the following:

FROM ubuntu:focal

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update -y
RUN apt-get install -y --no-install-recommends\
                    vim \
                    git \
                    gcc-9 \
                    g++ \
                    build-essential \
                    libboost-all-dev \
                    cmake \
                    unzip \
                    tar \
                    ca-certificates \
                    doxygen \
                    graphviz
                    
RUN apt-get install -y libgdal-dev g++ --no-install-recommends && \
    apt-get clean -y

ENV CPLUS_INCLUDE_PATH=/usr/include/gdal
ENV C_INCLUDE_PATH=/usr/include/gdal

RUN git clone --recurse-submodules https://github.com/Becheler/quetzal-EGGS \
&& cd quetzal-EGGS \
&&  mkdir Release \
&&  cd Release \
&& cmake .. -DCMAKE_INSTALL_PREFIX="/usr/local/quetzal-EGGS" \
&& cmake --build . --config Release --target install

RUN set -xe \
    apt-get update && apt-get install -y \
    python3-pip \
    --no-install-recommends

RUN pip3 install --upgrade pip
RUN pip3 install build twine pipenv numpy # for crumbs publishing
RUN pip3 install rasterio && \
    pip3 install matplotlib && \
    pip3 install imageio && \
    pip3 install imageio-ffmpeg && \
    pip3 install pyproj && \
    pip3 install shapely && \
    pip3 install fiona && \
    pip3 install scikit-learn && \ 
    pip3 install pyimpute && \ 
    pip3 install geopandas && \
    pip3 install pygbif

########## MAYAVI 

# xcb plugin 
RUN apt-get install -y --no-install-recommends libxkbcommon-x11-0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 && \
    apt-get clean -y
    
RUN python3 -m pip install vtk
RUN apt-get update && apt-get install -y python3-opencv && apt-get clean -y
RUN pip3 install opencv-python
RUN pip3 install mayavi PyQt5
  
RUN pip3 install GDAL==$(gdal-config --version) pyvolve==1.0.3 quetzal-crumbs==0.0.15

# Clean to make image smaller
RUN apt-get autoclean && \
    apt-get autoremove && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

• Am I missing some dependencies?
• Should I had a -X option somewhere?
• Should I deactivate the mlab.show() and imshow() calls in my library?

Solution

I missed a dependency, qt5-default. I ended up having these lines for Mayavi running on Docker/CircleCi:

########## MAYAVI 

# xcb plugin 
RUN apt-get install -y --no-install-recommends xvfb libxkbcommon-x11-0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 && \
    apt-get clean -y
# Trying to solve the weird xcfb error
RUN apt-get install -y --no-install-recommends qt5-default && \
    apt-get clean -y
RUN python3 -m pip install vtk
RUN apt-get update && apt-get install -y python3-opencv && apt-get clean -y
RUN pip3 install opencv-python
RUN pip3 install PyVirtualDisplay
RUN pip3 install mayavi PyQt5

I am pretty sure that many of these dependencies are redundant or not required. But it works, so I will leave it this way until I have some time to make a bit of cleanup.

I also added this line in my python library:

mlab.options.offscreen = True

The rational for it comes from the mayavi offscreen rendering documentation

Answered By - Arnaud Becheler

Answer Checked By - Timothy Miller (PHPFixing Admin)

Read More