Test using Pluma¶

Welma and the demos can be tested using the Pluma test automation tool.

Pluma reference and installation guide:

https://docs.pluma.witekio.com/

Prerequisites¶

The Device Under Test (DUT, either an emulated board or a real board) must have an interface on which the test automation tool can communicate (eg: Serial port, IP Network, etc.).

On Welma, development images are tested, with SSH:

DUT instrumented with SSH access (ie: development image), powered up and running
Host configured with SSH access to the DUT

Pluma runs ssh, which honors ~/.ssh/config. Example:

.ssh/config

Host qemu
    Hostname fe80::1%%enx0050b6b107a3
    User root
    Port 22
    HostbasedAuthentication no
    StrictHostKeyChecking no

target.yml

system:
  prompt_regex: "qemuarm-welma:~# "
  boot_timeout: 30s

console:
  ssh:
    target: qemu

Testing the Welma Qemu target with SSH access¶

Additional System Requirements:

qemu-system-arm
Listening TCP port 2222 available (used by Pluma for SSH redirection)

Setup:

Download the Welma test repository:

$ git clone git@gitlab.com:witekio/rnd/theembeddedkit/welma/welma-test.git

Run Pluma:

$ cd welma-test/machine-qemuarm-welma

$ export MACHINE=qemuarm-welma
$ export TESTFILES=$PWD/../files
$ export TOPDIR=$HOME/projects/welma/build-$MACHINE
$ export DEPLOYDIR=$TOPDIR/tmp/deploy/images/$MACHINE
$ export UBOOT=$DEPLOYDIR/u-boot.bin
$ export SDIMG=$DEPLOYDIR/demo-image-headless-dev-$MACHINE.wic
$ export PKG_BOOT=$DEPLOYDIR/demo-image-headless-dev-$MACHINE.boot.swu
$ export PKG_APPRO=$DEPLOYDIR/demo-image-headless-dev-$MACHINE.appro.swu
$ export PKG_SYSRO=$DEPLOYDIR/demo-image-headless-dev-$MACHINE.sysro.swu
$ export SSH_TARGET=device-under-test-qemu

$ pluma -t target.yml -c tests-minimal.yml
Running Plan "Welma Minimal Test"...
- Action "Setup: Start Qemu"                     PASS
- Action "Setup: Wait 30 s for the DUT to st...  PASS
- Action "Setup: Check that Qemu is running"     PASS
- Action "uname --kernel-release"                PASS
  Check:    output[:5] in ["5.10.", "5.15."]     OK
Test "01. The DUT is running a kernel 5.10"      PASS
- Action "Have the demo app fail (SEGV)"         PASS
- Action "Wait a few seconds for the app to ...  PASS
- Action "Check that the demo app is running"    PASS
Test "05. The watchdog restarts the demo a...    PASS
- Action "Create a control sample file in vo...  PASS
- Action "Kill the demo app several times so...  PASS
- Action "Wait 30 s for the DUT to reboot"       PASS
- Action "Read the control sample file (shou...  PASS
- Action "Check that the demo app is running"    PASS
Test "06. The watchdog reboots the DUT if ...    PASS
Group "A group of minimal tests"                 PASS
- Action "Teardown: Stop Qemu"                   PASS
Plan "Welma Minimal Test"                        PASS
All tests were successful.