Skip to content

Test using Pluma

Welma and the demos can be tested using the Pluma test automation tool.

Pluma reference and installation guide:

Prerequisites

The Device Under Test (DUT, either an emulated board or a real board) must have an interface on which the test automation tool can communicate (eg: Serial port, IP Network, etc.).

On Welma, development images are tested, with SSH:

  • DUT instrumented with SSH access (ie: development image), powered up and running
  • Host configured with SSH access to the DUT

Pluma runs ssh, which honors ~/.ssh/config. Example:

  • .ssh/config 

    Host qemu
    Hostname fe80::1%%enx0050b6b107a3
    User root
    Port 22
    HostbasedAuthentication no
    StrictHostKeyChecking no

  • target.yml 

    system:
  prompt_regex: "qemuarm-welma:~# "
  boot_timeout: 30s

console:
  ssh:
    target: qemu

Testing the Welma Qemu target with SSH access

Additional System Requirements:

  • qemu-system-arm
  • Listening TCP port 2222 available (used by Pluma for SSH redirection)

Setup:

  • Have a Yocto image built in TOPDIR.

  • Download the Welma test repository: 

    $ git clone git@gitlab.com:witekio/rnd/theembeddedkit/welma/welma-test.git

Run Pluma:

$ cd welma-test/machine/qemuarm-welma

$ export MACHINE=qemuarm-welma
$ export TESTFILES=$PWD/../../files
$ export TOPDIR=.../build
$ export DEPLOYDIR=$TOPDIR/tmp/deploy/images/$MACHINE
$ export UBOOT=$DEPLOYDIR/u-boot.bin
$ export SDIMG=$DEPLOYDIR/demo-image-headless-dev-$MACHINE.wic
$ export PKG_BOOT=$DEPLOYDIR/demo-image-headless-dev-$MACHINE.boot.swu
$ export PKG_APPRO=$DEPLOYDIR/demo-image-headless-dev-$MACHINE.appro.swu
$ export PKG_SYSRO=$DEPLOYDIR/demo-image-headless-dev-$MACHINE.sysro.swu
$ export SSH_TARGET=device-under-test-qemu

$ pluma -t target.yml -c tests-all.yml
Running Plan "Welma All Tests"...
- Action "Setup: Start Qemu"                     PASS
- Action "Setup: Wait 40 s for the DUT to st...  PASS
- Action "Setup: Check that Qemu is running"     PASS
- Action "set_console"                           PASS
- Action "uname --kernel-release"                PASS
  Check:    ".".join(output.split(".")[:2]...    OK
Test "01. The DUT is running either kernel...    PASS
- Action "Have the demo app fail (SEGV)"         PASS
- Action "Wait a few seconds for the app to ...  PASS
- Action "Check that the demo app is running"    PASS
Test "05. The watchdog restarts the demo a...    PASS
- Action "Create a control sample file in vo...  PASS
- Action "Kill the demo app several times so...  PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "Read the control sample file (shou...  PASS
- Action "Check that the demo app is running"    PASS
Test "06. The watchdog reboots the DUT if ...    PASS
Group "A group of minimal tests"                 PASS
- Action "set_console"                           PASS
- Action "Get a random value"                    PASS
- Action "Create log entries"                    PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 1 ...  PASS
- Action "Check that our previous log messag...  PASS
Test "T910. Persistent storage of logs"          PASS
Group "Miscellaneous tests"                      PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Save initial normal slots"             PASS
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Check that the control file is not...  PASS
- Action "Upload appro1.swu"                     PASS
- Action "Install appro1.swu"                    PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "under-test"               OK
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro has NOT changed"       PASS
- Action "Partition appro has changed"           PASS
- Action "Confirm update"                        PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
- Action "Check that the system is correctly...  PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro has NOT changed"       PASS
- Action "Partition appro has changed"           PASS
- Action "Check that the system still has th...  PASS
- Action "Restore initial state"                 PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
Test "101. Nominal SW Update of APPRO"           PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Install empty file"                    PASS
- Action "Install missing file"                  PASS
- Action "Upload sysro-bad-device.swu"           PASS
- Action "Install sysro-bad-device.swu"          PASS
Test "102. Update install error"                 PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Upload sysro-empty.swu"                PASS
- Action "Install sysro-empty.swu"               PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro is back to initial...  PASS
- Action "Partition appro is back to initial...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
Test "103. Rollback if unbootable sysro"         PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Check that the control file is not...  PASS
- Action "Initialize directory /tmp/TXXX for...  PASS
- Action "Upload SYSRO and APPRO"                PASS
- Action "Install SYSRO and APPRO"               PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "under-test"               OK
- Action "Check that the control file is pre...  PASS
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro has changed"           PASS
- Action "Partition appro has changed"           PASS
- Action "Abort installation"                    PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
- Action "Check that the control file is not...  PASS
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro is back to initial...  PASS
- Action "Partition appro is back to initial...  PASS
Test "104. Aborted SW Update"                    PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Check that the control file is not...  PASS
- Action "Initialize directory /tmp/TXXX for...  PASS
- Action "Upload appro1.swu"                     PASS
- Action "Install appro1.swu"                    PASS
- Action "Abort installation"                    PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
- Action "Check that the control file is NOT...  PASS
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro is the initial value"  PASS
- Action "Partition appro is the initial value"  PASS
Test "105. Update aborted before reboot"         PASS
- Action "set_console"                           PASS
- Action "dbus get ConfirmTimeoutSec"            PASS
  Check:    output.find("variant       int...    OK
- Action "dbus get State"                        PASS
  Check:    output.find("variant       str...    OK
Test "107. Updated DBus interface"               PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Check that the control file is not...  PASS
- Action "Upload appro1.swu"                     PASS
- Action "Install appro1.swu"                    PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "under-test"               OK
- Action "Check that the system is correctly...  PASS
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro has not changed"       PASS
- Action "Partition appro has changed"           PASS
- Action "Wait 2 min without confirming the ...  PASS
- Action "close_console"                         PASS
Could not close device console ssh, console is already closed
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
- Action "Check that the control file has di...  PASS
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro back to initial"       PASS
- Action "Partition appro back to initial"       PASS
Test "110. Unconfirmed update"                   PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Save initial normal slots"             PASS
- Action "Check that the control file is not...  PASS
- Action "Upload appro1.swu"                     PASS
- Action "Install appro1.swu"                    PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "under-test"               OK
- Action "Check that the system is correctly...  PASS
- Action "Confirm update"                        PASS
- Action "Wait 2 min"                            PASS
- Action "Check that the control file is sti...  PASS
- Action "Restore initial state"                 PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
Test "111. Confirmed update, wait"               PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Save initial active partitions"        PASS
- Action "Initialize directory /tmp/TXXX for...  PASS
- Action "Upload BOOT"                           PASS
- Action "Install BOOT"                          PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "Get active partitions"                 PASS
- Action "Check that the 1st active partitio...  PASS
- Action "Check that the 2nd active partitio...  PASS
- Action "Check that the 3rd active partitio...  PASS
- Action "Confirm installation of BOOT"          PASS
- Action "Initialize directory /tmp/TXXX for...  PASS
- Action "Upload SYSRO"                          PASS
- Action "Install SYSRO"                         PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "Get active partitions"                 PASS
- Action "Check that the 1st active partitio...  PASS
- Action "Check that the 2nd active partitio...  PASS
- Action "Check that the 3rd active partitio...  PASS
- Action "Confirm installation of SYSRO"         PASS
- Action "Initialize directory /tmp/TXXX for...  PASS
- Action "Upload APPRO"                          PASS
- Action "Install APPRO"                         PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "Get active partitions"                 PASS
- Action "Check that the 1st active partitio...  PASS
- Action "Check that the 2nd active partitio...  PASS
- Action "Check that the 3rd active partitio...  PASS
- Action "Confirm installation of APPRO"         PASS
- Action "Initialize directory /tmp/TXXX for...  PASS
- Action "Upload BOOT, SYSRO and APPRO"          PASS
- Action "Install BOOT, SYSRO and APPRO"         PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "Get active partitions"                 PASS
- Action "Check that all active partitions a...  PASS
- Action "Confirm installation"                  PASS
Test "115. Update slots one by one"              PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Save initial normal slots"             PASS
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Check that the control file is not...  PASS
- Action "Upload appro2.gz.swu"                  PASS
- Action "Install appro2.gz.swu"                 PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "under-test"               OK
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro has NOT changed"       PASS
- Action "Partition appro has changed"           PASS
- Action "Confirm update"                        PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
- Action "Check that the system is correctly...  PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
- Action "Get active partition: sysro"           PASS
- Action "Get active partition: appro"           PASS
- Action "Partition sysro has NOT changed"       PASS
- Action "Partition appro has changed"           PASS
- Action "Check that the system still has th...  PASS
- Action "Restore initial state"                 PASS
- Action "reboot"                                PASS
- Action "Wait for the DUT to reboot (max 2 ...  PASS
Test "116. Nominal SW Update of APPRO with...    PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Upload appro3-unsigned.swu"            PASS
- Action "Install appro3-unsigned.swu"           PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
Test "120. Install error with unsigned APPRO"    PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Upload appro4-compromised.swu"         PASS
- Action "Install appro4-compromised.swu"        PASS
- Action "updatectl status state"                PASS
  Check:    output == "normal"                   OK
Test "121. Install compromised APPRO"            PASS
- Action "set_console"                           PASS
- Action "Verify that the current boot is 'n...  PASS
  Check:    output == "normal"                   OK
- Action "Upload /home/fhoerni/projects/disk...  PASS
- Action "Upload /home/fhoerni/projects/disk...  PASS
- Action "Run the double-install test case"      PASS
- Action "Run the double-install-dbus test c...  PASS
- Action "Run the status-while-install test ...  PASS
- Action "Run the confirm-while-install test...  PASS
- Action "Run the submit-while-install test ...  PASS
- Action "Run the double-abort test case"        PASS
- Action "Wait for the system to be ready fo...  PASS
- Action "Run the abort-while-install test c...  PASS
- Action "Wait for the system to be ready fo...  PASS
- Action "Check the update status"               PASS
  Check:    output == "normal"                   OK
Test "122. Parallel install test"                PASS
Group "Swupdate"                                 PASS
- Action "Teardown: Stop Qemu"                   PASS
Plan "Welma All Tests"                           PASS