MultiNode API¶

Guidance in using the API¶

It is recommended to avoid doing a lot of calculation within the calls to the API. There are times when a script is needed to retrieve data from the test shell but resist the temptation to run that script in the call to the API. Always check the output of the script (e.g. with lava-test-case) and/or run the script separately in the test definition run steps so that the output appears in the test job logs. Preparing and outputting the data before sending it with the API will aid in debugging the test definition.

lava-self¶

lava-role¶

$ ./run-$(lava-role)

#!/bin/sh
for role in `lava-role list`; do
    echo $role
done

lava-group¶

12345     client
12346     loadbalancer
12347     backend
12348     backend

$ lava-group client
12345
$ lava-group backend
12347
12348

$ lava-group server ; echo $?
1

- lava-test-case check-server-role --shell lava-group server

#!/bin/sh
for device in `lava-group backend`; do
    echo $device
done

lava-send¶

Sends a message to the group, optionally passing associated key-value data pairs. Sending a message is a non-blocking operation. The message is guaranteed to be available to all members of the group, but some of them might never retrieve it.

The message-id will be persistent for the lifetime of the target group managing the entire multinode test job. Re-sending a different message with an existing message-id is not supported.

lava-wait¶

Waits until any other device in the group sends a message with the given ID. This call will block until such message is sent.

lava-wait-all¶

lava-wait-all operates in two distinct ways - with or without a role.

lava-wait-all <message-id> [<role>]

If data was sent by the other devices with the message, the key-value pairs will be printed in the cache file (/tmp/lava_multi_node_cache.txt in default), each in one line, prefixed with the target name and a colon.

Some examples for lava-send, lava-wait and lava-wait-all are given below.

The message returned can include data from other devices which sent a message with the relevant message ID, only the wait is dependent on particular devices with a specified role.

As with lava-wait, the message ID is persistent for the duration of the MultiNode group.

lava-sync¶

Global synchronization primitive. Sends a message, and waits for the same message from all of the other devices.

Example 1: simple client-server MultiNode test¶

Two devices, with roles client, server

LAVA Test Shell test definition (say, example1.yaml):

run:
    steps:
        - ./run-`lava-role`.sh

The test image or the test definition would then provide two scripts, with only one being run on each device, according to the role specified.

run-server.sh:

#!/bin/sh

SPACE=`df -h | grep "/$" | awk '{print $4}'`
echo $SPACE
lava-send server-ready free-space=$SPACE

Notes:

• To make use of the server-ready message, some kind of client needs to do a lava-wait server-ready

run-client.sh:

#!/bin/sh

lava-wait server-ready
free-space=$(cat /tmp/lava_multi_node_cache.txt | cut -d = -f 2)
echo "The free disk space on server is ${free-space}"

Notes:

• The client waits for the server-ready message then get the data which was sent by server from /tmp/lava_multi_node_cache.txt

Example 2: iperf client-server test¶

Two devices, with roles client, server

LAVA Test Shell test definition (say, example1.yaml):

run:
    steps:
        - ./run-`lava-role`.sh

The test image or the test definition would then provide two scripts, with only one being run on each device, according to the role specified.

run-server.sh:

#!/bin/sh

iperf -s &
echo $! > /tmp/iperf-server.pid
IP=`ip route get 8.8.8.8 | head -n 1 | awk '{print $NF}'`
echo $IP
lava-send server-ready server-ip=$IP
lava-wait client-done
kill -9 `cat /tmp/iperf-server.pid`

Notes:

• iperf server process needs to be run in the background to wait for the connection from the client and the process id will be stored somewhere for later use.
• To make use of the server-ready message, some kind of client needs to do a lava-wait server-ready
• There needs to be a support on a client to do the lava-send client-done or the wait will fail on the server.
• If there was more than one client, the server could call lava-wait-all client-done instead.
• iperf server process must be killed after getting client-done message, otherwise the test job will not proceed.

run-client.sh:

#!/bin/sh

lava-wait server-ready
server=$(cat /tmp/lava_multi_node_cache.txt | cut -d = -f 2)
iperf -c $server
# ... do something with output ...
lava-send client-done

Notes:

• The client waits for the server-ready message as it’s first task, then does some work, then sends a message so that the server can move on and do other tests.

Example 3: variable number of clients¶

run-server.sh:

#!/bin/sh

start-server
lava-sync ready
lava-sync done

run-client.sh:

#!/bin/sh

# refer to the server by name, assume internal DNS works
server=$(lava-group | grep 'server$' | cut -f 1)

lava-sync ready
run-client
lava-sync done

Example 4: peer-to-peer application¶

Single role: peer, any number of devices

run-peer.sh:

#!bin/sh

initialize-data
start-p2p-service
lava-sync running

push-data
for peer in $(lava-group | cut -f 1); then
    if [ $peer != $(lava-self) ]; then
        query-data $peer
    fi
fi

Using a flow table to plan the job¶

Synchronization of any type needs to be planned and the simplest way to manage the messages between roles within a group is to set out a strict table of the flow.

Set out the call and leave blank rows until that call is matched by the appropriate roles, to represent the time that the devices with that role will block in a wait loop with the coordinator.

In this overly simplistic table, the Observer role really has nothing useful to do but to demonstrate that it will spend most of it’s time in lava-sync fin.

All roles will wait in lava-sync start until all deploy and boot operations (or whatever other tasks are put ahead of the call to lava-sync) are complete. The flow table does not include this delay.

The Server role runs a script to start a service, sending ready when the script returns.

The Client role waits until all devices with the Server role have completed lava-send ready - Observer is unaffected and Server moves directly into the lava-sync fin. Once the Client completes lava-wait-all ready server, the Client can run the client tasks script. That script finally puts the devices with the Client role into lava-sync fin at which point, the Client role receives the message that everyone else is already in that sync, the sync completes and the flow table ends.

Tables like this also help visualize how long the timeouts need to be to allow the Observer role to wait for all the server tasks and all the client tasks to complete.