LAVA is generic and will just display results as generic results - LAVA does not have any specific intelligence to know how to do anything more or how to make those results meaningful to the developers who originally caused the CI system to spawn the test job.
The stages in light blue, (analyse, develop and commit) are the developer input. (The develop stage is here assumed to include some amount of local testing). Building the tree after a commit is typically handed off to a dedicated service like Jenkins. Jenkins in turn is able to hand off the testing of the build to LAVA.
The missing step here is the reporting. Each team needs customised reports and data based on custom criteria. LAVA cannot meet all these requirements at the same time. Instead, teams are advised to design a custom frontend which collects data from LAVA, the build system and the VCS and presents that data in a cohesive manner.
If the development team is very small, the entire team may be able to retain an overview of the entire CI system. This allows the team to get useful, relevant results directly. However, this model does not scale to larger teams.
Time - developers need test results in a timely fashion, before they have moved on to a completely unrelated topic.
Relevance - the results which the developer sees need to be directly relevant to the work which started the CI loop in the first place.
this can easily involve multiple variants of the final results report to cover the various topics which that development team needs to handle.
Usefulness - this includes not sending reports that everything is working when only failure reports are useful, or sending reports that only hint at the problem rather than provide access to the actual fault. This can be the hardest element to get right. The more time is spent here talking to the development team, the better the process will work for everyone.
Build tools like Jenkins can also do an amount of testing on the built files, for example unit tests. On the basis of always optimising the CI loop to fail early, it is always worth balancing the number of tests run after the build against how long those tests take to run. It may be pointless to test a build in LAVA when that a unit test on that build would have failed. It is also possible to execute the build, submit to LAVA and run the unit tests as a new Jenkins job in parallel if the unit tests are slow.
LAVA is best suited for those tests where the hardware is directly relevant. In some cases, the machines used for building the files will be faster than the machines used to test the files in LAVA. If those files can be tested on the faster build machine, omit that part of the testing from the submission to LAVA.
LAVA combines the benefits of ready access to a multiple types of device with a genuinely scalable scheduler. LAVA is capable of running thousands of test jobs a day across hundreds of devices on a single instance. With a custom frontend organising the submissions and collating the results, this can scale to larger groups using multiple LAVA instances.
Not all tests need to be run on every commit. Identify which tests can be run on a daily or weekly cycle or as a bespoke per-release test.
It is not necessarily appropiate for all commits to go through the entire CI loop. The hook in the version control system which triggers the Jenkins build could be based on merges rather than commits.
KernelCI is a build and boot automation tool for upstream Linux kernel trees. Under the hood, kernelci uses LAVA alongside other automation systems. The LAVA workload is based on booting each build of the kernel with a known working rootfs on as many devices as possible. KernelCI schedules builds of supported kernel configurations, then submits those builds to test instances. It imports the test results and generates a user interface which is specific to the needs of the upstream Linux kernel developer teams.
Development of KernelCI started in 2013, gathering the requirements from the kernel developers. This included a number of sessions covering what the developers wanted and needed from the project.
The specific details of the interface of KernelCI may not be directly relevant to other development teams, but it is a good example of the kind of custom frontend that the LAVA team recommend. Specific frontends may differ, but the ideas are common - using the results from LAVA effectively, targeting the needs of the development team.
One important feature from KernelCI is that the devices themselves are development boards and can fail for reasons other than the build being tested. Mitigating this problem requires a balance of having enough devices to smooth out the anomalous results against the risk of missing an unusual corner case which genuinely only affects devices in a specific set of circumstances.