kernelci.org update - 2020-09-23 #minutes


Guillaume Tucker
 

Summary of changes going into production
========================================

* fix branch names with slash characters "/"
* enable stable-rc queue/* branches
* enable stable 5.8 branches
* enable soc arm/fixes branch
* build linux-next with clang-10, drop clang-9
* use Linaro test-definitions for kselftest
* enable kselftest to run on a few initial devices
* add direct links to regressions on web dashboard
* improve log filtering to remove more LAVA messages


Technical Steering Committee minutes - 2020-09-08
=================================================

* LAVA test-definitions repository and version control

* No new tags upstream since “2019.11”, should we ask Linaro about restarting
that?
* Should we make a fork in kernelci GitHub with kernelci.org branch and
kernelci tags?
* Show we accept to use the head of upstream master branch, even if this
means hard-to-reproduce issues and unexpected change in results if new
commits are pushed at any time?

-> create fork for staging initially, and try to use upstream in production

* Finally fixed support for slashes in branch names, useful for stable-rc queue
branches in particular

* LLVM/CLang: still waiting for v11 to be released

* LAVA log filtering: tested on staging - seems to work fine

* Login test case fix: implemented more thorough testing needed

* preempt-rt: Linaro test-definitions repo getting updates from Daniel Wagner

* KCIDB

JSON stream parsing is implemented and tested in jq.py, upstreaming is in
progress. Still takes four times as long and uses twice the memory as the
stock JSON parser, but does streams and should be good enough for now.
Starting implementing report stream parsing in KCIDB.

* Notes: rework test email reports to show number of regressions in table
-> https://github.com/kernelci/kernelci-backend/issues/257


Technical Steering Committee minutes - 2020-09-15
=================================================

* LLVM/Clang

LLVM people not super interested in older clang versions or even clang-10.
Once clang-9 is out, when Mark updates to clang-11 he’ll send something also
dropping the older clang versions once clang-11 is live. Can reevaluate this
policy once there are stable distros with clang.

Nick maintains a clang-latest docker image, we should into integrating that
for potential inclusion in linux-next coverage - will need some evaluation to
see for example how noisy it is and if we need to do something about
segregating results for the bleeding edge compiler.

LLVM 11 release:
* Status tracked at https://bugs.llvm.org/show_bug.cgi?id=46725
* Several pending bugs need fixing, some look relevant

Android LLVM versions: Mark to ping Todd and ask him about using those for
Android branches.

Testing with clang-12: could take a short cut and deploy on staging rather
than sorting out fancier reporting

* KCIDB

Half of JSON stream parsing is merged into jq.py. Another half remains,
pending on the maintainer's attention.

KCIDB input stream parsing implemented locally, to be tested, and waiting on
the jq.py PR above.

KCIDB output splitting next.

* Web dashboard

Improving web UI to differentiate skips and tests that have always failed
See on https://staging.kernelci.org
(pie charts and small things left to tweak)

* Using Linaro test-definitions for LAVA jobs

Created test-definitions fork for kernelci.org and staging.kernelci.org
kernelci.org branch updated weekly with prod update
staging.kernelci.org branch updated with open PRs like other projects

* kselftests: build errors mixed with main kernel build

Long-term solution would be to have separate stages (kernel build, kselftest
build, runtime) with dependencies and pass/fail status

Short-term solution and necessary step is to build kselftests as a separate
“make” command and keep the output in a separate log file (like tuxmake does)

* login test case false-positives:

On LAVA 2020.08 kernel panic is detected, test case status marked as failed
LAVA job Incomplete


Technical Steering Committee minutes - 2020-09-22
=================================================

* KCIDB

The jq.py maintainer is slow to respond and keeps making requests, official
streaming support doesn't seem close anymore, so I made a release for
temporary use in my personal fork.

JSON streaming support is merged into KCIDB - integration work can be
started! You can now write reports one after another into any tool accepting
reports. Submitting 1000 reports containing 8000 objects through
kcidb-submit takes about 35 seconds now.

Kcidb-query, kcidb-db-query, and kcidb-db-dump now accept the
'-o/--objects-per-report' option specifying how many objects should be put
into each output JSON report. When that option is used, they can output
multiple reports.

All tools outputting JSON can be asked to not pretty-print it (with
'--indent=0'), outputting single-line, or to prepend each report with the RS
character (with '--seq'), complying with RFC 7464, either of which could be
easier to process with command-line tools.

Next KCIDB release soon. Theme: JSON streaming support.

Cristian Marussi from ARM is setting up sending their CI results, and managed
to push a bunch to the playground setup - looks great!

* Plumbers 2020 KernelCI blog post: ready to get published this week

* KernelCI TSC plan: Starting to make plan for 2020-Q4

* E-mail regression reports: fixing formatting when more than one regression

* User experience:

KernelCI LF board starting to brainstorm around next-gen dashboard /
visualization / analytics to potentially fund a 3rd party to develop web
tooling

* "unknown" failures on web UI

https://groups.io/g/kernelci/topic/user_interface_issues_with/76927781?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,76927781

-> Yes this looks very much like the web frontend issue described before,
being fixed right now (separating regressions, always-fail and skips)
https://github.com/kernelci/kernelci-frontend/pull/125

* EFI on QEMU

u-boot with EFI extensions on arm/arm64 to get coverage for the EFI boot
paths in the kernel - Might just do this via the FVP.

* clang-11

Can merge the docker update, will need to rebuild when the final LLVM 11
release lands (hopefully RSN, the tracking bug looks to have mostly no
dependencies).


Advisory Board minutes - 2020-09-16
===================================

* LPC recorded talks are now on YouTube, time to tweet about them

* Discussing priorities for 2020-Q4 TSC plan:

Native tests: kselftest, LTP, KUnit, device tree validation
KCIDB with production data from kernelci.org
“Polishing”: Improving docs, fixing long-standing bugs, LF project PDFs…
Metrics: discussed many times, now we should start implementing it
CIP KernelCI instance: already discussed, also needs some action now

* Discussing next important topics, for 2021

Define some work packages / RFQs
Testing patches from mailing lists
Improve regressions tracking for native KernelCI tests


Best wishes,
Guillaume


Nick Desaulniers
 

On Wed, Sep 23, 2020 at 7:50 AM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:

Summary of changes going into production
========================================

* fix branch names with slash characters "/"
* enable stable-rc queue/* branches
* enable stable 5.8 branches
* enable soc arm/fixes branch
* build linux-next with clang-10, drop clang-9
* use Linaro test-definitions for kselftest
* enable kselftest to run on a few initial devices
* add direct links to regressions on web dashboard
* improve log filtering to remove more LAVA messages


Technical Steering Committee minutes - 2020-09-08
=================================================

* LAVA test-definitions repository and version control

* No new tags upstream since “2019.11”, should we ask Linaro about restarting
that?
* Should we make a fork in kernelci GitHub with kernelci.org branch and
kernelci tags?
* Show we accept to use the head of upstream master branch, even if this
means hard-to-reproduce issues and unexpected change in results if new
commits are pushed at any time?

-> create fork for staging initially, and try to use upstream in production

* Finally fixed support for slashes in branch names, useful for stable-rc queue
branches in particular

* LLVM/CLang: still waiting for v11 to be released

* LAVA log filtering: tested on staging - seems to work fine

* Login test case fix: implemented more thorough testing needed

* preempt-rt: Linaro test-definitions repo getting updates from Daniel Wagner

* KCIDB

JSON stream parsing is implemented and tested in jq.py, upstreaming is in
progress. Still takes four times as long and uses twice the memory as the
stock JSON parser, but does streams and should be good enough for now.
Starting implementing report stream parsing in KCIDB.

* Notes: rework test email reports to show number of regressions in table
-> https://github.com/kernelci/kernelci-backend/issues/257


Technical Steering Committee minutes - 2020-09-15
=================================================

* LLVM/Clang

LLVM people not super interested in older clang versions or even clang-10.
clang-10 is the minimally supported version of Clang for building
Linux kernels; we do very much care about it:
https://lore.kernel.org/lkml/20200902225911.209899-1-ndesaulniers@google.com/T/#m61957eaada46dc8c51fdf2010859eb1976227005

Once clang-9 is out, when Mark updates to clang-11 he’ll send something also
dropping the older clang versions once clang-11 is live. Can reevaluate this
policy once there are stable distros with clang.

Nick maintains a clang-latest docker image, we should into integrating that
for potential inclusion in linux-next coverage - will need some evaluation to
see for example how noisy it is and if we need to do something about
segregating results for the bleeding edge compiler.
Nathan maintains it more than I do:
https://github.com/ClangBuiltLinux/dockerimage


LLVM 11 release:
* Status tracked at https://bugs.llvm.org/show_bug.cgi?id=46725
* Several pending bugs need fixing, some look relevant
Fixed! (oh boy, we were on the "chase list")


Android LLVM versions: Mark to ping Todd and ask him about using those for
Android branches.
android-mainline
android12-5.4
android-4.19-stable

are the 3 newest branches that I know of.


Testing with clang-12: could take a short cut and deploy on staging rather
than sorting out fancier reporting

* KCIDB

Half of JSON stream parsing is merged into jq.py. Another half remains,
pending on the maintainer's attention.

KCIDB input stream parsing implemented locally, to be tested, and waiting on
the jq.py PR above.

KCIDB output splitting next.

* Web dashboard

Improving web UI to differentiate skips and tests that have always failed
See on https://staging.kernelci.org
(pie charts and small things left to tweak)

* Using Linaro test-definitions for LAVA jobs

Created test-definitions fork for kernelci.org and staging.kernelci.org
kernelci.org branch updated weekly with prod update
staging.kernelci.org branch updated with open PRs like other projects

* kselftests: build errors mixed with main kernel build

Long-term solution would be to have separate stages (kernel build, kselftest
build, runtime) with dependencies and pass/fail status

Short-term solution and necessary step is to build kselftests as a separate
“make” command and keep the output in a separate log file (like tuxmake does)

* login test case false-positives:

On LAVA 2020.08 kernel panic is detected, test case status marked as failed
LAVA job Incomplete


Technical Steering Committee minutes - 2020-09-22
=================================================

* KCIDB

The jq.py maintainer is slow to respond and keeps making requests, official
streaming support doesn't seem close anymore, so I made a release for
temporary use in my personal fork.

JSON streaming support is merged into KCIDB - integration work can be
started! You can now write reports one after another into any tool accepting
reports. Submitting 1000 reports containing 8000 objects through
kcidb-submit takes about 35 seconds now.

Kcidb-query, kcidb-db-query, and kcidb-db-dump now accept the
'-o/--objects-per-report' option specifying how many objects should be put
into each output JSON report. When that option is used, they can output
multiple reports.

All tools outputting JSON can be asked to not pretty-print it (with
'--indent=0'), outputting single-line, or to prepend each report with the RS
character (with '--seq'), complying with RFC 7464, either of which could be
easier to process with command-line tools.

Next KCIDB release soon. Theme: JSON streaming support.

Cristian Marussi from ARM is setting up sending their CI results, and managed
to push a bunch to the playground setup - looks great!

* Plumbers 2020 KernelCI blog post: ready to get published this week

* KernelCI TSC plan: Starting to make plan for 2020-Q4

* E-mail regression reports: fixing formatting when more than one regression

* User experience:

KernelCI LF board starting to brainstorm around next-gen dashboard /
visualization / analytics to potentially fund a 3rd party to develop web
tooling

* "unknown" failures on web UI

https://groups.io/g/kernelci/topic/user_interface_issues_with/76927781?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,76927781

-> Yes this looks very much like the web frontend issue described before,
being fixed right now (separating regressions, always-fail and skips)
https://github.com/kernelci/kernelci-frontend/pull/125

* EFI on QEMU

u-boot with EFI extensions on arm/arm64 to get coverage for the EFI boot
paths in the kernel - Might just do this via the FVP.

* clang-11

Can merge the docker update, will need to rebuild when the final LLVM 11
release lands (hopefully RSN, the tracking bug looks to have mostly no
dependencies).


Advisory Board minutes - 2020-09-16
===================================

* LPC recorded talks are now on YouTube, time to tweet about them

* Discussing priorities for 2020-Q4 TSC plan:

Native tests: kselftest, LTP, KUnit, device tree validation
KCIDB with production data from kernelci.org
“Polishing”: Improving docs, fixing long-standing bugs, LF project PDFs…
Metrics: discussed many times, now we should start implementing it
CIP KernelCI instance: already discussed, also needs some action now

* Discussing next important topics, for 2021

Define some work packages / RFQs
Testing patches from mailing lists
Improve regressions tracking for native KernelCI tests


Best wishes,
Guillaume





--
Thanks,
~Nick Desaulniers


Guillaume Tucker
 

On 23/09/2020 22:08, Nick Desaulniers wrote:
On Wed, Sep 23, 2020 at 7:50 AM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:

Summary of changes going into production
========================================

* fix branch names with slash characters "/"
* enable stable-rc queue/* branches
* enable stable 5.8 branches
* enable soc arm/fixes branch
* build linux-next with clang-10, drop clang-9
^ more on this below :)

* use Linaro test-definitions for kselftest
* enable kselftest to run on a few initial devices
* add direct links to regressions on web dashboard
* improve log filtering to remove more LAVA messages


Technical Steering Committee minutes - 2020-09-08
=================================================

* LAVA test-definitions repository and version control

* No new tags upstream since “2019.11”, should we ask Linaro about restarting
that?
* Should we make a fork in kernelci GitHub with kernelci.org branch and
kernelci tags?
* Show we accept to use the head of upstream master branch, even if this
means hard-to-reproduce issues and unexpected change in results if new
commits are pushed at any time?

-> create fork for staging initially, and try to use upstream in production

* Finally fixed support for slashes in branch names, useful for stable-rc queue
branches in particular

* LLVM/CLang: still waiting for v11 to be released

* LAVA log filtering: tested on staging - seems to work fine

* Login test case fix: implemented more thorough testing needed

* preempt-rt: Linaro test-definitions repo getting updates from Daniel Wagner

* KCIDB

JSON stream parsing is implemented and tested in jq.py, upstreaming is in
progress. Still takes four times as long and uses twice the memory as the
stock JSON parser, but does streams and should be good enough for now.
Starting implementing report stream parsing in KCIDB.

* Notes: rework test email reports to show number of regressions in table
-> https://github.com/kernelci/kernelci-backend/issues/257


Technical Steering Committee minutes - 2020-09-15
=================================================

* LLVM/Clang

LLVM people not super interested in older clang versions or even clang-10.
clang-10 is the minimally supported version of Clang for building
Linux kernels; we do very much care about it:
https://lore.kernel.org/lkml/20200902225911.209899-1-ndesaulniers@google.com/T/#m61957eaada46dc8c51fdf2010859eb1976227005
Quite, we're now building linux-next with Clang 10 on
kernelci.org (production) and unreleased versions on
staging.kernelci.org.

Once clang-9 is out, when Mark updates to clang-11 he’ll send something also
dropping the older clang versions once clang-11 is live. Can reevaluate this
policy once there are stable distros with clang.

Nick maintains a clang-latest docker image, we should into integrating that
for potential inclusion in linux-next coverage - will need some evaluation to
see for example how noisy it is and if we need to do something about
segregating results for the bleeding edge compiler.
Nathan maintains it more than I do:
https://github.com/ClangBuiltLinux/dockerimage
Great, I think Mark wanted to take a look into it.

At the moment we're building with snapshots of the next Clang
version on staging.kernelci.org (clang-11, soon clang-12). It
would be nice to be able to use this container and test the very
latest instead.

LLVM 11 release:
* Status tracked at https://bugs.llvm.org/show_bug.cgi?id=46725
* Several pending bugs need fixing, some look relevant
Fixed! (oh boy, we were on the "chase list")
Nice! Erm, I believe there's one last blocker:

https://bugs.llvm.org/show_bug.cgi?id=47619

Android LLVM versions: Mark to ping Todd and ask him about using those for
Android branches.
android-mainline
android12-5.4
android-4.19-stable

are the 3 newest branches that I know of.
And which LLVM/Clang versions should be used to build those?


Thanks,
Guillaume


Nick Desaulniers
 

On Wed, Sep 23, 2020 at 2:26 PM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:

On 23/09/2020 22:08, Nick Desaulniers wrote:
On Wed, Sep 23, 2020 at 7:50 AM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:
LLVM 11 release:
* Status tracked at https://bugs.llvm.org/show_bug.cgi?id=46725
* Several pending bugs need fixing, some look relevant
Fixed! (oh boy, we were on the "chase list")
Nice! Erm, I believe there's one last blocker:

https://bugs.llvm.org/show_bug.cgi?id=47619
I'll let the release manager sort that out.


Android LLVM versions: Mark to ping Todd and ask him about using those for
Android branches.
android-mainline
android12-5.4
android-4.19-stable

are the 3 newest branches that I know of.
And which LLVM/Clang versions should be used to build those?
Will likely be clang-12 (or possibly clang-13). Those branches are
basically the kernel branches for "S" which will be released next
year. So they are building with a pre-clang-11 right now, but we're
about to upgrade them to something closer to ToT soon. Once the "S"
release of Android is closer, then we will lock down the toolchain
used for these kernels, but it's sure to change between now and then.
--
Thanks,
~Nick Desaulniers


Mark Brown
 

On Wed, Sep 23, 2020 at 03:06:52PM -0700, Nick Desaulniers via groups.io wrote:
On Wed, Sep 23, 2020 at 2:26 PM Guillaume Tucker
android-mainline
android12-5.4
android-4.19-stable
are the 3 newest branches that I know of.
And which LLVM/Clang versions should be used to build those?
Will likely be clang-12 (or possibly clang-13). Those branches are
basically the kernel branches for "S" which will be released next
year. So they are building with a pre-clang-11 right now, but we're
about to upgrade them to something closer to ToT soon. Once the "S"
release of Android is closer, then we will lock down the toolchain
used for these kernels, but it's sure to change between now and then.
The thought was that it might be worth using the actual Android
toolchains (including all the out of tree patches and whatnot) with
these branches as well as upstream clang.


Nick Desaulniers
 

On Thu, Sep 24, 2020 at 4:05 AM Mark Brown <broonie@kernel.org> wrote:

On Wed, Sep 23, 2020 at 03:06:52PM -0700, Nick Desaulniers via groups.io wrote:
On Wed, Sep 23, 2020 at 2:26 PM Guillaume Tucker
android-mainline
android12-5.4
android-4.19-stable
are the 3 newest branches that I know of.
And which LLVM/Clang versions should be used to build those?
Will likely be clang-12 (or possibly clang-13). Those branches are
basically the kernel branches for "S" which will be released next
year. So they are building with a pre-clang-11 right now, but we're
about to upgrade them to something closer to ToT soon. Once the "S"
release of Android is closer, then we will lock down the toolchain
used for these kernels, but it's sure to change between now and then.
The thought was that it might be worth using the actual Android
toolchains (including all the out of tree patches and whatnot) with
these branches as well as upstream clang.
That's...a brilliant idea, and would be much closer to what's being
used in production.

While Android ships its own distribution of LLVM; this is because
upstream LLVM's release process is simultaneously too slow for Android
(we need fixes and features faster) and would be too late for finding
compiler regressions (that's pretty much all we do, is fight
regressions). While there's potential there to try to ship secret
sauce, I *wish* we had the headcount to even consider that; our
distribution will carry cherry picks of reverts or backports of fixes
so that we minimize the number of known issues with the release. We
run at a manpower deficit, so we don't have the ability to even
develop any kind of secret sauce.

Would a Docker image containing AOSP LLVM probably be the easiest to
integrate? If so, I can look at providing such an image.
--
Thanks,
~Nick Desaulniers


Mark Brown
 

On Thu, Sep 24, 2020 at 12:25:42PM -0700, Nick Desaulniers wrote:
On Thu, Sep 24, 2020 at 4:05 AM Mark Brown <broonie@kernel.org> wrote:
The thought was that it might be worth using the actual Android
toolchains (including all the out of tree patches and whatnot) with
these branches as well as upstream clang.
Would a Docker image containing AOSP LLVM probably be the easiest to
integrate? If so, I can look at providing such an image.
Yes, a docker image would be great - ideally Debian based. Our current
docker images are built from:

https://github.com/kernelci/kernelci-core/tree/master/jenkins/dockerfiles

FWIW.


Sedat Dilek <sedat.dilek@...>
 

On Wed, Sep 23, 2020 at 11:26 PM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:

On 23/09/2020 22:08, Nick Desaulniers wrote:
On Wed, Sep 23, 2020 at 7:50 AM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:

Summary of changes going into production
========================================
...

LLVM 11 release:
* Status tracked at https://bugs.llvm.org/show_bug.cgi?id=46725
* Several pending bugs need fixing, some look relevant
Fixed! (oh boy, we were on the "chase list")
Nice! Erm, I believe there's one last blocker:

https://bugs.llvm.org/show_bug.cgi?id=47619
Fixed in release/11.x Git by...

commit 81eb1c1fa75c6407713b5657156d8d9149572bfe
"AArch64/GlobalISel: Reduced patch for bug 47619"

- Sedat -

[1] https://github.com/llvm/llvm-project/commit/81eb1c1fa75c6407713b5657156d8d9149572bfe