Kernel CI setup


Guillaume Tucker
 

+kernelci@groups.io

Hello,

On 06/04/2021 05:47, Krishnan, Neelima wrote:

Guillaume,

 

I am trying to setup the kernel ci local instance in my lab, as per instructions from: https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance
This documentation should still be mostly correct although it
might have a few inaccuracies and gaps to fill.  Also, the
installation steps are quite lengthy and complicated even for
someone familiar with the code.  Having new people like you
trying to install it and providing feedback is a great way of
improving things.

I am not using dockers for installation. Instead I am trying out the VM method. I was able to use the kernelci-backend-config git to install the backend on a VM. Now I am trying to generate the tokens as per instructions in the INSTALL.md @ https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance

 

Here is what I get.. This is not a proxy issue. I am not sure how to move forward. Can you please advise on how to proceed

 

* Connected to api.mydomain.local (127.0.0.1) port 8888 (#0)

POST /token HTTP/1.1
Host: api.mydomain.local:8888
User-Agent: curl/7.64.0
Accept: */*
Content-Type: application/json
Authorization: master-key
Content-Length: 63
 
* upload completely sent off: 63 out of 63 bytes

< HTTP/1.1 403 Operation not permitted: provided token is not authorized

< Content-Length: 12

< Vary: Accept-Encoding

< Server: TornadoServer/3.2.2

< Date: Tue, 06 Apr 2021 01:45:25 GMT

< Access-Control-Allow-Headers: authorization

< Content-Type: application/json; charset=UTF-8



* Connection #0 to host api.mydomain.local left intact
A few things to check:

** Is the kernelci-backend server process running?*

To start it manually (for example, on port 5001):

cd kernelci-backend/app
python server.py --port=5001

To start the Celery process manually:

cd kernelci-backend/app
sudo python \
    -OO \
    -R \
    /srv/.venv/kernelci-backend/bin/celery worker \
    -Ofair \
    --without-gossip \
    -autoscale=24,6 \
    --loglevel=INFO \
    --app=taskqueue \
    --pidfile=/tmp/kernelci-celery.pid

If you start them manually, you'll see the logs directly in the
terminals and that should help debugging issues.  On a real
deployment, these services are typically run with systemd and the
backend is behind a web server such as nginx.

** Is the Mongo DB service running?*

$ sudo systemctl | grep mongo
mongod.service  loaded active running   MongoDB Database Server
*
*** Which API tokens have been created?*

$ mongo kernel-ci
db['api-token'].find({}, {token: 1, properties: 1})
{ "_id" : ObjectId("5bc51d7bb8d49ee75dbd5a6e"), "properties" : [ 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0 ], "token" : "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" }
{ "_id" : ObjectId("5dd80903469d6ddc79e7f7cc"), "token" : "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "properties" : [ 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ] }
*
*** Can the token be used with the API?*

$ curl -X GET -H "Authorization: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" http://localhost:5001/version
{"code":200,"result":[{"full_version":"2020.11.2","version":"2020.11.2"}]}

If all these things work, then it's possible to start sending
build and test results to it manually.  Setting up a frontend
instance would be a logical next step, and potentially Jenkins to
have a fully automated pipeline although that's not really needed
in a local development setup.

Hope this helps!

Best wishes,
Guillaume


 


Krishnan, Neelima <neelima.krishnan@...>
 

Hi Guillaume,

 

Thank you for the quick reply. I was able to go forward with backend installation. The problem was setting of the url I had set in the curl command.

curl -v -XPOST -H "Content-Type: application/json" -H "Authorization: master-key" "http://localhost:8888/token" -d '{"email": "xxxx@...", "username": "xxx", "admin": 1}'

I had set this to point to http://api.mydomain.local:8888.

I got a token from the backend.

 

For setting up the front end, do I use the https://github.com/kernelci/kernelci-frontend-config.git ? Do I use the same VM as the one I had set up for back end or do I create a new VM “kernelci-frontend”?

 

Either way, when I run the ansible command I get this:

 

TASK [install-deps : Centos 7.3 nodejs Hack (bug 1481008 / 1481470)] ******************************************************************************************

fatal: [kernelci-frontend]: FAILED! => {"msg": "The conditional check 'ansible_distribution == \"CentOS\" and ansible_distribution_major_version == \"7\" and ansible_distribution_version | search(\"7.3\")' failed. The error was: template error while templating string: no filter named 'search'. String: {% if ansible_distribution == \"CentOS\" and ansible_distribution_major_version == \"7\" and ansible_distribution_version | search(\"7.3\") %} True {% else %} False {% endif %}\n\nThe error appears to be in '/home/labuser/kernelci/kbc/kernelci-frontend-config/roles/install-deps/tasks/main.yml': line 28, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Centos 7.3 nodejs Hack (bug 1481008 / 1481470)\n  ^ here\n"}

I did update the secret.yml with the token I for from backend.

Neelima

 

From: Guillaume Tucker <guillaume.tucker@...>
Sent: Tuesday, April 6, 2021 1:29 AM
To: Krishnan, Neelima <neelima.krishnan@...>
Cc: kernelci@groups.io
Subject: Re: Kernel CI setup

 

 

Hello,

 

On 06/04/2021 05:47, Krishnan, Neelima wrote:

Guillaume,

 

I am trying to setup the kernel ci local instance in my lab, as per instructions from: https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance

This documentation should still be mostly correct although it
might have a few inaccuracies and gaps to fill.  Also, the
installation steps are quite lengthy and complicated even for
someone familiar with the code.  Having new people like you
trying to install it and providing feedback is a great way of
improving things.

I am not using dockers for installation. Instead I am trying out the VM method. I was able to use the kernelci-backend-config git to install the backend on a VM. Now I am trying to generate the tokens as per instructions in the INSTALL.md @ https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance

 

Here is what I get.. This is not a proxy issue. I am not sure how to move forward. Can you please advise on how to proceed

 

* Connected to api.mydomain.local (127.0.0.1) port 8888 (#0)

> POST /token HTTP/1.1

> Host: api.mydomain.local:8888

> User-Agent: curl/7.64.0

> Accept: */*

> Content-Type: application/json

> Authorization: master-key

> Content-Length: 63

* upload completely sent off: 63 out of 63 bytes

< HTTP/1.1 403 Operation not permitted: provided token is not authorized

< Content-Length: 12

< Vary: Accept-Encoding

< Server: TornadoServer/3.2.2

< Date: Tue, 06 Apr 2021 01:45:25 GMT

< Access-Control-Allow-Headers: authorization

< Content-Type: application/json; charset=UTF-8

* Connection #0 to host api.mydomain.local left intact

A few things to check:

* Is the kernelci-backend server process running?

To start it manually (for example, on port 5001):

cd kernelci-backend/app
python server.py --port=5001


To start the Celery process manually:

cd kernelci-backend/app
sudo python \
    -OO \
    -R \
    /srv/.venv/kernelci-backend/bin/celery worker \
    -Ofair \
    --without-gossip \
    -autoscale=24,6 \
    --loglevel=INFO \
    --app=taskqueue \
    --pidfile=/tmp/kernelci-celery.pid


If you start them manually, you'll see the logs directly in the
terminals and that should help debugging issues.  On a real
deployment, these services are typically run with systemd and the
backend is behind a web server such as nginx.

* Is the Mongo DB service running?

$ sudo systemctl | grep mongo
mongod.service  loaded active running   MongoDB Database Server


* Which API tokens have been created?


$ mongo kernel-ci
> db['api-token'].find({}, {token: 1, properties: 1})
{ "_id" : ObjectId("5bc51d7bb8d49ee75dbd5a6e"), "properties" : [ 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0 ], "token" : "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" }
{ "_id" : ObjectId("5dd80903469d6ddc79e7f7cc"), "token" : "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "properties" : [ 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ] }


* Can the token be used with the API?


$ curl -X GET -H "Authorization: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" http://localhost:5001/version
{"code":200,"result":[{"full_version":"2020.11.2","version":"2020.11.2"}]}

If all these things work, then it's possible to start sending
build and test results to it manually.  Setting up a frontend
instance would be a logical next step, and potentially Jenkins to
have a fully automated pipeline although that's not really needed
in a local development setup.

Hope this helps!

Best wishes,
Guillaume

 

 

 


Krishnan, Neelima <neelima.krishnan@...>
 

Hi,

 

Following up here… I was able to install kernelci frontend and backend locally. I commented off the Centos 7.3 nodejs Hack In roles/install-deps/tasks/main.yml And installed libhttp-parser-dev

In Debian buster VM manually. I am now able to see frontend running at http://127.0.0.1:5000/

 

I will send a PR to update the documentation with some gaps I see.

 

I want to integrate the Jenkins and Lava to this local setup. Do I do this using the https://github.com/kernelci/kernelci-jenkins and https://github.com/kernelci/lava-docker? Is there a BKM I could use?

 

Thanks,

Neelima

From: Krishnan, Neelima
Sent: Tuesday, April 6, 2021 6:35 PM
To: Guillaume Tucker <guillaume.tucker@...>
Cc: kernelci@groups.io
Subject: RE: Kernel CI setup

 

Hi Guillaume,

 

Thank you for the quick reply. I was able to go forward with backend installation. The problem was setting of the url I had set in the curl command.

curl -v -XPOST -H "Content-Type: application/json" -H "Authorization: master-key" "http://localhost:8888/token" -d '{"email": "xxxx@...", "username": "xxx", "admin": 1}'

I had set this to point to http://api.mydomain.local:8888.

I got a token from the backend.

 

For setting up the front end, do I use the https://github.com/kernelci/kernelci-frontend-config.git ? Do I use the same VM as the one I had set up for back end or do I create a new VM “kernelci-frontend”?

 

Either way, when I run the ansible command I get this:

 

TASK [install-deps : Centos 7.3 nodejs Hack (bug 1481008 / 1481470)] ******************************************************************************************

fatal: [kernelci-frontend]: FAILED! => {"msg": "The conditional check 'ansible_distribution == \"CentOS\" and ansible_distribution_major_version == \"7\" and ansible_distribution_version | search(\"7.3\")' failed. The error was: template error while templating string: no filter named 'search'. String: {% if ansible_distribution == \"CentOS\" and ansible_distribution_major_version == \"7\" and ansible_distribution_version | search(\"7.3\") %} True {% else %} False {% endif %}\n\nThe error appears to be in '/home/labuser/kernelci/kbc/kernelci-frontend-config/roles/install-deps/tasks/main.yml': line 28, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Centos 7.3 nodejs Hack (bug 1481008 / 1481470)\n  ^ here\n"}

I did update the secret.yml with the token I for from backend.

Neelima

 

From: Guillaume Tucker <guillaume.tucker@...>
Sent: Tuesday, April 6, 2021 1:29 AM
To: Krishnan, Neelima <neelima.krishnan@...>
Cc: kernelci@groups.io
Subject: Re: Kernel CI setup

 

 

Hello,

 

On 06/04/2021 05:47, Krishnan, Neelima wrote:

Guillaume,

 

I am trying to setup the kernel ci local instance in my lab, as per instructions from: https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance

This documentation should still be mostly correct although it
might have a few inaccuracies and gaps to fill.  Also, the
installation steps are quite lengthy and complicated even for
someone familiar with the code.  Having new people like you
trying to install it and providing feedback is a great way of
improving things.

I am not using dockers for installation. Instead I am trying out the VM method. I was able to use the kernelci-backend-config git to install the backend on a VM. Now I am trying to generate the tokens as per instructions in the INSTALL.md @ https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance

 

Here is what I get.. This is not a proxy issue. I am not sure how to move forward. Can you please advise on how to proceed

 

* Connected to api.mydomain.local (127.0.0.1) port 8888 (#0)

> POST /token HTTP/1.1

> Host: api.mydomain.local:8888

> User-Agent: curl/7.64.0

> Accept: */*

> Content-Type: application/json

> Authorization: master-key

> Content-Length: 63

* upload completely sent off: 63 out of 63 bytes

< HTTP/1.1 403 Operation not permitted: provided token is not authorized

< Content-Length: 12

< Vary: Accept-Encoding

< Server: TornadoServer/3.2.2

< Date: Tue, 06 Apr 2021 01:45:25 GMT

< Access-Control-Allow-Headers: authorization

< Content-Type: application/json; charset=UTF-8

* Connection #0 to host api.mydomain.local left intact

A few things to check:

* Is the kernelci-backend server process running?

To start it manually (for example, on port 5001):

cd kernelci-backend/app
python server.py --port=5001


To start the Celery process manually:

cd kernelci-backend/app
sudo python \
    -OO \
    -R \
    /srv/.venv/kernelci-backend/bin/celery worker \
    -Ofair \
    --without-gossip \
    -autoscale=24,6 \
    --loglevel=INFO \
    --app=taskqueue \
    --pidfile=/tmp/kernelci-celery.pid


If you start them manually, you'll see the logs directly in the
terminals and that should help debugging issues.  On a real
deployment, these services are typically run with systemd and the
backend is behind a web server such as nginx.

* Is the Mongo DB service running?

$ sudo systemctl | grep mongo
mongod.service  loaded active running   MongoDB Database Server


* Which API tokens have been created?


$ mongo kernel-ci
> db['api-token'].find({}, {token: 1, properties: 1})
{ "_id" : ObjectId("5bc51d7bb8d49ee75dbd5a6e"), "properties" : [ 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0 ], "token" : "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" }
{ "_id" : ObjectId("5dd80903469d6ddc79e7f7cc"), "token" : "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "properties" : [ 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ] }


* Can the token be used with the API?


$ curl -X GET -H "Authorization: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" http://localhost:5001/version
{"code":200,"result":[{"full_version":"2020.11.2","version":"2020.11.2"}]}

If all these things work, then it's possible to start sending
build and test results to it manually.  Setting up a frontend
instance would be a logical next step, and potentially Jenkins to
have a fully automated pipeline although that's not really needed
in a local development setup.

Hope this helps!

Best wishes,
Guillaume

 

 

 


Guillaume Tucker
 

Hi Neelima,

Please see my replies inline.

On 08/04/2021 00:32, Krishnan, Neelima wrote:

Hi,

 

Following up here… I was able to install kernelci frontend and backend locally. I commented off the Centos 7.3 nodejs Hack In roles/install-deps/tasks/main.yml And installed /libhttp-parser-dev/

In Debian buster VM manually. I am now able to see frontend running at http://127.0.0.1:5000/
Great!

This sounds like something that should be fixed in the Ansible
configuration, it should work on Debian without any changes.  In
fact I thought all references to other distros (e.g. CentOS) had
been dropped since only Debian is being tested and used on
kernelci.org.  Please feel free to send a PR to fix that.

I will send a PR to update the documentation with some gaps I see.
Thank you.

I want to integrate the Jenkins and Lava to this local setup. Do I do this using the https://github.com/kernelci/kernelci-jenkins and https://github.com/kernelci/lava-docker? Is there a BKM I could use?
I think the next step would be to get a LAVA instance up and
running, then you can start using it with kci_test by hand to
check it's all working fine.  Once you have that you can set up
Jenkins if you want to create an actual CI pipeline to build
kernels and run tests automatically.

There is nothing specific to KernelCI when installing LAVA, so
following the regular documentation should be fine.  You can use
the Docker image from lava-docker but it's not required.  See
this part of the documentation for more details on how to get
started:

  https://kernelci.org/docs/labs/lava/

Initially you should be able to set it up with a QEMU instance to
run tests with no extra hardware.

Best wishes,
Guillaume


 

*From:* Krishnan, Neelima
*Sent:* Tuesday, April 6, 2021 6:35 PM
*To:* Guillaume Tucker <guillaume.tucker@collabora.com>
*Cc:* kernelci@groups.io
*Subject:* RE: Kernel CI setup

 

Hi Guillaume,

 

Thank you for the quick reply. I was able to go forward with backend installation. The problem was setting of the url I had set in the curl command.

curl -v -XPOST -H "Content-Type: application/json" -H "Authorization: master-key" "http://localhost:8888/token <http://localhost:8888/token>" -d
'{"email": "xxxx@xx.xx <mailto:xxxx@xx.xx>", "username": "xxx", "admin": 1}'

I had set this to point to http://api.mydomain.local:8888.

I got a token from the backend.

 

For setting up the front end, do I use the https://github.com/kernelci/kernelci-frontend-config.git ? Do I use the same VM as the one I had set up for back end or do I create a new VM “kernelci-frontend”?

 

Either way, when I run the ansible command I get this:

 

TASK [install-deps : Centos 7.3 nodejs Hack (bug 1481008 / 1481470)] ******************************************************************************************

fatal: [kernelci-frontend]: FAILED! => {"msg": "The conditional check
'ansible_distribution == \"CentOS\" and ansible_distribution_major_version == \"7\" and ansible_distribution_version | search(\"7.3\")' failed. The error was: template error while templating string: no filter named 'search'. String: {% if ansible_distribution == \"CentOS\" and ansible_distribution_major_version == \"7\" and ansible_distribution_version | search(\"7.3\") %} True {% else %} False {% endif %}\n\nThe error
appears to be in '/home/labuser/kernelci/kbc/kernelci-frontend-config/roles/install-deps/tasks/main.yml': line 28, column 3, but may\nbe elsewhere
in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Centos 7.3 nodejs Hack (bug 1481008 / 1481470)\n  ^ here\n"}

I did update the secret.yml with the token I for from backend.

Neelima

 

*From:* Guillaume Tucker <guillaume.tucker@collabora.com <mailto:guillaume.tucker@collabora.com>>
*Sent:* Tuesday, April 6, 2021 1:29 AM
*To:* Krishnan, Neelima <neelima.krishnan@intel.com <mailto:neelima.krishnan@intel.com>>
*Cc:* kernelci@groups.io <mailto:kernelci@groups.io>
*Subject:* Re: Kernel CI setup

 

+kernelci@groups.io <mailto:+kernelci@groups.io>

 

Hello,

 

On 06/04/2021 05:47, Krishnan, Neelima wrote:

Guillaume,

 

I am trying to setup the kernel ci local instance in my lab, as per
instructions from: https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance

This documentation should still be mostly correct although it
might have a few inaccuracies and gaps to fill.  Also, the
installation steps are quite lengthy and complicated even for
someone familiar with the code.  Having new people like you
trying to install it and providing feedback is a great way of
improving things.

I am not using dockers for installation. Instead I am trying out the VM method. I was able to use the kernelci-backend-config git to install
the backend on a VM. Now I am trying to generate the tokens as per instructions in the INSTALL.md @ https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance

 

Here is what I get.. This is not a proxy issue. I am not sure how to move forward. Can you please advise on how to proceed

 

* Connected to api.mydomain.local (127.0.0.1) port 8888 (#0)

> POST /token HTTP/1.1

> Host: api.mydomain.local:8888

> User-Agent: curl/7.64.0

> Accept: */*

> Content-Type: application/json

> Authorization: master-key

> Content-Length: 63



* upload completely sent off: 63 out of 63 bytes

< HTTP/1.1 403 Operation not permitted: provided token is not authorized

< Content-Length: 12

< Vary: Accept-Encoding

< Server: TornadoServer/3.2.2

< Date: Tue, 06 Apr 2021 01:45:25 GMT

< Access-Control-Allow-Headers: authorization

< Content-Type: application/json; charset=UTF-8



* Connection #0 to host api.mydomain.local left intact

A few things to check:

** Is the kernelci-backend server process running?*

To start it manually (for example, on port 5001):

cd kernelci-backend/app
python server.py --port=5001

To start the Celery process manually:

cd kernelci-backend/app
sudo python \
    -OO \
    -R \
    /srv/.venv/kernelci-backend/bin/celery worker \
    -Ofair \
    --without-gossip \
    -autoscale=24,6 \
    --loglevel=INFO \
    --app=taskqueue \
    --pidfile=/tmp/kernelci-celery.pid

If you start them manually, you'll see the logs directly in the
terminals and that should help debugging issues.  On a real
deployment, these services are typically run with systemd and the
backend is behind a web server such as nginx.

** Is the Mongo DB service running?*

$ sudo systemctl | grep mongo
mongod.service  loaded active running   MongoDB Database Server
*
* Which API tokens have been created?*

$ mongo kernel-ci
db['api-token'].find({}, {token: 1, properties: 1})
{ "_id" : ObjectId("5bc51d7bb8d49ee75dbd5a6e"), "properties" : [ 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0 ], "token" : "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" }
{ "_id" : ObjectId("5dd80903469d6ddc79e7f7cc"), "token" : "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "properties" : [ 0, 0, 0, 1, 1, 0, 0, 1, 0, 0,
0, 0, 0, 0, 0, 0 ] }
*
* Can the token be used with the API?*

$ curl -X GET -H "Authorization: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" http://localhost:5001/version <http://localhost:5001/version>
{"code":200,"result":[{"full_version":"2020.11.2","version":"2020.11.2"}]}

If all these things work, then it's possible to start sending
build and test results to it manually.  Setting up a frontend
instance would be a logical next step, and potentially Jenkins to
have a fully automated pipeline although that's not really needed
in a local development setup.

Hope this helps!

Best wishes,
Guillaume