Memory leak via node SSH shell


Eugene Aseev
 

Hi,

We have an issue with Corda 4.3 OS leaking memory on the SSH thread. We're running regular health checks by connecting to the node shell via SSH and running run exit command. After some time node memory consumption grows and node reboots (you can see these cycles on the chart attached).

Also, a brief investigation shows that every time we connect to the node over SSH, a new file descriptor is opened. So the number of opened files grows all the time.

The node is running in Docker under Kubernetes with the following JVM args: -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap.

Any ideas on how to mitigate this?



Regards,
Eugene Aseev


Przemyslaw Bak
 

Hi Eugene,

 

does the same happen when you do the health check without docker/k8s ?

 

Regards,

Przemek

 

 

From: <corda-dev@groups.io> on behalf of "ea via Groups.Io" <ea@...>
Reply to: "corda-dev@groups.io" <corda-dev@groups.io>
Date: Tuesday, 7 January 2020 at 02:34
To: "corda-dev@groups.io" <corda-dev@groups.io>
Subject: [corda-dev] Memory leak via node SSH shell

 

Hi,

We have an issue with Corda 4.3 OS leaking memory on the SSH thread. We're running regular health checks by connecting to the node shell via SSH and running run exit command. After some time node memory consumption grows and node reboots (you can see these cycles on the chart attached).

Also, a brief investigation shows that every time we connect to the node over SSH, a new file descriptor is opened. So the number of opened files grows all the time.

The node is running in Docker under Kubernetes with the following JVM args: -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap.

Any ideas on how to mitigate this?



Regards,
Eugene Aseev


Eugene Aseev
 

Yep, just tried locally, same result. Descriptor number keeps growing when I connect and exit using regular ssh client.

 

Regards,

Eugene Aseev

 

From: corda-dev@groups.io <corda-dev@groups.io> On Behalf Of Przemyslaw Bak via Groups.Io
Sent: Tuesday, 7 January 2020 4:27 PM
To: corda-dev@groups.io
Subject: Re: [corda-dev] Memory leak via node SSH shell

 

Hi Eugene,

 

does the same happen when you do the health check without docker/k8s ?

 

Regards,

Przemek

 

 

From: <corda-dev@groups.io> on behalf of "ea via Groups.Io" <ea@...>
Reply to: "corda-dev@groups.io" <corda-dev@groups.io>
Date: Tuesday, 7 January 2020 at 02:34
To: "corda-dev@groups.io" <corda-dev@groups.io>
Subject: [corda-dev] Memory leak via node SSH shell

 

Hi,

We have an issue with Corda 4.3 OS leaking memory on the SSH thread. We're running regular health checks by connecting to the node shell via SSH and running run exit command. After some time node memory consumption grows and node reboots (you can see these cycles on the chart attached).

Also, a brief investigation shows that every time we connect to the node over SSH, a new file descriptor is opened. So the number of opened files grows all the time.

The node is running in Docker under Kubernetes with the following JVM args: -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap.

Any ideas on how to mitigate this?



Regards,
Eugene Aseev


Mike Hearn