Date   
Re: [Ibmtpm20tss-users] tpm sessions

James Bottomley
 

On Mon, 2019-03-18 at 21:05 +0000, Doug Fraser wrote:
Ken,

We are interfacing to the TPM via openssl with engine = tpm2, so at
that level, do we have direct access to session control?

The test server requests openssl to encrypt a block of data, then
passes it to the client. The client requests openssl to decrypt the
block, and checks its content. At that point, at the application
level, the transaction is complete.

Is there some other operation that the application has to request
from openssl to flush the related context?
No as I explained in the other email, the openssl_tpm2_engine mostly
keeps the TPM clear when it's not doing anything even if the key is
present and referenced from an openssl point of view.

I'll go back to something James mentioned in his reply, the failure
seems to be percolating up from SPI device layer.
Is there a serialization point within the SPI layer to marshal
commands into and out of the TPM?
Yes, it's chip->tpm_mutex which is taken in tpm_common_write via
tpm_try_get_ops.

I noticed a flag based mutex() with tpm_command(), but I haven't
spilled data from that yet.
Is that the serialization point for the interface?

I don't suspect signal integrity at the low clock rate we are
currently running, particularly in regard to the extreme long term
test that was run over the weekend.
well something caused the -EIO in tpm_try_transmit and that can only
come from the underlying driver. For the tis driver it means that
STS_DATA_EXPECT wasn't signalled as it should for transmitting data.

James

Re: [Ibmtpm20tss-users] tpm sessions

James Bottomley
 

On Mon, 2019-03-18 at 15:48 -0500, Kenneth Goldman wrote:
From: Doug Fraser <doug.fraser@...>
To: Kenneth Goldman <kgoldman@...>
Cc: "Ibmtpm20tss-users@..." <Ibmtpm20tss-
users@...>, "openssl-tpm2-engine@groups.io"
<openssl-tpm2-engine@groups.io>
Date: 03/18/2019 03:48 PM
Subject: RE: [Ibmtpm20tss-users] tpm sessions

Ken,

Thank you for that information.

I don?t believe we should be exceeded that limit.

The nature of the test is that each thread spawns a shell that
invokes a client/server pair that each use openssl (and the tpm2
engine) to trade a piece of data. One side is encrypting, the
other decrypting.

So for any given thread, there are two processes, that both
completely exit and then the shell terminates, and the process
repeats, with a new shell invocation.

So for seven test threads, worst case is 2 * 7 simultaneous
applications active (14) which seems safely less than 21.
14 applications ... but how many session per application?
The openssl_tpm2_engine typically only uses 1 session for each command.
We use the same session for hmac and decryption and we don't run
audits. I think all of the commands we use only require at most one
authorization. We also don't keep sessions loaded, so every time you
use the key, we go through a start session/load key/do key op/flush
handles sequence.

That is, a typical operation will use 2-3 sessions - 2 for
authorization and perhaps one for audit.

If an application doesn't either explicitly flush,
or exit so that the resource manager will clean up, sessions
can hang around. Eventually the 64 session limit is reached.
There's another possibility as well, which is that an application that
keeps a session for a long time can cause a gap error (the TPM doesn't
allow session sequence numbers to wrap around).

However, openssl_tpm2_engine was coded with the old /dev/tpm0 interface
in mind, so it tries to keep the session and volatile handle use
periods as short as possible.

If it runs for while and then fails, perhaps there is a
'session leak'?

FWIW, when the RM was first coded, I regression tested it.
I ran 21 processes, and each created 3 session, did something,
and then flushed them, in a loop. It ran quite a while.

However, I did not test a process closing without explicitly
flushing. Should I do that?
We should already have tested that, but there's no harm in making sure.
The current smoke tests are in the kernel under
tools/testing/selftests/tpm2

James


Doug, James:

Should I rerun the test on a newer kernel? Perhaps there
was a regression.

I.e., if I can help, let me know.



Re: [Ibmtpm20tss-users] tpm sessions

Doug Fraser
 

Ken,

We are interfacing to the TPM via openssl with engine = tpm2, so at that level, do we have direct access to session control?

The test server requests openssl to encrypt a block of data, then passes it to the client. The client requests openssl to decrypt the block, and checks its content. At that point, at the application level, the transaction is complete.

Is there some other operation that the application has to request from openssl to flush the related context?

I'll go back to something James mentioned in his reply, the failure seems to be percolating up from SPI device layer.
Is there a serialization point within the SPI layer to marshal commands into and out of the TPM?

I noticed a flag based mutex() with tpm_command(), but I haven't spilled data from that yet.
Is that the serialization point for the interface?

I don't suspect signal integrity at the low clock rate we are currently running, particularly in regard to the extreme long term test that was run over the weekend.

Doug

From: Kenneth Goldman <kgoldman@...>
Sent: Monday, March 18, 2019 4:49 PM
To: Doug Fraser <doug.fraser@...>
Cc: Ibmtpm20tss-users@...; openssl-tpm2-engine@groups.io
Subject: RE: [Ibmtpm20tss-users] tpm sessions


From: Doug Fraser <doug.fraser@...<mailto:doug.fraser@...>>
To: Kenneth Goldman <kgoldman@...<mailto:kgoldman@...>>
Cc: "Ibmtpm20tss-users@...<mailto:Ibmtpm20tss-users@...>" <Ibmtpm20tss-
users@...<mailto:users@...>>, "openssl-tpm2-engine@groups.io<mailto:openssl-tpm2-engine@groups.io>"
<openssl-tpm2-engine@groups.io<mailto:openssl-tpm2-engine@groups.io>>
Date: 03/18/2019 03:48 PM
Subject: RE: [Ibmtpm20tss-users] tpm sessions

Ken,

Thank you for that information.

I don't believe we should be exceeded that limit.

The nature of the test is that each thread spawns a shell that
invokes a client/server pair that each use openssl (and the tpm2
engine) to trade a piece of data. One side is encrypting, the other
decrypting.

So for any given thread, there are two processes, that both
completely exit and then the shell terminates, and the process
repeats, with a new shell invocation.

So for seven test threads, worst case is 2 * 7 simultaneous
applications active (14) which seems safely less than 21.
14 applications ... but how many session per application?

That is, a typical operation will use 2-3 sessions - 2 for
authorization and perhaps one for audit.

If an application doesn't either explicitly flush,
or exit so that the resource manager will clean up, sessions
can hang around. Eventually the 64 session limit is reached.

If it runs for while and then fails, perhaps there is a
'session leak'?

FWIW, when the RM was first coded, I regression tested it.
I ran 21 processes, and each created 3 session, did something,
and then flushed them, in a loop. It ran quite a while.

However, I did not test a process closing without explicitly
flushing. Should I do that?

Doug, James:

Should I rerun the test on a newer kernel? Perhaps there
was a regression.

I.e., if I can help, let me know.

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a) any dissemination or copying of this message is strictly prohibited; and (b) immediately notify the sender by return message and destroy any copies of this message in any form (electronic, paper or otherwise) that you have. The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications.

Re: [Ibmtpm20tss-users] tpm sessions

Kenneth Goldman <kgoldman@...>
 


> From: Doug Fraser <doug.fraser@...>

> To: Kenneth Goldman <kgoldman@...>
> Cc: "Ibmtpm20tss-users@..." <Ibmtpm20tss-
> users@...>, "openssl-tpm2-engine@groups.io"
> <openssl-tpm2-engine@groups.io>

> Date: 03/18/2019 03:48 PM
> Subject: RE: [Ibmtpm20tss-users] tpm sessions
>
> Ken,

>  
> Thank you for that information.
>  
> I don’t believe we should be exceeded that limit.
>  
> The nature of the test is that each thread spawns a shell that
> invokes a client/server pair that each use openssl (and the tpm2
> engine) to trade a piece of data. One side is encrypting, the other
> decrypting.

>  
> So for any given thread, there are two processes, that both
> completely exit and then the shell terminates, and the process
> repeats, with a new shell invocation.

>  
> So for seven test threads, worst case is 2 * 7 simultaneous
> applications active (14) which seems safely less than 21.


14 applications ... but how many session per application?

That is, a typical operation will use 2-3 sessions - 2 for
authorization and perhaps one for audit.  

If an application doesn't either explicitly flush,
or exit so that the resource manager will clean up, sessions
can hang around.  Eventually the 64 session limit is reached.

If it runs for while and then fails, perhaps there is a
'session leak'?

FWIW, when the RM was first coded, I regression tested it.
I ran 21 processes, and each created 3 session, did something,
and then flushed them, in a loop.  It ran quite a while.

However, I did not test a process closing without explicitly
flushing.  Should I do that?

Doug, James:

Should I rerun the test on a newer kernel?  Perhaps there
was a regression.

I.e., if I can help, let me know.

Re: [Ibmtpm20tss-users] tpm sessions

Doug Fraser
 

Ken,

Thank you for that information.

I don't believe we should be exceeded that limit.

The nature of the test is that each thread spawns a shell that invokes a client/server pair that each use openssl (and the tpm2 engine) to trade a piece of data. One side is encrypting, the other decrypting.

So for any given thread, there are two processes, that both completely exit and then the shell terminates, and the process repeats, with a new shell invocation.

So for seven test threads, worst case is 2 * 7 simultaneous applications active (14) which seems safely less than 21.

I am currently rerunning with three active test threads, and so far that has issued about twice the number of openssl engine calls without leading to any failures. If that is still running error free at the end of my day, I am going to let it run through the night. If that runs without error, I am going to binary search between 3 and 7 to see if there is a 'cliff'

Thanks again for the high level view of the numbers and limitations coming into play.

Doug


From: Kenneth Goldman <kgoldman@...>
Sent: Monday, March 18, 2019 2:34 PM
To: Doug Fraser <doug.fraser@...>
Cc: Ibmtpm20tss-users@...; openssl-tpm2-engine@groups.io
Subject: Re: [Ibmtpm20tss-users] tpm sessions

From: Doug Fraser <doug.fraser@...<mailto:doug.fraser@...>>
To: "openssl-tpm2-engine@groups.io<mailto:openssl-tpm2-engine@groups.io>" <openssl-tpm2-engine@groups.io<mailto:openssl-tpm2-engine@groups.io>>,
Doug Fraser <doug.fraser@...<mailto:doug.fraser@...>>, "Ibmtpm20tss-
users@...<mailto:users@...>" <Ibmtpm20tss-users@...<mailto:Ibmtpm20tss-users@...>>
Date: 03/18/2019 02:19 PM
Subject: [Ibmtpm20tss-users] tpm sessions

So we have moved beyond the signaling issues on our TPM for now, but
in ramping up performance saturation testing, I am pounding on the
openssl engine with multiple threads of execution, and I am finding
this fault.

/var/log/messages:Mar 18 16:43:28 C05BCB00C0A000001153 kern.err
kernel: [11840.869864] tpm tpm0: tpm_try_transmit: tpm_send: error -5
/var/log/messages:Mar 18 16:43:28 C05BCB00C0A000001153 kern.err
kernel: [11840.878969] tpm tpm0: A TPM error (357) occurred flushing context

Within the kernel, reflect up through the applications as:

TPM2_StartAuthSession failed with 2309
TPM_RC_SESSION_HANDLES - out of session handles - a session must be
flushed before a new session may be created
Failed to get Key Handle in TPM EC key routines

The underlying tss code is build with:

CCFLAGS += -DTPM_POSIX \
-DTPM_INTERFACE_TYPE_DEFAULT="\"dev\"" \
-DTPM_DEVICE_DEFAULT="\"/dev/tpmrm0\"" \
$(BLD_SYSROOT)

So we should be using the tpmrm resource manager within the kernel.

If I run the test code as a single instance, this never occurs
(within the bounds of 64 hours of constant running)

Is there a practical limit to the openssl engine, underlying tpmrm,
or even the underlying physical block that I am ignoring here?
My view was that as long as you pass through the tpmrm, you might
stall, but the resources would be managed.
Background:

Sessions are different from keys, in that the TPM has to prevent a
replay attack on a saved session context. To do this, the TPM has a
table of active session contexts with a 'version number' to detect and
block the replay.

In TCG jargon (if I have it right):

A loaded session is actually on the TPM.
A saved session is off the TPM, but is still in the table.
An active session is either loaded or saved.

Result:

The table has typically 64 entries for active sessions. Thus,
threads cannot make an unlimited number of sessions, even with
a resource manager.

Again typically, an application needs at most 3 sessions, so
the TPM can handle 21 simultaneous applications.

Keys:

Since keys are not subject to a replay attack, the TPM does not
keep any state when a key is context saved. There is thus
no TPM defined limit to the number of saved key contexts.


I am going back to dig through tpm-tis, in particular, tpm2-cmd.c
and tpm-interface.c.

Doug

_______________________________________________
Ibmtpm20tss-users mailing list
Ibmtpm20tss-users@...<mailto:Ibmtpm20tss-users@...>
https://urldefense.proofpoint.com/v2/url?
u=https-3A__lists.sourceforge.net_lists_listinfo_ibmtpm20tss-2Dusers&d=DwICAg&c=jf_iaSHvJObTbx-
siA1ZOg&r=DZCVG43VcL8GTneMZb8k8lEwb-O1GZktFfre1-
mlmiA&m=XmhnfVlfhSYHtZr6FrXKzEkZPZOJHo_STtu4jislJQE&s=Fs921riOnezqzdagg8OMOP8iqow-
oOAQI5B0wwzwq2M&e=

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a) any dissemination or copying of this message is strictly prohibited; and (b) immediately notify the sender by return message and destroy any copies of this message in any form (electronic, paper or otherwise) that you have. The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications.

Re: [Ibmtpm20tss-users] tpm sessions

Kenneth Goldman <kgoldman@...>
 

> From: Doug Fraser <doug.fraser@...>
> To: "openssl-tpm2-engine@groups.io" <openssl-tpm2-engine@groups.io>,
> Doug Fraser <doug.fraser@...>, "Ibmtpm20tss-
> users@..." <Ibmtpm20tss-users@...>

> Date: 03/18/2019 02:19 PM
> Subject: [Ibmtpm20tss-users] tpm sessions
>
> So we have moved beyond the signaling issues on our TPM for now, but
> in ramping up performance saturation testing, I am pounding on the
> openssl engine with multiple threads of execution, and I am finding
> this fault.
>
> /var/log/messages:Mar 18 16:43:28 C05BCB00C0A000001153 kern.err
> kernel: [11840.869864] tpm tpm0: tpm_try_transmit: tpm_send: error -5
> /var/log/messages:Mar 18 16:43:28 C05BCB00C0A000001153 kern.err
> kernel: [11840.878969] tpm tpm0: A TPM error (357) occurred flushing context
>
> Within the kernel, reflect up through the applications as:
>
> TPM2_StartAuthSession failed with 2309
> TPM_RC_SESSION_HANDLES - out of session handles - a session must be
> flushed before a new session may be created
> Failed to get Key Handle in TPM EC key routines
>
> The underlying tss code is build with:
>
> CCFLAGS +=  -DTPM_POSIX \
>         -DTPM_INTERFACE_TYPE_DEFAULT="\"dev\""  \
>         -DTPM_DEVICE_DEFAULT="\"/dev/tpmrm0\"" \
>         $(BLD_SYSROOT)
>
> So we should be using the tpmrm  resource manager within the kernel.
>
> If I run the test code as a single instance, this never occurs
> (within the bounds of 64 hours of constant running)
>
> Is there a practical limit to the openssl engine, underlying tpmrm,
> or even the underlying physical block that I am ignoring here?
> My view was that as long as you pass through the tpmrm, you might
> stall, but the resources would be managed.


Background:  

Sessions are different from keys, in that the TPM has to prevent a
replay attack on a saved session context. To do this, the TPM has a
table of active session contexts with a 'version number' to detect and
block the replay.

In TCG jargon (if I have it right):

A loaded session is actually on the TPM.
A saved session is off the TPM, but is still in the table.
An active session is either loaded or saved.

Result:

The table has typically 64 entries for active sessions.  Thus,
threads cannot make an unlimited number of sessions, even with
a resource manager.

Again typically, an application needs at most 3 sessions, so
the TPM can handle 21 simultaneous applications.

Keys:

Since keys are not subject to a replay attack, the TPM does not
keep any state when a key is context saved.  There is thus
no TPM defined limit to the number of saved key contexts.

 
> I am going back to dig through tpm-tis, in particular, tpm2-cmd.c
> and tpm-interface.c.
>
> Doug
>
> _______________________________________________
> Ibmtpm20tss-users mailing list
> Ibmtpm20tss-users@...
>
https://urldefense.proofpoint.com/v2/url?
> u=https-3A__lists.sourceforge.net_lists_listinfo_ibmtpm20tss-2Dusers&d=DwICAg&c=jf_iaSHvJObTbx-
> siA1ZOg&r=DZCVG43VcL8GTneMZb8k8lEwb-O1GZktFfre1-
> mlmiA&m=XmhnfVlfhSYHtZr6FrXKzEkZPZOJHo_STtu4jislJQE&s=Fs921riOnezqzdagg8OMOP8iqow-
> oOAQI5B0wwzwq2M&e=
>

Re: tpm sessions

James Bottomley
 

On Mon, 2019-03-18 at 18:03 +0000, Doug Fraser wrote:
So we have moved beyond the signaling issues on our TPM for now, but
in ramping up performance saturation testing, I am pounding on the
openssl engine with multiple threads of execution, and I am finding
this fault.

/var/log/messages:Mar 18 16:43:28 C05BCB00C0A000001153 kern.err
kernel: [11840.869864] tpm tpm0: tpm_try_transmit: tpm_send: error -5
/var/log/messages:Mar 18 16:43:28 C05BCB00C0A000001153 kern.err
kernel: [11840.878969] tpm tpm0: A TPM error (357) occurred flushing
context
This sounds a bit serious. I've taken the liberty of cc'ing the linux-
integrity group which is the mailing list where kernel based TPM issues
get discussed. Error -5 is EIO which still points to a TPM
communications problem.

Within the kernel, reflect up through the applications as:

TPM2_StartAuthSession failed with 2309
TPM_RC_SESSION_HANDLES - out of session handles - a session must be
flushed before a new session may be created
Failed to get Key Handle in TPM EC key routines

The underlying tss code is build with:

CCFLAGS += -DTPM_POSIX \
-DTPM_INTERFACE_TYPE_DEFAULT="\"dev\"" \
-DTPM_DEVICE_DEFAULT="\"/dev/tpmrm0\"" \
$(BLD_SYSROOT)

So we should be using the tpmrm resource manager within the kernel.
The answer should be yes because without it you'll exhaust the TPM
resources in a multi-threaded environment. The TPM has severe limits
(like 3) on the number of keys which can be active at any given time.

What is happening in the tpmrm situation is that you get one resource
manager instance for every separate open of /dev/tpmrm0 but also every
TPM operation you try results in a resource manager context save and
load for ever volatile key handle and session ... essentially it will
be more than tripling the TPM transaction load, since the way the
openssl engine works, it usually needs the parent key, a session and
the actual key you're loading every time you do something.

once a resource manager context flush fails we actually get left with
whatever handle it was trying to flush stuck in the TPM which will lead
to resource exhaustion.

If I run the test code as a single instance, this never occurs
(within the bounds of 64 hours of constant running)

Is there a practical limit to the openssl engine, underlying tpmrm,
or even the underlying physical block that I am ignoring here?
My view was that as long as you pass through the tpmrm, you might
stall, but the resources would be managed.
Right, the resource manager is supposed to make the TPM scalable.
There is a hard limit (64 usually) on the number of active sessions you
can have even with a resource manager, but I don't think you're hitting
that.

James

I am going back to dig through tpm-tis, in particular, tpm2-cmd.c and
tpm-interface.c.

tpm sessions

Doug Fraser
 

So we have moved beyond the signaling issues on our TPM for now, but in ramping up performance saturation testing, I am pounding on the openssl engine with multiple threads of execution, and I am finding this fault.

/var/log/messages:Mar 18 16:43:28 C05BCB00C0A000001153 kern.err kernel: [11840.869864] tpm tpm0: tpm_try_transmit: tpm_send: error -5
/var/log/messages:Mar 18 16:43:28 C05BCB00C0A000001153 kern.err kernel: [11840.878969] tpm tpm0: A TPM error (357) occurred flushing context

Within the kernel, reflect up through the applications as:

TPM2_StartAuthSession failed with 2309
TPM_RC_SESSION_HANDLES - out of session handles - a session must be flushed before a new session may be created
Failed to get Key Handle in TPM EC key routines

The underlying tss code is build with:

CCFLAGS += -DTPM_POSIX \
-DTPM_INTERFACE_TYPE_DEFAULT="\"dev\"" \
-DTPM_DEVICE_DEFAULT="\"/dev/tpmrm0\"" \
$(BLD_SYSROOT)

So we should be using the tpmrm resource manager within the kernel.

If I run the test code as a single instance, this never occurs (within the bounds of 64 hours of constant running)

Is there a practical limit to the openssl engine, underlying tpmrm, or even the underlying physical block that I am ignoring here?
My view was that as long as you pass through the tpmrm, you might stall, but the resources would be managed.

I am going back to dig through tpm-tis, in particular, tpm2-cmd.c and tpm-interface.c.

Doug

Re: Periodic Openssl TPM2 engine failures

Doug Fraser
 

Final note on this issue. Took the clock down to 1Mhz. Sixty four hours of continuous openssl engine testing down to the device, zero errors from the test code and zero warnings or errors from the kernel driver.

I am going to look into signal integrity issues on the board, but clearly there is a signal problem.

Note: given the run time for cipher operations inside the TPM in comparison to data transfer time, there is very little effect on overall performance dropping from 10Mhz to 1Mhz on SPI.

-----Original Message-----
From: openssl-tpm2-engine@groups.io <openssl-tpm2-engine@groups.io> On Behalf Of Doug Fraser
Sent: Friday, March 15, 2019 11:45 AM
To: openssl-tpm2-engine@groups.io; James.Bottomley@...; Ibmtpm20tss-users@...
Subject: Re: [openssl-tpm2-engine] Periodic Openssl TPM2 engine failures

James,

So I had originally set the MAX SPI clock in my DTSI to 10Mhz.
Spec on the part is 40Mhz, and it seemed that 10Mhz would be lots of margin.

The errors were really infrequent even under heavy use, so it took a while to notice they were occurring.

Over the course of this morning I tested at various clock rates.

100Khz, fine,
1Mhz, fine,
5Mhz, fine,
8Mhz, failed more often than at 10Mhz!

So I need to look at the bus with a scope, but I am currently setting at 2Mhz to unblock my users, and then continuing to look at the signal.

Given that it is worse at 8 Mhz than at 10 Mhz, my guess is overshoot and ringing, not slow rise time, but again, that is just a guess at this point.

Thanks again.

Doug

-----Original Message-----
From: openssl-tpm2-engine@groups.io <openssl-tpm2-engine@groups.io> On Behalf Of James Bottomley
Sent: Thursday, March 14, 2019 4:31 PM
To: openssl-tpm2-engine@groups.io; Doug Fraser <doug.fraser@...>; Ibmtpm20tss-users@...
Subject: Re: [openssl-tpm2-engine] Periodic Openssl TPM2 engine failures

On Thu, 2019-03-14 at 19:42 +0000, Doug Fraser wrote:
Hello all.

Currently running ibm libtss 1.3.3.1 and openssl engine 2.1.0
connected to openssl 1.0.2 under Alpine 3.8 on 32 bit armhf.

I have tested with engine 2.3.0 and seen similar results.

At a run rate of about 1 of 40 down to 1 of 50 test cases, I get the
following failure feedback from the engine.

TPM2_ReadPublic failed with 720963
response code unknown
This isn't actually response code unknown, it's TSS_RC_MALFORMED_PUBLIC and it means the public structure doesn't match the name.

I've seen this before as a bug in one of the ECC routines where there was an unnecessary leading zero in a point which is what causes the name hash to mismatch. But in any case it seems to be a bug worth tracking down in either the hardware or the IBM TSS.

Probably all the rest of the errors below stem from this, so the above is what needs root causing.

James


TPM2_StartAuthSession failed with 720912 TSS_RC_FILE_OPEN - The file
could not be opened Failed to get Key Handle in TPM EC key routines

Is anyone else seeing these error codes pop up?

I'm going to enable tpm-tis tracing in the kernel and see what pops
out of there.

Sincerely,

Douglas Fraser






This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a) any dissemination or copying of this message is strictly prohibited; and (b) immediately notify the sender by return message and destroy any copies of this message in any form (electronic, paper or otherwise) that you have. The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications.
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a) any dissemination or copying of this message is strictly prohibited; and (b) immediately notify the sender by return message and destroy any copies of this message in any form (electronic, paper or otherwise) that you have. The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications.





This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a) any dissemination or copying of this message is strictly prohibited; and (b) immediately notify the sender by return message and destroy any copies of this message in any form (electronic, paper or otherwise) that you have. The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications.

Re: Periodic Openssl TPM2 engine failures

Doug Fraser
 

James,

So I had originally set the MAX SPI clock in my DTSI to 10Mhz.
Spec on the part is 40Mhz, and it seemed that 10Mhz would be lots of margin.

The errors were really infrequent even under heavy use, so it took a while to notice they were occurring.

Over the course of this morning I tested at various clock rates.

100Khz, fine,
1Mhz, fine,
5Mhz, fine,
8Mhz, failed more often than at 10Mhz!

So I need to look at the bus with a scope, but I am currently setting at 2Mhz to unblock my users, and then continuing to look at the signal.

Given that it is worse at 8 Mhz than at 10 Mhz, my guess is overshoot and ringing, not slow rise time, but again, that is just a guess at this point.

Thanks again.

Doug

-----Original Message-----
From: openssl-tpm2-engine@groups.io <openssl-tpm2-engine@groups.io> On Behalf Of James Bottomley
Sent: Thursday, March 14, 2019 4:31 PM
To: openssl-tpm2-engine@groups.io; Doug Fraser <doug.fraser@...>; Ibmtpm20tss-users@...
Subject: Re: [openssl-tpm2-engine] Periodic Openssl TPM2 engine failures

On Thu, 2019-03-14 at 19:42 +0000, Doug Fraser wrote:
Hello all.

Currently running ibm libtss 1.3.3.1 and openssl engine 2.1.0
connected to openssl 1.0.2 under Alpine 3.8 on 32 bit armhf.

I have tested with engine 2.3.0 and seen similar results.

At a run rate of about 1 of 40 down to 1 of 50 test cases, I get the
following failure feedback from the engine.

TPM2_ReadPublic failed with 720963
response code unknown
This isn't actually response code unknown, it's TSS_RC_MALFORMED_PUBLIC and it means the public structure doesn't match the name.

I've seen this before as a bug in one of the ECC routines where there was an unnecessary leading zero in a point which is what causes the name hash to mismatch. But in any case it seems to be a bug worth tracking down in either the hardware or the IBM TSS.

Probably all the rest of the errors below stem from this, so the above is what needs root causing.

James


TPM2_StartAuthSession failed with 720912 TSS_RC_FILE_OPEN - The file
could not be opened Failed to get Key Handle in TPM EC key routines

Is anyone else seeing these error codes pop up?

I'm going to enable tpm-tis tracing in the kernel and see what pops
out of there.

Sincerely,

Douglas Fraser






This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a) any dissemination or copying of this message is strictly prohibited; and (b) immediately notify the sender by return message and destroy any copies of this message in any form (electronic, paper or otherwise) that you have. The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications.
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a) any dissemination or copying of this message is strictly prohibited; and (b) immediately notify the sender by return message and destroy any copies of this message in any form (electronic, paper or otherwise) that you have. The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications.

Re: Periodic Openssl TPM2 engine failures

Doug Fraser
 

James, you totally rule!

Thanks. I am going to toss a scope on the SPI leads and poke around. Look at eye patterns and such.
If nothing else, I am going to see what happens if I 'de-clock' the bus. Maybe knock it down by half.

I'll feed back onto this thread whatever I find.

Thanks again!

Douglas

-----Original Message-----
From: James Bottomley <James.Bottomley@...>
Sent: Thursday, March 14, 2019 4:35 PM
To: openssl-tpm2-engine@groups.io; Doug Fraser <doug.fraser@...>; Ibmtpm20tss-users@...
Subject: Re: [openssl-tpm2-engine] Periodic Openssl TPM2 engine failures

On Thu, 2019-03-14 at 20:30 +0000, Doug Fraser wrote:
And from the kernel....


Mar 14 20:22:07 C05BUB0000A000004124 kern.warn kernel: [ 4876.887354]
tpm tpm0: tpm2_load_context: failed with a TPM error 0x01DF
Oh boy, this is a nasty one: it's TPM_RC_INTEGRITY. It means that the integrity check failed of the context we're trying to load into the TPM. I've never seen this in all my stress testing of the in-kernel resource manager, so it is likely a TPM communication issue, possibly a bus glitch.

Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4876.938901]
tpm tpm0: A TPM error (459) occurred flushing context Mar 14 20:22:07
C05BUB0000A000004124 kern.warn kernel: [ 4876.972604] tpm tpm0:
tpm2_load_context: failed with a TPM error 0x01DF Mar 14 20:22:07
C05BUB0000A000004124 kern.err kernel: [ 4876.993230] tpm tpm0: A TPM
error (459) occurred flushing context Mar 14 20:22:07
C05BUB0000A000004124 kern.err kernel: [ 4877.020956] tpm tpm0: A TPM
error (459) occurred flushing context Mar 14 20:22:07
C05BUB0000A000004124 kern.err kernel: [ 4877.030189] tpm tpm0:
tpm_try_transmit: tpm_send: error -5 Mar 14 20:22:07
C05BUB0000A000004124 kern.err kernel: [ 4877.040832] tpm tpm0: A TPM
error (357) occurred flushing context

At least I have something concrete to look at now....

Note: I have the libtss built with default TPM_DEVICE=/dev/tpmrm0

I am curious why the "tpm0" in the messages above instead of "tpmrm0",
but I guess that RM path eventually maps to plan vanilla tpm0.
It's because the dev_print is on the chip device which is always tpmN regardless of whether you use /dev/tpmrmN or /dev/tpmN

James


This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a) any dissemination or copying of this message is strictly prohibited; and (b) immediately notify the sender by return message and destroy any copies of this message in any form (electronic, paper or otherwise) that you have. The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications.

Re: Periodic Openssl TPM2 engine failures

James Bottomley
 

On Thu, 2019-03-14 at 20:30 +0000, Doug Fraser wrote:
And from the kernel....


Mar 14 20:22:07 C05BUB0000A000004124 kern.warn kernel: [ 4876.887354]
tpm tpm0: tpm2_load_context: failed with a TPM error 0x01DF
Oh boy, this is a nasty one: it's TPM_RC_INTEGRITY. It means that the
integrity check failed of the context we're trying to load into the
TPM. I've never seen this in all my stress testing of the in-kernel
resource manager, so it is likely a TPM communication issue, possibly a
bus glitch.

Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4876.938901]
tpm tpm0: A TPM error (459) occurred flushing context
Mar 14 20:22:07 C05BUB0000A000004124 kern.warn kernel: [ 4876.972604]
tpm tpm0: tpm2_load_context: failed with a TPM error 0x01DF
Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4876.993230]
tpm tpm0: A TPM error (459) occurred flushing context
Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4877.020956]
tpm tpm0: A TPM error (459) occurred flushing context
Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4877.030189]
tpm tpm0: tpm_try_transmit: tpm_send: error -5
Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4877.040832]
tpm tpm0: A TPM error (357) occurred flushing context

At least I have something concrete to look at now....

Note: I have the libtss built with default TPM_DEVICE=/dev/tpmrm0

I am curious why the "tpm0" in the messages above instead of
"tpmrm0", but I guess that RM path eventually maps to plan vanilla
tpm0.
It's because the dev_print is on the chip device which is always tpmN
regardless of whether you use /dev/tpmrmN or /dev/tpmN

James

Re: Periodic Openssl TPM2 engine failures

James Bottomley
 

On Thu, 2019-03-14 at 19:42 +0000, Doug Fraser wrote:
Hello all.

Currently running ibm libtss 1.3.3.1 and openssl engine 2.1.0
connected to openssl 1.0.2 under Alpine 3.8 on 32 bit armhf.

I have tested with engine 2.3.0 and seen similar results.

At a run rate of about 1 of 40 down to 1 of 50 test cases, I get the
following failure feedback from the engine.

TPM2_ReadPublic failed with 720963
response code unknown
This isn't actually response code unknown, it's TSS_RC_MALFORMED_PUBLIC
and it means the public structure doesn't match the name.

I've seen this before as a bug in one of the ECC routines where there
was an unnecessary leading zero in a point which is what causes the
name hash to mismatch. But in any case it seems to be a bug worth
tracking down in either the hardware or the IBM TSS.

Probably all the rest of the errors below stem from this, so the above
is what needs root causing.

James


TPM2_StartAuthSession failed with 720912
TSS_RC_FILE_OPEN - The file could not be opened
Failed to get Key Handle in TPM EC key routines

Is anyone else seeing these error codes pop up?

I'm going to enable tpm-tis tracing in the kernel and see what pops
out of there.

Sincerely,

Douglas Fraser


Re: Periodic Openssl TPM2 engine failures

Doug Fraser
 

And from the kernel....


Mar 14 20:22:07 C05BUB0000A000004124 kern.warn kernel: [ 4876.887354] tpm tpm0: tpm2_load_context: failed with a TPM error 0x01DF
Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4876.938901] tpm tpm0: A TPM error (459) occurred flushing context
Mar 14 20:22:07 C05BUB0000A000004124 kern.warn kernel: [ 4876.972604] tpm tpm0: tpm2_load_context: failed with a TPM error 0x01DF
Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4876.993230] tpm tpm0: A TPM error (459) occurred flushing context
Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4877.020956] tpm tpm0: A TPM error (459) occurred flushing context
Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4877.030189] tpm tpm0: tpm_try_transmit: tpm_send: error -5
Mar 14 20:22:07 C05BUB0000A000004124 kern.err kernel: [ 4877.040832] tpm tpm0: A TPM error (357) occurred flushing context

At least I have something concrete to look at now....

Note: I have the libtss built with default TPM_DEVICE=/dev/tpmrm0

I am curious why the "tpm0" in the messages above instead of "tpmrm0", but I guess that RM path eventually maps to plan vanilla tpm0.

Douglas

-----Original Message-----
From: Doug Fraser
Sent: Thursday, March 14, 2019 4:00 PM
To: openssl-tpm2-engine@groups.io; Ibmtpm20tss-users@...
Subject: RE: Periodic Openssl TPM2 engine failures

I should also have noted that the physical TPM is an Infineon SLB9670 20 updated to what I believe is the latest firmware, 7.63.3353

Kernel is 4.14.77-v7+

Modules are:
tpm_tis_spi 16384 0
tpm_tis_core 20480 1 tpm_tis_spi
tpm 57344 2 tpm_tis_spi,tpm_tis_core

-----Original Message-----
From: Doug Fraser
Sent: Thursday, March 14, 2019 3:42 PM
To: openssl-tpm2-engine@groups.io; Ibmtpm20tss-users@...
Subject: Periodic Openssl TPM2 engine failures

Hello all.

Currently running ibm libtss 1.3.3.1 and openssl engine 2.1.0 connected to openssl 1.0.2 under Alpine 3.8 on 32 bit armhf.

I have tested with engine 2.3.0 and seen similar results.

At a run rate of about 1 of 40 down to 1 of 50 test cases, I get the following failure feedback from the engine.

TPM2_ReadPublic failed with 720963
response code unknown
TPM2_StartAuthSession failed with 720912 TSS_RC_FILE_OPEN - The file could not be opened Failed to get Key Handle in TPM EC key routines

Is anyone else seeing these error codes pop up?

I'm going to enable tpm-tis tracing in the kernel and see what pops out of there.

Sincerely,

Douglas Fraser

Re: Periodic Openssl TPM2 engine failures

Doug Fraser
 

I should also have noted that the physical TPM is an Infineon SLB9670 20 updated to what I believe is the latest firmware, 7.63.3353

Kernel is 4.14.77-v7+

Modules are:
tpm_tis_spi 16384 0
tpm_tis_core 20480 1 tpm_tis_spi
tpm 57344 2 tpm_tis_spi,tpm_tis_core

-----Original Message-----
From: Doug Fraser
Sent: Thursday, March 14, 2019 3:42 PM
To: openssl-tpm2-engine@groups.io; Ibmtpm20tss-users@...
Subject: Periodic Openssl TPM2 engine failures

Hello all.

Currently running ibm libtss 1.3.3.1 and openssl engine 2.1.0 connected to openssl 1.0.2 under Alpine 3.8 on 32 bit armhf.

I have tested with engine 2.3.0 and seen similar results.

At a run rate of about 1 of 40 down to 1 of 50 test cases, I get the following failure feedback from the engine.

TPM2_ReadPublic failed with 720963
response code unknown
TPM2_StartAuthSession failed with 720912 TSS_RC_FILE_OPEN - The file could not be opened Failed to get Key Handle in TPM EC key routines

Is anyone else seeing these error codes pop up?

I'm going to enable tpm-tis tracing in the kernel and see what pops out of there.

Sincerely,

Douglas Fraser

Periodic Openssl TPM2 engine failures

Doug Fraser
 

Hello all.

Currently running ibm libtss 1.3.3.1 and openssl engine 2.1.0 connected to openssl 1.0.2 under Alpine 3.8 on 32 bit armhf.

I have tested with engine 2.3.0 and seen similar results.

At a run rate of about 1 of 40 down to 1 of 50 test cases, I get the following failure feedback from the engine.

TPM2_ReadPublic failed with 720963
response code unknown
TPM2_StartAuthSession failed with 720912
TSS_RC_FILE_OPEN - The file could not be opened
Failed to get Key Handle in TPM EC key routines

Is anyone else seeing these error codes pop up?

I'm going to enable tpm-tis tracing in the kernel and see what pops out of there.

Sincerely,

Douglas Fraser

Re: [tpm2] Support for EAP-TLS with openssl TPM2 engine

James Bottomley
 

On Thu, 2019-03-14 at 11:33 -0700, David Woodhouse wrote:
On Thu, 2019-03-14 at 10:50 -0700, James Bottomley wrote:
[...]
Although if you just wanted to use those keys with GnuTLS, you
could have done that directly. I already ported it all except the
new "importable" keys support.

http://git.infradead.org/users/dwmw2/openconnect.git/blob/HEAD:/g
nutls_tpm2_ibm.c
Well, you know, using engines with gnutls does mean we don't have
to write the same code twice over ...
I'm not convinced that an OpenSSL ENGINE is the right form for
implementing this kind of thing in the general case. PKCS#11 is much
better as an existing portable standard, although it doesn't fit the
TPMv2 usage model very well.
Actually, here I disagree: the faults in PKCS11 are why we have a
problem and a boat load of not quite functional solutions: it has no
key id API, so it can't take files, pkcs11: or any other URIs formats
as an argument for a key operation. OpenSSL engines can do all of that
and as evidence, the seamless construction of both a tpm engine and a
p11-kit URI based engine.

Even OpenSSL is moving away from ENGINEs to a different plugin
mechanism.
I'm not averse. My only objection is to the idea that the PKCS11 API
is better than the existing engine one; it's a more widely accepted
standard but it's not a better API. Although, Engines are by no means
perfect as and API as our whining about the engine file selector
problem shows. At least a newer plugin can finally solve the key
identification problem and I think it will fit seamlessly into my
current openssl-pkcs11-exporter.

James

Re: [tpm2] Support for EAP-TLS with openssl TPM2 engine

David Woodhouse
 

On Thu, 2019-03-14 at 10:50 -0700, James Bottomley wrote:
On Thu, 2019-03-14 at 09:57 -0700, David Woodhouse wrote:
On Thu, 2019-03-14 at 09:27 -0700, James Bottomley wrote:
On Thu, 2019-03-14 at 09:19 -0700, Andersen, John wrote:
On Wed, Mar 13, 2019 at 04:56:17PM -0700, David Woodhouse wrote:
Here's a quick hack to make it work by abusing the OpenSC
engine config, as a proof of concept. Making it work cleanly so
that it can be merged is left as an exercise for the reader, or
perhaps an interested party in one of the mailing lists I've
added to Cc.
Well, you can't have the engine name hard coded ... that really
needs to be some type of parameter, which is going to be 99% of the
hassle making a proper patch ...
And of course, it shouldn't have to be specified at all. If given a
PEM file which happens to look like a TPM2 engine key, then the
appropriate engine should be invoked automatically.
Hey don't beat me on the sore spot ...
:)

This isn't really that hard to do in applications. For those using
OpenSSL it's just a case of making them recognise the appropriate
-----BEGIN string and invoke the engine appropriately.

Once my support gets merged into GnuTLS, it really can be automatic
with the application not having to do anything at all. OpenSSL might
get there too, once we have STORE support working in applications.

Although if you just wanted to use those keys with GnuTLS, you could
have done that directly. I already ported it all except the new
"importable" keys support.

http://git.infradead.org/users/dwmw2/openconnect.git/blob/HEAD:/gnutls_tpm2_ibm.c
Well, you know, using engines with gnutls does mean we don't have to
write the same code twice over ...
I'm not convinced that an OpenSSL ENGINE is the right form for
implementing this kind of thing in the general case. PKCS#11 is much
better as an existing portable standard, although it doesn't fit the
TPMv2 usage model very well.

Even OpenSSL is moving away from ENGINEs to a different plugin
mechanism.

Re: [tpm2] Support for EAP-TLS with openssl TPM2 engine

James Bottomley
 

On Thu, 2019-03-14 at 09:57 -0700, David Woodhouse wrote:
On Thu, 2019-03-14 at 09:27 -0700, James Bottomley wrote:
On Thu, 2019-03-14 at 09:19 -0700, Andersen, John wrote:
On Wed, Mar 13, 2019 at 04:56:17PM -0700, David Woodhouse wrote:
Here's a quick hack to make it work by abusing the OpenSC
engine config, as a proof of concept. Making it work cleanly so
that it can be merged is left as an exercise for the reader, or
perhaps an interested party in one of the mailing lists I've
added to Cc.
Well, you can't have the engine name hard coded ... that really
needs to be some type of parameter, which is going to be 99% of the
hassle making a proper patch ...
And of course, it shouldn't have to be specified at all. If given a
PEM file which happens to look like a TPM2 engine key, then the
appropriate engine should be invoked automatically.
Hey don't beat me on the sore spot ...

Just on this particular part: I recently got annoyed with the
inability to use TPM keys on firefox. I did look at the tpm pkcs11
projects but they all looked deficient to say the least, so I put
together this

https://git.kernel.org/pub/scm/linux/kernel/git/jejb/openssl-pkcs11
-export.git

It's a generic engine key to pkcs11 exporter (will work for any
openssl engine) driven by a simple ini like config file. The big
advantage it has is that now I can use openssl engines with gnutls.
Nice. I like the fact that it interoperates with the key storage
format we agreed upon for the ENGINEs.
Actually, it doesn't ... that's the nice thing about the project: it's
entirely key format agnostic. It just takes the engine name and the
key file and hands it to the engine to get it to work. This means it
can work with *any* engine format whatsoever, including a completely
non-compliant TPM one should someone come up with one.

Although if you just wanted to use those keys with GnuTLS, you could
have done that directly. I already ported it all except the new
"importable" keys support.

http://git.infradead.org/users/dwmw2/openconnect.git/blob/HEAD:/gnutl
s_tpm2_ibm.c
Well, you know, using engines with gnutls does mean we don't have to
write the same code twice over ...

James


Going the pkcs11 route is definitely the heath robinson approach,
so the direct engine route is definitely much better.
:)

Re: [tpm2] Support for EAP-TLS with openssl TPM2 engine

David Woodhouse
 

On Thu, 2019-03-14 at 09:27 -0700, James Bottomley wrote:
On Thu, 2019-03-14 at 09:19 -0700, Andersen, John wrote:
On Wed, Mar 13, 2019 at 04:56:17PM -0700, David Woodhouse wrote:
Here's a quick hack to make it work by abusing the OpenSC engine
config, as a proof of concept. Making it work cleanly so that it
can be merged is left as an exercise for the reader, or perhaps an
interested party in one of the mailing lists I've added to Cc.
Well, you can't have the engine name hard coded ... that really needs
to be some type of parameter, which is going to be 99% of the hassle
making a proper patch ...
And of course, it shouldn't have to be specified at all. If given a PEM
file which happens to look like a TPM2 engine key, then the appropriate
engine should be invoked automatically.

Just on this particular part: I recently got annoyed with the inability
to use TPM keys on firefox. I did look at the tpm pkcs11 projects but
they all looked deficient to say the least, so I put together this

https://git.kernel.org/pub/scm/linux/kernel/git/jejb/openssl-pkcs11-export.git

It's a generic engine key to pkcs11 exporter (will work for any openssl
engine) driven by a simple ini like config file. The big advantage it
has is that now I can use openssl engines with gnutls.
Nice. I like the fact that it interoperates with the key storage format
we agreed upon for the ENGINEs.

Although if you just wanted to use those keys with GnuTLS, you could
have done that directly. I already ported it all except the new
"importable" keys support.

http://git.infradead.org/users/dwmw2/openconnect.git/blob/HEAD:/gnutls_tpm2_ibm.c

Going the pkcs11 route is definitely the heath robinson approach, so
the direct engine route is definitely much better.
:)