Deterministic JVM (DJVM)


Tommy Lillehagen <tommy.lillehagen@...>
 

Contract Verification on a Deterministic JVM (DJVM)

 

Hi all,

 

Why do we need a deterministic sandbox?

 

It is important that all nodes that process a transaction always agree on whether it is valid or not. Because transaction types in Corda are defined using JVM byte code, this means that the execution of that byte code must be fully deterministic. Out of the box, a standard JVM is not fully deterministic, thus we must make some modifications in order to satisfy our requirements.

 

So, what does it mean for a piece of code to be fully deterministic? Ultimately, it means that the code, when viewed as a function, is pure. In other words, given the same set of inputs, it will always produce the same set of outputs without inflicting any side-effects that might later affect the computation.

 

What’s the current state of affairs?

 

In our pursuit of enabling deterministic contract verification in Corda, we have worked on a deterministic sandbox (DJVM) which will provide a safe environment for such verification methods to run in. As part of that work, I have just pushed a pull request for the first iteration of our DJVM work to our Corda repository; see corda/corda#3386. This will form the foundation for the future deterministic sandbox used in Corda. Before we get there though, the code needs thorough review, some further improvements and modifications, etc.

 

As you will see from the PR, the code in the top-level DJVM module has not yet been integrated with the rest of the platform. It will eventually become a part of the node and enforce deterministic and secure execution of smart contract code, which is mobile and may propagate around the network without human intervention. However, for now it stands alone as an evaluation version.

 

Preview and request for comments

 

We want to give developers the ability to start trying it out and get used to developing deterministic code under the set of constraints that we envision will be placed on contract code in the future. For details around the concept, implementation, constraints, scope, etc., please refer to the documentation site. A snapshot of the added documentation can be found on the PR: docs/source/key-concepts-djvm.rst.

 

NOTE: This is quite a meaty piece of work, and the PR is indeed quite large. I expect there to be quite a few comments and rounds of feedback, and for the PR to be up for a while before merging… and this is indeed the reason for my email – it would be great to get this in front of as many of our users as possible (i.e., you!), for us to perform a thorough review of the work covered by this PR, but also to start thinking about the follow-up work beyond this initial version of the sandbox. This is why I’m calling on anyone interested to take a look and provide feedback.

 

It should be mentioned that this work ties in closely with our development efforts on the SGX team, working towards a deterministic sandbox for running contract verification inside an SGX enclave. If you’re curious to learn more about that work, have a look at the following material:

 

  • Mike Hearn’s presentation about SGX and Corda (video, blog post)
  • Andras Slemmer’s post on our SGX infrastructure design (post)
  • Note also that there will be another SGX blog post coming soon (so stay tuned)!

 

Have a great weekend! Look forward to hearing your feedback on this.

 

Best regards,

Tommy Lillehagen

www.r3.com/email-disclaimer


Tommy Lillehagen <tommy.lillehagen@...>
 

Further to my previous email, it’s also worth noting the great work that Chris Rankin has been doing on making deterministic versions of our core and serialisation modules. Here’s a couple of links for reference:

 

  • Create deterministic Core and Serialisation artefacts (corda#3262)
  • Deterministic Corda modules (docs site)

 

Make sure you check those out as well if you’re interested in this topic.

www.r3.com/email-disclaimer


Tommy Lillehagen <tommy.lillehagen@...>
 

I've posted two Medium stories on the topic:


richard@...
 

Thanks Tommy – this is great!  There seems to be interest beyond the Corda community judging by the number of retweets and comments I received over the weekend when I publicised your first post.  I’ll share it some more during working hours too.  I think it’s really important that we get the eyes of the broader Java community on this

 

Richard

 

Richard G Brown R3| Chief Technology Officer

2 London Wall Place | Floor 12 | London | EC2Y 5AU

M: +44 7764 666821 | T: @gendal

richard@... . www.r3.com

 

From: <corda-dev@groups.io> on behalf of "Tommy Lillehagen via Groups.Io" <tommy.lillehagen@...>
Reply-To: "corda-dev@groups.io" <corda-dev@groups.io>
Date: Sunday, 17 June 2018 at 13:13
To: "corda-dev@groups.io" <corda-dev@groups.io>
Subject: Re: [corda-dev] Deterministic JVM (DJVM)

 

I've posted two Medium stories on the topic:

·         Deterministic Contract Verification

·         DJVM Behind the Scenes

www.r3.com/email-disclaimer


Chris Rankin
 


Writing a Verifiable Contract for a Deterministic JVM

So Corda will verify contracts within a Deterministic JVM - but this just begs the question: "How do you write a contract that can run within the DJVM?"

We need to provide tools that will allow CorDapp developers to do this. And so to this end, we have been steadily chipping away at the Java APIs to identify a deterministic subset without APIs for accessing file systems, networks, the system clock, or any other sources of external entropy. We have also created new modules corda-core-deterministic and corda-serialization-deterministic which contain just the deterministic functionality from corda-core and corda-serialization respectively. Contract classes which use only these deterministic Corda modules and Java APIs should then execute successfully on the DJVM. (Obviously those classes not needed for contract verification can continue using the standard Corda modules as before.)

You can find more details about how we generate these deterministic modules here, including instructions on how to integrate the deterministic Java APIs with IntelliJ.
All the code for these modules has been merged with the master branch in the Corda repository for everyone to examine, and we expecting to continue fine-tuning the modules' contents as we further develop the DJVM.

We will also be providing a template CorDapp project which will demonstrate how to build a CorDapp that has both deterministic and non-deterministic classes.

Please take this code out for a spin, and let us know what you think.

Cheers,
Chris Rankin


From: corda-dev@groups.io <corda-dev@groups.io> on behalf of Tommy Lillehagen via Groups.Io <tommy.lillehagen@...>
Sent: 15 June 2018 15:06
To: corda-dev@groups.io
Subject: [corda-dev] Deterministic JVM (DJVM)
 

Contract Verification on a Deterministic JVM (DJVM)

 

Hi all,

 

Why do we need a deterministic sandbox?

 

It is important that all nodes that process a transaction always agree on whether it is valid or not. Because transaction types in Corda are defined using JVM byte code, this means that the execution of that byte code must be fully deterministic. Out of the box, a standard JVM is not fully deterministic, thus we must make some modifications in order to satisfy our requirements.

 

So, what does it mean for a piece of code to be fully deterministic? Ultimately, it means that the code, when viewed as a function, is pure. In other words, given the same set of inputs, it will always produce the same set of outputs without inflicting any side-effects that might later affect the computation.

 

What’s the current state of affairs?

 

In our pursuit of enabling deterministic contract verification in Corda, we have worked on a deterministic sandbox (DJVM) which will provide a safe environment for such verification methods to run in. As part of that work, I have just pushed a pull request for the first iteration of our DJVM work to our Corda repository; see corda/corda#3386. This will form the foundation for the future deterministic sandbox used in Corda. Before we get there though, the code needs thorough review, some further improvements and modifications, etc.

 

As you will see from the PR, the code in the top-level DJVM module has not yet been integrated with the rest of the platform. It will eventually become a part of the node and enforce deterministic and secure execution of smart contract code, which is mobile and may propagate around the network without human intervention. However, for now it stands alone as an evaluation version.

 

Preview and request for comments

 

We want to give developers the ability to start trying it out and get used to developing deterministic code under the set of constraints that we envision will be placed on contract code in the future. For details around the concept, implementation, constraints, scope, etc., please refer to the documentation site. A snapshot of the added documentation can be found on the PR: docs/source/key-concepts-djvm.rst.

 

NOTE: This is quite a meaty piece of work, and the PR is indeed quite large. I expect there to be quite a few comments and rounds of feedback, and for the PR to be up for a while before merging… and this is indeed the reason for my email – it would be great to get this in front of as many of our users as possible (i.e., you!), for us to perform a thorough review of the work covered by this PR, but also to start thinking about the follow-up work beyond this initial version of the sandbox. This is why I’m calling on anyone interested to take a look and provide feedback.

 

It should be mentioned that this work ties in closely with our development efforts on the SGX team, working towards a deterministic sandbox for running contract verification inside an SGX enclave. If you’re curious to learn more about that work, have a look at the following material:

 

  • Mike Hearn’s presentation about SGX and Corda (video, blog post)
  • Andras Slemmer’s post on our SGX infrastructure design (post)
  • Note also that there will be another SGX blog post coming soon (so stay tuned)!

 

Have a great weekend! Look forward to hearing your feedback on this.

 

Best regards,

Tommy Lillehagen

www.r3.com/email-disclaimer
www.r3.com/email-disclaimer


joel.dudley@...
 

Hi all,

Reviving this thread to let you know that the recent release of Corda 4.4 marks a major milestone for the DJVM:
  • Nodes can be configured to use the DJVM for every smart contract they evaluate, either via a configuration file or via the Corda DemoBench. Any nodes configured in this way will automatically detect and reject any non-deterministic transactions they receive
  • A new Gradle script plugin allows most non-determinism in smart contracts to be detected at compile time, before a node is even started
You can read more about it and see a quick demo here: https://medium.com/corda/the-djvm-in-action-34f3b852e051.

If you get a chance, please download Corda 4.4, enable the DJVM, and give it a test run! We look forward to hearing how this affects your existing contracts - let us know on this thread or at productfeedback@....

Thanks,
Joel


Sean
 

Hi Joel and team,
  I ran Corda jdvm that came with CE4.4 on my CorDapp. First, the gradle script plugin passed with no complaints. And then I ran CordApp, i got the following error on an enum class.
[ERROR] 15:27:02-0700 [flow-worker] corda.flow. - net.corda.node.internal.djvm.DeterministicVerificationException: NotSerializable: Deserializing obj as enum class sandbox.com.mycompany.membership.PartyRole with value CONTROLLER.0 but ordinality has changed, transaction: AE0324817AD08889C49FE5E270AC3086178ADB7D58E3ADC7AF3E50E5C41B195D {actor_id=user1, actor_owning_identity=CN=PartyC, O=PartyC, L=Dallas, C=US, actor_store_id=NODE_CONFIG, fiber-id=10000001, flow-id=50a2c0a5-e27e-4bf3-b070-d2c6816be7a6, invocation_id=2403e835-7f09-4eaf-a4a4-0773d036068b, invocation_timestamp=2020-04-16T22:26:51.472Z, origin=user1, session_id=e7d568c8-bf69-4d18-b995-ccc5545852dc, session_timestamp=2020-04-16T22:10:49.151Z, thread-id=736, trackingId=bc4b324d-ea3e-4ce7-adff-a8df3e3c9c94}

Here is the simple enum class.
@CordaSerializable
interface Role {
    val roleName: String
    val description: String
}
 
@CordaSerializable
enum class PartyRole(override val roleName: String, override val description: String) : Role {
    CONTROLLER(roleName = Controller", description = "Party that is authorized to control the process."),
    WORKER(roleName = "Worker", description = "Party that is authorized to do work only.");
 
    override fun toString(): String = roleName
}

Not sure what makes DJVM unhappy. Is there a fix?

Thanks.

\Sean
 


joel.dudley@...
 

Hey Sean,

Thanks for reporting this issue. We looked into it and it turns out it was a bug on our end - SandboxEnumSerializer uses Enum.toString() to determine the instance's name, but PartyRole overrides the toString() method. Hence deserialization fails. Sorry about that.

We've created a bug ticket to fix the issue. If you want to test it further, you should be able to work around it for now by removing your override of
toString().

Thanks again,
Joel


Sean
 

Ok, Joel. I removed my custom toString() and the flow moved past the previous error.
But now I have the following error. It looks like my existing data is not consistent with the default toString(). I may have to consume the existing states and start all over again. I will give that a try and continue testing from there. In the meantime, if you have any other ideas to solve the error, please advise.

[ERROR] 10:02:53-0700 [flow-worker] corda.flow. - net.corda.core.node.services.VaultQueryException: An error occurred while attempting to query the vault: Failed to deserialise group OUTPUTS_GROUP at index 0 in transaction: net.corda.core.contracts.TransactionState (erased) -> data(net.corda.core.contracts.ContractState) -> Cannot construct evolution serializer for remote type
com.aurum.membership.PartyRole: Cannot resolve local enum member Issuer to a member of [CONTROLLER, WORKER] using rules {}
 
Full type information:
com.mycompany.membership.PartyRole(Controller|Worker) {actor_id=user1, actor_owning_identity=CN=PartyC, O=PartyC, L=Dallas, C=US, actor_store_id=NODE_CONFIG, fiber-id=10000001, flow-id=720adaac-dff8-4575-9d7d-16f7654ecae0, invocation_id=3d77e91f-81ec-4f23-9fbf-80ac307106f7, invocation_timestamp=2020-04-20T17:02:53.181Z, origin=user1, session_id=8c4ad9d2-e844-45b9-8bc7-423593377d01, session_timestamp=2020-04-20T17:00:51.095Z, thread-id=587, trackingId=e6537dd1-a987-4b5d-b749-0b51e3a1c92b}