Tests more complex than the solution they drive


paulnmackintosh
 

I am currently working on what you could categorise as a workflow automation product.

The product is comprised of independently deployed front and back end components.


The core domain is realised by a single .Net Core solution.  The front end interacts with it via a REST API and it interacts with further components (and occasionally itself) using messaging.


Two core aspects of the internal design are Entity Framework for data access and extensive use of the mediator pattern.

Using mediator, every element of the solution is broken down into sets of 3 classes, request, handler and result.  Requests and results are POCOs whilst handlers implement behaviour and require few dependencies.  Often, in cases where the only job to do is orchestrate the sending and receiving of requests and results, the sole dependency is the mediator type itself.  Other requests fall into the two categories of query or command.


All told a well thought out and pleasant stack to work in.  Mediator leads to few dependencies, which in turn leads to simple classes that are easy to drive with tests.


Except for the fact that nothing is ever quite as simple as one would like.


There is a problem writing tests in this solution in that to test a simple isolated query the test setup must replay all the steps the application would theoretically take in order to reach the point whereby the query can return sensible results when issued against Entity Framework over an in memory database.

This is achieved by gathering and then issuing a collection of steps, where each step is a high-level mediator message, as the “Given”s and “When”s of a test scenario.  During the “Then” assertions are made against the result POCOs that were observed by a stub implementation of the mediator type, e.g. assert that the last created entity of type “A” should have such and such an “A.Name”.


I believe the goal of the approach is to use the real application code to avoid making assumptions about the state of the system when writing tests, however the more steps required to reach a the prerequisite snapshot of state, the greater the reliance on developers writing new tests to comprehend the set of steps that should be replayed in order to reach the point from which they want to proceed.


It feels to me as though what has gone wrong here is that although the application code has a nicely decoupled architecture the database and/or the Entity Framework domain model layer does not and as a result entities depend on other entities in a way that makes them intrinsically co-dependent.  This interdependency is revealed by tests which cannot drive simple code without themselves being complicated.


I thought perhaps I could lean on the experience in this group to gain some suggestions on how to reduce this kind of test friction?


Paul


Russell Gold
 

I’m a bit confused by your description, as it sounds as though you are doing system tests. That’s the most obvious case in which you’d need to “replay all the steps the application would theoretically take in order to reach the point whereby the query can return sensible results.” One of the advantages of unit tests is that you don’t do that, as you are testing each small part of the system.

Is the problem that even the small parts have to be brought to a state via complex steps? 

Russ
-----------------
Author, Getting Started with Apache Maven <http://www.packtpub.com/getting-started-with-apache-maven/video>
Author, HttpUnit <http://www.httpunit.org> and SimpleStub <http://simplestub.meterware.com>
Now blogging at <http://russgold.net/sw/>

Have you listened to Edict Zero <https://edictzero.wordpress.com>? If not, you don’t know what you’re missing!





On Sep 7, 2020, at 6:10 AM, paulnmackintosh <pnm@...> wrote:

I am currently working on what you could categorise as a workflow automation product.

The product is comprised of independently deployed front and back end components.

The core domain is realised by a single .Net Core solution.  The front end interacts with it via a REST API and it interacts with further components (and occasionally itself) using messaging.

Two core aspects of the internal design are Entity Framework for data access and extensive use of the mediator pattern.
Using mediator, every element of the solution is broken down into sets of 3 classes, request, handler and result.  Requests and results are POCOs whilst handlers implement behaviour and require few dependencies.  Often, in cases where the only job to do is orchestrate the sending and receiving of requests and results, the sole dependency is the mediator type itself.  Other requests fall into the two categories of query or command.

All told a well thought out and pleasant stack to work in.  Mediator leads to few dependencies, which in turn leads to simple classes that are easy to drive with tests.

Except for the fact that nothing is ever quite as simple as one would like.

There is a problem writing tests in this solution in that to test a simple isolated query the test setup must replay all the steps the application would theoretically take in order to reach the point whereby the query can return sensible results when issued against Entity Framework over an in memory database.
This is achieved by gathering and then issuing a collection of steps, where each step is a high-level mediator message, as the “Given”s and “When”s of a test scenario.  During the “Then” assertions are made against the result POCOs that were observed by a stub implementation of the mediator type, e.g. assert that the last created entity of type “A” should have such and such an “A.Name”.

I believe the goal of the approach is to use the real application code to avoid making assumptions about the state of the system when writing tests, however the more steps required to reach a the prerequisite snapshot of state, the greater the reliance on developers writing new tests to comprehend the set of steps that should be replayed in order to reach the point from which they want to proceed.

It feels to me as though what has gone wrong here is that although the application code has a nicely decoupled architecture the database and/or the Entity Framework domain model layer does not and as a result entities depend on other entities in a way that makes them intrinsically co-dependent.  This interdependency is revealed by tests which cannot drive simple code without themselves being complicated.

I thought perhaps I could lean on the experience in this group to gain some suggestions on how to reduce this kind of test friction?

Paul


paulnmackintosh
 

The in solution 'framework' that has evolved for these tests does have a bit of a system test feel to it, although it is being leant on to test drive new classes with new behaviours.
Here's an example in pseudo code in case it helps:

[Fact]
public async Task ExampleFact()
{
  await TestScenario

    .Given(builder => builder

      .UserCreatesEntityA("Entity A")

      .UserCreatesEntityB("Entity B", new[] new DetailForB { Name = "Detail B"}, })
      .SystemCreatesEntityC()
      .SystemActsOnAAndBAndC()
    .When(builder =>
      builder.ExerciseNewUnitUnderTest())
    .Then(results =>
    {
      // use results to assert
    });
}


Rob Park
 

For me, it's great that you have 2 independently deployable pieces. Though I would ensure I also had independently runnable tests. With the exception of having contract tests to test your expectations of the backend interactions. This will have benefits of being easier to reason about (at least once you start to feel comfortable with the concept); faster test suites; less brittle maintenance.

In a similar setup to what you have, I've had end-to-end system tests in the past, but I literally only had 1 or 2 validating the most important use case still worked after each deploy (of either side).

Good luck!


On Mon, Sep 7, 2020 at 7:17 AM paulnmackintosh <pnm@...> wrote:

The in solution 'framework' that has evolved for these tests does have a bit of a system test feel to it, although it is being leant on to test drive new classes with new behaviours.
Here's an example in pseudo code in case it helps:

[Fact]
public async Task ExampleFact()
{
  await TestScenario

    .Given(builder => builder

      .UserCreatesEntityA("Entity A")

      .UserCreatesEntityB("Entity B", new[] new DetailForB { Name = "Detail B"}, })
      .SystemCreatesEntityC()
      .SystemActsOnAAndBAndC()
    .When(builder =>
      builder.ExerciseNewUnitUnderTest())
    .Then(results =>
    {
      // use results to assert
    });
}


Avi Kessner
 

Do you have common use cases where you can use the builder pattern to create the context? Or is the set of steps different in each scenario?
brought to you by the letters A, V, and I
and the number 47


On Mon, Sep 7, 2020 at 4:55 PM Rob Park <robert.d.park@...> wrote:
For me, it's great that you have 2 independently deployable pieces. Though I would ensure I also had independently runnable tests. With the exception of having contract tests to test your expectations of the backend interactions. This will have benefits of being easier to reason about (at least once you start to feel comfortable with the concept); faster test suites; less brittle maintenance.

In a similar setup to what you have, I've had end-to-end system tests in the past, but I literally only had 1 or 2 validating the most important use case still worked after each deploy (of either side).

Good luck!


On Mon, Sep 7, 2020 at 7:17 AM paulnmackintosh <pnm@...> wrote:

The in solution 'framework' that has evolved for these tests does have a bit of a system test feel to it, although it is being leant on to test drive new classes with new behaviours.
Here's an example in pseudo code in case it helps:

[Fact]
public async Task ExampleFact()
{
  await TestScenario

    .Given(builder => builder

      .UserCreatesEntityA("Entity A")

      .UserCreatesEntityB("Entity B", new[] new DetailForB { Name = "Detail B"}, })
      .SystemCreatesEntityC()
      .SystemActsOnAAndBAndC()
    .When(builder =>
      builder.ExerciseNewUnitUnderTest())
    .Then(results =>
    {
      // use results to assert
    });
}


paulnmackintosh
 

Yes, I can see a fair number of tests that are currently using common setup steps.  These could be encapsulated at a higher level of abstraction and doing so will remove duplication in the test code.  I wonder if this will lead to slow test execution over time though as it will manage complexity consistently rather than reduce it.


Russell Gold
 

Is it possible to place an object in a desired state without all of the builder steps? If so, you could then give it stimuli and test just single state transitions. I would be concerned, if I worked on a system like this, than the testing time would become rather problematic.

On Sep 7, 2020, at 7:17 AM, paulnmackintosh <pnm@...> wrote:

The in solution 'framework' that has evolved for these tests does have a bit of a system test feel to it, although it is being leant on to test drive new classes with new behaviours.
Here's an example in pseudo code in case it helps:

[Fact]
public async Task ExampleFact()
{
  await TestScenario

    .Given(builder => builder

      .UserCreatesEntityA("Entity A")

      .UserCreatesEntityB("Entity B", new[] new DetailForB { Name = "Detail B"}, })
      .SystemCreatesEntityC()
      .SystemActsOnAAndBAndC()
    .When(builder =>
      builder.ExerciseNewUnitUnderTest())
    .Then(results =>
    {
      // use results to assert
    });
}



Dave Foley
 

Is the design such that handlers manipulate EF collections directly?

On Sep 7, 2020, at 3:10 AM, paulnmackintosh <pnm@...> wrote:



I am currently working on what you could categorise as a workflow automation product.

The product is comprised of independently deployed front and back end components.


The core domain is realised by a single .Net Core solution.  The front end interacts with it via a REST API and it interacts with further components (and occasionally itself) using messaging.


Two core aspects of the internal design are Entity Framework for data access and extensive use of the mediator pattern.

Using mediator, every element of the solution is broken down into sets of 3 classes, request, handler and result.  Requests and results are POCOs whilst handlers implement behaviour and require few dependencies.  Often, in cases where the only job to do is orchestrate the sending and receiving of requests and results, the sole dependency is the mediator type itself.  Other requests fall into the two categories of query or command.


All told a well thought out and pleasant stack to work in.  Mediator leads to few dependencies, which in turn leads to simple classes that are easy to drive with tests.


Except for the fact that nothing is ever quite as simple as one would like.


There is a problem writing tests in this solution in that to test a simple isolated query the test setup must replay all the steps the application would theoretically take in order to reach the point whereby the query can return sensible results when issued against Entity Framework over an in memory database.

This is achieved by gathering and then issuing a collection of steps, where each step is a high-level mediator message, as the “Given”s and “When”s of a test scenario.  During the “Then” assertions are made against the result POCOs that were observed by a stub implementation of the mediator type, e.g. assert that the last created entity of type “A” should have such and such an “A.Name”.


I believe the goal of the approach is to use the real application code to avoid making assumptions about the state of the system when writing tests, however the more steps required to reach a the prerequisite snapshot of state, the greater the reliance on developers writing new tests to comprehend the set of steps that should be replayed in order to reach the point from which they want to proceed.


It feels to me as though what has gone wrong here is that although the application code has a nicely decoupled architecture the database and/or the Entity Framework domain model layer does not and as a result entities depend on other entities in a way that makes them intrinsically co-dependent.  This interdependency is revealed by tests which cannot drive simple code without themselves being complicated.


I thought perhaps I could lean on the experience in this group to gain some suggestions on how to reduce this kind of test friction?


Paul


paulnmackintosh
 

It depends, query handlers tend to have a repository interface injected, the repository encapsulates the EF LINQ in a read only way.  Command handlers will generally perform operations on EF collections using methods exposed on the Domain Entities and then saving the changes.  Orchestrating handlers, in theory, shouldn't need to modify entities, their job is to pass query results to command handlers.