Usually, a system communicates with others within a small or large map of systems, which makes up a set dedicated to satisfying the needs for which this small universe of agents performing collaborative work was designed.
These needs are those that usually make up the business of the systems map owner, and at this level it no longer makes sense to talk about components, but rather business flows.
At this point we can ask ourselves: Is our responsibility as QA Assurance that all the components of all the systems, and all the systems of a system map work correctly?
No. Our responsibility is to ensure that as well as working correctly individually, it works as a whole. Ultimately, our responsibility is to ensure that the business is secure, and the way to ensure this is through E2E testing.
With these premises and methodology, at MTP we work on all the assignments in the Digital Business Assurance environment mainly, quality assurance (QA), but also in other areas such as DevOps & Agile, cybersecurity or user experience (UX).
Error Detection in E2E Testing
The aim of these tests is the same as any other type of test: to detect errors. But the E2E perspective allows us to go a step further and, apart from errors with a more or less immediate visibility, we will be able to determine the existence of functional ambiguity or hidden errors.
In E2E tests we mainly detect:
- Errors in communication definition between two systems: A system sends some parameters, but the system with which it communicates expects to receive others and an error is generated.
- Absence of a system: We do not have the updated version of a system that is part of the business flow to be tested, or it has not been considered that this system must participate or must be modified.
- Error in the definition of the functioning of the functional flow: All components and systems work, we have executed the test from start to finish without errors, but the final result is not as expected. This is the great power of E2E testing, the detection of definition errors and the solution usually involves a redefinition of the process.
Let’s get down to work
Having understood the particularities and the aim of these kind of tests, below we describe the relevant points when planning them.
1.- The test team.
For the composition of the E2E test team, we will select profiles that meet three main requirements:
- QA Expertise: Those designing and executing this type of testing should not be juniors, they should have experience in manual testing that they will apply to this more complex scenario.
- Functional analysts: We will prioritise functional profiles over technical ones. Profiles that easily learn and understand the process as a whole, as opposed to the parts.
- Good communicators: A vital aspect, the agile resolution of errors detected in a scenario with many different system agents, will only be possible if the profile testing is able to properly communicate the errors in the right way, and to the right agents.
2.- The design of the tests.
Now that we have the team in place, we want to design a test plan that contemplates all the necessary validations to be carried out. Concepts to consider in our design:
- Functional flow: Process in which a behaviour is defined from some inputs, and a processing of those inputs by all the systems or components, until reaching a final result.
- Critical path: A functional flow typically involves many systems or components. It would be a mistake to try to validate the whole, because on the one hand there is usually no material time to do so, and on the other hand, many of the systems are not relevant. The critical path is composed only of the systems to be validated in the E2E test.
- Systems to validate: We will encounter 3 types of systems to validate in the E2E test of the functional flow, and which will define the critical path:
- Systems that modify SW: Systems that are going to modify the behaviour of the flow and are therefore the main object of the test.
- Data input and output systems: Systems on which we operate and test the result. Regardless of whether SW is modified in them, they are necessary for the correct execution of the test.
- Intermediate systems with high criticality: Systems that without having a SW modification, nor being data input or output, have an important criticality in the process and their corroboration is necessary.
Our test plan will be designed with the objective of validating a functional flow, and step by step, the operation and validation of the systems contained within the critical path.
3.- The Execution Phase.
We have a team in place, the test plan has been designed, and it is time to execute. Our execution is already guided by our design, and all that remains is to validate and operate on the already defined systems. However, for the execution stage to be successful, it is necessary to go deeper into several important points:
- The test environment: Our scenario requires a stable test environment because any interference in one part of the systems can modify and invalidate our execution.
- Incident and crash reporting: A vital aspect is to clearly and immediately report the incidents detected to the team that must analyse them and facilitate communication with the teams that may be responsible for the resolution or those involved by providing information.
- Adequate and regular reporting of progress: Reporting the correct or incorrectness of a business flow with many systems involved often means that you do not have ‘correct cases’ until near the end of the testing time.
This may cause unease, so a good practice is to report progress within the process, i.e., report ‘OKs’ at intermediate steps, alongside reporting ‘OKs’ at full flow validation.
Are there any more key aspects to consider?
Well, there are many. The E2E scenario is heterogeneous depending on the type of client, type of project, extent of the systems map, existence of dedicated environments, etc. However, here are some additional key points to consider for a successful process:
- Definition of the testing process: A definition of the process that is known by all the agents will result in agility and efficiency. For example, it is very important to define the lifecycle of the incidents, where it is immediately clear which system and agent must review the reported error.
- Tools: We must choose tools that have centralised information and that serve to record and show progress and blockages in an agile way.
- Continuous training: E2E testing creates a very important need for training in the testing team. The head of the testing team should plan training sessions that make all the information available to all members of the team.
- E2E regression: Not only is it important to do E2E progression testing, the potential of this testing also lies in regression. Ultimately in ensuring that initially small or large changes do not generate errors in critical production flows.
- Automation: Alongside the previous point, by investing in the automation of regression E2E flows you are investing in the productivity of the testing team. It also makes it possible to validate complex scenarios remotely.
Sergio Peñalvo
Senior Manager at MTP