Those who are already in the IT sector are familiar with the concept of software testing. Testing is a fundamental part of the development lifecycle. Testers aim to find as many functional and non-functional bugs as they can, where business requirements have not been met, whilst also making detailed observations about the application’s usability and potential readiness for production release.

As delivery teams continue to strive to maximise both application quality and release frequency, testers and a team’s test strategy have had to become increasingly agile and adaptive. However, in any software delivery across any industry sector, there will continue to be the need for both manual and automated testing.

Companies with a solid automation strategy can hope to achieve a 70/30 or 80/20 ratio of automated to manual testing effort with the right test tooling choices and tester skill set. 

This article will provide a high-level understanding of what test automation in software testing is all about. 

Let's go over the basics of software testing and automation and why it is considered critically important for agile software delivery.

Introduction to Automation Testing

Let's begin with a formal definition. Automation is the procedure of automating a manual process with the use of technology. The main aim of automation is to eliminate human errors and introduce efficiency, repeatability and timeliness in the process.

In simple terms, automated testing is an approach to software testing that involves the use of test automation tools with which testers and/or developers can write and execute tests in a more consistent, more timely and more repeatable manner.

A test is executed, assertions are made against a variety of test conditions and when expected results do not match actual results, an error will be thrown. 

With an automated software testing approach, the tester develops and runs the automated tests locally or may schedule them for regular daily execution for example, or more likely upon the release of any new code to a test environment. We refer to this as continuous testing. 

So it helps bring down the testing time, resulting in faster launching of the product. 

Test automation engineers developers might choose to write their automated tests in a coded test automation framework with C#, Ruby, JavaScript, TypeScript or Java programming languages.

Alternatively, they may choose to use a test tool that offers them the ability to record a test simply by navigating the application under test in the browser.

These types of automation tools are often referred to as no-code or low-code testing tools.

Criteria for Automation

The primary aim of automation is to reduce human error and time, so the choice of exactly which tests to automate is also very important.

We should automate these tests:

• Critical, High risk end-to-end UI tests that quickly determine the latest application quality

• API tests, Desktop and Mobile browser, Native App

• Functional tests with included accessibility checks

• Functional tests with included lighthouse audits

• Cross browser, Cross device non-functional tests

• End to end user journeys for which data-driven iteration achieves significant coverage - e.g. Ecommerce with multi locales, multi product carts, multi payment methods, multi customer profiles

• Long flow, Multi application tests

• Tests that must consider overnight batch runs or other scheduled environmental events

• Extreme functional edge case tests

We shouldn’t automate these tests:

• A test that really only requires execution one time

• Low risk tests that are infrequently performed

• Exploratory tests

• In depth security and penetration tests

The Agile Test Automation Pyramid

The test automation pyramid is a good practice conceptual model for the types and volume of automated tests of an application at 3 core layers. It promotes increasing numbers of automated tests through each of the layers from top to bottom:

• UI and End-to-end Testing

• Integration and API Testing

• Unit Testing

Unit Testing

Unit tests are the strong base for the test pyramid. The main focus of unit testing is to test small components or units of the codebase. Unit testing makes sure that the components behave as predicted in isolation.

Integration Tests and API Testing

Whilst unit tests are required to test the code at an individual component level, they are not intended nor are they able to test integrated units of code when coupled together.

We use integration tests for this reason and they make up the middle layer of the automation pyramid. As we climb higher in the pyramid, the volume of tests will generally be less than the previous layer below it.

Integration tests are where we start to consider the external dependencies of our application. For an ecommerce application, we might need to mock a response from a payment provider for example or a product recommendations engine.

Whilst we do not plan to test these external services themselves, we do need to test that our application can send and receive messages and process the resultant outcomes correctly.

In a de-coupled or headless architecture with different development teams delivering the front and backend of the application under test, we will expect to see a middleware layer consisting of numerous APIs (usually REST Services) that we can call to perform certain functions.

Before the front end is even available for end-to-end testing, we can undertake significant functional testing at this middle REST layer of the application to ‘add a product to cart’ for example or to ‘create a customer profile’.

These types of automated tests do not require a browser and are not typically prone to bad performance issues and so it makes sense that we automate more tests at this level then we do at the next stage up - UI and End to End.

UI and End-to-End Testing

This is the top most layer of the automation pyramid. Ultimately, we want to ensure our whole application performs as expected and we will also want to consider it from the perspective of the target user, which is where end-to-end UI (user interface) tests will help. As the name implies, end-to-end tests test the functional completeness of your application-under-test from beginning to end.

They will drive the front end or user interface which in turn will call any middleware REST or SOAP services and then finally the backend or database layers.

Myths about Software Test Automation

Let’s consider some common myths about test automation:

Testers are dependent on developers to deliver good quality automation code

With the early involvement of testers in the tool selection process, the right training, an active test community and plenty of real world project experience on which to work on daily, there need be no assumed dependency on the developers to facilitate the testers’ automation contributions.

Automation testers should use the same coding language as the developers

Again, this assumes that testers do not have the knowledge to own and maintain their own automation strategy and at some point, they might require the developers to come to the rescue.

A high percentage of automation means you don’t need to undertake any manual testing

A high percentage of automated testing simply means that you can do more effective targeted and edge case based exploratory testing - the best kind of manual testing. This is how you find the nasty bugs.

Test recording tools produce tests that are as brittle or as flakey as they were 20 years ago and should be avoided

With the entry of low code recorders and use of fallback locators supporting the concept of self-healing tests, record and play tool offerings have improved dramatically in the last 5 years. The best solutions offer accelerators to get you started quickly and access to underlying code to upskill your testers and further enhance your framework and tests.

You can get rid of your manual testers and your developers should own the testing

In our experience, developers would prefer to develop code, supported with tests at the unit and integration level, but not be responsible for iterative data driven testing of the APIs or the front end. They certainly won’t have time to support the functional automation with manual exploratory testing.

A test team having familiarity with user acceptance criteria, with an independent remit for considering application quality and user experience from multiple perspectives, continues to be a very sensible investment for any quality-driven organisation.

It’s hard to onboard a manual tester in a short space of time

With the current breed of test recording tools and very accessible development languages like Ruby and JavaScript, manual testers moving to automation for the first time can be productive in the first sprint.

Automation comes with a high cost and it takes months to see a return on your investment

On a new project, a team embracing test automation with just a handful of tests to begin with, will still experience the immediate benefit of being able to run those tests as many times a day as they wish, as dictated by the new code builds to various environments. This is work that will no longer need to be done manually and the numbers of functional regression tests will grow very quickly.

The knowledge that those testers can then bring to subsequent organisation projects means that an early and upfront investment in your staff as well as your tools will be repaid many times over on a longer term basis.

It’s possible to automate all of your testing

This is not a realistic aspiration and indeed not possible. For many companies, getting to 40-50% automation of functional test cases is a massive stride forwards. With the right tooling and skilled testers, getting to +70% is absolutely achievable.

Benefits of Automated Testing

Let’s discuss some of the major advantages of a robust automated testing strategy

• Automation testing helps to significantly reduce test execution time.

• It can be continually repeated with a variety of different input data to enhance testing coverage in a way that a manual test team could never do in an acceptable time frame.

• It can drastically reduce the total cost of application ownership to the Product Owner.

• It will facilitate the most important role of any testing team which should always be to report any observations on application quality as soon as possible, to the development team.

Explore more benefits of Test automation

Automated Testing vs Manual Testing

• As the name suggests, manual testing is undertaken by humans, and automated testing is executed with the assistance of automation tools and frameworks.

• Manual testing brings a higher risk of human error with repeated cycles of repetitive functional testing. Automated testing is not free from human error if tests are badly coded, but code reviews and repeated test running will ensure that automated tests as a rule, are far more consistently applied with a more reliable set of outcomes for analysis.

• A single cycle of manual regression testing will take significantly more time and resources than multiple cycles of automated test runs providing the same functional coverage.

• Programming knowledge is often necessary for automation testing, but not for manual testing.

• You cannot automate an exploratory test

• Your manual testing is made more effective if you have a solid automation strategy in place.

General Test Automation Process

The process for continual automation testing and test pack enhancement will typically follow this repeating cycle:

Prepare your test environment (including any externally integrated applications and data)

The environment and the data must be in a suitable state for your tests to run. Having complete control over your environment and data is critical for consistent and accurate testing.

Write your automated tests

The environment and the data must be in a suitable state for your tests to run. Having complete control over your environment and data is critical for consistent and accurate testing.

Establish your build pipeline + run your tests

A build tool like Jenkins can be used to ‘trigger’ an automation run based upon a deployment event such as new code being released to the UAT environment. Whenever this happens, the latest versions of your tests will be ‘pulled’ from their source code repository and executed.

You may also choose to ‘schedule’ test runs to happen on a certain daily or weekly frequency against 1 or more environments

Report on your tests and analyse any failures

Failing tests need to be assessed as soon as possible. Real time results dashboards make this very straightforward to do with full stack trace information on failures and the tester not needing to wait hours for all results to be in. Dashboards should be shared continually within the wider team for full transparency of the testing process and to provide awareness of functional coverage achieved.

Raise bugs

A tester’s most important job is to raise legitimate functional bugs within new features or existing feature regression, as quickly as possible with the developers.

Address any tests that need to be made more robust to run more reliably

In the event that a test fails due to it not being resilient enough to handle changeable environment performance for example, that test should be refactored until it runs reliably. This might also apply for tests that are attempting to use data/parameters within the UI that cease to be available and a more flexible data choice should be considered.

Review new features and acceptance criteria - prepare your test coverage

Even if the new feature is not yet delivered to a test environment, testers at the earliest opportunity should immerse themselves in what’s to come. Basic positive and negative test coverage can be prepared, even if you are not ready to actually test yet.

Add new tests to your pack

Continuous agile testing means getting ahead with the automation of tests for new features as early as possible. If teams are working in agile/scrum based sprints, testers should adopt an approach of explore, raise quick win bugs and then write the automated tests in the same sprint as the developers.

With the right choice of tooling, pace can be maintained and there is no need for automation to follow several sprints behind in a mini waterfall fashion. The aspiration should be that for each build of new code, the tests are already there to demonstrate that the functionality delivered in the previous build is still functionally sound and has not regressed.

Retire any tests that are no longer relevant

Finally, features come and go. Don’t be reticent to retire tests that are no longer fit for purpose. Do not endure failing tests on your dashboards simply because that feature has been removed.

Your test pack and each and every test within it has been selected for good reason and it needs to continue adding value. When it doesn’t, it should be archived.

How to select your Test Automation Tool

With so many automation tools available in the market, it isn't always easy to select the most appropriate for your team.

There are a number of things to consider when looking at tools for your shortlist.

• Will the tool support all of the application technologies that you will need to test?

• How good is the support documentation, tutorials and user community?

• Does the tool support the use of the most common development languages today?

• Does the tool offer low-code accelerators for new automation engineers?

• If you move away from that tool, do you lose all of your tests? Is there a vendor lock-in with that provider?

• Can you trial the tool before committing?

• How easy is it to publish and share the test output from the tool?

• Aside from functional testing, does it offer other types of testing support - visual regression, accessibility, mobile apps?

• Can it integrate with other tools?

Getting Started with Automated Testing

There is no doubt that for a delivery team looking to increase quality and release cadence, test automation is an absolutely necessary investment in order to succeed with agile and continuous testing.

But always remember, both manual and automated testing go hand-in-hand to both find new bugs rapidly and ensure that long standing critical features remain fit for purpose without regression.

If you are looking to embrace automation for the first time or looking to consolidate your range of current tooling, why not consider evaluating Test Evolve. We know and understand what test teams are looking for and it is always foremost in our mind when we plan our feature releases.