Testing, types and terminology with examples

John D K -

Testing(some times can be used as QA(quality assurance) is very important part of the product and especially development life-cycle. Test can be reviewed as a exam for products - is this product, unit, software mature enough for usage. Good test scenario requires deep knowledge of used technologies, product area and same intuition. Below I'll try to share my knowledge on this topic. I know that this terms could vary a lot depending on the business, product, company or even teams in one company. It's common people to be confused with unit and integration tests, non regression and regression. This can have a huge impact on the final test results by skipping or overlapping some tests.
Another huge mistake is a misinformed(based on bad information) or uninformed( based on no information or inadequate information) tester or QA:

  • test on the wrong place - in production environment or real data(in software aspect);
  • testing with wrong version of the software
  • testing wrong scenario

Testing by type/level(in software point of view)

  • Unit test: verify and test one unit(only the unit without interaction to other units). Usually it has a focused and well defined scope. You are testing individual methods, services, classes or blocks of code without considering for surrounding infrastructure.

Examples:

  1. test method return result is not negative;
  2. verify class fields length;
  3. create new instance of a class(only by code no storing on DB)
  • Integration test: verify and test multiple units and their interactions and complex dependencies. You will have a full access to other functionalities and units. From testing integration between two classes, to testing integration with the production environment or testing interaction of two different systems - all this can be considered as integration tests.

Examples:

  1. test communication between methods;
  2. send and receive service;
  3. create new instance of a class( by GUI - graphical user interface and storing to DB)

  • Difference between Unit and Integration tests would be: unit tests does not inject or use any of the dynamic methods present during integration tests and at run-time. You can think of it as - a unit test is used to test the class in isolation, whereas the integration test allows you to test the class in a full, running environment. How to make the difference? Write a well defined scenario and check all related units.

  • Regression test: A bit confusing with non-regression test due to overlap of some aspects. In my experience regression test will verify that new modifications of the product or the code are not breaking something from the standard functionality. If a new change of the code breaks the functionality while fixing/changing something else - this is called regression.

Examples:

  1. class field size enlarged - test if UI(user interface) or DB is not crashing due to this change;
  2. change a method return type - test if impacted scenarios(already tested and working before) are still working as expected.
  • Non-regression test - It must ensures that this specific new change works as intended . It can also be considered as a normal test made after to changing an application to make sure the application provides the desired outcome.

Examples:

  1. class field size enlarged - test if the new size is enough to store the required sample data

According to Wikipedia:

The intent of regression testing is to assure that in the process of fixing a defect no existing functionality has been broken. Non-regression testing is performed to test that an intentional change has had the desired effect.

For many people the term non regression test is a non sense and they considered it as a normal testing.

Other tests

  • Automated test: Record a test scenario on a software and execute these scenarios before delivering new code. Some popular testing utilities: Selenium, CasperJs, PhantomJs

Examples:

Test scenario

  1. open web site
  2. login to the site
  3. do some action
  4. logout

  • Acceptance test maybe be referenced as UAT phase - user acceptance tests: Ask an user or a responsible person to test that a feature or use case is correctly implemented and working as expected. It differ from an integration test, because the focus is on the use case and it's on the side of the users/customers.

  • System test: Testing a system is performed on the entire system in the context of whole functionality.

  • Smoke/Sanity testing: Check for very basic problems that can break normal functioning of the tested unit. If this test fails further testing is stopped until the problems are fixed.

  • Alpha testing: similar to acceptance test on developer site or on target test group

Examples:

  1. a game is shared with selected group of gamers
  • Beta testing: like alpha testing done by real or potential customers/users
  1. a game is shared with the public or larger group
  • A/B testing - generally means testing two versions with some differences and comparing the results of both.

Examples:

  1. prepare two starting pages for a website. Show the pages on two equal groups and measure the results of the user satisfaction

Some general tips from experienced QA

  • define carefully your test plans and scenarios in order to cover whole functionality
  • automate everything that is possible and produce reports
  • do training for the test team
  • prepare dedicated environments for tests
  • if needed spent time for creating and validating test data(or ask the clients for it)
  • define and check error levels and thresholds - in case of a minor defect(display or not important format problem) whether to stop the delivery or continue
  • be sure that people use one and the same terminology! What is Unit and what is Integration test. And who is responsible for what: developer for Unit, QA for integration, user for UAT.

Wish you happy testing and less defects :)