Manual Testing tutorial - part3

Seven Principles of Software Testing

Software testing is an extremely creative and intellectually challenging task. When testing follows the principles given below, the creative element of test design and execution rivals any of the preceding software development steps.

Testing shows the presence of bugs

Testing an application can only reveal that one or more defects exist in the application, however, testing alone cannot prove that the application is error free. Therefore, it is important to design test cases which find as many defects as possible.

Exhaustive testing in impossible

Unless the application under test (UAT) has a very simple logical structure and limited input, it is not possible to test all possible combinations of data and scenarios. For this reason, risk and priorities are used to concentrate on the most important aspects to test.

Early testing

The sooner we start the testing activities the better we can utilize the available time. As soon as the initial products, such the requirement or design documents are available, we can start testing. It is common for the testing phase to get squeezed at the end of the development lifecycle, i.e. when development has finished, so by starting testing early, we can prepare testing for each level of the development lifecycle.

Another important point about early testing is that when defects are found earlier in the lifecycle, they are much easier and cheaper to fix. It is much cheaper to change an incorrect requirement than having to change a functionality in a large system that is not working as requested or as designed!

Defect clustering

During testing, it can be observed that most of the reported defects are related to small number of modules within a system. i.e. small number of modules contain most of the defects in the system. This is the application of the Pareto Principle to software testing: approximately 80% of the problems are found in 20% of the modules.

The pesticide paradox

If you keep running the same set of tests over and over again, chances are no more new defects will be discovered by those test cases. Because as the system evolves, many of the previously reported defects will have been fixed and the old test cases do not apply anymore. Anytime a fault is fixed or a new functionality added, we need to do regression testing to make sure the new changed software has not broken any other part of the software. However, those regression test cases also need to change to reflect the changes made in the software to be applicable and hopefully fine new defects.

Testing is context dependent

Different methodologies, techniques and types of testing is related to the type and nature of the application. For example, a software application in a medical device needs more testing than a games software. More importantly a medical device software requires risk based testing, be compliant with medical industry regulators and possibly specific test design techniques. By the same token, a very popular website, needs to go through rigorous performance testing as well as functionality testing to make sure the performance is not affected by the load on the servers.

Absense of errors fallacy

Just because testing didn’t find any defects in the software, it doesn’t mean that the software is ready to be shipped. Were the executed tests really designed to catch the most defects? or where they designed to see if the software matched the user’s requirements? There are many other factors to be considered before making a decision to ship the software.

The Five Essentials For Software Testing

Five essential elements are required for successful software testing: test strategy, testing plan, test cases, test data and a test environment. If any one of these five elements is missing or inadequate, your test effort will most likely fall far short of what you could otherwise achieve.

The following advice should help clarify your thinking about software testing and help you improve the effectiveness and efficiency of your testing. It is helpful to think about software testing in terms of five essential elements:

1.     A test strategy that tells you what types of testing and the amount of testing you think will work best at finding the defects that are lurking in the software

2.     A testing plan of the actual testing tasks you will need to execute to carry out that strategy

3.     Test cases that have been prepared in advance in the form of detailed examples you will use to check that the software will actually meet its requirements

4.     Test data consisting of both input test data and database test data to use while you are executing your test cases, and

5.     A test environment which you will use to carry out your testing.

If any one of these five elements is missing or inadequate, your test effort will most likely fall far short of what you could otherwise achieve.

Test Strategy
The purpose of testing is to find defects, not to pass easy tests. A test strategy basically tells you which types of testing seem best to do, the order in which to perform them, the proposed sequence of execution, and the optimum amount of effort to put into each test objective to make your testing most effective. A test strategy is based on the prioritized requirements and any other available information about what is important to the customers. Because you will always face time and resource constraints, a test strategy faces up to this reality and tells you how to make the best use of whatever resources you do have to locate most of the worst defects. Without a test strategy, you are apt to waste your time on less fruitful testing and miss using some of your most powerful testing options. You should create the test strategy at about the middle of the design phase as soon as the requirements have settled down.

Testing Plan
A testing plan is simply that part of your project plan that deals with the testing tasks. It details who will do which tasks, starting when, ending when, taking how much effort, and depending on which other tasks. It provides a complete list of all the things that need to be done for testing, including all the preparation work during all of the phases before testing. It shows the dependencies among the tasks to clearly create a critical path without surprises. You will be able to start filling in the details of your testing plan as soon as your test strategy is completed. Both your test strategy and testing plan are subject to change as the project evolves. Modify your strategy first, if you need to, and then your testing plan.

Test Cases
Your test cases (and automated test scripts if called for by your strategy) are prepared based on the strategy which tells you how much of each type of testing to do. Test cases are developed based on prioritized requirements and acceptance criteria for the software, keeping in mind the customer’s emphasis on quality dimensions and the project’s latest risk assessment of what could go wrong. Except for a small amount of ad hoc testing, all of your test cases should be prepared in advance of the start of testing. There are many different approaches to developing test cases. Test case development is an activity performed in parallel with software development. It is just as difficult to do a good job of coming up with test cases as it is to program the system itself. In addition to figuring out what steps to take to test the system, you need to know the requirements and business rules well enough to predict exactly what the expected results should be. Without expected results to compare to actual results, you will not be able to say whether a test will pass or fail. A good test case checks to make sure requirements are being met and has a good chance of uncovering defects.

Test Data
In addition to the steps to perform to execute your test cases, you also need to systematically come up with test data to use. This often equals sets of names, addresses, product orders, or whatever other information the system uses. Since you are probably going to test query functions, change functions and delete functions, you will most likely need a starting database of data in addition to the examples to input. Consider how many times you might need to go back to the starting point of the database to restart the testing and how many new customer names you will need for all the testing in your plan. Test data development is usually done simultaneously with test case development.

Test Environment
You will need a place to do the testing and the right equipment to use. Unless the software is very simple, one PC will not suffice. You will need all of the components of the system as close as possible to what it will eventually be. Test environments may be scaled-down versions of the real thing, but all the parts need to be there for the system to actually run. Building a test environment usually involves setting aside separate regions on mainframe computers and/or servers, networks and PCs that can be dedicated to the test effort and that can be reset to restart testing as often as needed. Sometimes lab rooms of equipment are set aside, especially for performance or usability testing. A wish list of components that will be needed is part of the test strategy, which then needs to be reality checked as part of the test planning process. Steps to set up the environment are part of the testing plan and need to be completed before testing begins.

There are five assignments for each workbench:

1.     Input: Each task requires certain input and output parameters. For each workbench we need specific inputs. Input form is the first stage of the workbench.

2.     Perform: The main goal of the workbench is to change the input forms on the expected output ones.

3.     Check: Check ensures that output after the performing achieves the desired result.

4.     Production output: If the check is done correctly the production output becomes the last stage of the workbench.

5.     Rework: If the result after the check doesn’t meet our expectations we need to start again from the step of performance.

In fact, scenarios are not made of one workbench but of many related workbenches. A workbench gives you an opportunity to execute any one task with appropriate software testing.

Let’s discuss the workbench concept:

Requirement phase

The input data – the requirements of clients; we perform a task – writing a document with the customer’s requirements, we check the suitability of a document to all needs of client, and receive the output – requirement document.

Design phase

The input data – the requirement document, we execute the preparing a technical document; review/test is performed to see if the design document is technically right and transfers all the requirements in the requirement document, and receive a technical document.

Execution phase

It is the actual performance of the project. The input data – the technical document; the performance is nothing but realization/ coding according to the technical document, and the output data – the source code.

Testing phase workbench

It is the stage of software testing. The input data – the source code which is required testing; the realization – implementation of the test case and the output – the results of software testing.

Distribution phase

There are two inputs for this step – the source code which requires of customers and the source code with the results of testing. The output of this project is the product which is ready for use.

Maintenance phase

The input – the results of distribution, execution – execution of the last customer requests, the running regression software testing after every changed customer request, and the output is a new release.

Important features of Testing

1.     Test for quality over quantity: “Here’s 10,000 bugs… good luck!” Testers, please don’t ever shoot for quantity. Identifying the most important bugs and glitches, and helping the company or developer make sense of the bugs is ten times more helpful then testing for mere volume.

2.     Learn to prioritize: In line with “quality over quantity”, prioritizing what you test is extremely important. Testing the mission critical parts of an application before the minute details of an app will help you to identify the most valuable bugs first. This will also allow the development team to fix the most imperative parts of their application as quickly as possible.

3.     Practice and improve your written communication skills: Everyone can right, write? (Ha!). Good testers must have excellent written communication skills in order to write good test cases, bug reports and so on. These testing artifacts are an essential part of QA and must be detailed and easy to consume.

4.     Learn from your own mistakes – and from others too:  Everyone makes mistakes, but learning from others and from your own will make you better tester. How can you improve your bug report next time? How can you prioritize better during the next test cycle? How can you communicate better with the development team? These are questions you should constantly be asking yourself, and your fellow testers.

5.     Be objective and professional: Every time you test, go at it with a fresh perspective. Look at the software you are testing without bias or past experience, and test with an open mind. Testers who think “Oh, I know this software” or “I’ve used this before” are at risk of overlooking important bugs. Objectivity is key.

6.     Don’t be humble with software… think out of the box: Explore the software, ‘test to break’ and be willing to suggest improvements; these are all attributes that make up the attitude of a good software tester.

7.     Question. Everything: Does this work as intended? Does it work on all devices? Does it work under every possible use-case, every time? Question. Everything.  

8.     Think like the user: Remember; your job is to find bugs before the software reaches the hands of users. Pair your technical skills with an end-user’s mindset and you will find the best, most valuable bugs possible.

9.     Increase the effectiveness of bug reports: Attaching screen shots and providing detailed bug reports will give the developer the information he or she needs to understand the bug and fix it. Where did it occur, when, how many times, on what devices, running what operating system and under what circumstances? Without the right details a bug is useless to a development team.

10.                        Be Passionate! To excel in any field, you need to be passionate for what you do. Read, seek new training opportunities, engage with your fellow testers, attend testing conference, classes – immerse yourself in all things QA.

Some of the myths are explained below, along with their related facts:

1.     MYTH: Quality Control = Testing.

§  FACT: Testing is just one component of software quality control. Quality Control includes other activities such as Reviews.

2.     MYTH: The objective of Testing is to ensure a 100% defect- free product.

§  FACT: The objective of testing is to uncover as many defects as possible. Identifying all defects and getting rid of them is impossible.

3.     MYTH: Testing is easy.

§  FACT: Testing can be difficult and challenging (sometimes, even more so than coding).

4.     MYTH: Anyone can test.

§  FACT: Testing is a rigorous discipline and requires many kinds of skills.

5.     MYTH: There is no creativity in testing.

§  FACT: Creativity can be applied when formulating test approaches, when designing tests, and even when executing tests.

6.     MYTH: Automated testing eliminates the need for manual testing.

§  FACT: 100% test automation cannot be achieved. Manual Testing, to some level, is always necessary.

7.     MYTH: When a defect slips, it is the fault of the Testers.

§  FACT: Quality is the responsibility of all members/stakeholders, including developers, of a project.

8.     MYTH: Software Testing does not offer opportunities for career growth.

§  FACT: Gone are the days when users had to accept whatever product was dished to them; no matter what the quality. With the abundance of competing software and increasingly demanding users, the need for software testers to ensure high quality will continue to grow.