TESTING METRICS

What is Test Metric?

Test metrics is a process for analyzing the current level of maturity while testing and predict future trends, finally meant for enhancing the testing activities which were missed in current testing will be added in next build to improve or enhance the Testing Process.
Metrics are the numerical data which will help us to measure the test effectiveness.
Metrics are produced in two forms
1. Base Metrics and
2. Derived Metrics.
Example of Base Metrics:

# Test Cases
# New Test Cases
# Test Cases Executed
# Test Cases Unexecuted
# Test Cases Re-executed
# Passes
# Fails
# Test Cases Under Investigation
# Test Cases Blocked
# 1st Run Fails
#Test Case Execution Time
# Testers

Examples of Derived Metrics:

% Test Cases Complete
% Test Cases Passed
% Test Cases Failed
% Test Cases Blocked
% Test Defects Corrected

Objective of Test Metrics
The objective of Test Metrics is to capture the planned and actual quantities the effort, time and resources required to complete all the phases of Testing of the software Project.
Test metrics usually covers 3 things:
1. Test coverage
2. Time for one test cycle.
3. Convergence of testing

Why Testing metrics?
As, we all know, a major percentage of software projects suffer from quality problems. Software testing provides visibility into product and process quality. Test metrics are key ”facts” that project managers can understand their current position and to prioritize their activities to reduce the risk of schedule over-runs on software releases.
Test metrics are a very powerful management tool. They help you to measure your current performance. Because today’s data becomes tomorrow’s historical data. its never too late to start recording key information on your project. This data can be used to improved future work estimates and quality levels. Without historical data, estimates will be guesses.
You cannot track the project status meaningfully unless you know the actual effort and time spent on each task as compared to your estimates. You cannot sensibly decide whether your product is stable enough to ship unless you track the rates at which your team is finding and fixing defects. You cannot quantify the performance of your new development processes without some statistics on your current performance and a baseline to compare it with. Metrics help you to better control your software projects. They enable you to learn more about the functioning of your organization by establishing a Process Capability baseline that can be used to better estimate and predict the quality of your projects in the future.
The benefits of having testing metrics
1. Test metrics data collection helps predict the long-term direction and scope for an organization and enables a more holistic view of business and identifies high-level goals
2. Provides a basis for estimation and facilitates planning for closure of the performance gap
3. Provides a means for control / status reporting
4. Identifies risk areas that requires more testing
5. Provides meters to flag actions for faster, more informed decision making
6. Quickly identifies and helps resolve potential problems and identifies areas of improvement
7. Test metrics provide an objective measure of the effectiveness and efficiency of testing
Key factors to bear in mind while setting up test metrics
1. Collect only the data that you will actually use/need to make informed decisions to alter your strategies, if you are not going to change your strategy regardless of the finding, your time is better spent in testing.
2. Do not base decisions solely on data that is variable or can be manipulated. For example, measuring testers on the number of tests they write per day can reward them for speeding through superficial tests or punish them for tracking trickier functionality.
3. Use statistical analysis to get a better understanding of the data. Difficult metrics data should be analyzed carefully. The templates used for presenting data should be self explanatory.
4. One of the key inputs to the metrics program is the defect tracking system in which the reported process and product defects are logged and tracked to closure. It is therefore very important to carefully decide on the fields that need per defect in the defect tracking systems and then generate customizable reports.
5. Metrics should be decided on the basis of their importance to stakeholders rather than ease of data collection. Metrics that are of not interest to the stakeholders should be avoided.
6. Inaccurate data should be avoided and complex data should be collected carefully. Proper benchmarks should be definite for the entire program
17.6 Deciding on the Metrics to Collect

There are literally thousands of possible software metrics to collect and possible things to measure about software development. There are many books and training programs available about software metrics, which of the many metrics are appropriate for your situation? One method is to start with one of the many available published suites of metrics and a vision of your own management problems and goals, and then customize the metrics list based on the following metrics collection checklist. For each metric, you must consider,
1) What are you trying to manage with this metric? Each metric must relate to a specific management area of interest in a direct way. The more convoluted the relationship between the measurement and the management goal, the less likely you are to be collecting the right thing.
2) What does this metric measure? Exactly what does this metric count? High-level attempts to answer this question (such as "it measures how much we've accomplished") may be misleading. The detailed answers (such as "it reports how much we had budgeted for design tasks that first-level supervisors are reporting as greater than 80 percent complete") is much more informative, and can provide greater insight regarding the accuracy and usefulness of any specific metric.
3) If your organization optimized this metric alone, what other important aspects of your software design, development, testing, deployment, and maintenance would be affected? Asking this question will provide a list of areas where you must check to be sure that you have a balancing metric. Otherwise, your metrics program may have unintended effects and drive your organization to undesirable behavior.
4) How hard/expensive is it to collect this information? This is where you actually get to identify whether collection of this metric is worth the effort. If it is very expensive or hard to collect, look for automation that can make the collection easier, or consider alternative metrics that can be substituted.
5) Does the collection of this metric interact with (or interfere with) other business processes? For example, does the metric attempt to gather financial information on a different periodic basis or with different granularity than your financial system collects and reports it? If so, how will the two quantitative systems be synchronized? Who will reconcile differences? Can the two collection efforts be combined into one and provide sufficient software metrics information?

6) How accurate will the information be after you collect it? Complex or manpower-intensive metrics collection efforts are often short circuited under time and schedule pressure by the people responsible for the collection. Metrics involving opinions (e.g., what percentage complete do you think you are?) are notoriously inaccurate. Exercise caution, and carefully evaluate the validity of metrics with these characteristics.
7) Can this management interest area be measured by other metrics? What alternatives to this metric exist? Always look for an easier-to-collect, more accurate, more timely metric that will measure relevant aspects of the management issue of concern.
Use of this checklist will help ensure the collection of an efficient suite of software development metrics that directly relates to management goals. Periodic review of existing metrics against this checklist is recommended.
Projects that are underestimated, over-budget, or that produce unstable products, have the potential to devastate the company. Accurate estimates, competitive productivity, and renewed confidence in product quality are critical to the success of the company.
Hoping to solve these problems as quickly as possible, the company management embarks on the 8-Step Metrics Program
Step 1: Document the Software Development Process
Integrated Software does not have a defined development process. However, the new metrics coordinator does a quick review of project status reports and finds that the activities of requirements analysis, design, code, review, recode, test, and debugging describe how the teams spend their time. The inputs, work performed, outputs and verification criteria for each activity have not been recorded. He decides to skip these details for this "test" exercise. The recode activity includes only effort spent addressing software action items (defects) identified in reviews.
Step 2: State the Goals
The metrics coordinator sets out to define the goals of the metrics program. The list of goals in Step 2 of the 7 -Step Metrics Program are broader than (yet still related to) the immediate concerns of Integrated Software. Discussion with development staff leads to some good ideas on how to tailor these goals into specific goals for the company.
1. Estimates
The development staff at Integrated Software considers past estimates to have been unrealistic as they were established using “finger in the wind” techniques. They suggest that current plan could benefit from past experience as the present project is very similar to past projects.
Goal: Use previous project experience to improve estimations of Productivity.
2. Productivity
Discussions about the significant effort spent in debugging center on a comment by one of the developers that defects found early on in reviews have been faster to repair than Defects discovered by the test group. It seems that both reviews and testing are needed, but the amount of effort to put into each is not clear.
Goal: Optimize defect detection and removal.
3. Quality
The test group at the company argues for exhaustive testing. This however, is Prohibitively expensive. Alternatively, they suggest looking at the trends of defects discovered and repaired over time to better understand the probable number of defects remaining.
Goal: Ensure that the defect detection rate during testing is converging towards a level that indicates that less than five defects per KSLOC will be discovered in the next year.
Step 3: Define Metrics Required to Reach Goals and Identify Data to Collect
Working from the Step 3 tables, the metrics coordinator chooses the following metrics for the metrics program.
Goal 1: Improve Estimates
• Actual effort for each type of software in PH
• Size of each type of software in SLOC
• Software product complexity (type)
• Labor rate (PH/SLOC) for each type
Goal 2: Improve Productivity
• Total number of person hours per activity
• Number of defects discovered in reviews
• Number of defects discovered in testing
• Effort spent repairing defects discovered in reviews
• Effort spent repairing defects discovered in testing
• Number of defects removed per effort spent in reviews and recode
• Number of defects removed per effort spent in testing and debug
Goal 3: Improve Quality
• total number of defects discovered
• total number of defects repaired
• number of defects discovered / schedule date
• number of defects repaired / schedule date


Types of test metrics
1. Product test metrics
i. Number of remarks
ii. Number of defects
ii. Remark status
iv. Defect severity
v. defect severity index
vi. Time to find a defect
vii. Time to solve a defect
viii. Test coverage
ix. defects/KLOC
2. Project test metrics
i. workload capacity ratio
ii. Test planning performance.
Iii. Test effort ratio.
iv. Defect category
3. Process test metrics
i. should be found in which phase
ii. Residual defect density
iii. Defect remark ratio
iv. Valid remark ratio
v. bad fix ratio
vi. Defect removal efficiency
vii. Phase yield
viii. Backlog development
ix. Backlog testing
x. scope changes

Product test metrics

I. Number of remarks
Definition
The total number of remarks found in a given time period/phase/test type. A remark is a claim made by test engineer that the application shows an undesired behavior. It may or may not result in software modification or changes to documentation.
Purpose
One of the earliest indicators to measure once the testing commences; provides initial indications about the stability of the software
Data to collect
Total number of remarks found.

II. Number of defects
Definition
The total number of remarks found in a given time period/phase/test type that resulted in software or documentation modifications.
Purpose
The total number of remarks found in a given time period/phase/test type that resulted in software or documentation modifications.
Data to collect
Only remarks that resulted in modifying the software or the documentation are counted.

III. Remark status
Definition
The status of the defect could vary depending upon the defect-tracking tool that is used. Broadly, the following statuses are available: To be solved: Logged by the test engineers and waiting to be taken over by the software engineer. To be retested: Solved by the developer, and waiting to be retested by the test engineer. Closed: The issue was retested by the test engineer and was approved.
Purpose
Track the progress with respect to entering, solving and retesting the remarks. During this phase, the information is useful to know the number of remarks logged, solved, waiting to be resolved and retested.
Data to collect
This information can normally be obtained directly from the defect tracking system based on the remark status.

IV. Defect severity
Definition
The severity level of a defect indicates the potential business impact for the end user (business impact = effect on the end user x frequency of occurrence).
Purpose
Provides indications about the quality of the product under test. A high-severity defect means low product quality, and vice versa. At the end of this phase, this information is useful to make the release decision based on the number of defects and their severity levels.
Data to collect
Every defect has severity levels attached to it. Broadly, these are Critical, Serious, Medium and Low.

V. Defect severity index
Definition
An index representing the average of the severity of the defects.
Purpose
Provides a direct measurement of the quality of the product—specifically, reliability, fault tolerance and stability.
Data to collect
Two measures are required to compute the defect severity index. A number is assigned against each severity level: 4 (Critical), 3 (Serious), 2 (Medium), 1 (Low). Multiply each remark by its severity level number and add the totals; divide this by the total number of defects to determine the defect severity index.

VI. Time to find a defect
Definition
The effort required to find a defect.
Purpose
Shows how fast the defects are being found. This metric indicates the correlation between the test effort and the number of defects found.
Data to collect
Divide the cumulative hours spent on test execution and logging defects by the number of defects entered during the same period.




VII. Time to solve a defect
Definition
Effort required resolving a defect (diagnosis and correction).
Purpose
Provides an indication of the maintainability of the product and can be used to estimate projected maintenance costs.
Data to collect
Divide the number of hours spent on diagnosis and correction by the number of defects resolved during the same period.

VIII. Test coverage
Definition
Defined as the extent to which testing covers the product’s complete functionality.
Purpose
This metric is an indication of the completeness of the testing. It does not indicate anything about the effectiveness of the testing. This can be used as a criterion to stop testing.
Data to collect
Coverage could be with respect to requirements, functional topic list, business flows, use cases, etc. It can be calculated based on the number of items that were covered vs. the total number of items.

IX. Test case effectiveness
Definition
The extent to which test cases are able to find defects
Purpose
This metric provides an indication of the effectiveness of the test cases and the stability of the software.
Data to collect
Ratio of the number of test cases that resulted in logging remarks vs. the total number of test cases.

X. Defects/KLOC
Definition
The number of defects per 1,000 lines of code.
Purpose
This metric indicates the quality of the product under test. It can be used as a basis for estimating defects to be addressed in the next phase or the next version.
Data to collect
Ratio of the number of defects found vs. the total number of lines of code (thousands)

Formula used

Uses of defect/KLOC
Defect density is used to compare the relative number of defects in various software components. This helps identifies candidates various for additional inspection or testing or for possible engineering or replacement. Identifying defect prone components allows the concentration of limited resources into areas with the highest potential return on investment.
Another use of defect density is to compare subsequent releases of a product to track the impact of defect reduction and quality improvement activities. Normalling by size allows releasing of various sizes to be compared. Differences between products or products lines can also be compared in this manner.

Project test metrics:
I. Workload capacity
Definition
Ratio of the planned workload and the gross capacity for the total test project or phase. Similar.
Purpose
This metric helps in detecting issues related to estimation and planning. It serves as an input for estimating similar projects as well.
Data to collect
Computation of this metric often happens in the beginning of the phase or project. Workload is determined by multiplying the number of tasks against their norm times. Gross capacity is nothing but planned working time, determined by workload divided by gross capacity.

II. Test planning performance
Definition
The planned value related to the actual value.
Purpose
Shows how well estimation was done.
Data to collect
The ratio of the actual effort spent to the planned effort

III. Test effort percentage
Definition
Test effort is the amount of work spent, in hours or days or weeks. Overall project effort is divided among multiple phases of the project: requirements, design, coding, testing and such. This metric can be computed by dividing the overall test effort by the total project effort.

Purpose
The effort spent in testing, in relation to the effort spent in the development activities, will give us an indication of the level of investment in testing. This information can also be used to estimate similar projects in the future.
Data to collect
This metric can be computed by dividing the overall test effort by the total project effort.

IV. Defect category
Definition
An attribute of the defect in relation to the quality attributes of the product. Quality attributes of a product include functionality, usability, documentation, performance, installation and internationalization.
Purpose
This metric can provide insight into the different quality attributes of the product.

Data to collect
This metric can be computed by dividing the defects that belong to a particular category by the total number of defects.

Process test metrics
I. Should be found in which phase
Definition
An attribute of the defect, indicating in which phase the remark should have been found.
Purpose
Are we able to find the right defects in the right phase as described in the test strategy? Indicates the percentage of defects that are getting migrated into subsequent test phases.
Data to collect
Computation of this metric is done by calculating the number of defects that should have been found in previous test phases.

II. Residual defect density:
Definition
An estimate of the number of defects that may have been unresolved in the product phase.
Purpose
The goal is to achieve a defect level that is acceptable to the clients. We remove defects in each of the test phases so that few will remain.
Data to collect
This is a tricky issue. Released products have a basis for estimation. For new versions, industry standards, coupled with project specifics, form the basis for estimation.


III. Defect remark ratio
Definition
Ratio of the number of remarks that resulted in software modification vs. the total number of remarks.
Purpose
Provides an indication of the level of understanding between the test engineers and the software engineers about the product, as well as an indirect indication of test effectiveness.
Data to collect
The number of remarks that resulted in software modification vs. the total number of logged remarks. Valid for each test type, during and at the end of test phases.

IV. Valid remark ratio
Definition
Percentage of valid remarks during a certain period.
Purpose
Indicates the efficiency of the test process.
Data to collect
Ratio of the total number of remarks that are valid to the total number of remarks found.
Formula used
Valid remarks = number of defects + duplicate remarks + number of remarks that will be resolved in the next phase or release.

V. Phase yield
Definition
Defined as the number of defects found during the phase of the development life cycle vs. the estimated number of defects at the start of the phase.
Purpose
Shows the effectiveness of the defect removal. Provides a direct measurement of product quality; can be used to determine the estimated number of defects for the next phase.
Data to collect
Ratio of the number of defects found by the total number of estimated defects. This can be used during a phase and also at the end of the phase.

VI. Backlog development
Definition
The number of remarks that are yet to be resolved by the development team.
Purpose
Indicates how well the software engineers are coping with the testing efforts.
Data to collect
The number of remarks that remain to be resolved.

VII. Backlog testing

Definition
The number of resolved remarks that are yet to be retested by the development team.
Purpose
Indicates how well the test engineers are coping with the development efforts.
Data to collect
The number of remarks that have been resolved.

VII. Scope changes
Definition
The number of changes that were made to the test scope.
Purpose
Indicates requirements stability or volatility, as well as process stability.
Data to collect
Ratio of the number of changed items in the test scope to the total number of items.

VIII. Defect removal efficiency
Definition
The number of defects that are removed per time unit (hours/days/weeks)
Purpose
Indicates the efficiency of defect removal methods, as well as indirect measurement of the quality of the product.
Data to collect
Computed by dividing the effort required for defect detection, defect resolution time and retesting time by the number of remarks. This is calculated per test type, during and across test phases.

TEST REPORT

TEST REPORT


The Software Test Report (STR) is a record of the qualification testing performed on a Computer Software Configuration Item (CSCI), a software system or subsystem, or other software-related item.
The STR enables the acquirer to assess the testing and its results.
APPLICATION/INTERRELATIONSHIP
The Data Item Description (DID) contains the format and content preparation instructions for the data product generated by specific and discrete task requirements as delineated in the contract.
This DID is used when the developer is tasked to analyze and record the results of CSCI qualification testing, system qualification testing of a software system, or other testing identified in the contract.
The Contract Data Requirements List (CDRL) should specify whether deliverable data are to be delivered on paper or electronic media; are to be in a given electronic form (such as ASCII, CALS, or compatible with a specified word processor or other support software); may be delivered in developer format rather than in the format specified herein; and may reside in a computer-aided software engineering (CASE) or other automated tool rather than in the form of a traditional document.

PREPARATION INSTRUCTIONS
General instructions.
a. Automated techniques. Use of automated techniques is encouraged. The term "document" in this DID means a collection of data regardless of its medium.
b. Alternate presentation styles. Diagrams, tables, matrices, and other presentation styles are acceptable substitutes for text when data required by this DID can be made more readable using these styles.
c. Title page or identifier. The document shall include a title page containing, as applicable: document number; volume number; version/revision indicator; security markings or other restrictions on the handling of the document; date; document title; name, abbreviation, and any other identifier for the system, subsystem, or item to which the document applies; contract number; CDRL item number; organization for which the document has been prepared; name and address of the preparing organization; and distribution statement. For data in a database or other alternative form, this information shall be included on external and internal labels or by equivalent identification methods.
d. Table of contents. The document shall contain a table of contents providing the number, title, and page number of each titled paragraph, figure, table, and appendix. For data in a database or other alternative form, this information shall consist of an internal or external table of contents containing pointers to, or instructions for accessing, each paragraph, figure, table, and appendix or their equivalents.
e. Page numbering/labeling. Each page shall contain a unique page number and display the document number, including version, volume, and date, as applicable. For data in a database or other alternative form, files, screens, or other entities shall be assigned names or numbers in such a way that desired data can be indexed and accessed.
f. Response to tailoring instructions. If a paragraph is tailored out of this DID, the resulting document shall contain the corresponding paragraph number and title, followed by "This paragraph has been tailored out." For data in a database or other alternative form, this representation need occur only in the table of contents or equivalent.
g. Multiple paragraphs and subparagraphs. Any section, paragraph, or subparagraph in this DID may be written as multiple paragraphs or subparagraphs to enhance readability.
h. Standard data descriptions. If a data description required by this DID has been published in a standard data element dictionary specified in the contract, reference to an entry in that dictionary is preferred over including the description itself.
i. Substitution of existing documents. Commercial or other existing documents may be substituted for all or part of the document if they contain the required data.
1. Content requirements. Content requirements begin on the following page. The numbers shown designate the paragraph numbers to be used in the document. Each such number is understood to have the prefix within this DID. For example, the paragraph numbered 1.1 is understood to be paragraph 10.2.1.1 within this DID.

Scope. This section shall be divided into the following paragraphs.
Identification. This paragraph shall contain a full identification of the system and the software to which this document applies, including, as applicable, identification number(s), title(s), abbreviation(s), version number(s), and release number(s).
System overview. This paragraph shall briefly state the purpose of the system and the software to which this document applies. It shall describe the general nature of the system and software; summarize the history of system development, operation, and maintenance; identify the project sponsor, acquirer, user, developer, and support agencies; identify current and planned operating sites; and list other relevant documents.
1. Document overview. This paragraph shall summarize the purpose and contents of this document and shall describe any security or privacy considerations associated with its use.

1. Referenced documents. This section shall list the number, title, revision, and date of all documents referenced in this report. This section shall also identify the source for all documents not available through normal Government stocking activities.

Overview of test results. This section shall be divided into the following paragraphs to provide an overview of test results.
Overall assessment of the software tested. This paragraph shall:
a. Provide an overall assessment of the software as demonstrated by the test results in this report
b. Identify any remaining deficiencies, limitations, or constraints that were detected by the testing performed. Problem/change reports may be used to provide deficiency information.
c. For each remaining deficiency, limitation, or constraint, describe:
1) Its impact on software and system performance, including identification of requirements not met
2) The impact on software and system design to correct it
3) A recommended solution/approach for correcting it
Impact of test environment. This paragraph shall provide an assessment of the manner in which the test environment may be different from the operational environment and the effect of this difference on the test results.
Recommended improvements. This paragraph shall provide any recommended improvements in the design, operation, or testing of the software tested. A discussion of each recommendation and its impact on the software may be provided. If no recommended improvements are provided, this paragraph shall state "None."

Detailed test results. This section shall be divided into the following paragraphs to describe the detailed results for each test. Note: The word "test" means a related collection of test cases.
(Project-unique identifier of a test). This paragraph shall identify a test by project-unique identifier and shall be divided into the following subparagraphs to describe the test results.
Summary of test results. This paragraph shall summarize the results of the test. The summary shall include, possibly in a table, the completion status of each test case associated with the test (for example, "all results as expected," "problems encountered," "deviations required"). When the completion status is not "as expected," this paragraph shall reference the following paragraphs for details.
Problems encountered. This paragraph shall be divided into subparagraphs that identify each test case in which one or more problems occurred.
(Project-unique identifier of a test case). This paragraph shall identify by project-unique identifier a test case in which one or more problems occurred, and shall provide:
a. A brief description of the problem(s) that occurred
b. Identification of the test procedure step(s) in which they occurred
c. Reference(s) to the associated problem/change report(s) and backup data, as applicable
d. The number of times the procedure or step was repeated in attempting to correct the problem(s) and the outcome of each attempt
e. Back¬up points or test steps where tests were resumed for retesting
Deviations from test cases/procedures. This paragraph shall be divided into subparagraphs that identify each test case in which deviations from test case/test procedures occurred.
(Project-unique identifier of a test case). This paragraph shall identify by project-unique identifier a test case in which one or more deviations occurred, and shall provide:
a. A description of the deviation(s) (for example, test case run in which the deviation occurred and nature of the deviation, such as substitution of required equipment, procedural steps not followed, schedule deviations). (Red-lined test procedures may be used to show the deviations)
b. The rationale for the deviation(s)
1. An assessment of the deviations' impact on the validity of the test case

Test log. This section shall present, possibly in a figure or appendix, a chronological record of the test events covered by this report. This test log shall include:
a. The date(s), time(s), and location(s) of the tests performed
b. The hardware and software configurations used for each test including, as applicable, part/model/serial number, manufacturer, revision level, and calibration date of all hardware, and version number and name for the software components used
c. The date and time of each test¬related activity, the identity of the individual(s) who performed the activity, and the identities of witnesses, as applicable
Notes. This section shall contain any general information that aids in understanding this document (e.g., background information, glossary, rationale). This section shall include an alphabetical listing of all acronyms, abbreviations, and their meanings as used in this document and a list of any terms and definitions needed to understand this document.

Boundary value analysis

What is boundary value analysis in software testing?
Concepts: Boundary value analysis is a methodology for designing test cases that concentrates software testing effort on cases near the limits of valid ranges Boundary value analysis is a method which refines equivalence partitioning. Boundary value analysis generates test cases that highlight errors better than equivalence partitioning. The trick is to concentrate software testing efforts at the extreme ends of the equivalence classes. At those points when input values change from valid to invalid errors are most likely to occur. As well, boundary value analysis broadens the portions of the business requirement document used to generate tests. Unlike equivalence partitioning, it takes into account the output specifications when deriving test cases.

How do you perform boundary value analysis?
Once again, you'll need to perform two steps:
1. Identify the equivalence classes.
2. Design test cases.
But the details vary. Let's examine each step.
Step 1: identify equivalence classes
Follow the same rules you used in equivalence partitioning. However, consider the output specifications as well. For example, if the output specifications for the inventory system stated that a report on inventory should indicate a total quantity for all products no greater than 999,999, then you d add the following classes
to the ones you found previously:
6. The valid class ( 0 < = total quantity on hand < = 999,999 )
7. The invalid class (total quantity on hand <0)
8. The invalid class (total quantity on hand> 999,999 )


Step 2: design test cases
In this step, you derive test cases from the equivalence classes. The process is similar to that of equivalence partitioning but the rules for designing test cases differ. With equivalence partitioning, you may select any test case within a range and any on either side of it with boundary analysis, you focus your attention on cases close to the edges of the range. The detailed rules for generating test cases follow:



Rules for test cases
1. If the condition is a range of values, create valid test cases for each end of the range and invalid test cases just beyond each end of the range. For example, if a valid range of quantity on hand is -9,999
through 9,999, write test cases that include:
1. the valid test case quantity on hand is -9,999,
2. the valid test case quantity on hand is 9,999,
3. the invalid test case quantity on hand is -10,000 and
4. the invalid test case quantity on hand is 10,000
You may combine valid classes wherever possible, just as you did with equivalence partitioning, and, once again, you may not combine invalid classes. Don�t forget to consider output conditions as well. In our inventory example the output conditions generate the following test cases:
1. the valid test case total quantity on hand is 0,
2. the valid test case total quantity on hand is 999,999
3. the invalid test case total quantity on hand is -1 and
4. the invalid test case total quantity on hand is 1,000,000

2. A similar rule applies where the, condition states that the number of values must lie within a certain range select two valid test cases, one for each boundary of the range, and two invalid test cases, one just below and one just above the acceptable range.
3. Design tests that highlight the first and last records in an input or output file.
4. Look for any other extreme input or output conditions, and generate a test for each of them.

Definition of Boundary Value Analysis from our Software Testing Dictionary:
Boundary Value Analysis (BVA). BVA is different from equivalence partitioning in that it focuses on "corner cases" or values that are usually out of range as defined by the specification. This means that if function
expects all values in range of negative 100 to positive 1000, test inputs would include negative 101 and positive 1001. BVA attempts to derive the value often used as a technique for stress, load or volume testing. This type of validation is usually performed after positive functional validation has completed (successfully) using requirements specifications and user documentation

Equivalence Class Partitioning

A definition of Equivalence Partitioning from our software testing dictionary:
Equivalence Partitioning: An approach where classes of inputs are categorized for product or function validation. This usually does not include combinations of input, but rather a single state value based by class. For example, with a given function there may be several classes of input that may be used for positive testing. If function expects an integer and receives an integer as input, this would be considered as positive test assertion. On the other hand, if a character or any other input class other than integer is provided, this would be considered a negative test assertion or condition.

WHAT IS EQUIVALENCE PARTITIONING?
Concepts: Equivalence partitioning is a method for deriving test cases. In this method, classes of input conditions called equivalence classes are identified such that each member of the class causes the same kind of processing and output to occur.
In this method, the tester identifies various equivalence classes for partitioning. A class is a set of input conditions that are is likely to be handled the same way by the system. If the system were to handle one case in the class erroneously, it would handle all cases erroneously.
WHY LEARN EQUIVALENCE PARTITIONING?
Equivalence partitioning drastically cuts down the number of test cases required to test a system reasonably. It is an attempt to get a good 'hit rate', to find the most errors with the smallest number of test cases.
DESIGNING TEST CASES USING EQUIVALENCE PARTITIONING
To use equivalence partitioning, you will need to perform two steps

 Identify the equivalence classes
 Design test cases


STEP 1: IDENTIFY EQUIVALENCE CLASSES
Take each input condition described in the specification and derive at least two equivalence classes for it. One class represents the set of cases which satisfy the condition (the valid class) and one represents cases which do not (the invalid class)
Following are some general guidelines for identifying equivalence classes:

a) If the requirements state that a numeric value is input to the system and must be within a range of values, identify one valid class inputs which are within the valid range and two invalid equivalence classes inputs which are too low and inputs which are too high. For example, if an item in inventory can have a quantity of - 9999 to + 9999, identify the following classes:

1. One valid class: (QTY is greater than or equal to -9999 and is less than or equal to 9999). This is written as (- 9999 < = QTY < = 9999)
2. the invalid class (QTY is less than -9999), also written as (QTY < -9999)
3. the invalid class (QTY is greater than 9999) , also written as (QTY >9999)

b) If the requirements state that the number of items input by the system at some point must lie within a certain range, specify one valid class where the number of inputs is within the valid range, one invalid class where there are too few inputs and one invalid class where there are too many inputs.

For example, specifications state that a maximum of 4 purchase orders can be registered against anyone product. The equivalence classes are: the valid equivalence class: (number of purchase an order is greater than or equal to 1 and less than or equal to 4, also written as (1 < = no. of purchase orders < = 4) the invalid class (no. of purchase orders> 4) the invalid class (no. of purchase orders < 1)
c) If the requirements state that a particular input item match one of a set of values and each case will be dealt with the same way, identify a valid class for values in the set and one invalid class representing values outside of the set.
Says that the code accepts between 4 and 24 inputs; each is a 3-digit integer
• One partition: number of inputs
• Classes “x<4”, “4<=x<=24”, “24• Chosen values: 3,4,5,14,23,24,25

TEST PLAN

• The test plan keeps track of possible tests that will be run on the system after coding.
• The test plan is a document that develops as the project is being developed.
• Record tests as they come up
• Test error prone parts of software development.
• The initial test plan is abstract and the final test plan is concrete.
• The initial test plan contains high level ideas about testing the system without getting into the details of exact test cases.
• The most important test cases come from the requirements of the system.
• When the system is in the design stage, the initial tests can be refined a little.
• During the detailed design or coding phase, exact test cases start to materialize.
• After coding, the test points are all identified and the entire test plan is exercised on the software.

Purpose of Software Test Plan:

• To achieve 100% CORRECT code. Ensure all Functional and Design Requirements are implemented as specified in the documentation.
• To provide a procedure for Unit and System Testing.
• To identify the documentation process for Unit and System Testing.
• To identify the test methods for Unit and System Testing.
Advantages of test plan

• Serves as a guide to testing throughout the development.
• We only need to define test points during the testing phase.
• Serves as a valuable record of what testing was done.
• The entire test plan can be reused if regression testing is done later on.
• The test plan itself could have defects just like software!

In software testing, a test plan gives detailed testing information regarding an upcoming testing effort, including
• Scope of testing
• Schedule
• Test Deliverables
• Release Criteria
• Risks and Contingencies

Process of the Software Test Plan

• Identify the requirements to be tested. All test cases shall be derived using the current Design Specification.
• Identify which particular test(s) you're going to use to test each module.
• Review the test data and test cases to ensure that the unit has been thoroughly verified and that the test data and test cases are adequate to verify proper operation of the unit.
• Identify the expected results for each test.
• Document the test case configuration, test data, and expected results. This information shall be submitted via the on-line Test Case Design(TCD) and filed in the unit's Software Development File(SDF). A successful Peer Technical Review baselines the TCD and initiates coding.
• Perform the test(s).
• Document the test data, test cases, and test configuration used during the testing process. This information shall be submitted via the on-line Unit/System Test Report(STR) and filed in the unit's Software Development File(SDF).
• Successful unit testing is required before the unit is eligible for component integration/system testing.
• Unsuccessful testing requires a Program Trouble Report to be generated. This document shall describe the test case, the problem encountered, its possible cause, and the sequence of events that led to the problem. It shall be used as a basis for later technical analysis.
• Test documents and reports shall be submitted on-line. Any specifications to be reviewed, revised, or updated shall be handled immediately.
Deliverables: Test Case Design, System/Unit Test Report, Problem Trouble Report(if any).
Test plan template
• Test Plan Identifier
• References
• Introduction of Testing
• Test Items
• Software Risk Issues
• Features to be Tested
• Features not to be Tested
• Approach
• Item Pass/Fail Criteria
• Entry & Exit Criteria
• Suspension Criteria and Resumption Requirements
• Test Deliverables
• Remaining Test Tasks
• Environmental Needs
• Staffing and Training Needs
• Responsibilities
• Schedule
• Planning Risks and Contingencies
• Approvals
• Glossary
Test plan identifier
Master test plan for the Line of Credit Payment System.
References
List all documents that support this test plan.
Documents that are referenced include:
• Project Plan
• System Requirements specifications.
• High Level design document.
• Detail design document.
• Development and Test process standards.
• Methodology guidelines and examples.
• Corporate standards and guidelines.
Objective of the plan
Scope of the plan
In relation to the Software Project plan that it relates to. Other items may include, resource and budget constraints, scope of the testing effort, how testing relates to other evaluation activities (Analysis & Reviews), and possibly the process to be used for change control and communication and coordination of key activities.
Test items (functions)
These are things you intend to test within the scope of this test plan. Essentially, something you will test, a list of what is to be tested. This can be developed from the software application inventories as well as other sources of documentation and information.
This can be controlled on a local Configuration Management (CM) process if you have one. This information includes version numbers, configuration requirements where needed, (especially if multiple versions of the product are supported). It may also include key delivery schedule issues for critical elements.
Remember, what you are testing is what you intend to deliver to the client.
This section can be oriented to the level of the test plan. For higher levels it may be by application or functional area, for lower levels it may be by program, unit, module or build.
Software risk issues
Identify what software is to be tested and what the critical areas are, such as:
• Delivery of a third party product.
• New version of interfacing software.
• Ability to use and understand a new package/tool, etc.
• Extremely complex functions.
• Modifications to components with a past history of failure.
• Poorly documented modules or change requests.
There are some inherent software risks such as complexity; these need to be identified.
• Safety.
• Multiple interfaces.
• Impacts on Client.
• Government regulations and rules.
Another key area of risk is a misunderstanding of the original requirements. This can occur at the management, user and developer levels. Be aware of vague or unclear requirements and requirements that cannot be tested.
The past history of defects (bugs) discovered during Unit testing will help identify potential areas within the software that are risky. If the unit testing discovered a large number of defects or a tendency towards defects in a particular area of the software, this is an indication of potential future problems. It is the nature of defects to cluster and clump together. If it was defect ridden earlier, it will most likely continue to be defect prone.
One good approach to define where the risks are is to have several brainstorming sessions.
Features to be tested
This is a listing of what is to be tested from the user's viewpoint of what the system does. This is not a technical description of the software, but a users view of the functions.
Set the level of risk for each feature. Use a simple rating scale such as (H, M, L): High, Medium and Low. These types of levels are understandable to a User. You should be prepared to discuss why a particular level was chosen.
Features not to be tested
This is a listing of what is 'not' to be tested from both the user's viewpoint of what the system does and a configuration management/version control view. This is not a technical description of the software, but a user's view of the functions.
Identify why the feature is not to be tested, there can be any number of reasons.
• Not to be included in this release of the Software.
• Low risk has been used before and was considered stable.
• Will be released but not tested or documented as a functional part of the release of this version of the software.
Approach (strategy)
This is your overall test strategy for this test plan; it should be appropriate to the level of the plan (master, acceptance, etc.) and should be in agreement with all higher and lower levels of plans. Overall rules and processes should be identified.
• Are any special tools to be used and what are they?
• Will the tool require special training?
• What metrics will be collected?
• Which level is each metric to be collected at?
• How is Configuration Management to be handled?
• How many different configurations will be tested?
• Hardware
• Software
• Combinations of HW, SW and other vendor packages
• What levels of regression testing will be done and how much at each test level?
• Will regression testing be based on severity of defects detected?
• How will elements in the requirements and design that do not make sense or are un testable be processed?

If this is a master test plan the overall project testing approach and coverage requirements must also be identified.
Specify if there are special requirements for the testing.
• Only the full component will be tested.
• A specified segment of grouping of features/components must be tested together.
Other information that may be useful in setting the approach are:
(3) MTBF, Mean Time Between Failures - if this is a valid measurement for the test involved and if the data is available.
(4) SRE, Software Reliability Engineering - if this methodology is in use and if the information is available.
How will meetings and other organizational processes be handled?
.Item pass/fail criteria
Show stopper issues. Specify the criteria to be used to determine whether each test item has passed or failed.
Entry & exit criteria
Entrance Criteria
The Entrance Criteria specified by the system test controller, should be fulfilled before System Test can commence. In the event, that any criterion has not been achieved, the System Test may commence if Business Team and Test Controller are in full agreement that the risk is manageable.
• All developed code must be unit tested. Unit and Link Testing must be completed and signed off by development team.
• System Test plans must be signed off by Business Analyst and Test Controller.
• All human resources must be assigned and in place.
• All test hardware and environments must be in place, and free for System test use.
• The Acceptance Tests must be completed, with a pass rate of not less than 80%.
Exit Criteria
The Exit Criteria detailed below must be achieved before the Phase 1 software can be recommended for promotion to Operations Acceptance status. Furthermore, I recommend that there be a minimum 2 days effort Final Integration testing AFTER the final fix/change has been retested.
• All High Priority errors from System Test must be fixed and tested
• If any medium or low-priority errors are outstanding - the implementation risk must be signed off as acceptable by Business Analyst and Business Expert
Suspension Criteria and Resumption Requirements:
This is a particular risk clause to define under what circumstances testing would stop and restart.

Resumption Criteria
In the event that system testing is suspended resumption criteria will be specified and testing will not re-commence until the software reaches these criteria.
• Project Integration Test must be signed off by Test Controller and Business Analyst.
• Business Acceptance Test must be signed off by Business Expert.

Risks and Contingencies
This defines all other risk events, their likelihood, impact and counter measures to over come them.
Summary
The goal of this exercise is to familiarize students with the process of creating test plans.
This exercise is divided into three tasks as described below.
• Devise a test plan for the group.
• Inspect the plan of another group who will in turn inspect yours.
• Improve the group's own plan based on comments from the review session
Task 1: Creating a Test Plan
The role of a test plan is to guide all testing activities. It defines what is to be tested and what is to be overlooked, how the testing is to be performed (described on a general level) and by whom. It is therefore a managerial document, not technical one - in essence, it is a project plan for testing. Therefore, the target audience of the plan should be a manager with a decent grasp of the technical issues involved.
Experience has shown that good planning can save a lot of time, even in an exercise, so do not underestimate the effort required for this phase.
The goal of all these exercises is to carry out system testing on Word Pad, a simple word processor. Your task is to write a thorough test plan in English using the above-mentioned sources as guidelines. The plan should be based on the documentation of Word Pad




Task 2: Inspecting a Test Plan

The role of a review is to make sure that a document (or code in a code review) is readable and clear and that it contains all the necessary information and nothing more. Some implementation details should be kept in mind:
• The groups will divide their roles themselves before arriving at the inspection. A failure to follow the roles correctly will be reflected in the grading. However, one of the assistants will act as the moderator and will not assume any other roles.
• There will be only one meeting with the other group and the moderator. All planning, overview and preparation is up to the groups themselves. You should use the suggested check lists in the lecture notes while preparing. Task 3 deals with the after-meeting activities.
• The meeting is rather short, only 60 minutes for a pair (that is, 30 minutes each). Hence, all comments on the language used in the other group's test plan are to be given in writing. The meeting itself concentrates on the form and content of the plan.

Task 3: Improved Test Plan and Inspection Report

After the meeting, each group will prepare a short inspection report on their test plan listing their most typical and important errors in the first version of the plan together with ideas for correcting them. You should also answer the following questions in a separate document:
• What is the role of the test plan in designing test cases?
• What were the most difficult parts in your test plan and why?
Furthermore, the test plan is to be revised according to the input from the inspection.

Test Case:
A set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
In software engineering, a test case is a set of conditions or variables under which a tester will determine if a requirement upon an application is partially or fully satisfied. It may take many test cases to determine that a requirement is fully satisfied. In order to fully test that all the requirements of an application are met, there must be at least one test case for each requirement unless a requirement has sub requirements. In that situation, each sub requirement must have at least one test case. Some methodologies recommend creating at least two test cases for each requirement. One of them should perform positive testing of requirement and other should perform negative testing.

If the application is created without formal requirements, then test cases are written based on the accepted normal operation of programs of a similar class.
What characterises a formal, written test case is that there is a known input and an expected output, which is worked out before the test is executed. The known input should test a precondition and the expected output should test a post condition.
Under special circumstances, there could be a need to run the test, produce results, and then a team of experts would evaluate if the results can be considered as a pass. This happens often on new products' performance number determination. The first test is taken as the base line for subsequent test / product release cycles.
Written test cases include a description of the functionality to be tested taken from either the requirements or use cases, and the preparation required to ensure that the test can be conducted.
Written test cases are usually collected into Test suites.
A variation of test cases are most commonly used in acceptance testing. Acceptance testing is done by a group of end-users or clients of the system to ensure the developed system meets their requirements. User acceptance testing is usually differentiated by the inclusion of happy path or positive test cases.

System Testing

System testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of Black box testing and as such, should require no knowledge of the inner design of the code or logic. System testing should be performed by testers who are trained to plan, execute, and report on application and system code. They should be aware of scenarios that might not occur to the end user, like testing for null, negative, and format inconsistent values.
System testing is actually done to the entire system against the Functional Requirement Specifications (FRS) and/or the System Requirement Specification (SRS). Moreover, the System testing is an investigatory testing phase, where the focus is to have almost a destructive attitude and test not only the design, but also the behavior and even the believed expectations of the customer. It is also intended to test up to and beyond the bounds defined in the software/hardware requirements specification.
Types of System Testing:

Types of Testing:
• Sanity Testing
• Compatibility Testing
• Recovery Testing
• Usability Testing
• Exploratory Testing
• Adhoc Testing
• Stress Testing
• Volume Testing
• Load Testing
• Performance Testing
• Security Testing

Sanity Testing
Testing the major working functionality of the system whether the system is working fine for the major testing effort. This testing is done before the testing phase and after the coding. The tests performed during sanity testing are
Compatibility Testing
Testing how well software performs in a particular hardware/software/operating system/network/environment etc.
Recovery Testing
Recovery testing is basically done in order to check how fast and better the application can recover against any type of crash or hardware failure etc. Type or extent of recovery is specified in the requirement specifications.
UsabilityTesting:
This testing is also called as ‘Testing for User-Friendliness’. This testing is done if User Interface of the application stands an important consideration and needs to be specific for the specific type of user.
ExploratoryTesting:
This testing is similar to the ad-hoc testing and is done in order to learn/explore the application.
Ad-hocTesting:
This type of testing is done without any formal Test Plan or Test Case creation. Ad-hoc testing helps in deciding the scope and duration of the various other testing and it also helps testers in learning the application prior starting with any other testing.
StressTesting:
The application is tested against heavy load such as complex numerical values, large number of inputs, large number of queries etc. which checks for the stress/load the applications can withstand.
VolumeTesting:
Volume testing is done against the efficiency of the application. Huge amount of data is processed through the application (which is being tested) in order to check the extreme limitations of theSystem.
LoadTesting:
The application is tested against heavy loads or inputs such as testing of web sites in order to find out at what point the web-site/application fails or at what point its performance degrades.
UserAcceptanceTesting:
In this type of testing, the software is handed over to the user in order to find out if the software meets the user expectations and works as it is expected to.
AlphaTesting:
In this type of testing, the users are invited at the development center where they use the application and the developers note every particular input or action carried out by the user. Any type of abnormal behavior of the system is noted and rectified by the developers.

BetaTesting:
In this type of testing, the software is distributed as a beta version to the users and users test the application at their sites. As the users explore the software, in case if any exception/defect occurs that is reported to the developers.








Testing Techniques

Manual Testing Technique:

This method of testing is done manually by the tester on the application under test
Why do manual testing?
Even in this age of short development cycles and automated-test-driven development, manual testing contributes vitally to the software development process. Here are a number of good reasons to do manual testing:
• By giving end users repeatable sets of instructions for using prototype software, manual testing allows them to be involved early in each development cycle and draws invaluable feedback from them that can prevent "surprise" application builds that fail to meet real-world usability requirements.
• Manual test cases gives testers something to use while awaiting for the construction of the application
• Manual test cases can be used to provide feedback to development teams in the form of a set of repeatable steps that lead to bugs or usability problems.
• If done thoroughly, manual test cases can also form the basis for help or tutorial files for the application under test.
• Finally, in a nod toward test-driven development, manual test cases can be given to the development staff to provide a clear description of the way the application should flow across use cases.
In summary, manual testing fills a gap in the testing repertoire and adds invaluably to the software development process.
Test automation is the use of software to control the execution of tests , the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.
Over the past few years , tools that help programmers quickly create applications with graphical user interfaces have dramatically improved programmer productivity. This has increased the pressure on testers, who are often perceived as bottlenecks to the delivery of software products. Testers are being asked to test more and more code in less and less time. Test automation is one way to do this, as manual testing is time consuming. As and when different versions of software are released, the new features will have to be tested manually time and again. But, now there are tools available that help the testers in the automation of the GUI which reduce the test time as well as the cost, other test automation tools support execution of performance tests.
Many test automation tools provide record and playback features that allow users to record interactively user actions and replay it back any number of times, comparing actual results to those expected. However, reliance on these features poses major reliability and maintainability problems. Most successful automators use a software engineering approach, and as such most serious test automation is undertaken by people with development experience.
A growing trend in software development is to use testing frameworks such as the xUnit frameworks (for example, JUnit and Nunit which allow the code to conduct unit tests to determine whether various sections of the code are acting as expected in various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected. All three aspects of testing can be automated.
Another important aspect of test automation is the idea of partial test automation, or automating parts but not all of the software testing process. If, for example, an oracle cannot reasonably be created, or if fully automated tests would be too difficult to maintain, then a software tools engineer can instead create testing tools to help human testers perform their jobs more efficiently. Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.
Test automation is expensive and it is an addition, not a replacement, to manual testing. It can be made cost-effective in the longer term though, especially in regression testing. One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.

Testing Types

This article explains about different testing types Unit Test. System Test, Integration Test, Functional Test, Performance Test, Beta Test and Acceptance Test.
Introduction:
The development process involves various types of testing. Each test type addresses a specific testing requirement. The most common types of testing involved in the development process are:

• Unit Test.
• System Test
• Integration Test
• Functional Test
• Performance Test
• Beta Test
• Acceptance Test.


Unit Test
The first test in the development process is the unit test. The source code is normally divided into modules, which in turn are divided into smaller units called units. These units have specific behavior. The test done on these units of code is called unit test. Unit test depends upon the language on which the project is developed. Unit tests ensure that each unique path of the project performs accurately to the documented specifications and contains clearly defined inputs and expected results.


System Test
Several modules constitute a project. If the project is long-term project, several developers write the modules. Once all the modules are integrated, several errors may arise. The testing done at this stage is called system test.


System testing ensures that the entire integrated software system meets requirements. It tests a configuration to ensure known and predictable results. System testing is based on process descriptions and flows, emphasizing pre-driven process links and integration points.


Testing a specific hardware/software installation. This is typically performed on a COTS (commercial off the shelf) system or any other system comprised of disparent parts where custom configurations and/or unique installations are the norm.


Functional Test
Functional test can be defined as testing two or more modules together with the intent of finding defects, demonstrating that defects are not present, verifying that the module performs its intended functions as stated in the specification and establishing confidence that a program does what it is supposed to do.


Acceptance Testing
Testing the system with the intent of confirming readiness of the product and customer acceptance.


Ad Hoc Testing
Testing without a formal test plan or outside of a test plan. With some projects this type of testing is carried out as an adjunct to formal testing. If carried out by a skilled tester, it can often find problems that are not caught in regular testing. Sometimes, if testing occurs very late in the development cycle, this will be the only kind of testing that can be performed. Sometimes ad hoc testing is referred to as exploratory testing.


Alpha Testing
Testing after code is mostly complete or contains most of the functionality and prior to users being involved. Sometimes a select group of users are involved. More often this testing will be performed in-house or by an outside testing firm in close cooperation with the software engineering department.





Automated Testing
Software testing that utilizes a variety of tools to automate the testing process and when the importance of having a person manually testing is diminished. Automated testing still requires a skilled quality assurance professional with knowledge of the automation tool and the software being tested to set up the tests.


Beta Testing
Testing after the product is code complete. Betas are often widely distributed or even distributed to the public at large in hopes that they will buy the final product when it is released.


Black Box Testing
Testing software without any knowledge of the inner workings, structure or language of the module being tested. Black box tests, as most other kinds of tests, must be written from a definitive source document, such as a specification or requirements document.
Compatibility Testing
Testing used to determine whether other system software components such as browsers, utilities, and competing software will conflict with the software being tested.


Configuration Testing
Testing to determine how well the product works with a broad range of hardware/peripheral equipment configurations as well as on different operating systems and software.


Independent Verification & Validation
The process of exercising software with the intent of ensuring that the software system meets its requirements and user expectations and doesn't fail in an unacceptable manner. The individual or group doing this work is not part of the group or organization that developed the software. A term often applied to government work or where the government regulates the products, as in medical devices.


Installation Testing
Testing with the intent of determining if the product will install on a variety of platforms and how easily it installs.


Integration Testing
Testing two or more modules or functions together with the intent of finding interface defects between the modules or functions. Testing completed at as a part of unit or functional testing, and sometimes, becomes its own standalone test phase. On a larger level, integration testing can involve a putting together of groups of modules and functions with the goal of completing and verifying that the system meets the system requirements. (see system testing)


Load Testing
Testing with the intent of determining how well the product handles competition for system resources. The competition may come in the form of network traffic, CPU utilization or memory allocation.


Performance Testing
Testing with the intent of determining how quickly a product handles a variety of events. Automated test tools geared specifically to test and fine-tune performance are used most often for this type of testing.


Pilot Testing
Testing that involves the users just before actual release to ensure that users become familiar with the release contents and ultimately accept it. Often is considered a Move-to-Production activity for ERP releases or a beta test for commercial products. Typically involves many users, is conducted over a short period of time and is tightly controlled. (see beta testing).


Regression Testing
Testing with the intent of determining if bug fixes have been successful and have not created any new problems. Also, this type of testing is done to ensure that no degradation of baseline functionality has occurred.


Security Testing
Testing of database and network software in order to keep company data and resources secure from mistaken/accidental users, hackers, and other malevolent attackers.


Software Testing
The process of exercising software with the intent of ensuring that the software system meets its requirements and user expectations and doesn't fail in an unacceptable manner. The organization and management of individuals or groups doing this work is not relevant. This term is often applied to commercial products such as internet applications. (contrast with independent verification and validation)


Stress Testing
Testing with the intent of determining how well a product performs when a load is placed on the system resources that nears and then exceeds capacity.


User Acceptance Testing
See Acceptance Testing.


White Box Testing
Testing in which the software tester has knowledge of the inner workings, structure and language of the software, or at least its purpose.