Rational TestManager User's Guide
Part 1: Using TestManager to Manage Testing Projects
Introducing Rational TestManager
What Is Rational TestManager
TestManager Workflow
Testing Workflow
Planning Tests
Test Inputs
Test Plans
Test Case Folders
Test Cases
Iterations
Configurations
Designing Tests
Implementing Tests
Executing Tests
Evaluating Tests
TestManager and Other Rational Products
The Rational Unified Process
Projects and the Rational Administrator
Rational Projects
Security and Privileges for the Rational Test Datastore
Automated Test Scripts and Rational Robot
Automated Test Scripts and Rational TestFactory
Component Testing and Rational QualityArchitect
Requirements and Rational RequisitePro
Model Elements and Rational Rose
Defects and Rational ClearQuest
Reports and Rational SoDA
TestManager and Extensibility
Extending Test Script Types
Defining Custom Test Input Types
Defining Custom Test Script Types
Functional and Performance Testing
Functional Testing
Performance Testing
Virtual Testers
Virtual Testers in Functional Tests
Virtual Testers in Performance Tests
Local and Agent Computers
The TestManager Environment
Suites
Starting Rational TestManager
Logging On to TestManager
Starting Other Rational Products and Components from TestManager
The TestManager Main Window
Test Asset Workspace
Planning Tab
Execution Tab
Results Tab
Analysis Tab
Other TestManager Windows
Planning Tests
About Test Planning
Identifying What to Test by Using Test Inputs
Built-In Test Input Types
Requirements from Rational RequisitePro
Values from Microsoft Excel
Registering a Test Input Source
Custom Test Input Types
Creating a Test Plan
Creating Test Plans
Properties of a Test Plan
Inheriting Iterations and Configurations from a Test Plan
Organizing Test Cases with Folders
Inheriting Iterations and Configurations from a Test Case Folder
Creating Test Cases
Properties of a Test Case
Specifying the Owner
Defining the Configurations to Test
Defining Configuration Attributes and Their Values
Viewing Built-In Configuration Attributes
Defining Custom Configuration Attributes
Setting Up Custom Attributes in tmsconfig.csv
Defining the Configurations You Need to Test
Viewing and Editing Your Configurations
Associating a Configuration with a Test Case
Inheriting Test Case Properties
Specifying When to Run Tests
Creating and Editing Iterations
Associating Iterations with a Test Case
Test Impact Analysis
Setting Up Traceability Using Test Inputs
Inherited Test Inputs
Setting Up Automatic Test Case Suspicion
Built-In Test Input Adapters
Custom Test Input Adapters
Test Case Suspicion
Inherited Test Case Suspicion
Managing Test Assets
Upgrading from Previous Versions of Rational TestManager
Verifying and Repairing Test Datastores Using the Datastore Doctor
Exporting and Importing Test Assets
Exporting Test Assets for Use in Other Test Datastores
Exporting Test Log Details to Third-Party Tools
Exporting Manual Test Scripts
Importing Test Assets
Importing Manual Test Scripts
Copy, Cut, and Paste Test Assets
Software Configuration Management
Test Script Version Control
Setting Up Test Script Version Control
Adding Test Scripts to Version Control
Designing Tests
About Designing Tests
Benefits of Good Test Design
Specifying the Testing Steps and Verification Points
Specifying Conditions and Acceptance Criteria of Test Cases
Example of a Test Design
Implementing Tests
About Implementing Tests
Test Scripts Window
Implementing Test Cases
Built-In Test Script Types
Automated Test Scripts Recorded in Rational Robot
Manual Test Scripts Created in Rational ManualTest
Custom Test Script Types
Command-Line and Custom Adapters
Command-Line Adapters
Custom Adapters
Defining a New Test Script Type
Suites Created in TestManager
Calling Test Script Services from Test Scripts
Test Script Services and Test Script Types
Test Script Services and TestManager
Creating Manual Test Scripts
Importing a Manual Test Script from Another Project or a File
Creating a Manual Test Script from a Test Case Design
Creating a Manual Test Script in Rational ManualTest
Starting Rational ManualTest
Example of a Manual Test Script
Setting the Default Editor for Manual Test Scripts
Creating Test Script Queries
Customizing Test Assets
Associating an Implementation with a Test Case
Implementing Tests as Suites
Using Suites in Functional Tests
Using Suites in Performance Tests
Activities Common to Performance and Functional Testing
Whether you are running performance tests or functional tests, you perform certain common activit...
Defining Agent Computers and Computer Lists
Adding an Agent Computer
Combining Computers into Lists
Changing the Settings of an Agent Computer
Creating a Suite
Creating a Suite from a Wizard
Opening a Suite
Editing a Test Script
Editing the Properties of a Test Script
Editing the Text of a Test Script
Editing the Properties of a Suite
Replacing Items in a Suite
Editing the Run Properties of Items in a Suite
Editing Information for All User and Computer Groups
Editing Settings for Virtual Testers
Initializing TSS Environment Variables
Disabling Test Script Services
Changing the Number of Start Test Scripts
Limiting the Number of Test Scripts
Changing the Way Random Numbers Are Generated
Initializing Shared Variables
Printing and Exporting a Suite
Saving a Suite
Executing Tests
About Running Tests
Built-In Support for Running Test Scripts
Running Automated Test Scripts
Running Manual Test Scripts
Example of Running a Manual Test Script
Running Test Cases
Viewing the Associated Implementations
Running a Test Case
Ignoring Configured Test Cases
Before You Run a Suite
Checking a Suite
Checking Agent Computers
Controlling Runtime Information of a Suite
Start Group Information
Suite Pass Criteria
Execution Order
Time Information
Seed
IP Aliasing
Controlling How a Suite Terminates
Specifying Virtual Testers and Configurations for the Suite Run
Running a Suite from TestManager
Running a Suite from the Command Line
Running Two Suites Sequentially
Scheduling a Suite to Run at a Specific Time
Monitoring Test Runs
The Progress Bar and the Default Views
Displaying the Suite Views
The Suite - Overall View
The Suite - Users and Suite - Computers Views
Displaying the State Histograms
Zooming In on Histogram Bars
Displaying the User and Computer Views
User/Computer View - Full
User/Computer View - Compact
User/Computer View - Results
User/Computer View - Source
User/Computer View - Message
Displaying the Shared Variables View
Changing the Value of a Shared Variable
Displaying the Virtual Testers Waiting on a Shared Variable
Displaying the Test Script View
Debugging a Test Script
Displaying the Sync Points View
Displaying Virtual Testers Waiting on a Synchronization Point
Releasing a Synchronization Point
Displaying the Computer View
Viewing Resource Usage During a Run
Graphing Resource Usage During a Run
Viewing Computers at the Start or End of a Run
Displaying the Transactor View
Displaying the Group Views
Filtering and Sorting Views
Sorting the Virtual Testers in a User or Computer View
Filtering a View
Filtering Virtual Testers
Filtering a Virtual Tester by Value
Filtering a Group View
Restoring the Default Views
Changing Monitor Defaults
Configuring Custom Histograms
Controlling a Suite During a Run
Suspending and Resuming Virtual Testers
Setting and Changing the Update Rate
Stopping Test Runs
Evaluating Tests
About Test Logs
Opening a Test Log in TestManager
The Test Log Window
Test Case Results Tab
Interpreting Test Case Results
Promoting Test Case Results
Details Tab
Viewing Test Log Results
Viewing Test Case Results
Viewing Event Details
About Log Filters
Creating and Editing a Log Filter
Applying a Test Log Filter
Viewing a Test Script
Working with Test Logs
About Test Logs
Suite Log
Virtual Tester Error File
Virtual Tester Output File
About Submitting and Modifying Defects
About the Rational TestStudio Schema
How to Submit and Modify a Defect
Printing a Test Log
Managing Log Event Property Types
Viewing Test Script Results Recorded with Rational Robot
Viewing a Verification Point in the Comparators
Playback/Environmental Differences
Intentional Changes to an Application Build
Reporting Results
About Test Case Reports
About Test Case Distribution Reports
About Test Case Results Distribution Reports
About Default Test Case Coverage Reports
About Reporting Test Case Suspicion
About Test Case Trend Reports
Filtering Test Input Source Information
Viewing Properties of Assets in a Test Case Report
About Listing Reports
Customizing Design Layouts for Listing Reports
About Performance Testing Reports
Selecting Which Reports to Use
Designing Your Own Reports
Additional Reports
Creating Reports
Creating a Test Case Distribution Report
Creating Test Case Suspicion Reports
Creating a Test Case Results Distribution Report
Creating a Test Case Trend Report
Creating a Listing Report
Creating Performance Testing Reports
Opening a Report
Running Reports
Running a Report from the Test Asset Workspace
Running a Report from the Menu
Printing, Saving, or Copying a Test Case Report
Printing, Exporting, or Zooming In on a Listing Report
Copying Reports to a New Project
Creating a Query
Queries for Test Case Distribution, Test Case Trend, and Performance Testing Reports
Queries for Listing Reports
Part 2: Functional Testing with Rational TestManager
Creating Functional Testing Suites
About Suites
Inserting a Computer Group into a Suite
Inserting a Test Script into a Suite
Inserting a Test Case into a Suite
Inserting Suites and Scenarios into Suites
Inserting a Suite into a Suite
Inserting a Scenario
Setting a Precondition Within a Suite
How to Set a Precondition
Relating Preconditions Set in Suites to Those in Test Cases
Inserting a Selector into a Suite
Inserting Other Items into a Suite
Inserting a Delay
Inserting a Synchronization Point
Using Events and Dependencies to Coordinate Execution
Distributing Tests Among Different Computers
Example of a Distributed Functional Test
Executing Suites
Using the Comparators
About the Four Comparators
Starting a Comparator
Using the Object Properties Comparator
The Main Window
The Objects Hierarchy and the Properties List
Changing the Window Focus
Working Within the Objects Hierarchy
Working Within the Properties List
Loading the Current Baseline
Locating and Comparing Differences
Viewing Verification Point Properties
Adding and Removing Properties
Editing the Baseline File
Saving the Baseline File
Using the Text Comparator
The Main Window
The Text Window
Locating and Comparing Differences
Viewing Verification Point Properties
Editing the Baseline File
Saving the Baseline File
Using the Grid Comparator
The Main Window
The Grid Window
Differences List
Setting Display Options
Locating and Comparing Differences
Viewing Verification Point Properties
Using Keys to Compare Data Files
Editing the Baseline File
Saving the Baseline File
Using the Image Comparator
The Main Window
The Image Window
Differences List
Mask/OCR List
The Status Bar
Locating and Comparing Differences
Changing How Differences Are Determined
Changing the Color of Masks, OCR Regions, or Differences
Moving and Zooming an Image
Viewing Image Properties
Working with Masks
Working with OCR Regions
Saving the Baseline File
Viewing Unexpected Active Window
Part 3: Performance Testing with Rational TestManager
Performance Testing Concepts
About Performance Testing
Rational TestManager and Performance Testing
Creating Test Scripts
Planning Performance Tests
Testing Response Times
Setting Pass and Fail Criteria for Performance Tests
Identifying Performance Testing Requirements
Designing a Realistic Workload
Implementing Performance Tests
Examples of Performance Tests
Number of Virtual Testers Supported Under Normal Conditions
Incrementally Increasing Virtual Testers
How a System Performs Under Stress Conditions
How Different System Configurations Affect Performance
Analyzing Performance Results
Comparing Results of Multiple Runs
Comparing Specific Requests and Responses
Determining the Cause of Performance Problems
Analyzing Results Statistically
Monitoring Computer Resources and Tuning Your System
Designing Performance Testing Suites
About Suites
Creating a Suite from a Robot Session
Inserting User Groups into a Suite
Inserting Test Scripts into a Suite
Setting a Precondition Within a Suite
How to Set a Precondition
Relating Preconditions Set in Suites to Those in Test Cases
Inserting Other Items into a Suite
Inserting a Test Case into a Suite
Inserting a Scenario
Inserting a Suite into a Suite
Inserting a Selector
Types of Selectors
Inserting a Delay
Inserting a Transactor
Inserting a Synchronization Point
How Synchronization Points Work
Why Use Synchronization Points?
Release Times and Time-Outs for Synchronization Points in Test Scripts
Scope of a Synchronization Point
Using Events and Dependencies to Coordinate Execution
Executing Suites
Working with Datapools
What Is a Datapool?
Datapool Tools
Managing Datapool Files
Datapool Cursor
Row Access Order
Datapool Limits
What Kinds of Problems Does a Datapool Solve?
Planning and Creating a Datapool
Data Types
Standard and User-Defined Data Types
Finding Out Which Data Types You Need
Creating User-Defined Data Types
Generating Unique Values from User-Defined Data Types
Generating Multi-Byte Characters
Managing Datapools
Creating a Datapool
If There Are Errors
Viewing Datapool Values
Making the Datapool Available to a Test Script
Defining Datapool Columns
Example of Datapool Column Definition
Example of Datapool Value Generation
Editing Datapool Column Definitions
Deleting a Datapool Column
Editing Datapool Values
Renaming or Copying a Datapool
Deleting a Datapool
Importing a Datapool
Datapool Location
Importing a Datapool from Another Project
Exporting a Datapool
Managing User-Defined Data Types
Editing User-Defined Data Type Values
Editing User-Defined Data Type Definitions
Importing a User-Defined Data Type
Renaming or Copying a User-Defined Data Type
Deleting a User-Defined Data Type
Generating and Retrieving Unique Datapool Rows
What You Can Do to Guarantee Unique Row Retrieval
Creating a Datapool Outside Rational Test
Datapool Structure
Example Datapool
Using Microsoft Excel to Create Datapool Data
Saving the Datapool in Excel
Matching Datapool Columns with Test Script Variables
Maximum Number of Imported Columns
Creating a Column of Values Outside Rational Test
Step 1. Create the File
Step 2. Assign the Files Values to the Datapool Column
Generating Unique Values
Reporting Performance Testing Results
About Reports
Running Reports
Running Reports from the Report Bar
Running Reports from the Menu Bar
Customizing Reports
Filtering Report Data
Setting Advanced Options
Eliminating Outliers
Reporting on a Stable Workload
Including Passed and Failed Commands
Reporting on a Particular Command ID
Mapping Computer Resource Usage onto Response Time
Changing a Graphs Appearance or Type
Changing a Graphs Appearance
Displaying and Clearing Data Point Information
Changing a Graphs Type
Enlarging and Rotating a Graph
Changing a Graphs Labels
Filtering Command IDs That Appear in a Graph
Exporting Report Data
Changing Report Defaults
Changing the Reports That Run Automatically
Changing the Reports That Run from the Report Bar
Types of Reports
Command Data Reports
Whats in Command Data Reports?
Command Status Reports
Whats in Command Status Reports?
Command Usage Reports
Whats in Command Usage Reports?
Cumulative Statistics
Summary Statistics
Command Trace
Whats in Command Trace Reports?
Performance Reports
Whats in Performance Reports?
Compare Performance Reports
Defining a Compare Performance Report
Whats in Compare Performance Reports?
Absolute Reports
Weighted Absolute Reports
Relative Reports
Weighted Relative Reports
N/A and Undefined Responses
Response vs. Time Reports
Whats in Response vs. Time Reports?
Configuring Local and Agent Computers
Running More Than 245 Virtual Testers
Running More Than 1000 Virtual Testers
Running More Than 1000 Virtual Testers on One NT Computer
High-Volume Performance Testing in Windows Platforms
Running More Than 24 Virtual Testers on a UNIX Agent
Controlling TCP Port Numbers
Setting Up IP Aliasing
Assigning Values to System Environment Variables
Standard Datapool Data Types
Standard Data Type Table
Data Type Ranges
ManualTest Web Execution
About ManualTest Web Execution
Overview of Tasks
About Shared Projects
How to Run a Test Case from a Web Browser
Viewing the Results
Troubleshooting
Rational Test Asset Parcel File Format
XML Standards for Rational Test Asset Parcel Files
Sample Rational Test Asset Parcel Files
Test Plan Test Asset Parcel File
Test Case Test Asset Parcel File
Test Case Folder Test Asset Parcel File
Build Test Asset Parcel File
Iteration Test Asset Parcel File
Computer Test Asset Parcel File
Computer List Test Asset Parcel File
Configuration Test Asset Parcel File
Configuration Attribute Test Asset Parcel File
Test Script Type Test Asset Parcel File
Test Input Type Test Asset Parcel File
Log Filter Test Asset Parcel File
Log Event Property Type Test Asset Parcel File
Test Log Details Test Asset Parcel File
Index