For this example, we will be looking at Focused Test setup on Web/Mobile Web in pre-production (staging). The focus of this test cycle is to test the functionality of new features.
Note: this is not a template or the only way to set up a test of this type; this is merely a detailed example.
Step 1
First, we will select the Test Type we will be running (in this case, Focused).
Next, we need to specify the Test Environment, which is the environment that we want the testers to evaluate.
Then, we need to specify the particulars of accessing this test.
First, we input the Title of the test. In this case, we are running the test on “Payments” in a pre-production (staging) environment, so the we’ve named the test “Payments, STAGING.”
It’s a website, so we’ve input the appropriate URL and made sure “Url,” not “Upload,” is selected.
Since it’s a staging environment, we need a username and password to access it.
After adding an environment, we need to select which bugs are in scope for this test.
For this test, we’ve selected only “high” and “critical” level severities, as low-severity bugs are out of scope for this test (seen later).
Closer Look: Bug Severities
Different test types include different severity levels in the test scope. Sometimes the bug severities you define internally may vary from test IO's default guidelines. To make sure our views on severity are in line, here’s how we classify bugs and issues.
Functional Bugs
Critical - Preventing a function of the app or website, causes a potential loss of income for the company running the app or website - i.e. an app crash or "not able to login."
High - Serious impact on user experience but doesn’t prevent the function of the app or website.
Low - Minimal impact on user experience.
Non-Functional Bugs
Usability - suggested improvements to existing features and functions that would make the product easier and more intuitive to use.
Visual - The user can accomplish a task, but the interface looks wrong, typically because of responsive design, CSS, HTML, or layout framework problems.
Content - Bugs relate to missing data, images, or broken links.
Note: Non-Functional Bugs are classified as low-severity bugs by definition.
Step 2
Next, we will select which Features we are testing. These are the Features we want to cover in this test cycle. Depending on the Test Type (Focused, in this case), you may have the option to select a variety of bug types that are in-scope for this test as well as a variety of features. At least one feature should be in-scope to be able to run a test.
Since we are only concerned with “payments” for this focused test, we will use the default Cart Feature.
Note: If you want to “Add feature,” you would simply follow the below instructions.
First, select the “Add feature” option at the bottom
Then, fill out the necessary input required to add the new Feature.
If we wanted to test the way customers can filter through our products, the name is that of that Feature could be, “Product filtering.”
Describe where on the site the feature can be found.
Define the expected behavior (the correct functionality) of this feature.
Closer Look: Features
For more in-depth information, best practices and tips on creating Features, check out this article.
Step 3
Moving down the page, we continue with the test setup.
First, we need to state the Goal of the test. This section can make the difference between a good and a great test cycle!
Note: This section should be your guide for our testers. Think about the desired results of the test and what information the testers need in order to discover these types of issues. Common examples include the release of a new feature, a sanity test for critical functional issues only, changes to a common workflow, etc.
It’s also common to include a brief description of your company and product so testers have some context about the software they’re testing. In other words, keep in mind that test instructions are the only way for you to communicate your needs and expectations to a crowd of testers who might have never had experience with your site/platform/app before.
“YourCompanyName.com is a platform where ‘buyers’ can purchase services from ‘providers.’ Both parties are individuals. Goal: Find issues related to the payment process from the perspectives of a buyer and a provider.”
Next, we state what/if any functionalities are Out of scope. This section should explicitly state any Features or functionality that are out of scope for this test. It’s also common to:
State undesired functional severities or bug types in this sections
Inform testers not to take particular steps in the flow (make final purchases, contact support, send out applications etc.)
Share known issues in a given Feature
“Low severity bugs are out of scope. Do not use discount codes.”
Then, we state any Additional requirements. This section should be used to provide any additional information the tester should know to achieve the desired results. Examples include:
Test accounts
Test payment information
Any special requirements to the environment access/ bug report format/ device usage/ access settings
“Use tester account information on the following Google Sheet to login as a buyer and provider. Enter your tester ID next to the accounts you use: https://docs.google.com/spreadsheets/…
Credit Card information: 3333333333334444, any 3 digit security code, any future expiration date”
We have no Attachments for this test cycle and will skip this step.
Step 4
Next, we will need to specify “Where to test” (on which devices) and the “Bug language."
When selecting “Where to test,” we will be choosing which devices our testers will use to run our test. We need to keep in mind our audience and their devices. Since we’re testing for Web and Mobile Web, we’ll include desktops, smartphones, and tablets.
Note: You can define device scope by editing device type, device model, OS, and/or browsers. Please keep in mind that any “unconventional” or outdated device, OS, and browser combinations may lead to irrelevant device specific bugs.
When choosing the Bug language, please keep in mind that testers get invited based on the report language; therefore, the instructions/Features and bug report languages should be consistent.
Note: Bug report language is defaulted to English.
Step 5
Lastly, we select our desired test date and start time. In this case, we are running a single Focused test, which is set to begin at 5:00pm Tuesday, 5th March, delivering results by 5:00pm the next day.
Note: Due to our Team Lead review and invitation process, the earliest a test can start is the next full hour +1 (e.g. current time: 9:40am / test time: 11:00am, current time: 5:05pm / test time: 7:00pm). Duration ranges vary based on Test Type, from two hours (Rapid Test) up to 48 hours.
Click Create Test.
This will essentially save the test in "draft." At this point:
•You should review all test information and make any adjustments as needed before you submit the test and tester/Team Leader invitations go out.
•If you need to leave and return to it, can find your test on the "All Tests" page in the same state.
To edit the Goal, Out of Scope, Additional Requirement fields before submitting follow the appropriate steps below:
On the test cycle page, click on Edit at the bottom of the page, make the desired changes, then click Update which will return you to the test cycle draft with updated fields.
To edit the Features before submitting follow the appropriate steps below:
Open the Manage Features page in a new tab
Make the change to the Feature in the new tab
Click Submit
If you also need to make additional edits to the instructions, click Edit > Update.
Once all information has been confirmed you can Submit the test.