For this example, we will be looking at Coverage Test setup on Android in pre-production (staging). The focus of this test cycle is to run pre-production regression.
Note: this is not a template or the only way to set up a test of this type; this is merely a detailed example.
First, we will select the Test Type we will be running (in this case, Coverage).
Next, we need to specify the Test Environment, which is the environment we want testers to evaluate.
Then, we need to specify the particulars of accessing this test.
- First, we input the Title of the test environment. In this case, we are running the test on Android (APK) 3.2 in pre-production (staging).
- It’s a staging environment, so we must make sure to upload the appropriate APK file by selecting “New File.”
After adding an environment, we need to select which bugs are in scope for this test.
Note: Since this is a Coverage Test, all levels of severity bugs will be reported.
Closer Look: Bug Severities
Different test types include different severity levels in the test scope. Sometimes the bug severities you define internally may vary from test IO's default guidelines. To make sure our views on severity are in line, here’s how we classify bugs and issues.
Critical - Preventing a function of the app or website, causes a potential loss of income for the company running the app or website - i.e. an app crash or "not able to login."
High - Serious impact on user experience but doesn’t prevent the function of the app or website.
Low - Minimal impact on user experience.
Usability - suggested improvements to existing features and functions that would make the product easier and more intuitive to use.
Visual - The user can accomplish a task, but the interface looks wrong, typically because of responsive design, CSS, HTML, or layout framework problems.
Content - Bugs relate to missing data, images, or broken links.
Note: Non-Functional Bugs are classified as low-severity bugs by definition.
Next, we will select which Features we are testing. These are the Features we want to get cover in this test cycle. Depending on the Test Type (Coverage, in this case), you may have the option to select a variety of bug types that are in-scope for this test as well as a variety of features. At least one Feature should be in-scope to be able to run a test.
- In this case, we will need to add features. To do so, you would simply follow the below instructions.
First, select the “Add feature” option.
Then, fill out the necessary input required to add the new feature
- If we want to test the way customers move through the sign-up user flow, we’d name the Feature “Sign Up.”
- Describe where on the site the feature can be found.
- Define the expected behavior (the correct functionality) of this feature.
Closer Look: Features
For more in-depth information, best practices, and tips on creating Features, check out this article.
Here, we are concerned with “pre-release regression,” particularly in terms of international coverage, so we checked all the Features that are relevant to our changes and have enabled Functional, Usability, Content, and Visual bugs.
Moving down the page, we continue with the test setup.
First, we need to state the Goal of the test. This section can make the difference between a good and a great test cycle!
Note: This section should be your guide for our testers. Think about the desired results of the test and what information the testers need in order to discover these types of issues. Common examples include the release of a new feature, a sanity test for critical functional issues only, changes to a common workflow, etc.
It’s also common to include a brief description of your company and product so testers have some context about the software they’re testing. In other words, keep in mind that test instructions are the only way for you to communicate your needs and expectations to a crowd of testers who might have never had experience with your site/platform/app before.
- “Please test general app navigation. One bug we have trouble re-creating is that when you take a screenshot outside the app and then attach it to a message, it won't show the screenshot.”
Next, we state what/if any functionalities are Out of scope. This section should explicitly state any Features or functionality out of scope for this test. It’s also common to:
- State undesired functional severities or bug types in this sections
- Inform testers not to take particular steps in the flow (make final purchases, contact support, send out applications etc.)
- Share known issues in a given Feature
- “We know phone numbers outside of USA and Canada do not work for SMS verification. We know that search results sometimes crash randomly.”
Then, we state any Additional requirements. This section should be used to provide any additional information testers should know to achieve the desired results. Examples include:
- Test accounts
- Test payment information
- Any special requirements to the environment access/ bug report format/ device usage/ access settings
- Test on wifi and cellular networks. Include network info at the end of ‘Actual Results’ section.”
We have no Attachments for this test cycle and will skip this step.
Next, we will need to specify “Where to test” (on which devices) and the “Bug language.”
When selecting “Where to test,” we will be choosing which devices our testers will use to run our test. We need to keep in mind our audience and their devices. Since we’re testing for Web and Mobile Web, we’ll include desktops, smartphones, and tablets.
Note: You can define device scope by editing device type, device model, OS, and/or browsers. Please keep in mind that any “unconventional” or outdated device, OS, and browser combinations may lead to irrelevant device specific bugs.
- When choosing the Bug language, please keep in mind that testers get invited based on the report language; therefore, the instructions/Features and bug report languages should be consistent.
Note: Bug report language is defaulted to English.
Lastly, we select our desired test date and start time. In this case, we are running a single Coverage test, which is set to begin at 7:00pm Wednesday, 6th March, delivering results by 7:00pm the following day.
Note: Due to our Team Lead review and invitation process, the earliest a test can start is the next full hour +1 (e.g. current time: 9:40am / test time: 11:00am, current time: 5:05pm / test time: 7:00pm). Duration ranges vary based on Test Type, from two hours (Rapid Test) up to 48 hours.
Click Create Test.
This will essentially save the test in "draft." At this point:
•You should review all test information and make any adjustments as needed before you submit the test and tester/Team Leader invitations go out.
•If you need to leave and return to it, can find your test on the "All Tests" page in the same state.
To edit the Goal, Out of Scope, Additional Requirement fields before submitting follow the appropriate steps below:
- On the test cycle page, click on Edit at the bottom of the page, make the desired changes, then click Update which will return you to the test cycle draft with updated fields.
To edit the Features before submitting follow the appropriate steps below:
- Open the Manage Features page in a new tab
- Make the change to the Feature in the new tab
- Click Submit
- If you also need to make additional edits to the instructions, click Edit > Update.
Once all information has been confirmed you can Submit the test.