For this example, we will be looking at Coverage Test setup on Web/Mobile Web in production (live). The focus of this test cycle is to run post-release regression.

Note: this is not a template or the only way to set up a test of this type; this is merely a detailed example.

Step 1

First, we will select the Test Type we will be running (in this case, Coverage).

Next, we need to specify the Test Environment, which is the environment that we want the testers to evaluate.

Then, we need to specify the particulars of accessing this test.

First, we input the Title of the test. In this case, we are running the test on post-release regression on a production (live) environment, so we’ve named the test “Web, PROD.”
It’s a website, so we’ve input the appropriate URL and made sure “Url,” not “Upload,” is selected.
Since it’s a live environment, we do not need a username or password to access staging.

Note: Since this is a Coverage Test, all levels of severity bugs will be reported.

Closer Look: Bug Severities

Different test types include different severity levels in the test scope. Sometimes the bug severities you define internally may vary from test IO's default guidelines. To make sure our views on severity are in line, here’s how we classify bugs and issues.

Functional Bugs

Critical - Preventing a function of the app or website, causes a potential loss of income for the company running the app or website - i.e. an app crash or "not able to login."

High - Serious impact on user experience but doesn’t prevent the function of the app or website.

Low - Minimal impact on user experience.

Non-Functional Bugs

Visual - The user can accomplish a task, but the interface looks wrong, typically because of responsive design, CSS, HTML, or layout framework problems.

Content - Bugs relate to missing data, images, or broken links.

Note: Non-Functional Bugs are classified as low-severity bugs by definition.

Step 2

Next, we will select which Features we are testing. These are the Features we want to cover in this test cycle. Depending on the Test Type (Coverage, in this case), you may have the option to select a variety of bug types that are in-scope for this test as well as a variety of Features. At least one Feature should be in-scope to be able to run a test.

Since we are concerned with “post-release regression,” particularly in terms of international coverage, we checked all the Features that are relevant to our changes and have enabled Functional, Content, and Visual bugs (see “Goal of test” below).

Note: If you want to “Add feature,” you would simply follow the below instructions.

First, select the “Add feature” option at the bottom

Then, fill out the necessary input required to add the new feature.

If we wanted to test the way customers can filter through our products, the name is that of that Feature could be, “Product filtering.”
Describe where on the site the feature can be found.
Define the expected behavior (the correct functionality) of this feature.

Closer Look: Features

For more in-depth information, best practices and tips on creating Features, check out this article.

Step 3

Moving down the page, we continue with the test setup.

First, we need to state the Goal of the test. This section can make the difference between a good and a great test cycle!

Note: This section should be your guide for our testers. Think about the desired results of the test and what information the testers need in order to discover these types of issues. Common examples include the release of a new feature, a sanity test for critical functional issues only, changes to a common workflow, etc.

It’s also common to include a brief description of your company and product so testers have some context about the software they’re testing. In other words, keep in mind that test instructions are the only way for you to communicate your needs and expectations to a crowd of testers who might have never had experience with your site/platform/app before.

“YourCompanyName.com operates in the USA, Canada, France, Germany, and UK. The goal of this test is to ensure that country-specific items are correct on all pages, based on the selected country. These items include, but are not limited to: currency, language, flag images, accepted/rejected area codes, and products.”

Next, we state what/if any functionalities are Out of scope. This section should explicitly state any Features or functionality out of scope for this test. It’s also common to:

State undesired functional severities or bug types in this sections
Inform testers not to take particular steps in the flow (make final purchases, contact support, send out applications etc.)
Share known issues in a given Feature

“This is a live site, do not place any orders. Low Severity bugs are out of scope.”

Then, we state any Additional requirements. This section should be used to provide any additional information the testers should know to achieve the desired results. Examples include:

Test accounts
Test payment information
Any special requirements to the environment access/ bug report format/ device usage/ access settings

“To ensure coverage across all countries, use the Google Sheet below. https://docs.google.com/spreadsheets/...
Only use QA.team emails to login. Please include country location the tester is located in at the end of "Actual Results" section. Please ensure that your pop-up blocker is turned off.”

We have no Attachments for this test cycle and will skip this step.

Step 4

Next, we will need to specify “Where to test” (on which devices) and the “Bug language.”

When selecting “Where to test,” we will be choosing which devices our testers will use to run our test. We need to keep in mind our audience and their devices. Since we’re testing for Web and Mobile Web, we’ll include desktops, smartphones, and tablets.

Note: You can define device scope by editing device type, device model, OS, and/or browsers. Please keep in mind that any “unconventional” or outdated device, OS, and browser combinations may lead to irrelevant device specific bugs.

When choosing the Bug language, please keep in mind that testers get invited based on the report language; therefore, the instructions/Features and bug report languages should be consistent.

Note: Bug report language is defaulted to English.

Step 5

Lastly, we select our desired test date and start time. In this case, we are running a single Coverage test, which is set to begin at 6:00pm Tuesday, 5th March, delivering results by 6:00pm the following day.

Note: Due to our Team Lead review and invitation process, the earliest a test can start is the next full hour +1 (e.g. current time: 9:40am / test time: 11:00am, current time: 5:05pm / test time: 7:00pm). Duration ranges vary based on Test Type, from two hours (Rapid Test) up to 48 hours.

Click Create Test.
This will essentially save the test in "draft." At this point:

•You should review all test information and make any adjustments as needed before you submit the test and tester/Team Leader invitations go out.
•If you need to leave and return to it, can find your test on the "All Tests" page in the same state.

To edit the Goal, Out of Scope, Additional Requirement fields before submitting follow the appropriate steps below:

On the test cycle page, click on Edit at the bottom of the page, make the desired changes, then click Update which will return you to the test cycle draft with updated fields.

To edit the Features before submitting follow the appropriate steps below:

Open the Manage Features page in a new tab
Make the change to the Feature in the new tab
Click Submit
If you also need to make additional edits to the instructions, click Edit > Update.

Once all information has been confirmed you can Submit the test.