Step 1 - Test Environment
After selecting a Test Type (i.e. Rapid, Focused, Coverage, Usability, Custom), you must choose the Test Environment that you want the testers to evaluate. In the drop-down you can see any previously tested builds, or you can create a new one by clicking "add new." Name your test environment something easy to remember, like “Beta 3.2” or “Staging.”
Choose the type of environment, either the URL of a website/app or upload an .IPA or .APK file. Be sure to enter credentials to access the environment if needed, including the use of a proxy server.
For additional information on security access requirements to non-public environments, please contact your Customer Success Manager to discuss other solutions, including setup of a VPN tunnel from test IO servers (note: this service is for Professional and Elite customer only).
Step 2 - Features & User stories
Pick the Features and User stories you want covered in this test cycle. Depending on the Test Type, you may have the option of selecting a variety of bug types in scope for this test as well as a varying amount of features.
At least one Feature should be in scope to be able to run a test. For Rapid and Focused tests there is a limit of four Features per test cycle.
You can only edit Features on the Manage Features page. In other words, you can’t edit the Features on any test wizard page. The best practice would be to edit Features on the Manage Features page before creating a new test cycle.
If you need to make a change to a Feature while in the test wizard, follow the appropriate steps below:
There are three possible pages in the test setup wizard. Each can be identified by the call to action button in the bottom left of the wizard. (left column in the table)
Step 3 - Instructions
Important! This section can make the difference between a good and a great test!
Goal of this test
This section should be the guiding star for testers. Think about the desired results of the test and what information the testers need in order to discover these types of issues. Common examples include release of a new feature, a sanity test for “critical functional issues only,” and changes to a common workflow.
It’s also common to include a brief description of your company and product so the tester has some context of the software they’re testing. In other words, keep in mind that test instructions is the only way for you to communicate your needs and expectations to a crowd of testers who might have never had experience with your site/platform/app before.
Out of scope
This section should explicitly state any Features or functionality out of scope for this test. It’s also common to:
State undesired functional severities or bug types in this section
Inform the tester not to take particular steps in the flow (make final purchases, contact support, send out applications, etc.)
Share known issues in a given feature
This section should be used to provide any additional information the tester should know to achieve the desired results. Examples include:
Test payment information
Any special requirements to the environment access/bug report format/device usage/access settings
Consult with your CSM for applicable scenarios. A common use case is the provision of screenshots for the testers as a reference for testing, be it a module, newsletter, or page layout.
Any test guidelines, access settings, user stories, additional details, or payment data can be attached to the instructions for testers to download.
Step 4 - Devices & Language
You can define device scope by editing device type, device model, OS and/or browsers. Customers often leverage the crowd for device coverage alone.
Please keep in mind that any “unconventional” or outdated device, OS, and browser combinations may lead to irrelevant device-specific bugs.
Bug Report Language
Bug report language is always defaulted to English.
When choosing the bug language, please keep in mind that testers get invited based on the report language; therefore, the instructions/Features and bug report languages should be consistent.
Step 5 - Launch
Test Launch Date and Duration
Select your desired test date and start time. Note that due to our Team Lead review and invitation process, the earliest a test can start is the next full hour +1 (e.g. current time: 9:40am / test time: 11:00am, current time: 5:05pm / test time: 7:00pm). Duration ranges vary based on Test Type, from two hours (Rapid Test) up to 48 hours.
The final step in the process is to review all test information and make
any adjustments as needed. Once all information has been confirmed you
can Submit the Test.
Editing the Test Cycle
To provide an opportunity for Team Leader review, the option to edit the test it is not available once the test has been submitted. If you notice that there is a mistake in the test setup, please reach out to your CSM, but please keep in mind that the test environment and instructions can only be edited by a CSM within one hour before the test starts. After the test has started, no further changes can be made. If you feel the test will not yield appropriate results based on the current criteria, please contact your CSM to cancel the test and work with you on resubmission.
If the test cannot be accepted for any reason (problem with the test environment, inconsistent instructions, etc), you will get notified via email. Please correct or provide clarity based on the test rejection reason included in the email before re-submitting the test.
For more in-depth explanations of each step, please refer to the next few articles, beginning here.