Rerun failed tests via the CLI and CircleCI

Now when a Rainforest run reports failed tests in CI/CD - retest only those failing tests.

The latest version of our CLI, v2.16, adds a rerun command which behaves exactly like the “Rerun Failed Tests” button on the Test Results page in the application.

As with the rest of our CLI this was built with CI in mind. Use the rerun command when retrying CI builds to significantly shorten your time to release.

This functionality is baked into the latest version of our CircleCI orb, v2.0, to automatically rerun failed Rainforest tests when rerunning a failed CircleCI workflow.

Rerunning a single failed test rather than our full release suite

Read more about our CLI and CircleCI integration in our Help Center.

Rainforest Automation Content Matching

We are reducing brittleness by adding additional ways to identify elements beyond the pixel matching we do today.

When you create a new action or recapture an element in a Rainforest Automation, we will automatically analyze the image and use our content matching if possible. Content matching means that as long as the text content stays the same, the test will pass, even if the appearance of the changes.

Screen Recording 2020-07-02 at 05.20 PM.gif

If exact visual appearance is important, you can turn off content matching for that step and we will only look for the element using pixel-based matching. We're continuing to add flexibility and intelligence to reduce brittleness and test maintenance.

Screen Recording 2020-07-06 at 09.07 AM.gif

We've made image capture easier too. No simply click on Screen Capture. No need to click the camera when capturing an image!

For more information see our help center.

VM Notification to Support

As part of our continuing focus on delivering great customer experience, we’ve introduced a program to investigate all the test failures that you categorize as a “System issue” with “Rainforest virtual machines or devices” to see whether the failure was an issue with the Rainforest platform and see what we can do to stop it from happening again.

We’re doing this to reduce the number of false failures and save you more time to concentrate on the important things in life!

You can find out more about categorizing test failures in this document.

New and Enhanced Charts

More metrics and charts to help you monitor and improve quality.

People measure and track Quality differently. Rainforest's new and more robust reporting charts help you slice and dice the data that’s most important to you. You want to track the health of test suites, team engagement, and value over time.

We've improved charts by:

  • adding time filtering - so you can zoom and hone in on the time periods you care about to spot patterns and trends over time
  • adding more metrics that you have wanted. You now have details on average run times, %failures by jobgroup (rather than by run), % failures categorized over time, and browser/platform usage.

We've added new reports so you can further dive into your test suite:

  • Tests created/updated over time - RFA vs. Plain English
  • Average run times - RFA vs. Plain English
  • % Failures - RFA vs. Plain English

We have also updated how we represent failed runs so you have a clearer insight into failed tests. Previously, we provided % failed RUNS (which means that if 1 test in a run of 500 tests fails, the whole run is counted as a failure). One test failing in a group of 500 is much different than 50, 100, or more. Now you can see the details more accurately.

For more detailed information on Rainforest Reporting, see Rainforest Help Center.

Rainforest Automation

We had previously called our no-code automation language "Rainforest Test Language". We renamed it "Rainforest Automation" so it's clear whether you are using our crowd for manual testing or our virtual machines for automation tests.

Rainforest Automation is an easy to use, no-code automation solution that is fully integrated with the Rainforest platform. You may create tests yourself or have our Test Authors create them for you via our Test Writing Services.

Additionally, Rainforest Automation tests may be executed via our automation bots or by the Rainforest Tester Crowd, giving you additional options and flexibility on how you manage your test suite. For more information, please check our help center here.

Test Writing Service

We have renamed Test Designer to reflect what it actually is - a complete test writing service.

Test Writing Service allows you to offload test writing, refactoring, and rewriting tasks to our highly trained Rainforest Test Authors. We step you through a wizard to submit tests. Our Test Authors review your request and create custom tests for you.

Most recently, customers have also used the Test Writing Service to convert your tests from traditional Rainforest Plain English crowd tests to Rainforest Automation tests.

For more information on Test Writing Service see our help documentation here.

Schedule Runs in Local Time Zones

Use local (or any) time zone when scheduling runs!

Scheduling runs using UTC can create a lot of confusion and inevitably leads to issues with runs getting scheduled earlier or later, especially after daylight savings time changeovers. Rainforest wants you to be able to schedule runs without having to think about timezones or DST. If you want to schedule a run for every night at 3:00 am local, you can now just set it and forget it.

Rainforest gives you the ability to schedule runs using your local timezone (or whatever timezone you prefer). Runs will get scheduled exactly when you expect them to be. Anywhere you can edit the schedule for a run group (Run Group’s page, Scheduled Run Group’s page in settings), you will now see a dropdown to select the timezone for that particular schedule.

Timezones can be set as simple GMT offsets or can be location-based (e.g. America/New_York). If you choose a location that observes daylight savings time then it will be taken into account when future runs get scheduled. Since timezones can now be set separately for each schedule it allows you to create one schedule for 1:00 AM in New York and a different schedule for 1:00 AM in Los Angeles without having to do any timezone math.

Make fixing failing actions for RFA tests easier

Streamlined maintenance workflow for RFA tests, minimizing the effort needed to keep tests up-to-date as your application evolves.

Keeping RFA tests up-to-date just got easier: once you have a working test suite, the bulk of your interactions with the tests happen when your product changes and the tests have to be updated correspondingly. This often requires looking at the latest test result to understand what changed and fixing screenshots for selected elements.

Both of these tasks are now streamlined: the result view contains easy-to-apply suggestions for screenshot updates, making common changes a breeze. In addition we’ve also refreshed the appearance of the result view to make it more compact, easier on the eye and consistent with test editing.

Additionally, the test editing view now includes context information about the latest run, giving you the relevant information exactly where you need it - so you can update your tests faster.

Android Phone 10.0 available

Android Phone 10.0 is now available as a Mobile Browser VM. It will show up automatically as an available platform option.

Automatically add new VMs to client accounts

Default VMs will automatically be added to Rainforest accounts. You will no longer need to request new VMs to be added. As new VMs are available, they will also be added automatically.