Ranorex Forum

I would like to share with the Ranorex community a recent challenge I encountered (and the solution I developed) to implement Fail Fast in my test automation framework.

My organization uses Azure DevOps for all things CI/CD. In my release definition, I leverage the Visual Studio Test task to execute my Test Suite. My Test Cases contain Associated Automation, pointing to my CodedUI solution, which executes my Ranorex tests.

• More simply, Azure DevOps > VsTest > Test Suite > Test Case > CodedUI > Ranorex

My Sanity Test Suite (which runs on each release) is a full end-to-end flow. It currently takes about 40 minutes to complete execution. Tests must pass @ 100% for the deployment to proceed to the QA environments.

My goal for Fail Fast was simple: If an automated test failure occurs, I want to know about it right away, so I can triage and fix the issue.

The first thing I tried was to set the errorbehavior="Stop" for all of my Ranorex Test Cases and Smart Folders. This tells the Ranorex Test Suite to stop on error, and it works perfectly when run from Ranorex. However, when run from CodedUI, the test cases are invoked individually, with no insight into a previous test's results. This approach was partially successful, and an individual test does stop as soon as a failure is detected, but the remaining tests in the suite are still executed.

I then looked into the VsTest task which executes my Test Suite in Azure DevOps. It does not contain the ability to stop on test failure.

I also looked into the Azure DevOps Test Suite, itself, but it does not contain any configuration properties to stop on failure.

After taking some time to think about this, I architected a solution to drop a "failFast.txt" file in a known location on the target machine if a test failure occurs. All tests will look for the presence of this file to determine if they should run or terminate.

Here is the code snippet:

using System;
using System.IO;
using Microsoft.VisualStudio.TestTools.UITesting;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Ranorex;
using Ranorex.Core.Testing;

namespace RanorexCodedUITests
{
    [CodedUITest]

    public class RanorexSmokeTests
    {

        public RanorexSmokeTests()
        {
        }

        public void StartRanorexTestCase(string tc)
        {
            Ranorex.Keyboard.AbortKey = System.Windows.Forms.Keys.Pause;
            int error = 0;
            string failFast = "C:\\Temp\\Test\\failFast.txt";
            string failedTest = "";

            // Cleanup old failFast file
            if (tc == "SmokeTest1" && File.Exists(failFast))
            {
                File.Delete(failFast);
            }

            // If a test failure has not already occurred, run test
            if (!File.Exists(failFast))
            {
                String param = string.Format(@"/zr /zrf:report.rxzlog /tc:" + tc);
                error = TestSuiteRunner.Run(typeof(RanorexTest.Program), param);
                Delay.Seconds(5);
                TestContext.AddResultFile("report.rxzlog");
            }
            else
            {
                failedTest = File.ReadAllText(failFast);
                throw new RanorexException("This test was skipped because " + failedTest + " failed!");
            }

            // If a test failure occurs, create failFast file, record failing test name, and throw exception
            if (error != 0)
            {
                File.WriteAllText(failFast, tc);
                throw new RanorexException(tc + " failed! See report.rxzlog for further details!");
            }
        }

        #region Smoke Tests
        // End to End Smoke Tests
        [TestMethod]
        public void SmokeTest1()
        {
            StartRanorexTestCase("SmokeTest1");
        }

        [TestMethod]
        public void SmokeTest2()
        {
            StartRanorexTestCase("SmokeTest2");
        }

        [TestMethod]
        public void SmokeTest3()
        {
            StartRanorexTestCase("SmokeTest3");
        }
        #endregion

        /// <summary>
        ///Gets or sets the test context which provides
        ///information about and functionality for the current test run.
        ///</summary>
        public TestContext TestContext
        {
            get
            {
                return testContextInstance;
            }
            set
            {
                testContextInstance = value;
            }
        }
        private TestContext testContextInstance;
    }
}

The failing test will contain a detailed Ranorex exception and the expected attachments (Ranorex report, log, screenshot).

Downstream skipped tests will contain a Ranorex exception which calls out the failing test.

With the Fail Fast enhancement, if any failure is detected, test execution will stop right away. Instead of waiting 40 minutes to learn of an issue, I will now be notified immediately. This has resulted in faster feedback for my team and the entire development organization.

Please let me know your thoughts on this implementation, if it was helpful for you, and if you have any other ideas. Thanks!