Equivalency instrumented vs non-instrumented code?

Mobile Testing, Android App Testing.
merklh
Posts: 24
Joined: Thu Jan 03, 2013 6:55 pm

Equivalency instrumented vs non-instrumented code?

Post by merklh » Tue Sep 20, 2016 9:40 pm

My company works in the medical device field so we have to validate all tools we are using and report to the FDA. I am kind of stuck how we could prove to the FDA that the instrumented app we are testing with Ranorex is equivalent to the non-instrumented app we will ship to the user.

Has anybody else encountered this problem or is there any guidance from Ranorex?

krstcs
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Equivalency instrumented vs non-instrumented code?

Post by krstcs » Wed Sep 21, 2016 2:07 pm

Please provide more information.

What type of application are you testing?
Is it on a mobile device? Web-based? Flash/Flex?
Shortcuts usually aren't...

merklh
Posts: 24
Joined: Thu Jan 03, 2013 6:55 pm

Re: Equivalency instrumented vs non-instrumented code?

Post by merklh » Wed Sep 21, 2016 6:52 pm

krstcs wrote:Please provide more information.

What type of application are you testing?
Is it on a mobile device? Web-based? Flash/Flex?

Sorry, I was really unclear. It's about mobile apps on iOS and Android.

I am trying to come up with a line of reasoning that demonstrates that doing most of the testing with an instrumented app is equivalent to testing the non-instrumented app.

I would guess this is a common problem for medical companies that want to go into mobile.

If we automated with Appium for example we don't have to instrument the app so it's easier to explain.

User avatar
N612
Posts: 135
Joined: Mon Jul 11, 2016 4:01 pm

Re: Equivalency instrumented vs non-instrumented code?

Post by N612 » Wed Sep 21, 2016 8:09 pm

Ranorex instrumentation attaches their own libraries to assist with automation for mobile applications. The behavior of the application should be exactly the same but on a bit level, they are different. Do not release a instrumented application either, as the library is not designed for end-user use, only automation.

krstcs
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Equivalency instrumented vs non-instrumented code?

Post by krstcs » Wed Sep 21, 2016 8:24 pm

In my opinion, it would be exceedingly difficult to justify to an investigator that they are equivalent.

Ranorex does, indeed, change the app's files, even if they don't really change the functionality. Therefore, it would be really difficult, without doing a very extensive binary analysis, to say for certain that injecting Ranorex into the app didn't change something, for good or bad, that makes it different than what is released to your production environment. Even then, it might not be truly possible.

But, that is my opinion, having worked in the US Defense industry for 15 years before starting where I'm at now.
Shortcuts usually aren't...

merklh
Posts: 24
Joined: Thu Jan 03, 2013 6:55 pm

Re: Equivalency instrumented vs non-instrumented code?

Post by merklh » Wed Sep 21, 2016 9:11 pm

krstcs wrote:In my opinion, it would be exceedingly difficult to justify to an investigator that they are equivalent.

Ranorex does, indeed, change the app's files, even if they don't really change the functionality. Therefore, it would be really difficult, without doing a very extensive binary analysis, to say for certain that injecting Ranorex into the app didn't change something, for good or bad, that makes it different than what is released to your production environment. Even then, it might not be truly possible.

But, that is my opinion, having worked in the US Defense industry for 15 years before starting where I'm at now.
Is there a statement from Ranorex on this? It seems they are losing a big target market that way. I don't understand why they even need instrumentation. Appium doesn't for example.

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Equivalency instrumented vs non-instrumented code?

Post by Support Team » Wed Sep 21, 2016 10:23 pm

merklh wrote:I am kind of stuck how we could prove to the FDA that the instrumented app we are testing with Ranorex is equivalent to the non-instrumented app we will ship to the user.
The instrumented and the non-instrumented app are definitely not equivalent, not on a binary nor on a functional level (e.g. Ranorex "inserts" code to get notified of events).

That said, every automated testing tool (instrumenting and non-instrumenting) will have influences on the AUT that a human user would not have (and vice versa). For example, even if Appium does not instrument an app, it uses special APIs to identify controls when searching for the control and to interact with it (touch). So even in this case, other code will be executed in the AUT as if a human user would interact with the AUT.

On the other hand, human testers tend to make systematic (and random) mistakes, too.
And with automation, you may be able to do more tests or increase test coverage.

IMHO it's just a matter of risk management: you either "risk" doing automated testing and possibly influence the AUT or not automate your testing at all, with all consequences. My choice is clear (automate everything!), but I may be biased :wink:
merklh wrote:If we automated with Appium for example we don't have to instrument the app so it's easier to explain.
AFAIK Appium uses the Android MonkeyRunner API under the hood, which Ranorex also uses to automate system apps (see this chapter in the user guide). So, this way could work with Ranorex, too, but without instrumenting your app you cannot use the recorder to record actions.

Regards,
Alex
Ranorex Team

merklh
Posts: 24
Joined: Thu Jan 03, 2013 6:55 pm

Re: Equivalency instrumented vs non-instrumented code?

Post by merklh » Thu Sep 22, 2016 6:34 pm

Support Team wrote: AFAIK Appium uses the Android MonkeyRunner API under the hood, which Ranorex also uses to automate system apps (see this chapter in the user guide). So, this way could work with Ranorex, too, but without instrumenting your app you cannot use the recorder to record actions.
Alex, Can you elaborate on this sentence a little more? We would be fine without recording at all or doing recording with instrumented apps. Could we automate our own app without instrumentation by treating it like a system app? How about iOS?

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Equivalency instrumented vs non-instrumented code?

Post by Support Team » Mon Sep 26, 2016 3:00 pm

Hi merklh,

the monkeyrunner API was initially designed to control an Android device from outside of an Android application. This approach allows you to test/control applications on Android based devices without instrumenting the application itself. Please find more information about the monkeyrunner on the Android Developer side.
Of course, you can also use this approach to test your application without instrumenting. Just follow the steps mentioned in our User Guide. Please also note the points which are mentioned under the Restrictions header.
Thank you.

On iOS, the situation is a bit more complicated since Apple is very restricted and does not allow to interact with the device outside it's sand boxed environment. In this case, our instrumentation ensures a reliable and stable object recognition within the application.

Please let me know if you have any further questions.

Regards,
Markus (S)

Speedboat
Certified Professional
Certified Professional
Posts: 31
Joined: Wed Mar 25, 2015 4:44 pm
Location: Switzerland

Re: Equivalency instrumented vs non-instrumented code?

Post by Speedboat » Wed Nov 16, 2016 3:15 pm

Hi

If you need to further test your mobile application with ranorex I would propose the following steps:

1) Create a list of regression test cases (most critical, mostly used by your customers
2) Write a test plan (Title: Validation of Ranorex and Instrumented Apps) where you compare the not instrumented and instrumented version
3) Perform all test cases manually with the original (not instrumented) version
4) Perform all test cases manually with the instrumented version (deployed by Ranorex)
5) Automate the test cases and execute (with Ranorex)
--> All results have to be identical
7) Write a test report with all the results
8) Conclusion: The behavior of all 3 test environments is the same - testing on the instrumented apps shows equal results

For medical devices you have to validate your test equipment before the results created with this test equipment can used as formal correct product validation results. This above procedure is equal to validate an external temperature sensor for example, where you prof that it delivers correct results.

If your App is FDA rated as 'high level of concern', I would discuss that with FDA representatives before.
Maybe they have additional request to prove equality

Hope this helps
Speedboat