Memory Leak in Target Silverlight App via AutomatonPeer objs

[markgarner]

Hello,

We're seeing memory leaks in our Silverlight OOB application whilst running tests using Ranorex 5.1.2.19431 . Using memory profilers we're seeing that the managed object table is holding references to the AutomationPeer objects (which are then referencing our controls - causing a leak). It appears there was a memory leak linked to Silverlight a couple of years ago: /forum/flex-memory-leak-issue-t3097.html [can't post full urls - this link is for the ranorex site]. The memory leaks aren’t present when Ranorex isn’t attached, as the AutomationPeer objects used are never accessed.

We've replicated using Silverlight 5.1.30214.0 as an out of browser application (with elevated permissions) on windows server 2008. To check it wasn't our code - we put together a small PoC to check, and replicated. Also, we can see a similar leak in same test app ported to WPF. Note in the Silverlight app - I've tried seeing if I can hint to release the AutomationPeers on the repeatedly instantiating control.

We wanted to use Ranorex to assist in ensuring our application stays within memory bounds - but it seems that this isn't feasible if the UI automation framework resources aren't being released.

(Also, I've tried this with the repository enabled app folder caching setting as true and false.)

Many thanks,
Mark

[Support Team]

There is a known issue with the base implementation of UIAutomation (UIA), which is the native accessibility framework that Ranorex uses to recognize WPF/Silverlight applications, that AutomationPeer instances stay in memory for three minutes after use:
http://stackoverflow.com/questions/1883 ... 5#19039905

So, it is not a memory leak, but kind of a laggy memory release. Sadly those 3 minutes are hard-coded into the .NET WPF framework source code (not Ranorex) and you cannot change that value from the outside.

Have you tried waiting for a few minutes before doing your memory comparison?
I know it's a workaround, but currently Ranorex depends on the UIA interface to automate WPF/Silverlight applications.

Regards,
Alex
Ranorex Team

[markgarner]

Hi Alex,

Unfortunately, we do still see the retained objects after 3 minutes (we've given it over 10 minutes, to be on the safe side), both with the Ranorex designer open and after closing.

Are you seeing the memory being released after 3 minutes on the test solution uploaded?

If it helps, we're running things as 32bit, on 64bit machines.

I'm happy to run a test build of ranorex with additional logging and such, if that will help diagnose the issue.

Many thanks,
Mark

[Support Team]

Are you seeing the memory being released after 3 minutes on the test solution uploaded?

I see the memory being released for the objects being described in the link I posted, all residing in the System.Threading namespace.

The AutomationPeer instances are first released when the control using it goes out of scope. I guess the problem is the cache you use internally in your sample application. Because of that cache, all of your custom ContentItem controls stay in memory, including the associated AutomationPeer elements.
When I removed the cache, I only saw the AutomationPeer elements for the currently displayed controls being in memory.

In general, this is not a Ranorex only issue; you would see the same memory behavior when you interact with the application using a screen reader or any other accessibility tool.

Regards,
Alex
Ranorex Team

[griffithsm]

Hi Alex,

Please will you attach your solution with the cache modifications? I've also removed the caching and the custom automation peer logic (not sure why that was included), but am still seeing unbounded memory growth running a Ranorex script that simply selects each item in the listbox. I also noticed that the memory usage does not decrease noticeably after setting the main page's content to null (i.e. forcing no controls to be in scope anymore) even after waiting for several minutes. All automation peers remain rooted.

Please find the associated Ranorex project and results from memory profiling attached. Unfortunately it seems like the full export from dotMemory is too large to attach, but I have it saved so let me know if you require any further information regarding it.

Note that this is using Silverlight 5.1.30214 on a 64-bit Windows 7 virtual machine.

Thanks,

Michael

[griffithsm]

Additionally, please find attached an example retention path for a control in my test case. This is the retention path of all 65 DataGrids that were created during the course of the test.

Thanks,

Michael

[Support Team]

Please will you attach your solution with the cache modifications?

All I did was commenting out the "_cache" member in the MainWindow class and remove all usages of that member:

private void uiListBox_SelectionChanged(object sender, SelectionChangedEventArgs e)
{
    while (uiPanel.Children.Count > 0)
    {
        uiPanel.Children.RemoveAt(0);
    }
    foreach (var item in e.AddedItems)
    {
        var name = item as string;
        uiPanel.Children.Add(new ContentItem(name));
    }
    GC.Collect();
}

I did not profile the Silverlight application, though, but the WPF application. (I currently don't have a Silverlight SDK on my machine.)

I then ran your Ranorex project by double clicking the EXE file (after correcting the repo item paths), without having Ranorex Studio, Spy, or any other Ranorex tool open and having the memory profiler running. After the run finished and the executable exited, I let the app sit for 3 minutes, then I took a memory snapshot with the profiler. Only a single DataGrid object was in memory then.

As I said before, for WPF/Silverlight applications Ranorex just uses UIAutomation which is a common WPF accessibility interface. It might take some time to get the GC step in and release all memory, but eventually it should happen, especially when the Ranorex executable exits and all proxies to the AutomationPeer instances are gone.

Regards,
Alex
Ranorex Team

[griffithsm]

Hi Alex,

Sorry to be a pain, but please will you confirm you do not see any leaks with the SL version (with the caching removed)?

Our production application is a SL application - I'm not sure why Mark attached a WPF example as well. If you do not see any leaks with the SL application, then we'll have to figure out what's different between our environments. I'm still seeing controls in memory after waiting for an hour and forcing GC in the memory profiler.

Thanks,

Michael

[Support Team]

please will you confirm you do not see any leaks with the SL version (with the caching removed)?

I installed the Silverlight SDK, commented out the "_cache" in your sample and ran the profiler on it.
And I can confirm that I could see no memory leak, only objects for the currently visible controls. The results were even better than with the WPF application, since obviously the 3 minute delay for the classes in the Threading namespace does not apply for Silverlight.

What I did:
I started the profiler with your Silverlight app and clicked on a few listbox entries to get the startup memory out of the equation. I took a memory snapshot.
Then I ran your Ranorex automation executable against the Silverlight app and again took a snapshot.
The second snapshot only contained a single DataGrid instance and a few AutomationPeer instances, but only for the controls currently visible in the application.
Then I ran the automation executable again and took a third snapshot.
When comparing the second and third snapshot, there were no differences in the number of instances currently in memory. The process memory for the Silverlight app stayed constant as well.

So what my memory profiler tells me, is that there is no memory leak

Sorry to be a pain

You are not a pain and I'm sorry if my last post made you think your are one
I'm not sure why your memory profiler delivers different results. It's just that it would be very strange to have memory leaks in a standard .NET component like UIAutomation.

Regards,
Alex
Ranorex Team

[griffithsm]

Hi Alex,

I've done some further testing and have managed to rule out the profiler I'm using - even running the application without profiling, running the automation executable causes the memory used by sllauncher.exe to jump by ~300MB each time (using the Private Working Set in Task Manager as a rough guide).

This has been replicated on machines with the following two configurations - which versions do you have for the below? This has to be down to something odd about our environment. In both cases, I followed the same steps as you - clicked on each listbox entry once, then ran the automation executable directly.

* Virtual machine
* Windows 7 Enterprise SP1 (64-bit)
* Silverlight 5.1.30214.0
* Silverlight SDK 5.0.6.1118.0
* Ranorex 5.1.2.19431

* Physical machine
* Windows Server 2008 R2 Standard (64-bit)
* Silverlight 5.1.30514.0
* No Silverlight SDK
* Ranorex 5.1.2.19431

Is there any further information that will help?

Thanks,

Michael

[Support Team]

Hi Michael,

...running the automation executable causes the memory used by sllauncher.exe to jump by ~300MB each time (using the Private Working Set in Task Manager as a rough guide).

I don't think this is a clear indication of a memory leak caused by automation, nor an indication of a general memory problem. I just tried to click quickly on the automation items manually and got your Silverlight application to 300+MB in just a few seconds (although I don't think it has a memory leak). However, the memory is released again afterwards - it just might take some time to get the GC jump in.

I can reproduce this behavior both by manually clicking through the items and by executing your automation executable (running 100 iterations) on two machines. Private memory may go up if you click quickly through the items, resulting in the processor to reach 100% (for multi-core processors, only one core will be fully consumed). If there's memory pressure (e.g. physical memory runs low) or the application has more processing time, the GC jumps in and cleans up, resulting in the private memory to go down again.

Reproduced on:
* Windows 7 SP1 & Windows 8.1
* Silverlight Version: 5.1.30514.0
* Both with Silverlight SDK and without
* Ranorex 5.2.0

Sorry, I don't see a memory leak there, just normal managed memory behavior with the GC cleaning up whenever it has time or pressure to do so.

Regards,
Alex
Ranorex Team

[griffithsm]

Hi Alex,

I don't doubt that you're not seeing a memory leak - however, I'm very certain that we are.

Just to note, I upgraded my Ranorex version to 5.2.0 to bring my environment in line with yours.

* After waiting for upwards of an hour, no amount of memory is freed
* In an environment with memory pressure (90%+ memory used in Task Manager), no amount of memory is freed
* If I run the automation executable several times in a row, the memory continues to grow unboundedly
** Once the process hits approximately 1GiB private working set, it becomes very slow to respond
** Once the process hits approximately 2GiB private working set, it crashes - debugging the crash in Visual Studio results in the (fairly unhelpful) error:

Code: Select all

Unhandled exception at 0x7673C42D (KernelBase.dll) in sllauncher.exe: 0xE0434352 (parameters: 0x8007000E, 0x00000000, 0x00000000, 0x00000000, 0x78790000).

Is there anything else we're missing? Any other potential differences in our environments that could be causing this? We're at a total loss as to what to do.

Thanks again for your help so far,

Michael

[Support Team]

** Once the process hits approximately 2GiB private working set, it crashes

OK, now that's what I call a clear indication of a memory leak

I can see your point now - sorry for all the back and forth! Frankly, to me it was almost impossible that there is a memory leak, since I did exactly what you did on two different machines and did not experience any problems. Sorry for not opening my eyes sooner

Is there anything else we're missing? Any other potential differences in our environments that could be causing this?

That's the question now... The good thing is that there almost has to be a solution, since I'm not experiencing this problem - and it's all standard .NET functionality Ranorex is using, so if there was a fundamental problem, a lot of people would have issues.

When you debug your application, you see that AutomationPeer objects are staying behind?
And you do not see those objects if you manually perform the same steps you do automated?
Is there any other software running that could also use UIA and cause the object to not be released? E.g. screen narrator, accessibility or automation software, anti-virus program, ...? (This blog points out that the Tablet PC Input Services can also cause AutomationPeer objects not to be cleaned up.)
You may want to start your OS in protected mode and try to reproduce the leak again to rule out influence of other software...
In the original post, Mark mentioned you start the Silverlight app "with elevated permissions". Have you tried running it with default permissions? Is Ranorex running with the same rights?

Sorry, for not being of too much help. Right now, I can only guess what the problem could be...

Regards,
Alex
Ranorex Team

[tringhamj]

Hi Alex,

We've just had a development - creating a fake AutomationPeer in the MainPage codebehind in order to shield the others seems to prevent the memory leak in both WPF and Silverlight, while still allowing Ranorex to record:

For WPF, in MainPage.xaml.cs:

Code: Select all

class _FakeWindowsPeer : WindowAutomationPeer
        {
            public _FakeWindowsPeer(Window window)
                : base(window)
            { }

            protected override List<AutomationPeer> GetChildrenCore()
            {
                return null;
            }
    
        }
        protected override AutomationPeer OnCreateAutomationPeer()
        {
            return new _FakeWindowsPeer(this);
        }

(WindowAutomationPeer doesn't exist in Silverlight, but replacing it with FrameworkElementAutomationPeer seems to do the trick)

While this stops the program leaking memory, recording and playing back in Ranorex works seemingly fine. However, Ranorex now doesn't know what object is being interacted with in the recordings, as this causes it to think the main window has no automatable children [see attachment]. The fact that Ranorex records the relative mouse position and keyboard inputs means that it is able to reproduce what was recorded, but this method isn't exactly ideal.

Obviously this is just a work around, but I'd just thought we should keep you updated in case you have any thoughts or suggestions, or whether it helps you narrow down what is causing the problem.

Thanks,
John

[Support Team]

Hi everyone,

However, Ranorex now doesn't know what object is being interacted with in the recordings, as this causes it to think the main window has no automatable children

Well, so at least we are pretty sure that the memory leak is not caused by the MainForm UIA element, but by it's children. In terms of automation robustness, the change you made won't be acceptable, though.

* When you debug your application, you see that AutomationPeer objects are staying behind?
* And you do not see those objects if you manually perform the same steps you do automated?
* Is there any other software running that could also use UIA and cause the object to not be released? E.g. screen narrator, accessibility or automation software, anti-virus program, ...? (This blog points out that the Tablet PC Input Services can also cause AutomationPeer objects not to be cleaned up.)
* You may want to start your OS in protected mode and try to reproduce the leak again to rule out influence of other software...
* In the original post, Mark mentioned you start the Silverlight app "with elevated permissions". Have you tried running it with default permissions? Is Ranorex running with the same rights?

Did you go through the questions from my previous post? As we cannot reproduce the memory leak on our side, we assumed that the environment could be a problem.

Regards,
Alex
Ranorex Team