Click here to Skip to main content
15,867,895 members
Articles / Programming Languages / C#

Speed up Selenium Tests through RAM Facts and Myths

Rate me:
Please Sign up or sign in to vote.
4.82/5 (12 votes)
11 Aug 2015Ms-PL8 min read 18.3K   9   1
Research how to speed up Selenium tests through putting browsers files on RAM Drive. Benchmarks and time comparison between all major browsers.

Introduction

I have been always curious if I could speed up my UI tests if the browser’s files (cache, profiles) are stored in RAM. When I was starting to test my idea, I was planning to put some majestic headline like “Speed up Your WebDriver Tests 120% Through RAM”. However, after a full day of hard research work, I found out that I was going to lie to you if I put a headline like this. For my benchmarks, I used all the major browsers- Chrome, Firefox, Internet Explorer and the new Edge. The tests were conducted against Hard Drive, SSD Drive, and RAM Drive. Find below all facts that break down the myth that you could make your browser fly if you put its files on an RAM drive.

Image 1

Prerequisites for the Research

1. Download and install an RAM Drive Software (for my tests I used a trial version of GiliSoft RAMDisk)

2. Create a virtual RAM Drive (at least 1 GB)

Image 2

Keep in mind that the program will take a real RAM of your machine so be sure that you have enough before proceeding with the experiments.

Benchmark Selenium Tests Execution Time- Browser’s Files Stored on Hard Drive

All quality performance tests should contain a baseline against the rest of the results are going to be compared. The baseline for this research will be the selenium tests execution time against browsers that store their files on the Hard Drive.

The following code is going to be used to measure the selenium tests speed against different browser configurations.

C#
[TestClass]
public class RamDiscTests
{
    private IWebDriver driver;

    [TestInitialize]
    public void SetupTest()
    {
        this.driver = new FirefoxDriver();
        this.driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(30));
    }

    [TestCleanup]
    public void TeardownTest()
    {
        this.driver.Quit();
    }

    [TestMethod]
    public void NavigateToUrlsTest()
    {
        this.Profile("Chrome-HardDrive", 10,
            () =>
            {
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878773/Implement-Copy-Paste-Csharp-Code");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878205/Read-Write-Windows-Registry-Csharp-VB-NET-Reusable");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/997954/SSRS-SQL-Server-Reporting-Services-Subscriptions-f");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/877859/Generic-Properties-Validator-Csharp-VB-NET-Code");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/886894/Reduced-AutoMapper-Auto-Map-Objects-Faster");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878910/Change-config-File-at-Runtime-Csharp-VB-NET-Code");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/877860/Assert-DateTime-the-Right-Way-MSTest-NUnit-Csharp");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878584/Manage-TFS-Test-Cases-in-MS-Test-Manager-Csharp-VB");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878774/Manage-TFS-Test-Suites-in-MS-Test-Manager-Csharp-V");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878583/Associate-Automated-Test-with-TFS-Test-Case-Csharp");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878912/Connect-to-TFS-Team-Project-Csharp-VB-NET-Code");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/design-techniques-for-enhancing-unit-tests");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/observer-design-pattern-design");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/types-code-coverage-examples-c");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/reduced-automapper-auto-map");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/advanced-page-object-pattern-0");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/design-patterns-in-automation-testing");
                this.WaitUntilLoaded();
                this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/using-selenium-webdriver-tor-c");
                this.WaitUntilLoaded();
            });
    }

    private void WaitUntilLoaded()
    {
        WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
        wait.Until((x) =>
        {
            return ((IJavaScriptExecutor)this.driver).ExecuteScript("return document.readyState").Equals("complete");
        });
    }

    private void Profile(string testResultsFileName, int iterations, Action actionToProfile)
    {
        string desktopPath = Environment.GetFolderPath(Environment.SpecialFolder.Desktop);
        string resultsPath = Path.Combine(desktopPath, "BenchmarkTestResults", string.Concat(testResultsFileName, "_", Guid.NewGuid().ToString(), ".txt"));
        StreamWriter writer = new StreamWriter(resultsPath);

        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();

        var watch = new Stopwatch();
        watch.Start();
        for (int i = 0; i < iterations; i++)
        {
            actionToProfile();
        }
        watch.Stop();
        writer.WriteLine("Total: {0:0.00} ms ({1:N0} ticks) (over {2:N0} iterations)",
            watch.ElapsedMilliseconds, watch.ElapsedTicks, iterations);
        var avgElapsedMillisecondsPerRun = watch.ElapsedMilliseconds / iterations;
        var avgElapsedTicksPerRun = watch.ElapsedMilliseconds / iterations;
        writer.WriteLine("AVG: {0:0.00} ms ({1:N0} ticks) (over {2:N0} iterations)",
            avgElapsedMillisecondsPerRun, avgElapsedTicksPerRun, iterations);
        writer.Flush();
        writer.Close();
    }
}

The test itself is straightforward- it only navigates to several URLs. In order WebDriver to wait for a full page load, the code uses the WaitUntilLoaded method. There the page’s document state is waited to be fully loaded though the WebDriver’s IJavaScriptExecuter.

Another fascinating piece of code that is used is the Profile method. First the garbage collector is called to clean up. After that, a stopwatch starts measuring the selenium tests execution time. The specified test is executed a predefined number of times (10). Finally, the results are stored in a file.

Firefox Driver Using Hard Drive

The initial version of the code is using FirefoxDriver. By the way by using the hard drive, I meant that the Firefox is storing all necessary files on the hard disk like cache files and user profiles.

The default folder for Windows is %APPDATA%\Mozilla\Firefox\Profiles\

Selenium Tests Execution Time- 56.112 seconds

Chrome Driver Using Hard Drive

Default Chrome user specific files location- C:\Users\%USERNAME%\AppData\Local\Google\Chrome\User Data\Default

To use the ChromeDriver, the TestInitialize should be changed a little bit. Also, you need to specify the location of the ChromeDriver.exe in the constructor.

[TestInitialize]
public void SetupTest()
{
    this.driver = new ChromeDriver(@"D:\Projects\PatternsInAutomation.Tests\WebDriver.Series.Tests\Drivers");
    this.driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(30));
}

Selenium Tests Execution Time- 29.174 seconds

Internet Explorer Driver Using Hard Drive

Default Internet Explorer cache locations:

Windows Vista and 7- %userprofile%\AppData\Local\Microsoft\Windows\Temporary Internet Files

Windows 8- %userprofile%\AppData\Local\Microsoft\Windows\INetCache

To use the InternetExplorerDriver, the TestInitialize should be changed. Also, you need to specify the location of the InternetExplorerDriver.exe in the constructor.

[TestInitialize]
public void SetupTest()
{
    var options = new InternetExplorerOptions();
    options.EnsureCleanSession = true;
    options.IgnoreZoomLevel = true;
    options.EnableNativeEvents = true;
    options.PageLoadStrategy = InternetExplorerPageLoadStrategy.Eager;
    this.driver = new InternetExplorerDriver(@"D:\Projects\PatternsInAutomation.Tests\WebDriver.Series.Tests\Drivers", options);
    this.driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(30));
}

Additionally, the test should be modified to be able to pass. Through the usage of InternetExplorerOptions, the InternetExplorerDriver is capable of waiting the pages to be fully loaded. This means that we don’t need the WaitUntilLoaded method anymore.

C#
[TestMethod]
public void NavigateToUrlsTestIE()
{
    this.Profile("IE-SSD-Drive", 10,
        () =>
        {
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878773/Implement-Copy-Paste-Csharp-Code");
            this.driver.Url = @"http://www.codeproject.com/Articles/878205/Read-Write-Windows-Registry-Csharp-VB-NET-Reusable";
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/997954/SSRS-SQL-Server-Reporting-Services-Subscriptions-f");
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/877859/Generic-Properties-Validator-Csharp-VB-NET-Code");
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/886894/Reduced-AutoMapper-Auto-Map-Objects-Faster");
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878910/Change-config-File-at-Runtime-Csharp-VB-NET-Code");
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/877860/Assert-DateTime-the-Right-Way-MSTest-NUnit-Csharp");
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878584/Manage-TFS-Test-Cases-in-MS-Test-Manager-Csharp-VB");
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878774/Manage-TFS-Test-Suites-in-MS-Test-Manager-Csharp-V");
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878583/Associate-Automated-Test-with-TFS-Test-Case-Csharp");
            this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878912/Connect-to-TFS-Team-Project-Csharp-VB-NET-Code");
            this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/design-techniques-for-enhancing-unit-tests");
            this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/observer-design-pattern-design");
            this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/types-code-coverage-examples-c");
            this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/reduced-automapper-auto-map");
            this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/advanced-page-object-pattern-0");
            this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/design-patterns-in-automation-testing");
            this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/using-selenium-webdriver-tor-c");
        });
}

Selenium Tests Execution Time- 36.324 seconds

Edge Driver Using Hard Drive

To use the InternetExplorerDriver, the TestInitialize should be changed. Furthermore, you can find out how to fully setup the Edge Driver in my article- Microsoft Edge WebDriver- What Everybody Ought to Know About.

C#
[TestInitialize]
public void SetupTest()
{
    if (System.Environment.Is64BitOperatingSystem)
    {
        serverPath = Path.Combine(System.Environment.ExpandEnvironmentVariables("%ProgramFiles(x86)%"), serverPath);
    }
    else
    {
        serverPath = Path.Combine(System.Environment.ExpandEnvironmentVariables("%ProgramFiles%"), serverPath);
    }
    EdgeOptions options = new EdgeOptions();
    options.PageLoadStrategy = EdgePageLoadStrategy.Eager;
    this.driver = new EdgeDriver(serverPath, options);
    this.driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(5));
}

Selenium Tests Execution Time- 14.197 seconds

Benchmark Selenium Tests Execution Time- Browser’s Files Stored on RAM Drive

The same code base is used for these series of tests. However, there is initial setup for every browser in order to move its files to the RAM Drive.

Image 3

Firefox Driver Using RAM Drive (Cache)

The first test that I performed was with Firefox, which cache files were moved to the RAM Drive.

Steps to Move Firefox Cache to RAM Drive

1. Type about:config into the location bar and press enter.

2. Accept the warning message that appears, you will be taken to a list of preferences.

Image 4

3. Right-click somewhere in the list and select “New > String”

Image 5

4. For the name of the preference type browser.cache.disk.parent_directory

5. For its value type the path to where you want to store the cache (path to the RAM Drive)

6. Next locate the preference browser.cache.disk.enable, it must be set to true, if it is not, double-click on it to change its value

Image 6

Selenium Tests Execution Time- 54.713 seconds

Steps to Move Firefox Profiles to RAM Drive

The next phase of the speed up process is to move the Firefox profiles files to the RAM Drive.

1. If Firefox is open, close Firefox.

2. Press Windows +R on the keyboard. A Run dialog will open.

3. In the Run dialog box, type in firefox.exe -P

Note: You can use -P or -ProfileManager (either one should work).

Image 7

4. Click OK.

5. Create a new profile and sets its location to the RAM Drive

Image 8

Selenium Tests Execution Time- 56.856 seconds

Move Firefox Installation, Cache and Profiles to RAM Drive

1. Uninstall Firefox

2. Download it again

3. Choose installation location RAM Drive

Image 9

With this setup, there was a real improvement of almost 7%. It was the best that I achieved through the usage of the RAM Drive from all tested browsers.

Selenium Tests Execution Time- 48.970 seconds

Chrome Driver Using RAM Drive (Cache)

To set up Chrome to load its cache from the RAM Drive, follow the next steps.

1.  Press Windows+R on the keyboard. A Run dialog will open.

2. Type regedit

3. Locate the Registry key – HKEY_CLASSES_ROOT\ChromeHTML\shell\open\command

You should find a path to the Chrome executable there. All you need to do is to append the cache location and size to the path so that Chrome uses the right caching information when links are clicked, and Chrome is not open at that time.

4. Add –disk-cache-dir=”D:\Browsers-Caches\Chrome-Cache” after chrome.exe, so that it looks like the following:

C:\Users\aangelov\AppData\Local\Google\Chrome\Application\chrome.exe” –disk-cache-dir=”D:\Browsers-Caches\Chrome-Cache”  — “%1

Image 10

The next time when you start Chrome it will load from your RAM Disk.

The execution of the tests against Chrome was the fastest when the cache is stored on the RAM. However, the improvement was only 1%.

Selenium Tests Execution Time- 28.077 seconds

Internet Explorer Driver Using RAM Drive (Cache)

Internet Explorer offers the easiest and most convenient way to change the default cache folder location.

1. Click Tools – Internet Options – under ‘Browsing History’ section

2. Choose ‘Settings’

3. Click ‘Move folder’ and provide the new location, e.g. Z:\IECache

After providing the new temp folder and clicking OK to save the new settings, Internet Explorer forcefully closes the current user session and logs the user off.

Image 11

Instead of speeding up the tests, they were executed even slower compared to the execution when the cached resources were present on the hard drive.

Selenium Tests Execution Time- 36.511 seconds

Benchmark Selenium Tests Execution Time- Browser’s Files Stored on SSD Drive

The setup for the tests against SSD drive is identical to the one against RAM Drive. So I will just share the test execution results.

Image 12

Firefox Driver Using SSD Drive (Cache, Profiles, Installation)

Selenium Tests Execution Time- 56.579 seconds

Chrome Driver Using SSD Drive

Selenium Tests Execution Time- 32.273 seconds

Internet Explorer Driver Using SSD Drive

Selenium Tests Execution Time- 33.287 seconds

Test Results Analysis

Below you can find a table with all execution times from the conducted experiments. The average execution time of all browsers is ~41 seconds, so the second column shows the improvement against it. The third column presents the advance of each test against the baseline for every browser that was the test execution on a hard drive.

Image 13

I think from the above results we can conclude that it is a myth that you can speed up your browser if you move some of its files to an RAM Drive. The same is valid for the SSD drives. As you can see from the picture, there was a real improvement with Chrome and Firefox. However, it is so small that I think it is safe to say that it doesn’t worth the time and resources to deal with it.

So Far in the 'Pragmatic Automation with WebDriver' Series

License

This article, along with any associated source code and files, is licensed under The Microsoft Public License (Ms-PL)


Written By
CEO Automate The Planet
Bulgaria Bulgaria
CTO and Co-founder of Automate The Planet Ltd, inventor of BELLATRIX Test Automation Framework, author of "Design Patterns for High-Quality Automated Tests: High-Quality Test Attributes and Best Practices" in C# and Java. Nowadays, he leads a team of passionate engineers helping companies succeed with their test automation. Additionally, he consults companies and leads automated testing trainings, writes books, and gives conference talks. You can find him on LinkedIn every day.

Comments and Discussions

 
QuestionYour article Pin
Anda Cristea11-Nov-15 1:19
Anda Cristea11-Nov-15 1:19 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.