Configuration file for Chrome perf and endure tests.

chrome/test/functional/perf_endure.py

 #!/usr/bin/env python
 # Copyright (c) 2012 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 """Performance tests for Chrome Endure (long-running perf tests on Chrome).
 
 This module accepts the following environment variable inputs:
   TEST_LENGTH: The number of seconds in which to run each test.
   PERF_STATS_INTERVAL: The number of seconds to wait in-between each sampling
@@ skipping 584 matching lines @@
 
   _WEBAPP_NAME = 'Gmail'
   _TAB_TITLE_SUBSTRING = 'Gmail'
   _FRAME_XPATH = 'id("canvas_frame")'
 
   def setUp(self):
     ChromeEndureBaseTest.setUp(self)
 
     # Log into a test Google account and open up Gmail.
     self._LoginToGoogleAccount(account_key='test_google_account_gmail')
-    self.NavigateToURL('http://www.gmail.com')
+    self.NavigateToURL(self._GetConfig().get('gmail_url'))
     loaded_tab_title = self.GetActiveTabTitle()
     self.assertTrue(self._TAB_TITLE_SUBSTRING in loaded_tab_title,
                     msg='Loaded tab title does not contain "%s": "%s"' %
                         (self._TAB_TITLE_SUBSTRING, loaded_tab_title))

[fdeng1, 2012/08/16 17:42:49]

I found that if gmail_url is set to 'https://mail.google.com'
The tab loaded_tab_title will not contain "Gmail" and lead to an assertion
exception. But if gmail_url is set to 'www.gmail.com' or
'https://mail.google.com/?chat=0&pre=0', it is fine. I can see after Navigating
to 'gmail_url', the tab title refreshes several times, the word 'Gmail' is
always in the title once the inbox page is loaded. But I guess
'loaded_tab_title' may not point to that tab title?

[dennis_jeffrey, 2012/08/16 18:49:58]

The problem is that line 606 above executes as soon as the call to NavigateToURL
in line 605 returns.  The call to NavigateToURL returns as soon as chrome gets
word that the page load has finished.  Sometimes when redirections happen, the
first page load may complete, then we get that active tab title (which may say
something like "redirecting..."), then we check that title before it actually
completes the redirection and changes to something with "Gmail" in it.

See line 944-949 below for how we can work around this problem with Google Docs.
 Maybe we could add the same workaround here for now, to improve reliability.

[fdeng1, 2012/08/17 06:39:34]

Thanks! Use the same workaround.

 
     self._driver = self.NewWebDriver()
     # Any call to wait.until() will raise an exception if the timeout is hit.
     # TODO(dennisjeffrey): Remove the need for webdriver's wait using the new
     # DOM mutation observer mechanism.
     self._wait = WebDriverWait(self._driver, timeout=60)
 
     # Wait until Gmail's 'canvas_frame' loads and the 'Inbox' link is present.
     # TODO(dennisjeffrey): Check with the Gmail team to see if there's a better
     # way to tell when the webpage is ready for user interaction.
@@ skipping 18 matching lines @@
       return True
     except selenium.common.exceptions.NoSuchFrameException:
       return False
 
   def _GetLatencyDomElement(self):
     """Returns a reference to the latency info element in the Gmail DOM."""
     latency_xpath = (
         '//span[starts-with(text(), "Why was the last action slow?")]')
     self.WaitForDomNode(latency_xpath, frame_xpath=self._FRAME_XPATH)
     return self._GetElement(self._driver.find_element_by_xpath, latency_xpath)
 

[fdeng1, 2012/08/16 17:42:49]

If we rely on checking the exception that is thrown when trying to access a
non-exist element, I think we will need to move the self.WaitForDomNode here
(correct me if I am wrong). If so, a problem is that it may think the latency
element is not available but in fact it just takes long time to load. Will this
situation happen very often?

[dennis_jeffrey, 2012/08/16 18:49:58]

We might be able to avoid this problem in the first place by doing something
simpler.  The default Gmail test account used by this test has latency dom
elements, but user-defined Gmail accounts may not.  When latency dom elements do
exist, they should exist during the entire test, since they are associated with
the Gmail account in use.  In other words, they will either always exist, or
never exist, for a given test run.

How about this idea.  At the start of a Gmail test, let's wait up to 3 or 5
seconds for a latency dom element to appear.  If it appears within that time,
assume that they will always be present and do what the tests currently do.  If
it does *not* appear in that time, assume they will never exist for this test,
and then simply refrain from accessing the latency values during the test run.

Do you think that sounds reasonable?  This way, we don't need to check on every
test iteration whether the latency element exists on the page.

[fdeng1, 2012/08/17 06:39:34]

I think we can do this. I implemented this and set the timeout to 3secs. I
tested with 100 milliseconds and it works with my current bandwidth. I hope 3 is
able to tolerant most of the cases where the element is available.

[dennis_jeffrey, 2012/08/17 17:35:44]

If you think it'll be safer with 5 seconds, we can do that too.  It just means a
slight delay at test startup before the test starts.  But if the test runs for
hours anyhow, an extra 5 seconds shouldn't matter that much.

Also, the only potential problem that might happen is we may not find a latency
DOM element within 5 seconds, even though one will eventually appear.  If that
case ever happens, the only effect is that the test simply doesn't record
latency values during the run.  I think that's ok.

   def _WaitUntilDomElementRemoved(self, dom_element):
     """Waits until the given element is no longer attached to the DOM.
 
     Args:
       dom_element: A selenium.webdriver.remote.WebElement object.
     """
     def _IsElementStale():
       try:
         dom_element.tag_name
       except selenium.common.exceptions.StaleElementReferenceException:
@@ skipping 274 matching lines @@
   """Long-running performance tests for Chrome using Google Docs."""
 
   _WEBAPP_NAME = 'Docs'
   _TAB_TITLE_SUBSTRING = 'Google Drive'
 
   def setUp(self):
     ChromeEndureBaseTest.setUp(self)
 
     # Log into a test Google account and open up Google Docs.
     self._LoginToGoogleAccount()
-    self.NavigateToURL('http://docs.google.com')
+    self.NavigateToURL(self._GetConfig().get('docs_url'))
     self.assertTrue(
         self.WaitUntil(lambda: self._TAB_TITLE_SUBSTRING in
                                self.GetActiveTabTitle(),
                        timeout=60, expect_retval=True, retry_sleep=1),
         msg='Timed out waiting for Docs to load. Tab title is: %s' %
             self.GetActiveTabTitle())
 
     self._driver = self.NewWebDriver()
 
   def testDocsAlternatelyClickLists(self):
@@ skipping 51 matching lines @@
   """Long-running performance tests for Chrome using Google Plus."""
 
   _WEBAPP_NAME = 'Plus'
   _TAB_TITLE_SUBSTRING = 'Google+'
 
   def setUp(self):
     ChromeEndureBaseTest.setUp(self)
 
     # Log into a test Google account and open up Google Plus.
     self._LoginToGoogleAccount()
-    self.NavigateToURL('http://plus.google.com')
+    self.NavigateToURL(self._GetConfig().get('plus_url'))
     loaded_tab_title = self.GetActiveTabTitle()
     self.assertTrue(self._TAB_TITLE_SUBSTRING in loaded_tab_title,
                     msg='Loaded tab title does not contain "%s": "%s"' %
                         (self._TAB_TITLE_SUBSTRING, loaded_tab_title))
 
     self._driver = self.NewWebDriver()
 
   def testPlusAlternatelyClickStreams(self):
     """Alternates between two different streams.
 
@@ skipping 92 matching lines @@
         self._num_errors += 1
 
       time.sleep(1)
 
     self._RunEndureTest(self._WEBAPP_NAME, self._TAB_TITLE_SUBSTRING,
                         test_description, scenario)
 
 
 if __name__ == '__main__':
   pyauto_functional.Main()