Adding pyauto-based memory usage tests for ChromeOS.

chrome/test/functional/perf.py

 #!/usr/bin/env python
 # Copyright (c) 2012 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 """Basic pyauto performance tests.
 
 For tests that need to be run for multiple iterations (e.g., so that average
 and standard deviation values can be reported), the default number of iterations
 run for each of these tests is specified by |_DEFAULT_NUM_ITERATIONS|.
@@ skipping 72 matching lines @@
     self._seen_graph_lines = {}
 
     pyauto.PyUITest.setUp(self)
 
     # Flush all buffers to disk and wait until system calms down.  Must be done
     # *after* calling pyauto.PyUITest.setUp, since that is where Chrome is
     # killed and re-initialized for a new test.
     # TODO(dennisjeffrey): Implement wait for idle CPU on Windows/Mac.
     if self.IsLinux():  # IsLinux() also implies IsChromeOS().
       os.system('sync')
-      self._WaitForIdleCPU(60.0, 0.03)
+      self._WaitForIdleCPU(60.0, 0.05)

[ilja, 2012/04/25 19:50:23]

Any reason you need to change this?

[dennis_jeffrey, 2012/04/25 21:31:58]

Yes - I mentioned about it as one of the notes in the original review e-mail
that was sent out:

"* For some reason, restarting the session manager (as part of logout and login)
causes CPU usage to go from < 3% to 4 or 5%.  I had to increase the CPU usage
threshold from 3% to 5% to get this test to start up."

 
   def _WaitForIdleCPU(self, timeout, utilization):
     """Waits for the CPU to become idle (< utilization).
 
     Args:
       timeout: The longest time in seconds to wait before throwing an error.
       utilization: The CPU usage below which the system should be considered
           idle (between 0 and 1.0 independent of cores/hyperthreads).
     """
     time_passed = 0.0
@@ skipping 1654 matching lines @@
   def testIntl2File(self):
     self._RunPageCyclerTest('intl2', self._num_iterations, 'Intl2File')
 
   def testMozFile(self):
     self._RunPageCyclerTest('moz', self._num_iterations, 'MozFile')
 
   def testMoz2File(self):
     self._RunPageCyclerTest('moz2', self._num_iterations, 'Moz2File')
 
 
+class MemoryTest(BasePerfTest):
+  """Tests to measure memory consumption under different usage scenarios."""
+
+  def setUp(self):
+    pyauto.PyUITest.setUp(self)
+
+    if self.GetLoginInfo()['is_logged_in']:
+      self.Logout()
+      self.assertFalse(self.GetLoginInfo()['is_logged_in'],
+                      msg='Failed to log out.')
+
+    # Log in.
+    credentials = self.GetPrivateInfo()['test_google_account']
+    self.Login(credentials['username'], credentials['password'])
+    self.assertTrue(self.GetLoginInfo()['is_logged_in'],
+                    msg='Failed to log in.')
+
+  def _GetMemoryStats(self):
+    """Identifies and returns different kinds of current memory usage stats.
+
+    Returns:
+      A dictionary containing current memory usage information.  All values are
+      given in KB.
+      {
+        'gtt': GPU memory usage (graphics translation table),
+        'mem_free': CPU free memory,
+      }
+    """
+    stats = {}
+

[ilja, 2012/04/25 19:50:23]

I spoke with Stephane about your numbers. He thinks you should pause 20 seconds
before taking a final measurement to allow the graphics driver to reclaim memory
(which will only happen when the system is under pressure or idle).

But he is also interested in peak numbers, which would imply continuous
sampling. Maybe like the WaitForIdleCPU we could sample these numbers once per
second, wait until things stabilize and analyze it as post process?

[dennis_jeffrey, 2012/04/25 21:31:58]

I was confused at first about exactly when you wanted to do this sampling, but
in reading your comment at line 1827 below, it looks like you're suggesting that
every call to GetMemoryStats should sample until things stabilize (or up to 20
seconds).

So what do you recommend as a definition for when numbers "stabilize"?  Maybe if
each measured value does not change by more than 3% from the value measured in
the last second?  (and maybe we just give up and let the test proceed if nothing
stabilizes before 20 seconds?)

+    # GPU memory.
+    p = subprocess.Popen('grep bytes /sys/kernel/debug/dri/0/i915_gem_gtt',
+                         stdout=subprocess.PIPE,
+                         shell=True)
+    stdout = p.communicate()[0]
+    gtt_used = re.search('Total [\d]+ objects, ([\d]+) bytes', stdout).group(1)
+    stats['gtt'] = int(gtt_used) / 1024.0
+
+    # CPU memory.
+    stdout = ''
+    with open('/proc/meminfo') as f:
+      stdout = f.read()
+    mem_free = re.search('MemFree:\s*([\d]+) kB', stdout).group(1)

[James Cook, 2012/04/25 19:15:56]

I recommend using available memory as a stat, which is what Luigi uses for the
low memory signal for the tab discarder.  AvailableMem = (ActiveFileMem +
InactiveFileMem) - DirtyMem + FreeMem.  You could record it in addition, or
instead of, this value.

We also got burned in the past with a Flash bug that was leaking "Shmem:", so
that might be good to track also.

[ilja, 2012/04/25 19:50:23]

Lets track as James suggest AvailableMem instead of FreeMem as this is used in
the tab discard decision.

Tracking shared memory sounds like a good idea.

[dennis_jeffrey, 2012/04/25 21:31:58]

Sure, we can track this data.  Puneet also said that he's interested in tracking
a wider variety of data, but for a start we can at least get available/shared
memory.

+    stats['mem_free'] = int(mem_free)
+
+    return stats
+
+  def _RecordMemoryStats(self, description, when):
+    mem = self._GetMemoryStats()
+    self._OutputPerfGraphValue('%s-GTT-%s' % (description, when), mem['gtt'],
+                               'KB', '%s-GTT' % description)
+    self._OutputPerfGraphValue('%s-MemFree-%s' % (description, when),
+                               mem['mem_free'], 'KB', '%s-GTT' % description)
+
+  def _RunTest(self, tabs, description):
+    self._RecordMemoryStats(description, 'Start')
+
+    for iteration_num in xrange(2):
+      for site in tabs:
+        self.AppendTab(pyauto.GURL(site))

[James Cook, 2012/04/25 19:15:56]

Depending on how realistic you want this to be, versus the speed of the test,
you might want to sleep a second or two between loading each tab.  Also, does
this wait for the tab to finish loading before continuing?

[ilja, 2012/04/25 19:50:23]

I think it does not. But maybe we don't need to wait here if we change
GetMemoryStats to sample once per second for at least 20 seconds (or until
numbers stabilize)?

[dennis_jeffrey, 2012/04/25 21:31:58]

AppendTab should return only when notified of the
content::NOTIFICATION_LOAD_STOP event for the given tab.  Note that this doesn't
necessarily mean that all content on the tab has loaded though (e.g., in Gmail,
which uses ajax).

Tabs like those to about:blank open up nearly instantaneously, but you'll see
noticeable waits in-between each call to AppendTab if loading live sites like
Gmail, Plus, etc.

+
+      self._RecordMemoryStats(description, 'Open%d' % (iteration_num + 1))
+
+      for _ in xrange(len(tabs)):
+        self.GetBrowserWindow(0).GetTab(1).Close(True)
+
+      self._RecordMemoryStats(description, 'Close%d' % (iteration_num + 1))
+
+  def testOpenCloseTabsControl(self):
+    tabs = ['about:blank'] * 10
+    self._RunTest(tabs, 'MemControl')
+
+
 class PerfTestServerRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
   """Request handler for the local performance test server."""
 
   def _IgnoreHandler(self, unused_args):
     """A GET request handler that simply replies with status code 200.
 
     Args:
       unused_args: A dictionary of arguments for the current GET request.
           The arguments are ignored.
     """
@@ skipping 212 matching lines @@
     """Identifies the port number to which the server is currently bound.
 
     Returns:
       The numeric port number to which the server is currently bound.
     """
     return self._server.server_address[1]
 
 
 if __name__ == '__main__':
   pyauto_functional.Main()