Adding pyauto-based memory usage tests for ChromeOS.

chrome/test/functional/perf.py

Index: chrome/test/functional/perf.py
diff --git a/chrome/test/functional/perf.py b/chrome/test/functional/perf.py
index 174f73131618b659a1e2bb39407460cda8360210..ca318a4805f66979399bee70f8a4ffe7e602acf8 100755
--- a/chrome/test/functional/perf.py
+++ b/chrome/test/functional/perf.py
@@ -90,7 +90,7 @@ class BasePerfTest(pyauto.PyUITest):
     # TODO(dennisjeffrey): Implement wait for idle CPU on Windows/Mac.
     if self.IsLinux():  # IsLinux() also implies IsChromeOS().
       os.system('sync')
-      self._WaitForIdleCPU(60.0, 0.03)
+      self._WaitForIdleCPU(60.0, 0.05)

[ilja, 2012/04/25 19:50:23]

Any reason you need to change this?

[dennis_jeffrey, 2012/04/25 21:31:58]

Yes - I mentioned about it as one of the notes in the original review e-mail
that was sent out:

"* For some reason, restarting the session manager (as part of logout and login)
causes CPU usage to go from < 3% to 4 or 5%.  I had to increase the CPU usage
threshold from 3% to 5% to get this test to start up."

 
   def _WaitForIdleCPU(self, timeout, utilization):
     """Waits for the CPU to become idle (< utilization).
@@ -1765,6 +1765,79 @@ class PageCyclerTest(BasePerfTest):
     self._RunPageCyclerTest('moz2', self._num_iterations, 'Moz2File')
 
 
+class MemoryTest(BasePerfTest):
+  """Tests to measure memory consumption under different usage scenarios."""
+
+  def setUp(self):
+    pyauto.PyUITest.setUp(self)
+
+    if self.GetLoginInfo()['is_logged_in']:
+      self.Logout()
+      self.assertFalse(self.GetLoginInfo()['is_logged_in'],
+                      msg='Failed to log out.')
+
+    # Log in.
+    credentials = self.GetPrivateInfo()['test_google_account']
+    self.Login(credentials['username'], credentials['password'])
+    self.assertTrue(self.GetLoginInfo()['is_logged_in'],
+                    msg='Failed to log in.')
+
+  def _GetMemoryStats(self):
+    """Identifies and returns different kinds of current memory usage stats.
+
+    Returns:
+      A dictionary containing current memory usage information.  All values are
+      given in KB.
+      {
+        'gtt': GPU memory usage (graphics translation table),
+        'mem_free': CPU free memory,
+      }
+    """
+    stats = {}
+

[ilja, 2012/04/25 19:50:23]

I spoke with Stephane about your numbers. He thinks you should pause 20 seconds
before taking a final measurement to allow the graphics driver to reclaim memory
(which will only happen when the system is under pressure or idle).

But he is also interested in peak numbers, which would imply continuous
sampling. Maybe like the WaitForIdleCPU we could sample these numbers once per
second, wait until things stabilize and analyze it as post process?

[dennis_jeffrey, 2012/04/25 21:31:58]

I was confused at first about exactly when you wanted to do this sampling, but
in reading your comment at line 1827 below, it looks like you're suggesting that
every call to GetMemoryStats should sample until things stabilize (or up to 20
seconds).

So what do you recommend as a definition for when numbers "stabilize"?  Maybe if
each measured value does not change by more than 3% from the value measured in
the last second?  (and maybe we just give up and let the test proceed if nothing
stabilizes before 20 seconds?)

+    # GPU memory.
+    p = subprocess.Popen('grep bytes /sys/kernel/debug/dri/0/i915_gem_gtt',
+                         stdout=subprocess.PIPE,
+                         shell=True)
+    stdout = p.communicate()[0]
+    gtt_used = re.search('Total [\d]+ objects, ([\d]+) bytes', stdout).group(1)
+    stats['gtt'] = int(gtt_used) / 1024.0
+
+    # CPU memory.
+    stdout = ''
+    with open('/proc/meminfo') as f:
+      stdout = f.read()
+    mem_free = re.search('MemFree:\s*([\d]+) kB', stdout).group(1)

[James Cook, 2012/04/25 19:15:56]

I recommend using available memory as a stat, which is what Luigi uses for the
low memory signal for the tab discarder.  AvailableMem = (ActiveFileMem +
InactiveFileMem) - DirtyMem + FreeMem.  You could record it in addition, or
instead of, this value.

We also got burned in the past with a Flash bug that was leaking "Shmem:", so
that might be good to track also.

[ilja, 2012/04/25 19:50:23]

Lets track as James suggest AvailableMem instead of FreeMem as this is used in
the tab discard decision.

Tracking shared memory sounds like a good idea.

[dennis_jeffrey, 2012/04/25 21:31:58]

Sure, we can track this data.  Puneet also said that he's interested in tracking
a wider variety of data, but for a start we can at least get available/shared
memory.

+    stats['mem_free'] = int(mem_free)
+
+    return stats
+
+  def _RecordMemoryStats(self, description, when):
+    mem = self._GetMemoryStats()
+    self._OutputPerfGraphValue('%s-GTT-%s' % (description, when), mem['gtt'],
+                               'KB', '%s-GTT' % description)
+    self._OutputPerfGraphValue('%s-MemFree-%s' % (description, when),
+                               mem['mem_free'], 'KB', '%s-GTT' % description)
+
+  def _RunTest(self, tabs, description):
+    self._RecordMemoryStats(description, 'Start')
+
+    for iteration_num in xrange(2):
+      for site in tabs:
+        self.AppendTab(pyauto.GURL(site))

[James Cook, 2012/04/25 19:15:56]

Depending on how realistic you want this to be, versus the speed of the test,
you might want to sleep a second or two between loading each tab.  Also, does
this wait for the tab to finish loading before continuing?

[ilja, 2012/04/25 19:50:23]

I think it does not. But maybe we don't need to wait here if we change
GetMemoryStats to sample once per second for at least 20 seconds (or until
numbers stabilize)?

[dennis_jeffrey, 2012/04/25 21:31:58]

AppendTab should return only when notified of the
content::NOTIFICATION_LOAD_STOP event for the given tab.  Note that this doesn't
necessarily mean that all content on the tab has loaded though (e.g., in Gmail,
which uses ajax).

Tabs like those to about:blank open up nearly instantaneously, but you'll see
noticeable waits in-between each call to AppendTab if loading live sites like
Gmail, Plus, etc.

+
+      self._RecordMemoryStats(description, 'Open%d' % (iteration_num + 1))
+
+      for _ in xrange(len(tabs)):
+        self.GetBrowserWindow(0).GetTab(1).Close(True)
+
+      self._RecordMemoryStats(description, 'Close%d' % (iteration_num + 1))
+
+  def testOpenCloseTabsControl(self):
+    tabs = ['about:blank'] * 10
+    self._RunTest(tabs, 'MemControl')
+
+
 class PerfTestServerRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
   """Request handler for the local performance test server."""