Retry tests on other bots if the device is unresponsive/offline

build/android/pylib/base_test_sharder.py

 # Copyright (c) 2012 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 
+import android_commands
 import logging
 import multiprocessing
 
 from test_result import TestResults
 
 
 def _ShardedTestRunnable(test):
   """Standalone function needed by multiprocessing.Pool."""
   log_format = '[' + test.device + '] # %(asctime)-15s: %(message)s'
   if logging.getLogger().handlers:
@@ skipping 63 matching lines @@
     logging.warning('*' * 80)
     final_results = TestResults()
     for retry in xrange(self.retries):
       logging.warning('Try %d of %d', retry + 1, self.retries)
       self.SetupSharding(self.tests)
       test_runners = []
       for index, device in enumerate(self.attached_devices):
         logging.warning('*' * 80)
         logging.warning('Creating shard %d for %s', index, device)
         logging.warning('*' * 80)
         test_runner = self.CreateShardedTestRunner(device, index)

[Yaron, 2012/10/26 18:48:44]

Based on:
http://build.chromium.org/p/chromium.linux/builders/Android%20Tests%20%28dbg%...
and
http://build.chromium.org/p/chromium.linux/builders/Android%20Tests%20%28dbg%...

It looks like the creation of sharded test runner needs to guarded with catching
DeviceUnresponsiveError or else we can fail without attempting any retries.

I'm concerned though that we aren't catching this issue more generally and we
need to ensure that all code run by a shard can enter the new retry logic below.

[yongsheng, 2012/10/28 01:41:03]

yes, i don't try to catch exceptions here for i'm not sure all of these errors
are due to device problems or others.
It seems that we have to do this kind of thing here.

[bulach, 2012/10/29 14:45:42]

yaron: just to clarify, I kindly asked Yongsheng (thanks btw! :) to help us with
the case where the device was going down when running the tests, which he fixed
here..

the sharding was originally done with speed in mind, not reliability. this is
the very first patch to try and make it failsafe / hot reuse devices if they
become ill during the test cycle, so I share your concern and we'll have to work
through them...

so two questions then:
yongsheng: would you be able to continue to help us here?
if not: yaron, would there be anyone on your side?

[bulach, 2012/10/29 14:51:23]

btw, the issue mentioned above I guess is in run_tests.py, TestSharder __init__
where it tries to GetAllTests().. we need to add some retry mechanism in there
should that call fail..

         test_runners += [test_runner]
       logging.warning('Starting...')
       pool = multiprocessing.Pool(len(self.attached_devices),
                                   SetTestsContainer,
                                   [BaseTestSharder.tests_container])
       # map can't handle KeyboardInterrupt exception. It's a python bug.
       # So use map_async instead.
       async_results = pool.map_async(_ShardedTestRunnable, test_runners)
       results_lists = async_results.get(999999)
+
       test_results = TestResults.FromTestResults(results_lists)
-      if retry == self.retries - 1:
+      # Re-check the attached devices for some devices may
+      # become offline
+      retry_devices = set(android_commands.GetAttachedDevices())
+      # Remove devices that had exceptions.
+      retry_devices -= TestResults.DeviceExceptions(results_lists)
+      # Retry on devices that didn't have any exception.
+      self.attached_devices = list(retry_devices)
+      if (retry == self.retries - 1 or
+          len(self.attached_devices) == 0):
         all_passed = final_results.ok + test_results.ok
         final_results = test_results
         final_results.ok = all_passed
         break
       else:
         final_results.ok += test_results.ok
         self.tests = []
         for t in test_results.GetAllBroken():
           self.tests += [t.name]
         if not self.tests:
           break
     self.OnTestsCompleted(test_runners, final_results)
     return final_results