Android: allows ignoring results of flaky sharded perf tests.

Android: allows ignoring results of flaky sharded perf tests.

In order to keep the tree green, it may be necessary to temporarily ignore
exit_code for some of the perf tests.
Adds this capability to bb_run_sharded_steps.py.

BUG=178438
TEST=

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=184703

[bulach, 2013-02-26 16:29:26]

ptal

[Tom Hudson, 2013-02-26 16:42:36]

lgtm

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
File build/android/bb_run_sharded_steps.py (right):

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:24: The JSON config contains is a file
containing a dictionary in the format:
While you're in here, fix the grammar of the comment?

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:85: end_time, step['device'])
You're changing the format of the output string here; I assume we don't have any
scripts scraping it.

[Sami, 2013-02-26 16:48:00]

Awesome, just question below.

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
File build/android/bb_run_sharded_steps.py (right):

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:82: exit_msg = '%s %s' % (exit_code or 0,
Maybe we should have a line above this one:

    exit_code = exit_code or 0

so we don't need to repeat the condition.

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:98: def _RunShardedSteps(steps, devices,
flaky):
Call this something like flaky_steps to make it clearer?

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:202: flaky = json.load(f)
Sort super duper bonus points should we sort the steps so that flaky tests
always run last? That way the bot would show feedback sooner. Or will changing
the order break something?

[bulach, 2013-02-26 17:14:15]

thanks both! clarified inline, another look please?

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
File build/android/bb_run_sharded_steps.py (right):

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:24: The JSON config contains is a file
containing a dictionary in the format:
On 2013/02/26 16:42:36, Tom Hudson wrote:
> While you're in here, fix the grammar of the comment?

ouch! fixed..

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:82: exit_msg = '%s %s' % (exit_code or 0,
On 2013/02/26 16:48:00, Sami wrote:
> Maybe we should have a line above this one:
> 
>     exit_code = exit_code or 0
> 
> so we don't need to repeat the condition.

Done.

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:85: end_time, step['device'])
On 2013/02/26 16:42:36, Tom Hudson wrote:
> You're changing the format of the output string here; I assume we don't have
any
> scripts scraping it.

that's right, this part is purely informational.

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:98: def _RunShardedSteps(steps, devices,
flaky):
On 2013/02/26 16:48:00, Sami wrote:
> Call this something like flaky_steps to make it clearer?

that looks better, thanks! fixed, and also changed the order so it's "steps,
flaky_steps, devices"... renamed everywhere.

https://codereview.chromium.org/12321138/diff/2001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:202: flaky = json.load(f)
On 2013/02/26 16:48:00, Sami wrote:
> Sort super duper bonus points should we sort the steps so that flaky tests
> always run last? That way the bot would show feedback sooner. Or will changing
> the order break something?

hmmm... that wouldn't make much difference in terms of getting feedback,
unfortunately... :-/

the way it works, *all* of the tests run as a single step first, persist the
result to disk, then once they are all done, the buildbot will print each one of
these results...

sure, the "single" step would have the stdout sooner, but that is fairly
unreadable as it's not synchronized across the shards... (the _second_ step
guarantees that each step will be printed sequentially)....

leave as is?

[Sami, 2013-02-26 17:19:13]

Ah, I knew there had to be a catch about getting the results sooner. Regardless,
this is a huge step in the right direction, lgtm, thanks!

https://codereview.chromium.org/12321138/diff/7001/build/android/bb_run_shard...
File build/android/bb_run_sharded_steps.py (right):

https://codereview.chromium.org/12321138/diff/7001/build/android/bb_run_shard...
build/android/bb_run_sharded_steps.py:92: 'exit_code': exit_code or 0,
No need for the "or 0" anymore :)

[Tom Hudson, 2013-02-26 17:20:00]

On 2013/02/26 17:14:15, bulach wrote:
> On 2013/02/26 16:48:00, Sami wrote:
> > Sort super duper bonus points should we sort the steps so that flaky tests
> > always run last? That way the bot would show feedback sooner. Or will
changing
> > the order break something?
> 
> hmmm... that wouldn't make much difference in terms of getting feedback,
> unfortunately... :-/
> 
> the way it works, *all* of the tests run as a single step first, persist the
> result to disk, then once they are all done, the buildbot will print each one
of
> these results...
> 
> sure, the "single" step would have the stdout sooner, but that is fairly
> unreadable as it's not synchronized across the shards... (the _second_ step
> guarantees that each step will be printed sequentially)....
> 
> leave as is?

The idea is so cool, let's memorialize it for future generations by filing a
low-priority feature request.

[bulach, 2013-02-26 17:34:50]

done! :) crbug.com/178438

CQing..

[commit-bot: I haz the power, 2013-02-26 17:36:35]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/bulach@chromium.org/12321138/4002

[commit-bot: I haz the power, 2013-02-26 19:43:44]

Change committed as 184703