Telemetry: shard tests on android.

Telemetry: shard tests on android.

This script allows to:
1. run all telemetry tests in N devices as a single step in the buildbot.
Each run will save its results in the buildbot.
2. A separate step will then essentially print them.

This is very similar mechanism to the existing one downstream, which will be deprecated
and replaced by this as soon as it lands.

BUG=163503
TEST=build/android/bb_run_sharded_steps.py


Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=172044

[tonyg, 2012-12-04 23:57:56]

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
File build/android/bb_run_sharded_steps.py (right):

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:14: - First, without params: all steps
will be executed in parallel using all
Please explain the JSON steps file format here.

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:64: end_time, step['device'])
indentation

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:75: def _RunShardedSteps(steps, devices):
assert steps
assert devices

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:86: device = devices[i]
above two lines can be collapsed into:
for i, device in enumerate(devices):

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:92: ' --keep_test_server_ports'}]
Telemetry doesn't support --keep_test_server_ports, we'll have to add that.

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:118: def _PrintResults(step_name):
_PrintStepOutput?

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:125: print result['output']
The buildbot output colors stdout in black and stderr in red, so it must know
the difference at some level. Should we worry about that for now?

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:150:
test_options_parser.AddBuildTypeOption(parser)
This looks unused

[bulach, 2012-12-07 13:53:00]

thanks tony! all comments addressed, another look please?

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
File build/android/bb_run_sharded_steps.py (right):

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:14: - First, without params: all steps
will be executed in parallel using all
On 2012/12/04 23:57:56, tonyg wrote:
> Please explain the JSON steps file format here.

Done.

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:64: end_time, step['device'])
On 2012/12/04 23:57:56, tonyg wrote:
> indentation

Done.

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:75: def _RunShardedSteps(steps, devices):
On 2012/12/04 23:57:56, tonyg wrote:
> assert steps
> assert devices

Done.

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:86: device = devices[i]
On 2012/12/04 23:57:56, tonyg wrote:
> above two lines can be collapsed into:
> for i, device in enumerate(devices):

Done.

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:92: ' --keep_test_server_ports'}]
On 2012/12/04 23:57:56, tonyg wrote:
> Telemetry doesn't support --keep_test_server_ports, we'll have to add that.

ahn, very good point! added, and fixed the port allocation in browser_android.py
as well.. note though that I'll only be able to fully test it next week when I'm
back home..

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:118: def _PrintResults(step_name):
On 2012/12/04 23:57:56, tonyg wrote:
> _PrintStepOutput?

Done.

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:125: print result['output']
On 2012/12/04 23:57:56, tonyg wrote:
> The buildbot output colors stdout in black and stderr in red, so it must know
> the difference at some level. Should we worry about that for now?

oh, that's a good point.. we use pexpect.run, in line 58. that unfortunately
collapses stdout and stderr.. :( we could obviously roll our own "run with
timeout and stream output", but I'd rather live with a single color for the time
being, wdyt?

https://codereview.chromium.org/11308344/diff/1/build/android/bb_run_sharded_...
build/android/bb_run_sharded_steps.py:150:
test_options_parser.AddBuildTypeOption(parser)
On 2012/12/04 23:57:56, tonyg wrote:
> This looks unused

Done.

[tonyg, 2012-12-07 17:01:15]

lgtm

[commit-bot: I haz the power, 2012-12-10 10:42:42]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/bulach@chromium.org/11308344/9001

[commit-bot: I haz the power, 2012-12-10 12:49:28]

Change committed as 172044