[pinpoint] Limit executions to one test run each + device sharding.

[pinpoint] Limit executions to one test run each + device sharding.

* Hard-code --pageset-repeat to 1 (or 2 for startup benchmarks).
* Limit one RunTest execution to one test run.
* Do repeat_count by adding additional Attempts at the beginning.
* Add device sharding with cross-change device affinity.
* Re-use the results of previous builds.

Having a 1:1 mapping between Execution and test runs makes failure handling
easier -- each Execution can only throw one Exception. And doing repeats with
Attempts allows functional bisects to use the same repeat mechanism as
performance bisects.

Removing --pageset-repeat means extra Swarming task overhead, but if the default
repeat count is 10 and there are 5 bots, then it's just 2 Swarming tasks instead
of 1.

BUG=catapult:#3665

Review-Url: https://chromiumcodereview.appspot.com/3010873003
Committed: https://chromium.googlesource.com/external/github.com/catapult-project/catapult/+/80c47581b19fb3cf92d19b0f34eb0e9518aa7eaa

[dtu, 2017-09-01 10:13:47]

dtu@chromium.org changed reviewers:
+ perezju@chromium.org, simonhatch@chromium.org, sullivan@chromium.org

[dtu, 2017-09-01 10:13:47]

How about something like this?
This appears to solve many problems at once.

Pros:
* Device sharding is done in a really easy way by shifting all the tracking to
the RunTest Quest singleton.
* A 1:1 mapping between RunTest Executions and swarming tasks means that we
eliminate confusion about result_values, don't have to worry about handling
multiple exceptions (which was incorrect previously), and simplify the
Executions overall.
* We don't have multiple places with repeat logic -- it's all handled at the
Attempt level. This means that functional bisects work the same way as
performance bisects.
* Mitigate intra-task noise by having a separate Swarming task for each repeat.

Cons:
* We take on extra Swarming overhead. But if our default repeat count is 10 and
we have 5 bots, then each bot is running 2 Swarming tasks instead of 1. Not bad
at all.

[dtu, 2017-09-01 10:19:38]

On 2017/09/01 10:13:47, dtu wrote:
> How about something like this?
> This appears to solve many problems at once.
> 
> Pros:
> * Device sharding is done in a really easy way by shifting all the tracking to
> the RunTest Quest singleton.
> * A 1:1 mapping between RunTest Executions and swarming tasks means that we
> eliminate confusion about result_values, don't have to worry about handling
> multiple exceptions (which was incorrect previously), and simplify the
> Executions overall.
> * We don't have multiple places with repeat logic -- it's all handled at the
> Attempt level. This means that functional bisects work the same way as
> performance bisects.
> * Mitigate intra-task noise by having a separate Swarming task for each
repeat.
> 
> Cons:
> * We take on extra Swarming overhead. But if our default repeat count is 10
and
> we have 5 bots, then each bot is running 2 Swarming tasks instead of 1. Not
bad
> at all.

The Mac Air bot looks like it has about 7s of overhead per task. That would mean
7s of additional overhead per Change, or about 70s for a Job.

https://chromium-swarm.appspot.com/bot?id=build197-b4

[perezju, 2017-09-05 14:01:28]

On 2017/09/01 10:19:38, dtu wrote:
> On 2017/09/01 10:13:47, dtu wrote:
> > How about something like this?
> > This appears to solve many problems at once.
> > 
> > Pros:
> > * Device sharding is done in a really easy way by shifting all the tracking
to
> > the RunTest Quest singleton.
> > * A 1:1 mapping between RunTest Executions and swarming tasks means that we
> > eliminate confusion about result_values, don't have to worry about handling
> > multiple exceptions (which was incorrect previously), and simplify the
> > Executions overall.
> > * We don't have multiple places with repeat logic -- it's all handled at the
> > Attempt level. This means that functional bisects work the same way as
> > performance bisects.
> > * Mitigate intra-task noise by having a separate Swarming task for each
> repeat.
> > 
> > Cons:
> > * We take on extra Swarming overhead. But if our default repeat count is 10
> and
> > we have 5 bots, then each bot is running 2 Swarming tasks instead of 1. Not
> bad
> > at all.
> 
> The Mac Air bot looks like it has about 7s of overhead per task. That would
mean
> 7s of additional overhead per Change, or about 70s for a Job.
> 
> https://chromium-swarm.appspot.com/bot?id=build197-b4

I would be happy with a change like this. Particularly "1:1 mapping between
RunTest Executions" and "a separate Swarming task for each repeat" sound like
big wins to me.

[shatch, 2017-09-05 21:04:35]

Thanks for the walkthrough, happy with this approach as well, lg2m, mentioned
small gotcha with pageset repeat and some benchmarks offline (startup is broken
and won't produce warm values).

[dtu, 2017-09-06 00:01:53]

Patchset #4 (id:60001) has been deleted

[dtu, 2017-09-06 00:01:58]

Patchset #4 (id:80001) has been deleted

[dtu, 2017-09-06 00:02:14]

Patchset #1 (id:1) has been deleted

[dtu, 2017-09-06 00:02:31]

Patchset #1 (id:20001) has been deleted

[dtu, 2017-09-06 00:07:48]

Description was changed from

==========
NOT READY TO SUBMIT: [pinpoint] Device sharding.

BUG=catapult:#3665
==========

to

==========
[pinpoint] Limit executions to one test run each + device sharding.

* Hard-code --pageset-repeat to 1 (or 2 for startup benchmarks).
* Limit one RunTest execution to one test run.
* Do repeat_count by adding additional Attempts at the beginning.
* Add device sharding with cross-change device affinity.
* Re-use the results of previous builds.

Having a 1:1 mapping between Execution and test runs makes failure handling
easier -- each Execution can only throw on Exception. And doing repeats with
Attempts allows functional bisects to use the same repeat mechanism as
performance bisects.

Removing --pageset-repeat means extra Swarming task overhead, but if the default
repeat count is 10 and there are 5 bots, then it's just 2 Swarming tasks instead
of 1.

BUG=catapult:#3665
==========

[dtu, 2017-09-06 00:08:07]

Tests added! Ready for review!

[perezju, 2017-09-06 14:24:56]

lgtm

https://codereview.chromium.org/3010873003/diff/100001/dashboard/dashboard/pi...
File dashboard/dashboard/pinpoint/handlers/quest_generator.py (right):

https://codereview.chromium.org/3010873003/diff/100001/dashboard/dashboard/pi...
dashboard/dashboard/pinpoint/handlers/quest_generator.py:90: swarming_extra_args
+= ('--pageset-repeat', '2')
ugh :(

Just saw that bug. Hope that gets eventually fixed.

[shatch, 2017-09-06 14:41:09]

lgtm

[dtu, 2017-09-06 15:48:44]

Description was changed from

==========
[pinpoint] Limit executions to one test run each + device sharding.

* Hard-code --pageset-repeat to 1 (or 2 for startup benchmarks).
* Limit one RunTest execution to one test run.
* Do repeat_count by adding additional Attempts at the beginning.
* Add device sharding with cross-change device affinity.
* Re-use the results of previous builds.

Having a 1:1 mapping between Execution and test runs makes failure handling
easier -- each Execution can only throw on Exception. And doing repeats with
Attempts allows functional bisects to use the same repeat mechanism as
performance bisects.

Removing --pageset-repeat means extra Swarming task overhead, but if the default
repeat count is 10 and there are 5 bots, then it's just 2 Swarming tasks instead
of 1.

BUG=catapult:#3665
==========

to

==========
[pinpoint] Limit executions to one test run each + device sharding.

* Hard-code --pageset-repeat to 1 (or 2 for startup benchmarks).
* Limit one RunTest execution to one test run.
* Do repeat_count by adding additional Attempts at the beginning.
* Add device sharding with cross-change device affinity.
* Re-use the results of previous builds.

Having a 1:1 mapping between Execution and test runs makes failure handling
easier -- each Execution can only throw one Exception. And doing repeats with
Attempts allows functional bisects to use the same repeat mechanism as
performance bisects.

Removing --pageset-repeat means extra Swarming task overhead, but if the default
repeat count is 10 and there are 5 bots, then it's just 2 Swarming tasks instead
of 1.

BUG=catapult:#3665
==========

[dtu, 2017-09-06 15:49:11]

The CQ bit was checked by dtu@chromium.org

[commit-bot: I haz the power, 2017-09-06 15:49:21]

CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-09-06 16:23:30]

CQ is committing da patch.

Bot data: {"patchset_id": 100001, "attempt_start_ts": 1504712951994180,
"parent_rev": "724bb776bcdea642a5bc9cc20fe14bb0282b4900", "commit_rev":
"80c47581b19fb3cf92d19b0f34eb0e9518aa7eaa"}

[commit-bot: I haz the power, 2017-09-06 16:23:34]

Description was changed from

==========
[pinpoint] Limit executions to one test run each + device sharding.

* Hard-code --pageset-repeat to 1 (or 2 for startup benchmarks).
* Limit one RunTest execution to one test run.
* Do repeat_count by adding additional Attempts at the beginning.
* Add device sharding with cross-change device affinity.
* Re-use the results of previous builds.

Having a 1:1 mapping between Execution and test runs makes failure handling
easier -- each Execution can only throw one Exception. And doing repeats with
Attempts allows functional bisects to use the same repeat mechanism as
performance bisects.

Removing --pageset-repeat means extra Swarming task overhead, but if the default
repeat count is 10 and there are 5 bots, then it's just 2 Swarming tasks instead
of 1.

BUG=catapult:#3665
==========

to

==========
[pinpoint] Limit executions to one test run each + device sharding.

* Hard-code --pageset-repeat to 1 (or 2 for startup benchmarks).
* Limit one RunTest execution to one test run.
* Do repeat_count by adding additional Attempts at the beginning.
* Add device sharding with cross-change device affinity.
* Re-use the results of previous builds.

Having a 1:1 mapping between Execution and test runs makes failure handling
easier -- each Execution can only throw one Exception. And doing repeats with
Attempts allows functional bisects to use the same repeat mechanism as
performance bisects.

Removing --pageset-repeat means extra Swarming task overhead, but if the default
repeat count is 10 and there are 5 bots, then it's just 2 Swarming tasks instead
of 1.

BUG=catapult:#3665

Review-Url: https://chromiumcodereview.appspot.com/3010873003
Committed:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...
==========

[commit-bot: I haz the power, 2017-09-06 16:23:35]

Committed patchset #2 (id:100001) as
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...