Rerun each CL 10 times to smooth out noise in perf testing.

Rerun each CL 10 times to smooth out noise in perf testing.
Committed: https://code.google.com/p/dart/source/detail?r=10086

[Emily Fortuna, 2012-07-30 21:07:05]

This change finally implements rerunning past CLs so what we have 10 results for
each CL for performance results. This change also resulted in me modifying
get_drt (somewhat substantially) to be a more general purpose google-storage
download tool.

[vsm, 2012-07-30 23:22:05]

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/get_archive.py
File tools/get_archive.py (right):

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/get_archive.p...
tools/get_archive.py:269: 
If the revision is set and positional[0] != 'sdk', you should return an error
rather than ignore.

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/testing/perf_...
File tools/testing/perf_testing/run_perf_tests.py (right):

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/testing/perf_...
tools/testing/perf_testing/run_perf_tests.py:1075: # been checked in.
Isn't this going to run (some) CLs well more than 10 times?  E.g., if there are
no checkins for the weekend, it'll just run the last CL repeatedly?

[vsm, 2012-07-30 23:27:53]

lgtm

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/testing/perf_...
File tools/testing/perf_testing/run_perf_tests.py (right):

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/testing/perf_...
tools/testing/perf_testing/run_perf_tests.py:1075: # been checked in.
Nevermind.  I see what I missed.  Perhaps a comment on number_of_results below?

On 2012/07/30 23:22:05, vsm wrote:
> Isn't this going to run (some) CLs well more than 10 times?  E.g., if there
are
> no checkins for the weekend, it'll just run the last CL repeatedly?

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/testing/perf_...
tools/testing/perf_testing/run_perf_tests.py:1091:
a_test.file_processor.get_score_type(benchmark_name))
Not sure if perf is a concern, but consider caching the number of results per
revision.

[Emily Fortuna, 2012-07-31 20:16:20]

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/get_archive.py
File tools/get_archive.py (right):

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/get_archive.p...
tools/get_archive.py:269: 
On 2012/07/30 23:22:05, vsm wrote:
> If the revision is set and positional[0] != 'sdk', you should return an error
> rather than ignore.

Done.

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/testing/perf_...
File tools/testing/perf_testing/run_perf_tests.py (right):

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/testing/perf_...
tools/testing/perf_testing/run_perf_tests.py:1075: # been checked in.
On 2012/07/30 23:27:53, vsm wrote:
> Nevermind.  I see what I missed.  Perhaps a comment on number_of_results
below?
> 
> On 2012/07/30 23:22:05, vsm wrote:
> > Isn't this going to run (some) CLs well more than 10 times?  E.g., if there
> are
> > no checkins for the weekend, it'll just run the last CL repeatedly?
> 

Clarified (hopefully).

https://chromiumcodereview.appspot.com/10828077/diff/6001/tools/testing/perf_...
tools/testing/perf_testing/run_perf_tests.py:1091:
a_test.file_processor.get_score_type(benchmark_name))
On 2012/07/30 23:27:53, vsm wrote:
> Not sure if perf is a concern, but consider caching the number of results per
> revision.

That's a good idea. I'll change my code to do that.

[ricow1, 2012-08-06 11:10:03]

https://chromiumcodereview.appspot.com/10828077/diff/1005/tools/testing/perf_...
File tools/testing/perf_testing/run_perf_tests.py (right):

https://chromiumcodereview.appspot.com/10828077/diff/1005/tools/testing/perf_...
tools/testing/perf_testing/run_perf_tests.py:1110: while revision_num > 3000 and
not has_run_extra:
This will take us back in time really really slow, and this might very well
starve out even recent revisions since I don't think that the current setup with
just 2 macs can keep up with our commit pace. This will make it harder to track
down regressions, and I doubt that we will ever get back historic results. How
does the distribution of download build/run benchmark look, i.e., if fetching
the build is dominating the time to run the benchmarks I see why you would do
this

[Emily Fortuna, 2012-08-06 17:39:34]

https://chromiumcodereview.appspot.com/10828077/diff/1005/tools/testing/perf_...
File tools/testing/perf_testing/run_perf_tests.py (right):

https://chromiumcodereview.appspot.com/10828077/diff/1005/tools/testing/perf_...
tools/testing/perf_testing/run_perf_tests.py:1110: while revision_num > 3000 and
not has_run_extra:
On 2012/08/06 11:10:03, ricow1 wrote:
> This will take us back in time really really slow, and this might very well
> starve out even recent revisions since I don't think that the current setup
with
> just 2 macs can keep up with our commit pace. This will make it harder to
track
> down regressions, and I doubt that we will ever get back historic results. How
> does the distribution of download build/run benchmark look, i.e., if fetching
> the build is dominating the time to run the benchmarks I see why you would do
> this

Right, so when this was originally designed, I thought we had SDK builds going
back for a long time so switching between revisions would be cheap. After
writing this I discovered this is not the case -- we only have the last 100
revisions saved on Google Storage, and building/switching between random
revisions is much more expensive. So, I was planning on starting at revision
9000 instead, and from this point onward, we'll be storing SDK revisions, so
switching between revisions in the future (should, I hope) be cheap. Does this
sound reasonable?

[ricow1, 2012-08-06 18:51:58]

On 2012/08/06 17:39:34, Emily Fortuna wrote:
>
https://chromiumcodereview.appspot.com/10828077/diff/1005/tools/testing/perf_...
> File tools/testing/perf_testing/run_perf_tests.py (right):
> 
>
https://chromiumcodereview.appspot.com/10828077/diff/1005/tools/testing/perf_...
> tools/testing/perf_testing/run_perf_tests.py:1110: while revision_num > 3000
and
> not has_run_extra:
> On 2012/08/06 11:10:03, ricow1 wrote:
> > This will take us back in time really really slow, and this might very well
> > starve out even recent revisions since I don't think that the current setup
> with
> > just 2 macs can keep up with our commit pace. This will make it harder to
> track
> > down regressions, and I doubt that we will ever get back historic results.
How
> > does the distribution of download build/run benchmark look, i.e., if
fetching
> > the build is dominating the time to run the benchmarks I see why you would
do
> > this
> 
> Right, so when this was originally designed, I thought we had SDK builds going
> back for a long time so switching between revisions would be cheap. After
> writing this I discovered this is not the case -- we only have the last 100
> revisions saved on Google Storage, and building/switching between random
> revisions is much more expensive. So, I was planning on starting at revision
> 9000 instead, and from this point onward, we'll be storing SDK revisions, so
> switching between revisions in the future (should, I hope) be cheap. Does this
> sound reasonable?
Sure, I am fine with whatever design you have, I just thought this could be an
issue. You could consider having a local cache on the mac for cheap switching,
e.g, two caches:
one cache with the last 100 revisions
one cache with a window of older revisions for which you are currently producing
results

You can do simple split, i.e, use 70% of the time running on the new cache
entries and 30% of the time on recalculating older ones (and some heuristics for
deciding this window based on existing result counts)

If you really want to be efficient you can have another machine locally in the
lab be cache, i.e., if you have an old machine and buy an external drive you
have a pretty efficient cache setup - we do this in the aarhus lab. You might
want to tell the cache upfront what to fetch instead of just fetching on demand
as we do, but feel free to reuse our cache functionality if you go down this
road - ping me or anders if you need info on this.