Android: replace temporary test runner with the instrumentation runner.

Android: replace temporary test runner with the instrumentation runner.

Follow up from http://codereview.chromium.org/10826071/

BUG=139365
TEST=Android FYI tester runs instrumentation tests.


Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=149679

[Yaron, 2012-07-31 16:26:17]

https://chromiumcodereview.appspot.com/10828094/diff/1008/build/android/build...
File build/android/buildbot_fyi.sh (right):

https://chromiumcodereview.appspot.com/10828094/diff/1008/build/android/build...
build/android/buildbot_fyi.sh:18: bb_compile_experimental
I think this bot was turned down and we can remove the config. Isaac?

https://chromiumcodereview.appspot.com/10828094/diff/1008/build/android/build...
File build/android/buildbot_fyi_tester.sh (right):

https://chromiumcodereview.appspot.com/10828094/diff/1008/build/android/build...
build/android/buildbot_fyi_tester.sh:19:
bb_run_content_shell_instrumentation_test
I think we should keep this ordering throughout (unit_tests before
instrumentation tests). That would be consistent with other ports.

[bulach, 2012-07-31 16:42:44]

thanks yaron! comments addressed, ptal

https://chromiumcodereview.appspot.com/10828094/diff/1008/build/android/build...
File build/android/buildbot_fyi.sh (right):

https://chromiumcodereview.appspot.com/10828094/diff/1008/build/android/build...
build/android/buildbot_fyi.sh:18: bb_compile_experimental
On 2012/07/31 16:26:18, Yaron wrote:
> I think this bot was turned down and we can remove the config. Isaac?

sgtm! I'll delete this file once Isaac confirms..

https://chromiumcodereview.appspot.com/10828094/diff/1008/build/android/build...
File build/android/buildbot_fyi_tester.sh (right):

https://chromiumcodereview.appspot.com/10828094/diff/1008/build/android/build...
build/android/buildbot_fyi_tester.sh:19:
bb_run_content_shell_instrumentation_test
On 2012/07/31 16:26:18, Yaron wrote:
> I think we should keep this ordering throughout (unit_tests before
> instrumentation tests). That would be consistent with other ports.

done all over the place..

[Yaron, 2012-07-31 16:51:48]

lgtm mod the q for Isaac

[Isaac (away), 2012-07-31 16:57:55]

yeah you can delete it.  I have a CL in limbo that did the same thing
(https://chromiumcodereview.appspot.com/10802033/ )

Also, I can't find this email anymore but I recall you said something about main
waterfall tests yesterday?  Those are not active yet, my best guess for landing
time is 2 weeks.  Unit_tests is causing serious pain for capacity planning :-(

[bulach, 2012-07-31 17:48:39]

thanks both!

isaac: removed the file (please check is the right one! :)

as for the main waterfall: the idea here is to keep this only on the FYI / try
bot for a while then graduate later on, is that ok?

I think this was captured in one of the docs, but just document it here. at some
point, we'd need to split "native" tests from "instrumentation" tests.
downstream, all these "instrumentation" tests take ~20mins (sharded), but I
don't think we'll be upstreaming the "large tests" (~8mins) any time soon..

so hopefully in the next couple of weeks this can piggyback on the unittests,
and then later on we split in separate tester bots? how does it sound?

[Isaac (away), 2012-07-31 17:52:14]

sg, lgtm

The review is listing buildbot_fyi.sh as modified, not deleted...  Could be just
an issue interpreting the patch though.

[nilesh, 2012-08-01 13:52:33]

On 2012/08/01 11:52:51, bulach wrote:

LGTM

[bulach, 2012-08-02 11:30:23]

ilevy, nilesh, yaron: I haven't landed this yet because, sigh, we haven't got a
successful run of the new instrumentation tests yet.. :(
not that the temporary test runner look any better anyway... ;)

yesterday there was a busted device (afaict), and the day before there was a
legitimate failed test..

so before landing this, I just want to double check: this is NOT adding any tree
closures or CQ blockers, right? it'll just show up in "android_test" try and
fyi, is that correct?

I'm a bit confused on the "buildbot_try_tester.sh", as it mentions "android" try
bot, which I think is the one used by CQ?

[nilesh, 2012-08-02 15:18:18]

On 2012/08/02 11:30:23, bulach wrote:
> ilevy, nilesh, yaron: I haven't landed this yet because, sigh, we haven't got
a
> successful run of the new instrumentation tests yet.. :(
> not that the temporary test runner look any better anyway... ;)
> 
> yesterday there was a busted device (afaict), and the day before there was a
> legitimate failed test..
> 
> so before landing this, I just want to double check: this is NOT adding any
tree
> closures or CQ blockers, right? it'll just show up in "android_test" try and
> fyi, is that correct?

Yes.
> 
> I'm a bit confused on the "buildbot_try_tester.sh", as it mentions "android"
try
> bot, which I think is the one used by CQ?

I think this script is in preparation to adding tests on the main waterfall and
CQ (and currently not used). Isaac can confirm.

[Isaac (away), 2012-08-02 18:02:25]

Only the main builder is on CQ right now.  I will give plenty of clank-team
heads up before making the switch.

There's a couple issues for getting tests on the main waterfall -- won't be up
for a couple weeks.

android_try_tester is run by
http://build.chromium.org/p/tryserver.chromium/builders/android_tester

but this is inactive presently.

[bulach, 2012-08-02 18:06:02]

cool, thanks for the clarification! CQing it.. (the last try failure was on unit
tests, unrelated... the instrumentation ran just fine).

[commit-bot: I haz the power, 2012-08-02 18:06:28]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/bulach@chromium.org/10828094/5004

[commit-bot: I haz the power, 2012-08-02 19:45:47]

Change committed as 149679