Buildbot enhancements: Builder/Tester split

Adding buildbot main scripts for fyi builders and testers.
I'm removing the experimental compiler step from the builder to
reduce zip size.
    

BUG=132707
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=143149

[John Grabowski, 2012-06-14 18:26:32]

Strongly feel you should have distinct entrypoint scripts.
Simple simple.

[Isaac (away), 2012-06-15 05:57:08]

On 2012/06/14 18:26:32, John Grabowski wrote:
> Strongly feel you should have distinct entrypoint scripts.
> Simple simple.

done.  ptal

[John Grabowski, 2012-06-18 18:20:04]

Overall like but...

https://chromiumcodereview.appspot.com/10533144/diff/11004/build/android/buil...
File build/android/buildbot_functions.sh (right):

https://chromiumcodereview.appspot.com/10533144/diff/11004/build/android/buil...
build/android/buildbot_functions.sh:29: local
script="${1-$SRC_ROOT}/build/install-build-deps-android-sdk.sh"
Args to this function say you must pass in a SRC_ROOT, not "if you want or we'll
figure it out".

Trying hard to not reference globals, especially since the test mode for this
script is "in your current shell environment".


Do you feel strongly?

https://chromiumcodereview.appspot.com/10533144/diff/11004/build/android/buil...
build/android/buildbot_functions.sh:47: export SRC_ROOT=${1-$SRC_ROOT}
ditto

[Isaac (away), 2012-06-18 18:59:46]

https://chromiumcodereview.appspot.com/10533144/diff/11004/build/android/buil...
File build/android/buildbot_functions.sh (right):

https://chromiumcodereview.appspot.com/10533144/diff/11004/build/android/buil...
build/android/buildbot_functions.sh:29: local
script="${1-$SRC_ROOT}/build/install-build-deps-android-sdk.sh"
On 2012/06/18 18:20:04, John Grabowski wrote:
> Args to this function say you must pass in a SRC_ROOT, not "if you want or
we'll
> figure it out".
> 
> Trying hard to not reference globals, especially since the test mode for this
> script is "in your current shell environment".
> 
> 
> Do you feel strongly?
> 

Was trying to keep backwards compatibility for scripts that didn't create
SRC_ROOT.  I can put back to before if you'd prefer.

The other issue I realized was that by defining SRC_ROOT in my fyi_tester.sh
script, I could affect the individual test functions.

[Isaac (away), 2012-06-19 04:32:47]

ptal

[John Grabowski, 2012-06-19 20:06:55]

LGTM

[cmp, 2012-06-19 20:46:42]

lgtm with nits, added a note about using *PROPERTIES

https://chromiumcodereview.appspot.com/10533144/diff/21001/build/android/buil...
File build/android/buildbot_functions.sh (right):

https://chromiumcodereview.appspot.com/10533144/diff/21001/build/android/buil...
build/android/buildbot_functions.sh:18: # and assign them global variable names
nit: please append periods to the end of all sentences in comments (here, line
68, line 72, 270, 276, 285).

https://chromiumcodereview.appspot.com/10533144/diff/21001/build/android/buil...
build/android/buildbot_functions.sh:23: FACTORY_PROPERTIES="$(echo "$1" | sed
's/^[^=]*=//')"
The content of FACTORY_PROPERTIES and BUILD_PROPERTIES has been designed by us
to be easily marshallable into a programming language that supports JSON
imports.  Python has loads() for example.  JS could eval it.  etc.  Shell
doesn't have these things, though.  Ideally, this script using the data would be
Python to harmonize with other parts of the system, but I understand a
conversion probably won't happen soon here.

So when it comes to actually using the contents of these *PROPERTIES values, it
may be worth adding a Python helper script that takes one of them on the
command-line as one arg and a key as a second arg for which value to extract. 
Then you can get back the string value of that key's entry in the dictionary. 
The caller here could look like:

  build_url=$(helper.py "$FACTORY_PROPERTIES" "build_url")

This would give you maximum flexibility: you won't need to do sh acrobatics to
dig out the right key value, you can reuse our methods for de-serializing, and
you can open the door to switching more parts of this over to Python as-needed.

[Isaac (away), 2012-06-20 00:34:53]

https://chromiumcodereview.appspot.com/10533144/diff/21001/build/android/buil...
File build/android/buildbot_functions.sh (right):

https://chromiumcodereview.appspot.com/10533144/diff/21001/build/android/buil...
build/android/buildbot_functions.sh:23: FACTORY_PROPERTIES="$(echo "$1" | sed
's/^[^=]*=//')"
Right now zip_build and extract_build already parse factory_properties and
build_properties properly.  While this seems like a good idea if I need to use
infomation from this variables in bash, I don't see the need to extract
information before passed the blobs to extract build.

Can you clarify the value in this?

On 2012/06/19 20:46:42, cmp wrote:
> The content of FACTORY_PROPERTIES and BUILD_PROPERTIES has been designed by us
> to be easily marshallable into a programming language that supports JSON
> imports.  Python has loads() for example.  JS could eval it.  etc.  Shell
> doesn't have these things, though.  Ideally, this script using the data would
be
> Python to harmonize with other parts of the system, but I understand a
> conversion probably won't happen soon here.
> 
> So when it comes to actually using the contents of these *PROPERTIES values,
it
> may be worth adding a Python helper script that takes one of them on the
> command-line as one arg and a key as a second arg for which value to extract. 
> Then you can get back the string value of that key's entry in the dictionary. 
> The caller here could look like:
> 
>   build_url=$(helper.py "$FACTORY_PROPERTIES" "build_url")
> 
> This would give you maximum flexibility: you won't need to do sh acrobatics to
> dig out the right key value, you can reuse our methods for de-serializing, and
> you can open the door to switching more parts of this over to Python
as-needed.

[Isaac (away), 2012-06-20 04:13:08]

OK I think this is good to go.  I've got a working test setup at levy.mtv:8011
(some tests are failing because it is running a new set of tests)

[commit-bot: I haz the power, 2012-06-20 04:15:04]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/ilevy@chromium.org/10533144/24002

[commit-bot: I haz the power, 2012-06-20 05:34:35]

Change committed as 143149

[cmp, 2012-06-20 17:30:06]

https://chromiumcodereview.appspot.com/10533144/diff/21001/build/android/buil...
File build/android/buildbot_functions.sh (right):

https://chromiumcodereview.appspot.com/10533144/diff/21001/build/android/buil...
build/android/buildbot_functions.sh:23: FACTORY_PROPERTIES="$(echo "$1" | sed
's/^[^=]*=//')"
On 2012/06/20 00:34:53, Isaac wrote:
> Right now zip_build and extract_build already parse factory_properties and
> build_properties properly.  While this seems like a good idea if I need to use
> infomation from this variables in bash, I don't see the need to extract
> information before passed the blobs to extract build.

Right, I didn't expect you to need to extract anything from it for this change,
but maybe in the future.

> Can you clarify the value in this?

It is just a note considering how to extract data from here in the future if
needed.  Feel free to ignore for now.

[cmp, 2012-06-20 17:30:47]

lgtm