Add build slave side support for the new swarming.py script.

Add build slave side support for the new swarming.py script.

R=ilevy@chromium.org
BUG=

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=219778

[M-A Ruel, 2013-08-20 20:22:35]

This needs to be committed before https://codereview.chromium.org/22980008, so
the exact revision can't be known ahead of time and can be easily updated with a
follow up docstring-only CL.

This is dependent on https://codereview.chromium.org/23176003.

https://codereview.chromium.org/22909021/diff/3001/scripts/slave/swarming/get...
File scripts/slave/swarming/get_swarm_results.py (right):

https://codereview.chromium.org/22909021/diff/3001/scripts/slave/swarming/get...
scripts/slave/swarming/get_swarm_results.py:133: return subprocess2.call(
In practice, this means https://codereview.chromium.org/22980008 should be fixed
to wrap the output nicely.

[M-A Ruel, 2013-08-26 18:14:05]

This CL was tested with:

1. Sync chromium to ToT
2. Put https://codereview.chromium.org/22980008 in place of tools/swarm_client.

ninja -C out/Release base_unittests_run

tools/swarm_client/isolate.py hashtable --isolated
out/Release/base_unittests.isolated -o https://isolateserver-dev.appspot.com

# Noted the sha1.

cd ..

PYTHONPATH=~/src/b/build/scripts
~/src/b/build/scripts/slave/swarming/trigger_swarm_shim.py --swarming
https://chromium-swarm-dev.appspot.com --isolate-server
https://isolateserver-dev.appspot.com --task-prefix=yummy --type fyi --task
3a2cc2ec85689cfcca8a73ff32506a4cfa67869d base_unittests 1 '*'

PYTHONPATH=~/src/b/build/scripts
~/src/b/build/scripts/slave/swarming/get_swarm_results_shim.py --swarming
https://chromium-swarm-dev.appspot.com yummybase_unittests

Then I tried but by intentionally breaking a test and then forcing 2 shards.


... aaaand I found bugs, fixed in patchset #5!

The important things were verified to work:
1. the data was properly streamed.
2. the test case summary was properly printed.

Note that swarming.py has no understanding of googletest test cases, it's all
driven by get_swarm_results_shim.py.

[Isaac (away), 2013-08-27 01:35:20]

lgtm w/ comment

https://codereview.chromium.org/22909021/diff/12001/scripts/slave/swarming/sw...
File scripts/slave/swarming/swarming_utils.py (right):

https://codereview.chromium.org/22909021/diff/12001/scripts/slave/swarming/sw...
scripts/slave/swarming/swarming_utils.py:46: return map(int, version.split('.'))
This returns a generator -- convert to a list?

[M-A Ruel, 2013-08-27 12:23:37]

thanks

https://codereview.chromium.org/22909021/diff/12001/scripts/slave/swarming/sw...
File scripts/slave/swarming/swarming_utils.py (right):

https://codereview.chromium.org/22909021/diff/12001/scripts/slave/swarming/sw...
scripts/slave/swarming/swarming_utils.py:46: return map(int, version.split('.'))
On 2013/08/27 01:35:20, Isaac wrote:
> This returns a generator -- convert to a list?

http://docs.python.org/2/library/functions.html#map
"The iterable arguments may be a sequence or any iterable object; the result is
always a list."

Note that I could have used:
return [int(i) for i in version.split('.')]
but it's longer to type. :) I don't really mind either way.

[commit-bot: I haz the power, 2013-08-27 12:23:48]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/maruel@chromium.org/22909021/12001

[commit-bot: I haz the power, 2013-08-27 12:24:35]

Presubmit check for 22909021-12001 failed and returned exit status 1.

Running presubmit commit checks ...

** Presubmit ERRORS **
tests/masters_test.py failed
1 masters failed:
master.chromium.webrtc.fyi
49 masters succeeded, 1 failed, 1 skipped in 27.3s.

=== Error running master.chromium.webrtc.fyi === 
2013-08-27 05:24:10-0700 [-] Setting up http.log rotating 10 files of 1000000
bytes each
2013-08-27 05:24:10-0700 [-] WebStatus using
(/b/commit-queue/workdir/tools/build/masters/master.chromium/public_html)
2013-08-27 05:24:10-0700 [-] removing 0 old schedulers, updating 0, and adding
12
2013-08-27 05:24:10-0700 [-] cros_every_4_hours_scheduler: missed scheduled
build time, so building immediately
2013-08-27 05:24:11-0700 [-] mac_every_4_hours_scheduler: missed scheduled build
time, so building immediately
2013-08-27 05:24:11-0700 [-] added buildset 1 to database
2013-08-27 05:24:11-0700 [-] added buildset 2 to database
2013-08-27 05:24:15-0700 [-] win_every_4_hours_scheduler: missed scheduled build
time, so building immediately
2013-08-27 05:24:15-0700 [-] added buildset 3 to database
2013-08-27 05:24:15-0700 [-] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.internet.defer.FirstError: FirstError[#7, [Failure instance:
Traceback: <class 'sqlalchemy.exc.OperationalError'>: (OperationalError)
database is locked u'INSERT INTO schedulers (name, state, class_name) VALUES (?,
?, ?)' ('mac_webrtc_stable_scheduler', '{}',
'buildbot.schedulers.basic.Scheduler')
	/usr/lib/python2.7/threading.py:524:__bootstrap
	/usr/lib/python2.7/threading.py:551:__bootstrap_inner
	/usr/lib/python2.7/threading.py:504:run
	--- <exception caught here> ---
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/threadpool.py:207:_worker
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/context.py:59:callWithContext
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/context.py:37:callWithContext
	/b/commit-queue/workdir/tools/build/third_party/buildbot_8_4p1/buildbot/db/pool.py:112:thd
	/b/commit-queue/workdir/tools/build/third_party/buildbot_8_4p1/buildbot/db/schedulers.py:165:thd
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1358:execute
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1491:_execute_clauseelement
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1599:_execute_context
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1592:_execute_context
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/default.py:325:do_execute
	]]



Presubmit checks took 40.9s to calculate.

[commit-bot: I haz the power, 2013-08-27 12:42:08]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/maruel@chromium.org/22909021/12001

[commit-bot: I haz the power, 2013-08-27 12:43:14]

Change committed as 219778

[Isaac (use chromium), 2013-08-27 17:43:00]

https://chromiumcodereview.appspot.com/22909021/diff/12001/scripts/slave/swar...
File scripts/slave/swarming/swarming_utils.py (right):

https://chromiumcodereview.appspot.com/22909021/diff/12001/scripts/slave/swar...
scripts/slave/swarming/swarming_utils.py:46: return map(int, version.split('.'))
Hmm, I must be remembering correctly.  Weird.

I use map for situations like this too.