Add buildbot state machine and master_manager tool.

Add buildbot state machine and master_manager tool.

Depends on the following CLs:

https://codereview.chromium.org/1094253002/
https://codereview.chromium.org/1096683003/
https://codereview.chromium.org/1097233002/

BUG=479059
R=iannucci@chromium.org

Committed: https://chromium.googlesource.com/infra/infra/+/5671cc4c4675271767bf2eef050769367c19ff2b

[ghost stip (do not use), 2015-04-24 07:06:55]

ptal. this is the 'last' CL in the chain to building up a functional tool.

if this CL is too large I can happily break it down, but the master manager tool
and buildbot state machine seemed like two great tastes that taste great
together

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
File infra/tools/master_manager/__main__.py (right):

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:20: def parse_args():  # pragma: no cover
this file basically has no unittests. I'm not sure what the value in adding
tests to it would be, but I can if you think it necessary

[ghost stip (do not use), 2015-04-24 07:07:15]

stip@chromium.org changed reviewers:
+ agable@chromium.org, iannucci@chromium.org

[ghost stip (do not use), 2015-04-24 07:07:30]

(again, with reviewers this time)

[agable, 2015-04-27 20:23:08]

https://codereview.chromium.org/1108523002/diff/1/infra/services/master_lifec...
File infra/services/master_lifecycle/buildbot_state.py (right):

https://codereview.chromium.org/1108523002/diff/1/infra/services/master_lifec...
infra/services/master_lifecycle/buildbot_state.py:23: 'desired_buildbot_state':
[
As noted in the comments below, the desired states don't make much sense. It
seems like there should be a common set of states that a buildbot can be in
(offline, starting, running, draining, drained, error/other), and that should be
the *same* as the set of possible future desired states. It doesn't make sense
to me to have disjoint sets of "current" states and "desired" states.

https://codereview.chromium.org/1108523002/diff/1/infra/services/master_lifec...
infra/services/master_lifecycle/buildbot_state.py:25: 'future',
I have no earthly idea what "desired buildbot state: future" could possibly
mean.

https://codereview.chromium.org/1108523002/diff/1/infra/services/master_lifec...
infra/services/master_lifecycle/buildbot_state.py:26: 'needs_reboot',
Desired state: needs reboot? I would like this buildbot to require a reboot.
What?

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
File infra/tools/master_manager/__main__.py (right):

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:26: help='location of the master to
manage')
Um

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:30: help='perform a gclient sync before
each master start')
Before each? Before any? At least once during the whole process?

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:33: help='perform a gclient sync before
each master start')
Um

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:33: help='perform a gclient sync before
each master start')
Um

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:35: help='actually run commands instead
of printing them.')
Note loudly in the --help that this script runs in dry-run mode *by default*

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:51: def run_state_machine_pass(
Seems like this function could be tested with a fake checkout (like the other
CL), a mock of the accepting_builds json endpoint, and a bunch of testing
desired_state.json files.

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:74: print 'current state', state
Use infra.libs.logs

[ghost stip (do not use), 2015-04-29 01:30:33]

addressed comments, ptal!

https://codereview.chromium.org/1108523002/diff/1/infra/services/master_lifec...
File infra/services/master_lifecycle/buildbot_state.py (right):

https://codereview.chromium.org/1108523002/diff/1/infra/services/master_lifec...
infra/services/master_lifecycle/buildbot_state.py:23: 'desired_buildbot_state':
[
On 2015/04/27 20:23:08, agable wrote:
> As noted in the comments below, the desired states don't make much sense. It
> seems like there should be a common set of states that a buildbot can be in
> (offline, starting, running, draining, drained, error/other), and that should
be
> the *same* as the set of possible future desired states. It doesn't make sense
> to me to have disjoint sets of "current" states and "desired" states.

After online discussion, adjusted to three dimensions: buildbot,
desired_buildbot_state, and desired_transition_time_utc.

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
File infra/tools/master_manager/__main__.py (right):

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:26: help='location of the master to
manage')
On 2015/04/27 20:23:08, agable wrote:
> Um

Acknowledged.

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:30: help='perform a gclient sync before
each master start')
On 2015/04/27 20:23:08, agable wrote:
> Before each? Before any? At least once during the whole process?

Done.

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:33: help='perform a gclient sync before
each master start')
On 2015/04/27 20:23:08, agable wrote:
> Um

Acknowledged.

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:35: help='actually run commands instead
of printing them.')
On 2015/04/27 20:23:08, agable wrote:
> Note loudly in the --help that this script runs in dry-run mode *by default*

Done.

https://codereview.chromium.org/1108523002/diff/1/infra/tools/master_manager/...
infra/tools/master_manager/__main__.py:74: print 'current state', state
On 2015/04/27 20:23:08, agable wrote:
> Use infra.libs.logs

Done.

[iannucci, 2015-04-29 01:55:12]

lgtm

https://codereview.chromium.org/1108523002/diff/40001/infra/services/master_l...
File infra/services/master_lifecycle/buildbot_state.py (right):

https://codereview.chromium.org/1108523002/diff/40001/infra/services/master_l...
infra/services/master_lifecycle/buildbot_state.py:31: 'future_transition',
ready, done, future

https://codereview.chromium.org/1108523002/diff/40001/infra/services/master_l...
infra/services/master_lifecycle/buildbot_state.py:59: @matchlist.add_match(
what about a

@matchlist.add_match(
  desired_transition_time="future")

?

https://codereview.chromium.org/1108523002/diff/40001/infra/services/master_l...
infra/services/master_lifecycle/buildbot_state.py:62:
exclusions={'desired_transition_time_utc': ['transition_hasnt_happened']})
what about

  desired_transition_time_utc=matchlist.NOT('transition_hasnt_happened')

? That would read a lot more cleanly than the exclusions thing

https://codereview.chromium.org/1108523002/diff/40001/infra/services/master_l...
infra/services/master_lifecycle/buildbot_state.py:138: if
data['accepting_builds'] is None:
later: let's invert this and call it 'draining_builds' since that's really more
like what it indicates.

[ghost stip (do not use), 2015-04-29 02:49:00]

The CQ bit was checked by stip@chromium.org

[ghost stip (do not use), 2015-04-29 02:49:00]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://chromiumcodereview.appspot.com/1108523002/#ps60001
(title: "Address iannucci's final comments.")

[commit-bot: I haz the power, 2015-04-29 02:49:18]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1108523002/60001

[ghost stip (do not use), 2015-04-29 02:55:29]

Committed patchset #4 (id:60001) manually as
5671cc4c4675271767bf2eef050769367c19ff2b (presubmit successful).

[agable, 2015-04-30 21:23:30]

https://codereview.chromium.org/1108523002/diff/60001/infra/services/master_l...
File infra/services/master_lifecycle/buildbot_state.py (right):

https://codereview.chromium.org/1108523002/diff/60001/infra/services/master_l...
infra/services/master_lifecycle/buildbot_state.py:14: STATES = {
Love it, this set of states is great.