Add CIPD GAE SDK recipe module and packager.

Add CIPD GAE SDK recipe module and packager.

Add the "gae_sdk" recipe module. This manages the checkout and
installation of CIPD-bundled Google AppEngine SDK packages for Python
and Go based on the current platform.

Add a packager recipe which generates those CIPD bundles and add it to
the "chromium.infra.cron" waterfall.

BUG=chromium:648635
TEST=local, expectations
  - Expectations look sane.
  - Ran "infra/gae_sdk_cipd_packager" locally and successfully generated
    and uploaded CIPD packages for all platforms.

Committed: https://chromium.googlesource.com/chromium/tools/build/+/7a6464e367517a4e6d02eaaa1445cbbc99fee6db

[dnj, 2016-09-20 16:15:03]

Description was changed from

==========
Add CIPD GAE SDK recipe module and packager.

BUG=None
TEST=expectations
==========

to

==========
Add CIPD GAE SDK recipe module and packager.

Add the "gae_sdk" recipe module. This manages the checkout and
installation of CIPD-bundled Google AppEngine SDK packages for Python
and Go based on the current platform.

Add a packager recipe which generates those CIPD bundles and add it to
the "chromium.infra.cron" waterfall.

BUG=chromium:648635
TEST=local, expectations
  - Expectations look sane.
  - Ran "infra/gae_sdk_cipd_packager" locally and successfully generated
    and uploaded CIPD packages for all platforms.
==========

[dnj, 2016-09-20 16:40:45]

dnj@chromium.org changed reviewers:
+ iannucci@chromium.org

[dnj, 2016-09-20 16:40:45]

PTAL! The continuous recipe works locally, haven't tested "fetch" yet, but
that's the thing we'd integrate into other scripts that want to check out the
SDK.

[dnj, 2016-09-27 18:19:52]

(ping)

[iannucci, 2016-09-30 21:05:21]

this looks ok, with the exception of code location... took me longer to review
because it's multiple logical changes bunched together :/. I was confused about
download_and_unpack for a while, because I thought it was intended for user
recipes (and thought that we were embedding a zip inside of a cipd package).

Don't need to split up the CL, but I think the non-user-facing code should not
live in that module (maybe a different module, but honestly I would really just
put it in the recipe, since it's only going to be used by that one recipe
anyway)

https://chromiumcodereview.appspot.com/2355483004/diff/20001/masters/master.c...
File masters/master.chromium.infra.cron/master.cfg (right):

https://chromiumcodereview.appspot.com/2355483004/diff/20001/masters/master.c...
masters/master.chromium.infra.cron/master.cfg:1: # -*- python -*-
This master/slave change should have been a separate CL

https://chromiumcodereview.appspot.com/2355483004/diff/20001/masters/master.c...
masters/master.chromium.infra.cron/master.cfg:232: hour=range(0, 24, 4),
so... once every 4 hours?

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
File scripts/slave/recipe_modules/gae_sdk/api.py (right):

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/gae_sdk/api.py:41: 32: '386',
hm... I didn't realize it was 386 instead of x86. That's an interesting choice.

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/gae_sdk/api.py:119: def download_and_unpack(self,
plat, arch, version):
can we move this to the packaging recipe? Otherwise users may be tempted to call
this directly.... I was initially confused and thought that this was intended to
be a user-facing interface.

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/gae_sdk/api.py:144: def fetch(self, plat, dst):
it should somehow be made clear that this is really intended to be the ONLY
relevant function for users in this module. All the rest is really private to
the recipe (I would probably recommend putting that logic in the recipe until we
have multiple recipes that actually need it).

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
File scripts/slave/recipe_modules/zip/api.py (right):

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/zip/api.py:1: # Copyright 2014 The Chromium
Authors. All rights reserved.
this zip change should have been a separate CL

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
File scripts/slave/recipe_modules/zip/resources/unzip.py (right):

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/zip/resources/unzip.py:65: quiet =
data.get('quiet', False)
this script will only ever be invoked by recipes. Please just have the recipe
emit this value explicitly instead of having an implicit default.

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
File scripts/slave/recipes/infra/gae_sdk_cipd_packager.py (right):

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipes/infra/gae_sdk_cipd_packager.py:1: # Copyright 2016 The
Chromium Authors. All rights reserved.
This recipe should have been a separate CL

[dnj, 2016-09-30 22:54:55]

https://chromiumcodereview.appspot.com/2355483004/diff/20001/masters/master.c...
File masters/master.chromium.infra.cron/master.cfg (right):

https://chromiumcodereview.appspot.com/2355483004/diff/20001/masters/master.c...
masters/master.chromium.infra.cron/master.cfg:1: # -*- python -*-
On 2016/09/30 21:05:21, iannucci wrote:
> This master/slave change should have been a separate CL

Acknowledged.

https://chromiumcodereview.appspot.com/2355483004/diff/20001/masters/master.c...
masters/master.chromium.infra.cron/master.cfg:232: hour=range(0, 24, 4),
On 2016/09/30 21:05:21, iannucci wrote:
> so... once every 4 hours?

It's a no-op if it fails, so I think that's not unreasonable.

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
File scripts/slave/recipe_modules/gae_sdk/api.py (right):

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/gae_sdk/api.py:41: 32: '386',
On 2016/09/30 21:05:21, iannucci wrote:
> hm... I didn't realize it was 386 instead of x86. That's an interesting
choice.

I've generally seen i386 / i686 vs. amd64 / x86_64. I think one is GNU and the
other is Intel or something.


... like assembly!

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/gae_sdk/api.py:119: def download_and_unpack(self,
plat, arch, version):
On 2016/09/30 21:05:21, iannucci wrote:
> can we move this to the packaging recipe? Otherwise users may be tempted to
call
> this directly.... I was initially confused and thought that this was intended
to
> be a user-facing interface.

Done.

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/gae_sdk/api.py:144: def fetch(self, plat, dst):
On 2016/09/30 21:05:21, iannucci wrote:
> it should somehow be made clear that this is really intended to be the ONLY
> relevant function for users in this module. All the rest is really private to
> the recipe (I would probably recommend putting that logic in the recipe until
we
> have multiple recipes that actually need it).

Done.

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
File scripts/slave/recipe_modules/zip/api.py (right):

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/zip/api.py:1: # Copyright 2014 The Chromium
Authors. All rights reserved.
On 2016/09/30 21:05:21, iannucci wrote:
> this zip change should have been a separate CL

Acknowledged.

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
File scripts/slave/recipe_modules/zip/resources/unzip.py (right):

https://chromiumcodereview.appspot.com/2355483004/diff/20001/scripts/slave/re...
scripts/slave/recipe_modules/zip/resources/unzip.py:65: quiet =
data.get('quiet', False)
On 2016/09/30 21:05:21, iannucci wrote:
> this script will only ever be invoked by recipes. Please just have the recipe
> emit this value explicitly instead of having an implicit default.

Done.

[iannucci, 2016-09-30 23:00:21]

lgtm

[dnj, 2016-09-30 23:21:08]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-09-30 23:21:13]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Vadim Sh., 2016-09-30 23:30:05]

vadimsh@chromium.org changed reviewers:
+ vadimsh@chromium.org

[Vadim Sh., 2016-09-30 23:30:06]

https://chromiumcodereview.appspot.com/2355483004/diff/40001/scripts/slave/re...
File scripts/slave/recipes/infra/gae_sdk_cipd_packager.expected/packages.json
(right):

https://chromiumcodereview.appspot.com/2355483004/diff/40001/scripts/slave/re...
scripts/slave/recipes/infra/gae_sdk_cipd_packager.expected/packages.json:166:
"[SLAVE_BUILD]/cipd/cipd",
you need to pass --service-account-json with path to the credentials file
deployed by puppet.

See other recipes.

[Vadim Sh., 2016-09-30 23:30:14]

The CQ bit was unchecked by vadimsh@chromium.org

[dnj, 2016-10-01 00:06:32]

oops, done. PTAL?

[Vadim Sh., 2016-10-01 00:30:46]

lgtm

[dnj, 2016-10-03 15:57:34]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-10-03 15:57:34]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://chromiumcodereview.appspot.com/2355483004/#ps60001
(title: "All expectations, use service account.")

[commit-bot: I haz the power, 2016-10-03 15:57:40]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-03 16:06:09]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-03 16:06:10]

Try jobs failed on following builders:
  Build Presubmit on luci.infra.try (JOB_FAILED,
https://luci-milo.appspot.com/swarming/task/31a41928504dd010)

[dnj, 2016-10-03 16:16:31]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-10-03 16:16:31]

The patchset sent to the CQ was uploaded after l-g-t-m from
vadimsh@chromium.org, iannucci@chromium.org
Link to the patchset: https://chromiumcodereview.appspot.com/2355483004/#ps80001
(title: "Regenerate :x")

[commit-bot: I haz the power, 2016-10-03 16:16:40]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-03 16:24:49]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-03 16:24:50]

Try jobs failed on following builders:
  Build Presubmit on luci.infra.try (JOB_FAILED,
https://luci-milo.appspot.com/swarming/task/31a426ef2bbb0010)

[dnj, 2016-10-04 15:34:56]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-10-04 15:34:57]

The patchset sent to the CQ was uploaded after l-g-t-m from
vadimsh@chromium.org, iannucci@chromium.org
Link to the patchset:
https://chromiumcodereview.appspot.com/2355483004/#ps100001 (title: "pylint")

[commit-bot: I haz the power, 2016-10-04 15:35:08]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-04 15:39:48]

Description was changed from

==========
Add CIPD GAE SDK recipe module and packager.

Add the "gae_sdk" recipe module. This manages the checkout and
installation of CIPD-bundled Google AppEngine SDK packages for Python
and Go based on the current platform.

Add a packager recipe which generates those CIPD bundles and add it to
the "chromium.infra.cron" waterfall.

BUG=chromium:648635
TEST=local, expectations
  - Expectations look sane.
  - Ran "infra/gae_sdk_cipd_packager" locally and successfully generated
    and uploaded CIPD packages for all platforms.
==========

to

==========
Add CIPD GAE SDK recipe module and packager.

Add the "gae_sdk" recipe module. This manages the checkout and
installation of CIPD-bundled Google AppEngine SDK packages for Python
and Go based on the current platform.

Add a packager recipe which generates those CIPD bundles and add it to
the "chromium.infra.cron" waterfall.

BUG=chromium:648635
TEST=local, expectations
  - Expectations look sane.
  - Ran "infra/gae_sdk_cipd_packager" locally and successfully generated
    and uploaded CIPD packages for all platforms.

Committed:
https://chromium.googlesource.com/chromium/tools/build/+/7a6464e367517a4e6d02...
==========

[commit-bot: I haz the power, 2016-10-04 15:39:49]

Committed patchset #6 (id:100001) as
https://chromium.googlesource.com/chromium/tools/build/+/7a6464e367517a4e6d02...