Teach build.py to cross-compile go-based packages.

Teach build.py to cross-compile go-based packages.

build.py now understands GOOS and GOARCH env vars and builds the packages for
corresponding target platform. Packages that support cross-compilation must be
declared via 'supports_cross_compilation' attribute in package definition yaml.

R=nodir@chromium.org
BUG=

Committed: https://chromium.googlesource.com/infra/infra/+/43ed7bea9bd3195083779ab566b5b64ab3b6b058

[Vadim Sh., 2016-06-25 03:31:33]

Patchset #1 (id:1) has been deleted

[Vadim Sh., 2016-06-25 03:40:26]

Patchset #1 (id:20001) has been deleted

[Vadim Sh., 2016-06-25 03:51:04]

PTAL. Tested manually with different combinations of GOOS and GOARCH.

Example of execution: https://paste.googleplex.com/6177740315164672

https://codereview.chromium.org/2095173002/diff/40001/build/build.py
File build/build.py (right):

https://codereview.chromium.org/2095173002/diff/40001/build/build.py#newcode79
build/build.py:79: if builder and builders and builder not in builders:
this is existing logic (moved from 'should_process_on_builder' function that no
longer exists).

https://codereview.chromium.org/2095173002/diff/40001/build/build.py#newcode83
build/build.py:83: if is_cross_compiling():
this is new

https://codereview.chromium.org/2095173002/diff/40001/build/build.py#newcode166
build/build.py:166: def hacked_workspace(go_workspace, goos=None, goarch=None):
this is mostly existing logic, moved into separate function

(GOOS and GOARCH modifications are new)

https://codereview.chromium.org/2095173002/diff/40001/build/build.py#newcode394
build/build.py:394: # We use 'mac' instead of 'darwin'.
I can't remember why :(

https://codereview.chromium.org/2095173002/diff/40001/build/build.py#newcode544
build/build.py:544: # To avoid rebuilding cipd client all the time, we cache it
in out/*, using
this is the most hacky part of this CL

we need CIPD client for the host platform, somewhere outside of go/bin/* (which
is cleared at each rebuild). And compiling it each time is sort of slow. So use
approximate list of its dependencies to come up with a cache key.

https://codereview.chromium.org/2095173002/diff/40001/build/build.py#newcode554
build/build.py:554: os.path.join(ROOT, 'go', 'bootstrap.py'),
indirectly defines stuff like Go toolset version

https://codereview.chromium.org/2095173002/diff/40001/build/build.py#newcode674
build/build.py:674: # Remove old build artifacts to avoid stale files in case
the script crashes
it used to remove everything under out/*, now build.py can be called multiple
times with different combinations of GOOS and GOARCH. Each invocation will add
more stuff into out/*, so we can't clear it entirely anymore.

https://codereview.chromium.org/2095173002/diff/40001/build/packages/cipd_cli...
File build/packages/cipd_client.yaml (right):

https://codereview.chromium.org/2095173002/diff/40001/build/packages/cipd_cli...
build/packages/cipd_client.yaml:9: supports_cross_compilation: true
I'll add other ones when they are needed

https://codereview.chromium.org/2095173002/diff/40001/build/test_packages.py
File build/test_packages.py (left):

https://codereview.chromium.org/2095173002/diff/40001/build/test_packages.py#...
build/test_packages.py:102: paths = glob.glob(os.path.join(package_out_dir,
'*.cipd'))
this method no longer works, since package_out_dir now may contain packages for
some other platform that the host can't test (since it can't run code in them).

[Vadim Sh., 2016-06-25 06:19:47]

Added some more stuff:
 * Support ARM as a host too (being careful with what instruction set is being
used, we need armv6l even if running on armv7l).
 * Detect 32-bit userland on 64-bit Linux.
 * Add a check that Go toolset builds what we expect it to build (e.g. 32-bit
binaries on Linux with 32-bit userland).

[nodir, 2016-06-26 05:00:48]

https://codereview.chromium.org/2095173002/diff/210001/build/README.md
File build/README.md (right):

https://codereview.chromium.org/2095173002/diff/210001/build/README.md#newcod...
build/README.md:104: `GOBIN`.
cool, it won't compile everything
an improvement

https://codereview.chromium.org/2095173002/diff/210001/build/build.py
File build/build.py (right):

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode64
build/build.py:64: return self.pkg_def.get('uses_python_env')
consider `return bool(...)` here like in other methods

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode75
build/build.py:75: package definition file.
the

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode83
build/build.py:83: # If cross compiling, pick only packages that supports cross
compilation.
support

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode91
build/build.py:91: """Returns true if using GOOS or GOARCH env vars.
True

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode222
build/build.py:222: for k in [key for key in os.environ if key not in
orig_environ]:
fwiu this is complicated because you don't want to iterate and mutate os.environ
at the same time
I think it would be easier if you iterated os.environ.keys

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode248
build/build.py:248: env[k] = v.strip('"')
strictly speaking v may end with \"
Maybe do that only if starts with "

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode270
build/build.py:270: ] + list(packages),
this may hit cmd length limit on win

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode279
build/build.py:279: package: package to build.
document go_workspace

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode307
build/build.py:307: shutil.rmtree(os.path.join(go_workspace, 'bin'),
ignore_errors=True)
is this necessary? fwiu if I want infra/go/bin/foo to exist and I happen to need
to run `build.py bar.yaml` it will delete foo?
and if I run `build.py` I have to have foo.yaml for infra/go/bin/foo to survive?

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode364
build/build.py:364: abs_path = os.path.join(package_def_dir, name)
not necessarily absolute though

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode416
build/build.py:416: """Returns a dict with variables that describe cross
compilation target env.
cross-compilation
here and other places

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode420
build/build.py:420: It contains only 'platform' and 'exe_suffix'.
s/It/Returned dict/

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode435
build/build.py:435: if goarm == '6':
invert and remove else

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode712
build/build.py:712: 
check that GOARM env var is '6' or None

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode718
build/build.py:718: tags.append('build_host_fqdn:' + socket.getfqdn())
maybe only first component? full name may contain something we don't want to
expose. shrug

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode775
build/build.py:775: if go_env['GOHOSTARCH'] != expected_arch:
GOHOST* vars are hardly documented. Can minimally explain them here?

[Vadim Sh., 2016-06-27 21:37:29]

https://codereview.chromium.org/2095173002/diff/210001/build/build.py
File build/build.py (right):

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode64
build/build.py:64: return self.pkg_def.get('uses_python_env')
On 2016/06/26 05:00:47, nodir wrote:
> consider `return bool(...)` here like in other methods

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode75
build/build.py:75: package definition file.
On 2016/06/26 05:00:47, nodir wrote:
> the

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode83
build/build.py:83: # If cross compiling, pick only packages that supports cross
compilation.
On 2016/06/26 05:00:47, nodir wrote:
> support

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode91
build/build.py:91: """Returns true if using GOOS or GOARCH env vars.
On 2016/06/26 05:00:47, nodir wrote:
> True

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode222
build/build.py:222: for k in [key for key in os.environ if key not in
orig_environ]:
On 2016/06/26 05:00:47, nodir wrote:
> fwiu this is complicated because you don't want to iterate and mutate
os.environ
> at the same time
> I think it would be easier if you iterated os.environ.keys

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode248
build/build.py:248: env[k] = v.strip('"')
On 2016/06/26 05:00:47, nodir wrote:
> strictly speaking v may end with \"
> Maybe do that only if starts with "

Done. It can also be different on Windows (and plan9, haha). Changed this code a
bit. https://github.com/golang/go/blob/master/src/cmd/go/env.go

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode270
build/build.py:270: ] + list(packages),
On 2016/06/26 05:00:47, nodir wrote:
> this may hit cmd length limit on win

Acknowledged. Not a problem in the near future though.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode279
build/build.py:279: package: package to build.
On 2016/06/26 05:00:48, nodir wrote:
> document go_workspace

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode307
build/build.py:307: shutil.rmtree(os.path.join(go_workspace, 'bin'),
ignore_errors=True)
On 2016/06/26 05:00:47, nodir wrote:
> is this necessary? fwiu if I want infra/go/bin/foo to exist and I happen to
need
> to run `build.py bar.yaml` it will delete foo?
> and if I run `build.py` I have to have foo.yaml for infra/go/bin/foo to
survive?

Using "go clean" now and not passing '-a' ("rebuild") to "go install" and "go
build". I think it works correctly with multiple GOPATHs. And it cuts total
build time when cross-compiling since now stdlib and common dependencies are
built only once.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode364
build/build.py:364: abs_path = os.path.join(package_def_dir, name)
On 2016/06/26 05:00:47, nodir wrote:
> not necessarily absolute though

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode416
build/build.py:416: """Returns a dict with variables that describe cross
compilation target env.
On 2016/06/26 05:00:48, nodir wrote:
> cross-compilation
> here and other places

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode420
build/build.py:420: It contains only 'platform' and 'exe_suffix'.
On 2016/06/26 05:00:47, nodir wrote:
> s/It/Returned dict/

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode435
build/build.py:435: if goarm == '6':
On 2016/06/26 05:00:47, nodir wrote:
> invert and remove else

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode712
build/build.py:712: 
On 2016/06/26 05:00:47, nodir wrote:
> check that GOARM env var is '6' or None

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode718
build/build.py:718: tags.append('build_host_fqdn:' + socket.getfqdn())
On 2016/06/26 05:00:47, nodir wrote:
> maybe only first component? full name may contain something we don't want to
> expose. shrug

Done.

https://codereview.chromium.org/2095173002/diff/210001/build/build.py#newcode775
build/build.py:775: if go_env['GOHOSTARCH'] != expected_arch:
On 2016/06/26 05:00:47, nodir wrote:
> GOHOST* vars are hardly documented. Can minimally explain them here?

IMHO, it's pretty obvious what they mean.

[nodir, 2016-06-27 22:41:56]

lgtm

https://codereview.chromium.org/2095173002/diff/230001/build/build.py
File build/build.py (right):

https://codereview.chromium.org/2095173002/diff/230001/build/build.py#newcode277
build/build.py:277: # Above command is either silent (with '-x') or too verbose
(with '-x').
did you mean it is silent without -x?

https://codereview.chromium.org/2095173002/diff/230001/build/build.py#newcode279
build/build.py:279: print '<nothing to see here>'
'Done'?

[Vadim Sh., 2016-06-27 22:54:00]

Also running "go install std" now before building packages in cross-compiling
mode (otherwise stdlib is rebuilt each time).

https://codereview.chromium.org/2095173002/diff/230001/build/build.py
File build/build.py (right):

https://codereview.chromium.org/2095173002/diff/230001/build/build.py#newcode277
build/build.py:277: # Above command is either silent (with '-x') or too verbose
(with '-x').
On 2016/06/27 22:41:56, nodir wrote:
> did you mean it is silent without -x?

Done.

https://codereview.chromium.org/2095173002/diff/230001/build/build.py#newcode279
build/build.py:279: print '<nothing to see here>'
On 2016/06/27 22:41:56, nodir wrote:
> 'Done'?

Done.

[Vadim Sh., 2016-06-27 22:54:42]

The CQ bit was checked by vadimsh@chromium.org

[Vadim Sh., 2016-06-27 22:54:42]

The patchset sent to the CQ was uploaded after l-g-t-m from nodir@chromium.org
Link to the patchset: https://codereview.chromium.org/2095173002/#ps250001
(title: "prebuild stdlib")

[commit-bot: I haz the power, 2016-06-27 22:54:56]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-06-27 23:16:21]

Description was changed from

==========
Teach build.py to cross-compile go-based packages.

build.py now understands GOOS and GOARCH env vars and builds the packages for
corresponding target platform. Packages that support cross-compilation must be
declared via 'supports_cross_compilation' attribute in package definition yaml.

R=nodir@chromium.org
BUG=
==========

to

==========
Teach build.py to cross-compile go-based packages.

build.py now understands GOOS and GOARCH env vars and builds the packages for
corresponding target platform. Packages that support cross-compilation must be
declared via 'supports_cross_compilation' attribute in package definition yaml.

R=nodir@chromium.org
BUG=

Committed:
https://chromium.googlesource.com/infra/infra/+/43ed7bea9bd3195083779ab566b5b...
==========

[commit-bot: I haz the power, 2016-06-27 23:16:22]

Committed patchset #12 (id:250001) as
https://chromium.googlesource.com/infra/infra/+/43ed7bea9bd3195083779ab566b5b...