Selective build clobbering feature (compile.py).

Selective build clobbering feature (compile.py).

Adds the ability for devs/troopers/etc. to set 'landmines' in the tree so that
the build will selectively clobber when a builder moves over a revision with such
a change.

This cl hooks the clobber mechanism to the compile.py build script.

The related cl for android's build scripts and an initial landmines.py is here:
https://chromiumcodereview.appspot.com/11175016/

R=cmp,maruel@chromium.org
BUG=121897
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=167551

[iannucci, 2012-11-05 21:58:07]

PTAL. This is the part which actually implements the clobbering for our
non-android builds.

[M-A Ruel, 2012-11-06 15:20:01]

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
File scripts/slave/compile.py (right):

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:562: clang_dir =
os.path.abspath(os.path.join(options.src_dir,
I guess adding "options.src_dir = os.path.abspath(options.src_dir)" in main
would be a good idea.

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:790: new_build = open(manifest+'.goma', 'w')
spacing

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1074: 
2 lines between file-level symbols

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1078: print "Setting clobber due to triggered
landmines:"
The file use single-quotes

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1080: sys.stdout.writelines(f.readlines())
You are doing that because you want to strip CR, right?

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1084: def get_target_build_dir(build_tool, src_dir,
target, is_iphone=False):
"""Keep in sync with foo_bar.py"""

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1183: 
options.build_dir = os.path.abspath(options.build_dir) ?

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1186: options.target_output_dir =
get_target_build_dir(options.build_tool,
This alignment hurts. Please everything at +4 or at (.

[iannucci, 2012-11-13 20:09:51]

All fixed (with new patch, too :D). PTAL. I rebased the other patch too, so if
you lgtm this, I'll cq them both.

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
File scripts/slave/compile.py (right):

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:562: clang_dir =
os.path.abspath(os.path.join(options.src_dir,
On 2012/11/06 15:20:01, Marc-Antoine Ruel wrote:
> I guess adding "options.src_dir = os.path.abspath(options.src_dir)" in main
> would be a good idea.

I believe this is already essentially implemented in main. I'll take out all the
unnecessary abspaths.

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:790: new_build = open(manifest+'.goma', 'w')
On 2012/11/06 15:20:01, Marc-Antoine Ruel wrote:
> spacing

Done.

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1074: 
On 2012/11/06 15:20:01, Marc-Antoine Ruel wrote:
> 2 lines between file-level symbols

Done.

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1078: print "Setting clobber due to triggered
landmines:"
On 2012/11/06 15:20:01, Marc-Antoine Ruel wrote:
> The file use single-quotes

Done.

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1080: sys.stdout.writelines(f.readlines())
On 2012/11/06 15:20:01, Marc-Antoine Ruel wrote:
> You are doing that because you want to strip CR, right?

Doing this to print the file as-is (and it's more efficient than
write(f.read())). It will only ever be generated on the same system, so the line
endings should always match.

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1084: def get_target_build_dir(build_tool, src_dir,
target, is_iphone=False):
On 2012/11/06 15:20:01, Marc-Antoine Ruel wrote:
> """Keep in sync with foo_bar.py"""

Done. I also updated the other function to refer to this file.

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1183: 
On 2012/11/06 15:20:01, Marc-Antoine Ruel wrote:
> options.build_dir = os.path.abspath(options.build_dir) ?

Sure, why not! :D

Would be useful for optparse to have a 'path' type which did this
automatically...

https://chromiumcodereview.appspot.com/11234013/diff/6001/scripts/slave/compi...
scripts/slave/compile.py:1186: options.target_output_dir =
get_target_build_dir(options.build_tool,
On 2012/11/06 15:20:01, Marc-Antoine Ruel wrote:
> This alignment hurts. Please everything at +4 or at (.

Done.

[M-A Ruel, 2012-11-13 21:40:57]

lgtm

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/comp...
File scripts/slave/compile.py (right):

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/comp...
scripts/slave/compile.py:745: command = ['ninja', '-C',
options.target_output_dir]
maybe relpath? I guess it'll work an absolute path anyway.

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/comp...
scripts/slave/compile.py:1077: sys.stdout.writelines(f.readlines())
I fail to see how better it is from:
print f.read()

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/zip_...
File scripts/slave/zip_build.py (right):

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/zip_...
scripts/slave/zip_build.py:136: all_platforms = ['.landmines']
You could add 'obj' and 'lib' to this list and remove from each of the
condition.

[iannucci, 2012-11-13 22:04:23]

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/comp...
File scripts/slave/compile.py (right):

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/comp...
scripts/slave/compile.py:745: command = ['ninja', '-C',
options.target_output_dir]
On 2012/11/13 21:40:57, Marc-Antoine Ruel wrote:
> maybe relpath? I guess it'll work an absolute path anyway.

absolute is fine. it's just doing a chdir ->
https://github.com/martine/ninja/blob/561780e6951f3ce8db44e10b3b8b072755ec8a4...

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/comp...
scripts/slave/compile.py:1077: sys.stdout.writelines(f.readlines())
On 2012/11/13 21:40:57, Marc-Antoine Ruel wrote:
> I fail to see how better it is from:
> print f.read()

Actually this is a fair point. I thought that readlines() returned a generator,
but it actually returns a list of strings (bleah). I'll fix.

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/zip_...
File scripts/slave/zip_build.py (right):

https://chromiumcodereview.appspot.com/11234013/diff/10001/scripts/slave/zip_...
scripts/slave/zip_build.py:136: all_platforms = ['.landmines']
On 2012/11/13 21:40:57, Marc-Antoine Ruel wrote:
> You could add 'obj' and 'lib' to this list and remove from each of the
> condition.

Done.

[commit-bot: I haz the power, 2012-11-14 01:10:40]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/iannucci@chromium.org/11234013/16001

[commit-bot: I haz the power, 2012-11-14 01:12:33]

Change committed as 167551

[Michael Starzinger, 2012-11-14 08:57:05]

https://chromiumcodereview.appspot.com/11234013/diff/16001/scripts/slave/comp...
File scripts/slave/compile.py (right):

https://chromiumcodereview.appspot.com/11234013/diff/16001/scripts/slave/comp...
scripts/slave/compile.py:713: ' in use any more. Asserting from orbit. It\'s the
only way to be sure')
Yes, we (the V8 waterfall) are using this feature and jkummerow@ explicitly
introduced that when we switched to the GYP build. Our builders just turned red.
I propose to revert this part of the change (didn't look at the rest).

http://build.chromium.org/p/client.v8/console

[Jakob Kummerow, 2012-11-14 09:12:05]

https://chromiumcodereview.appspot.com/11234013/diff/16001/scripts/slave/comp...
File scripts/slave/compile.py (right):

https://chromiumcodereview.appspot.com/11234013/diff/16001/scripts/slave/comp...
scripts/slave/compile.py:1097: raise NotImplementedError()
This gets hit by V8 Win builders:
http://build.chromium.org/p/client.v8/builders/V8%20Win32/builds/4630/steps/c...
We're setting --build_tool=vs as is required by the build_tool_map in line 1172
below.