Added virtualenv for depot_tools

Added virtualenv for depot_tools

R=pgervais@chromium.org
BUG=503067
TEST=tested on Mac, Windows and Linux bots

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=295811

[Sergiy Byelozyorov, 2015-06-22 17:22:13]

I've mostly copied the content of the bootstrap dir from infra, except that in
bootstrap/bootstrap.py I've added a block on lines 197-205. PTAL

[Sergiy Byelozyorov, 2015-06-22 17:26:33]

https://codereview.chromium.org/1200843003/diff/120001/bootstrap/bootstrap.py
File bootstrap/bootstrap.py (right):

https://codereview.chromium.org/1200843003/diff/120001/bootstrap/bootstrap.py...
bootstrap/bootstrap.py:197: # Create bin\python.bat on Windows to unify path
where Python is found.
This block below is new and wan't present in infra/bootstrap. This is needed
because python.exe is located in Scripts/ folder as opposed to bin/ on Mac and
Linux. Robbie has figured out that this was necessary:
https://codereview.chromium.org/479583002,
https://virtualenv.pypa.io/en/latest/userguide.html#windows-notes.

To still allow reach the Python in bin/python, I create a batch file here and
references the one in Scripts. I've tested it on a Windows machine and it works.

[Sergiy Byelozyorov, 2015-06-22 17:29:34]

I haven't tested this on Mac yet, but will do so tomorrow.

[iannucci, 2015-06-22 17:44:20]

On 2015/06/22 17:29:34, Sergiy Byelozyorov wrote:
> I haven't tested this on Mac yet, but will do so tomorrow.

Isn't this what glyco is supposed to be for?

[pgervais, 2015-06-22 17:50:37]

On 2015/06/22 17:44:20, iannucci wrote:
> On 2015/06/22 17:29:34, Sergiy Byelozyorov wrote:
> > I haven't tested this on Mac yet, but will do so tomorrow.
> 
> Isn't this what glyco is supposed to be for?

Yes this is what Glyco is for. We're just anticipating here. I suppose this copy
of bootstrap will be dropped as soon as Glyco is available.

[iannucci, 2015-06-22 17:52:28]

On 2015/06/22 17:50:37, pgervais wrote:
> On 2015/06/22 17:44:20, iannucci wrote:
> > On 2015/06/22 17:29:34, Sergiy Byelozyorov wrote:
> > > I haven't tested this on Mac yet, but will do so tomorrow.
> > 
> > Isn't this what glyco is supposed to be for?
> 
> Yes this is what Glyco is for. We're just anticipating here. I suppose this
copy
> of bootstrap will be dropped as soon as Glyco is available.

Okay, just checking :)

[pgervais, 2015-06-22 18:56:06]

I think you should trim down the bootstrap directory to leave only code used to
generate the virtualenv, and drop all that is related to generating and
uploading packages. So far the source of truth is infra.git and it should stay
this way until we have Glyco and a better process to generate packages.

[Dirk Pranke, 2015-06-22 20:46:24]

On 2015/06/22 18:56:06, pgervais wrote:
> I think you should trim down the bootstrap directory to leave only code used
to
> generate the virtualenv, and drop all that is related to generating and
> uploading packages. So far the source of truth is infra.git and it should stay
> this way until we have Glyco and a better process to generate packages.

What are the implications (if any), of using virtualenv in depot_tools? I don't
think I understand the rationale for this change
(I'm not necessarily against it, just don't know what's driving the change ...).

[pgervais, 2015-06-22 22:59:11]

On 2015/06/22 20:46:24, Dirk Pranke wrote:
> On 2015/06/22 18:56:06, pgervais wrote:
> > I think you should trim down the bootstrap directory to leave only code used
> to
> > generate the virtualenv, and drop all that is related to generating and
> > uploading packages. So far the source of truth is infra.git and it should
stay
> > this way until we have Glyco and a better process to generate packages.
> 
> What are the implications (if any), of using virtualenv in depot_tools? I
don't
> think I understand the rationale for this change
> (I'm not necessarily against it, just don't know what's driving the change
...).

This is part of a more general move toward (almost) self-contained environments.
This technique has been used in infra.git successfully for a couple months,
we're just using it more widely. Another long-term goal is to stop using
depot_tools as the source of truth for anything: it should just be assembled
from pieces living elsewhere (infra, luci-py, build, etc.)

From a user's perspective nothing should change. One thing that will break in
the long run though (if we've not moved to Go in the meantime) is the ability to
import a depot_tools package from outside depot_tools. This is intended: we're
trying to get back to the state where depot_tools is just a collection of
executables, nothing more.

[Dirk Pranke, 2015-06-22 23:00:46]

On 2015/06/22 22:59:11, pgervais wrote:
> From a user's perspective nothing should change. 

Okay, this was the part I cared about. As long as this is true, things seem fine
to me.

[Sergiy Byelozyorov, 2015-06-23 08:21:44]

On 2015/06/22 18:56:06, pgervais wrote:
> I think you should trim down the bootstrap directory to leave only code used
to
> generate the virtualenv, and drop all that is related to generating and
> uploading packages. So far the source of truth is infra.git and it should stay
> this way until we have Glyco and a better process to generate packages.

Done. Only left files that are necessary.

[Sergiy Byelozyorov, 2015-06-23 08:49:57]

Tested on Mac, Windows and Linux. Works as expected.

[pgervais, 2015-06-24 00:55:57]

lgtm with one small fix required.

https://chromiumcodereview.appspot.com/1200843003/diff/140001/update_depot_tools
File update_depot_tools (right):

https://chromiumcodereview.appspot.com/1200843003/diff/140001/update_depot_to...
update_depot_tools:160: python -u ./bootstrap/bootstrap.py --deps_file
bootstrap/deps.pyl ENV
This must be run even if ENV already exists, to update the content if
bootstrap/deps.pyl has changed. Which is what you do in the Windows script btw.

[Sergiy Byelozyorov, 2015-06-24 05:12:32]

https://chromiumcodereview.appspot.com/1200843003/diff/140001/update_depot_tools
File update_depot_tools (right):

https://chromiumcodereview.appspot.com/1200843003/diff/140001/update_depot_to...
update_depot_tools:160: python -u ./bootstrap/bootstrap.py --deps_file
bootstrap/deps.pyl ENV
On 2015/06/24 00:55:56, pgervais wrote:
> This must be run even if ENV already exists, to update the content if
> bootstrap/deps.pyl has changed. Which is what you do in the Windows script
btw.

Done.

[Sergiy Byelozyorov, 2015-06-24 05:13:18]

The CQ bit was checked by sergiyb@chromium.org

[Sergiy Byelozyorov, 2015-06-24 05:13:18]

The patchset sent to the CQ was uploaded after l-g-t-m from
pgervais@chromium.org
Link to the patchset:
https://chromiumcodereview.appspot.com/1200843003/#ps160001 (title: "Addressed
comments")

[commit-bot: I haz the power, 2015-06-24 05:13:23]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1200843003/160001

[commit-bot: I haz the power, 2015-06-24 05:15:50]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-06-24 05:15:51]

Try jobs failed on following builders:
  depot_tools_presubmit on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/depot_tools_pre...)

[Sergiy Byelozyorov, 2015-06-24 05:18:34]

The CQ bit was checked by sergiyb@chromium.org

[Sergiy Byelozyorov, 2015-06-24 05:18:34]

The patchset sent to the CQ was uploaded after l-g-t-m from
pgervais@chromium.org
Link to the patchset:
https://chromiumcodereview.appspot.com/1200843003/#ps180001 (title: "Rebase")

[commit-bot: I haz the power, 2015-06-24 05:18:47]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1200843003/180001

[Sergiy Byelozyorov, 2015-06-24 05:20:26]

The CQ bit was unchecked by sergiyb@chromium.org

[Sergiy Byelozyorov, 2015-06-24 05:23:10]

Committed patchset #10 (id:180001) manually as 295811.

[Adrian Kuegel, 2015-06-24 08:38:58]

A revert of this CL (patchset #10 id:180001) has been created in
https://chromiumcodereview.appspot.com/1202843005/ by akuegel@chromium.org.

The reason for reverting is: Doesn't work, virtualenv is missing in depot_tools
checkout..