Adds --config option to download_from_google_storage.py

Adds --config option to download_from_google_storage.py

In order to support both developer workflow and not breaking bots, if the script
fails on a 403 in a bucket, it'll print a message asking developers to run
"download_from_google_storage --config" in order to create a new boto file.

This is not done automatically because it would break bots (Imagine hitting a 403,
and then gsutil wiping the .boto file, waiting for input, then dying).

BUG=231699, 176331
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=208806

[Ryan Tseng, 2013-06-24 21:41:38]

PTAL - Robbie and Maruel for review, Maruel for approval.

I'm also going to send out this PSA once this gets landed:

Subject: PSA: Having a src-internal checkout will now require logging into
Google Storage.

tl;dr: Update your depot_tools checkout, run "download_from_google_storage
--config", follow onscreen directions, and you'll be all good.

In an effort to reduce the amount of space binary files take up in our SVN/Git
repositories, we're moving binaries over to be hosted on Google Storage buckets,
and replacing them with hashes checked into the repository.

What this means is that there are hooks inside the DEPS file of src-internal
that calls a script (download_from_google_storage.py) which scans the checkout
for files ending with *.sha1, and fetching those files from Google Storage. 
Think of these .sha1 files as torrent files - It just contains a 40 character
sha1 hash of the file to fetch, and the real file is stored in some bucket on
Google Storage.

Because some of these buckets will be restricted to Googlers only, you'll need
to log into Google Storage with your google.com@ credentials in order to access
these repositories.

Steps:
First, update your depot_tools checkout.
Then:
===
$ download_from_google_storage --config
This script will create a boto config file at
<Your Home Directory>/.boto
containing your credentials, based on your responses to the following questions.

Please navigate your browser to the following URL:
https://accounts.google.com/o/oauth2/auth?<Some Long URL> In your browser you
should see a page that requests you to authorize gsutil to access
Google Cloud Storage on your behalf. After you approve, an authorization code
will be displayed. (NOTE: Use your Google.com credential here, not your
Chromium.org credential)

Enter the authorization code: 
<Your Auth Code, should look like 4/somelongstring>

Please navigate your browser to http://code.google.com/apis/console,
then click "Services" on the left side panel and ensure you have Google Cloud
Storageactivated, then click "Google Cloud Storage" on the left side panel and
find the "x-goog-project-id" on that page.
What is your project-id? <Leave empty or enter a project ID>
No default project ID entered. You will need to edit the default_project_id
value
in your boto config file before using "gsutil ls gs://" or "mb" commandswith the
default API version (2).

Boto config file "<Your Home Directory>/.boto" created.
If you need to use a proxy to use a proxy to access the Internet please see the
instructions in that file.
===

All Googlers have read access for the specified buckets.  If you run into any
issues, or have any special cases, please email me.

Your Chrome Infraite
Ryan

[iannucci, 2013-06-25 20:48:26]

On 2013/06/24 21:41:38, Ryan T. wrote:
> PTAL - Robbie and Maruel for review, Maruel for approval.
> 
> I'm also going to send out this PSA once this gets landed:
> 
> Subject: PSA: Having a src-internal checkout will now require logging into
> Google Storage.
> 
> tl;dr: Update your depot_tools checkout, run "download_from_google_storage
> --config", follow onscreen directions, and you'll be all good.
> 
> In an effort to reduce the amount of space binary files take up in our SVN/Git
> repositories, we're moving binaries over to be hosted on Google Storage
buckets,
> and replacing them with hashes checked into the repository.
> 
> What this means is that there are hooks inside the DEPS file of src-internal
> that calls a script (download_from_google_storage.py) which scans the checkout
> for files ending with *.sha1, and fetching those files from Google Storage. 
> Think of these .sha1 files as torrent files - It just contains a 40 character
> sha1 hash of the file to fetch, and the real file is stored in some bucket on
> Google Storage.
> 
> Because some of these buckets will be restricted to Googlers only, you'll need
> to log into Google Storage with your google.com@ credentials in order to
access
> these repositories.
> 
> Steps:
> First, update your depot_tools checkout.
> Then:
> ===
> $ download_from_google_storage --config
> This script will create a boto config file at
> <Your Home Directory>/.boto
> containing your credentials, based on your responses to the following
questions.
> 
> Please navigate your browser to the following URL:
> https://accounts.google.com/o/oauth2/auth?%3CSome Long URL> In your browser
you
> should see a page that requests you to authorize gsutil to access
> Google Cloud Storage on your behalf. After you approve, an authorization code
> will be displayed. (NOTE: Use your http://Google.com credential here, not your
> http://Chromium.org credential)
> 
> Enter the authorization code: 
> <Your Auth Code, should look like 4/somelongstring>
> 
> Please navigate your browser to http://code.google.com/apis/console,
> then click "Services" on the left side panel and ensure you have Google Cloud
> Storageactivated, then click "Google Cloud Storage" on the left side panel and
> find the "x-goog-project-id" on that page.
> What is your project-id? <Leave empty or enter a project ID>
> No default project ID entered. You will need to edit the default_project_id
> value
> in your boto config file before using "gsutil ls gs://" or "mb" commandswith
the
> default API version (2).
> 
> Boto config file "<Your Home Directory>/.boto" created.
> If you need to use a proxy to use a proxy to access the Internet please see
the
> instructions in that file.
> ===
> 
> All Googlers have read access for the specified buckets.  If you run into any
> issues, or have any special cases, please email me.
> 
> Your Chrome Infraite
> Ryan

lgtm % comments

[iannucci, 2013-06-25 20:48:34]

comments...

https://chromiumcodereview.appspot.com/17590010/diff/2001/download_from_googl...
File download_from_google_storage.py (right):

https://chromiumcodereview.appspot.com/17590010/diff/2001/download_from_googl...
download_from_google_storage.py:277: 'current have Google Storage credentials
stored.')
Run this if you want to initialize your saved Google Storage credentials.

https://chromiumcodereview.appspot.com/17590010/diff/2001/download_from_googl...
download_from_google_storage.py:286: if os.path.exists(path) and 'gsutil' in
os.listdir(path):
I don't think this will find e.g. the .bat wrapper on windows.

[Ryan Tseng, 2013-06-25 21:02:36]

https://chromiumcodereview.appspot.com/17590010/diff/2001/download_from_googl...
File download_from_google_storage.py (right):

https://chromiumcodereview.appspot.com/17590010/diff/2001/download_from_googl...
download_from_google_storage.py:277: 'current have Google Storage credentials
stored.')
On 2013/06/25 20:48:34, iannucci wrote:
> Run this if you want to initialize your saved Google Storage credentials.

Done.

https://chromiumcodereview.appspot.com/17590010/diff/2001/download_from_googl...
download_from_google_storage.py:286: if os.path.exists(path) and 'gsutil' in
os.listdir(path):
On 2013/06/25 20:48:34, iannucci wrote:
> I don't think this will find e.g. the .bat wrapper on windows.

gsutil is actually a python file, and is called using (sys.executable,
self.path) + [args...].  So it should actually work.

[commit-bot: I haz the power, 2013-06-25 21:03:16]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/hinoka@chromium.org/17590010/7002

[commit-bot: I haz the power, 2013-06-25 21:04:54]

Presubmit check for 17590010-7002 failed and returned exit status 1.

Running presubmit commit checks ...
Checking out rietveld...
Running save-description-on-failure.sh
Running push-basic.sh
Running upstream.sh
Running submit-from-new-dir.sh
Running abandon.sh
Running submodule-merge-test.sh
Running upload-local-tracking-branch.sh
Running hooks.sh
Running post-dcommit-hook-test.sh
Running upload-stale.sh
Running patch.sh
Running basic.sh

** Presubmit ERRORS **
push-basic.sh failed
Command /b/commit-queue/workdir/tools/depot_tools/tests/push-basic.sh returned
non-zero exit status 1 in /b/commit-queue/workdir/tools/depot_tools/tests
Setting up test upstream git repo...
Setting up test git repo...
TESTING: git-cl upload wants a server
TESTING: git-cl status has no issue
TESTING: upload succeeds (needs a server running on localhost)
TESTING: git-cl status now knows the issue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
100    77    0     0  100    77      0   1371 --:--:-- --:--:-- --:--:--  1400
TESTING: Base URL contains branch name
TESTING: git-cl push ok
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chrome-bot/.codereview_upload_cookies
Running presubmit commit checks ...

Presubmit checks passed.
Description: 'foo-quux\n\nReview URL: http://localhost:8080/5629499534213120'
Closing issue (you may be prompted for your codereview password)...
TESTING: committed code has proper description
TESTING: issue no longer has a branch
FAILURE: issue no longer has a branch


basic.sh failed
Command /b/commit-queue/workdir/tools/depot_tools/tests/basic.sh returned
non-zero exit status 1 in /b/commit-queue/workdir/tools/depot_tools/tests
Setting up test SVN repo...
Setting up test git-svn repo...
TESTING: git-cl upload wants a server
TESTING: git-cl status has no issue
TESTING: upload succeeds (needs a server running on localhost)
TESTING: git-cl status now knows the issue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
100    77    0     0  100    77      0   1602 --:--:-- --:--:-- --:--:--  1638
TESTING: git-cl dcommits ok
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chrome-bot/.codereview_upload_cookies
Running presubmit commit checks ...

Presubmit checks passed.
Description: 'foo-quux\n\nReview URL: http://localhost:8080/5840605766746112'
Closing issue (you may be prompted for your codereview password)...
TESTING: dcommitted code has proper description
TESTING: issue no longer has a branch
FAILURE: issue no longer has a branch


Presubmit checks took 94.6s to calculate.

[commit-bot: I haz the power, 2013-06-25 21:34:36]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/hinoka@chromium.org/17590010/7002

[commit-bot: I haz the power, 2013-06-25 21:36:34]

Presubmit check for 17590010-7002 failed and returned exit status 1.

Running presubmit commit checks ...
Checking out rietveld...
Running save-description-on-failure.sh
Running push-basic.sh
Running upstream.sh
Running submit-from-new-dir.sh
Running abandon.sh
Running submodule-merge-test.sh
Running upload-local-tracking-branch.sh
Running hooks.sh
Running post-dcommit-hook-test.sh
Running upload-stale.sh
Running patch.sh
Running basic.sh

** Presubmit ERRORS **
push-basic.sh failed
Command /b/commit-queue/workdir/tools/depot_tools/tests/push-basic.sh returned
non-zero exit status 1 in /b/commit-queue/workdir/tools/depot_tools/tests
Setting up test upstream git repo...
Setting up test git repo...
TESTING: git-cl upload wants a server
TESTING: git-cl status has no issue
TESTING: upload succeeds (needs a server running on localhost)
TESTING: git-cl status now knows the issue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
100    77    0     0  100    77      0   1405 --:--:-- --:--:-- --:--:--  1425
TESTING: Base URL contains branch name
TESTING: git-cl push ok
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chrome-bot/.codereview_upload_cookies
Running presubmit commit checks ...

Presubmit checks passed.
Description: 'foo-quux\n\nReview URL: http://localhost:8080/5629499534213120'
Closing issue (you may be prompted for your codereview password)...
TESTING: committed code has proper description
TESTING: issue no longer has a branch
FAILURE: issue no longer has a branch


basic.sh failed
Command /b/commit-queue/workdir/tools/depot_tools/tests/basic.sh returned
non-zero exit status 1 in /b/commit-queue/workdir/tools/depot_tools/tests
Setting up test SVN repo...
Setting up test git-svn repo...
TESTING: git-cl upload wants a server
TESTING: git-cl status has no issue
TESTING: upload succeeds (needs a server running on localhost)
TESTING: git-cl status now knows the issue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
100    77    0     0  100    77      0   1521 --:--:-- --:--:-- --:--:--  1571
TESTING: git-cl dcommits ok
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chrome-bot/.codereview_upload_cookies
Running presubmit commit checks ...

Presubmit checks passed.
Description: 'foo-quux\n\nReview URL: http://localhost:8080/5840605766746112'
Closing issue (you may be prompted for your codereview password)...
TESTING: dcommitted code has proper description
TESTING: issue no longer has a branch
FAILURE: issue no longer has a branch


Presubmit checks took 116.9s to calculate.

[commit-bot: I haz the power, 2013-06-25 21:46:37]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/hinoka@chromium.org/17590010/7002

[commit-bot: I haz the power, 2013-06-25 21:48:21]

Presubmit check for 17590010-7002 failed and returned exit status 1.

Running presubmit commit checks ...
Checking out rietveld...
Running save-description-on-failure.sh
Running push-basic.sh
Running upstream.sh
Running submit-from-new-dir.sh
Running abandon.sh
Running submodule-merge-test.sh
Running upload-local-tracking-branch.sh
Running hooks.sh
Running post-dcommit-hook-test.sh
Running upload-stale.sh
Running patch.sh
Running basic.sh

** Presubmit ERRORS **
push-basic.sh failed
Command /b/commit-queue/workdir/tools/depot_tools/tests/push-basic.sh returned
non-zero exit status 1 in /b/commit-queue/workdir/tools/depot_tools/tests
Setting up test upstream git repo...
Setting up test git repo...
TESTING: git-cl upload wants a server
TESTING: git-cl status has no issue
TESTING: upload succeeds (needs a server running on localhost)
TESTING: git-cl status now knows the issue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
100    77    0     0  100    77      0   1441 --:--:-- --:--:-- --:--:--  1480
TESTING: Base URL contains branch name
TESTING: git-cl push ok
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chrome-bot/.codereview_upload_cookies
Running presubmit commit checks ...

Presubmit checks passed.
Description: 'foo-quux\n\nReview URL: http://localhost:8080/5629499534213120'
Closing issue (you may be prompted for your codereview password)...
TESTING: committed code has proper description
TESTING: issue no longer has a branch
FAILURE: issue no longer has a branch


basic.sh failed
Command /b/commit-queue/workdir/tools/depot_tools/tests/basic.sh returned
non-zero exit status 1 in /b/commit-queue/workdir/tools/depot_tools/tests
Setting up test SVN repo...
Setting up test git-svn repo...
TESTING: git-cl upload wants a server
TESTING: git-cl status has no issue
TESTING: upload succeeds (needs a server running on localhost)
TESTING: git-cl status now knows the issue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
100    77    0     0  100    77      0   1607 --:--:-- --:--:-- --:--:--  1638
TESTING: git-cl dcommits ok
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chrome-bot/.codereview_upload_cookies
Running presubmit commit checks ...

Presubmit checks passed.
Description: 'foo-quux\n\nReview URL: http://localhost:8080/5840605766746112'
Closing issue (you may be prompted for your codereview password)...
TESTING: dcommitted code has proper description
TESTING: issue no longer has a branch
FAILURE: issue no longer has a branch


Presubmit checks took 102.6s to calculate.

[commit-bot: I haz the power, 2013-06-26 22:15:04]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/hinoka@chromium.org/17590010/7002

[commit-bot: I haz the power, 2013-06-26 22:16:54]

Presubmit check for 17590010-7002 failed and returned exit status 1.

Running presubmit commit checks ...
Checking out rietveld...
Running save-description-on-failure.sh
Running push-basic.sh
Running upstream.sh
Running submit-from-new-dir.sh
Running abandon.sh
Running submodule-merge-test.sh
Running upload-local-tracking-branch.sh
Running hooks.sh
Running post-dcommit-hook-test.sh
Running upload-stale.sh
Running patch.sh
Running basic.sh

** Presubmit ERRORS **
push-basic.sh failed
Command /b/commit-queue/workdir/tools/depot_tools/tests/push-basic.sh returned
non-zero exit status 1 in /b/commit-queue/workdir/tools/depot_tools/tests
Setting up test upstream git repo...
Setting up test git repo...
TESTING: git-cl upload wants a server
TESTING: git-cl status has no issue
TESTING: upload succeeds (needs a server running on localhost)
TESTING: git-cl status now knows the issue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
100    77    0     0  100    77      0   1325 --:--:-- --:--:-- --:--:--  1350
TESTING: Base URL contains branch name
TESTING: git-cl push ok
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chrome-bot/.codereview_upload_cookies
Running presubmit commit checks ...

Presubmit checks passed.
Description: 'foo-quux\n\nReview URL: http://localhost:8080/5629499534213120'
Closing issue (you may be prompted for your codereview password)...
TESTING: committed code has proper description
TESTING: issue no longer has a branch
FAILURE: issue no longer has a branch


basic.sh failed
Command /b/commit-queue/workdir/tools/depot_tools/tests/basic.sh returned
non-zero exit status 1 in /b/commit-queue/workdir/tools/depot_tools/tests
Setting up test SVN repo...
Setting up test git-svn repo...
TESTING: git-cl upload wants a server
TESTING: git-cl status has no issue
TESTING: upload succeeds (needs a server running on localhost)
TESTING: git-cl status now knows the issue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
100    77    0     0  100    77      0   1578 --:--:-- --:--:-- --:--:--  1604
TESTING: git-cl dcommits ok
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chrome-bot/.codereview_upload_cookies
Running presubmit commit checks ...

Presubmit checks passed.
Description: 'foo-quux\n\nReview URL: http://localhost:8080/5840605766746112'
Closing issue (you may be prompted for your codereview password)...
TESTING: dcommitted code has proper description
TESTING: issue no longer has a branch
FAILURE: issue no longer has a branch


Presubmit checks took 108.4s to calculate.

[commit-bot: I haz the power, 2013-06-26 22:21:47]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/hinoka@chromium.org/17590010/7002

[commit-bot: I haz the power, 2013-06-26 22:23:57]

Change committed as 208806

[M-A Ruel, 2013-07-09 15:20:19]

Small nit, the rest is fine.

https://chromiumcodereview.appspot.com/17590010/diff/7002/download_from_googl...
File download_from_google_storage.py (right):

https://chromiumcodereview.appspot.com/17590010/diff/7002/download_from_googl...
download_from_google_storage.py:288: gsutil = Gsutil(os.path.join(path,
'gsutil'), boto_path=options.boto)
Add a "break" statement after, no need to continue searching.

You could also use for/else instead, removing the need for line 285. E.g.
for path in os.environ["PATH"].split(os.pathsep):
  if os.path.exists(path) and 'gsutil' in os.listdir(path):
    gsutil = Gsutil(os.path.join(path, 'gsutil'), boto_path=options.boto)
    break
else:
  parser.error(...)