Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(1103)

Issue 18328003: Add a git cache for gclient sync operations. (Closed)

Created:
7 years, 5 months ago by iannucci
Modified:
7 years, 5 months ago
CC:
chromium-reviews, Dirk Pranke, cmp-cc_chromium.org, M-A Ruel, iannucci+depot_tools_chromium.org
Visibility:
Public.

Description

Add a git cache for gclient sync operations. Instead of cloning straight into place, clones are made to a global cache dir, and then local (using --shared) clones are made from the cache to the final resting place. This means the 'final' clones are full repos with no shenanigans, meaning that branches, commits, etc. all work, which should allow the rest of the gclient ecosystem to work without change as well. The primary benefit is, of course, reduced network IO, and a much lower cost for 'clobber' operations (assuming we don't clobber the cache). It also means that a given bot can have a greater number of checkouts, since the entire git history will only be stored once per machine, instead of once per checkout. R=dpranke@chromium.org, szager@chromium.org BUG= Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=210024

Patch Set 1 #

Patch Set 2 : Use long form of --shared #

Patch Set 3 : extra newline #

Total comments: 18

Patch Set 4 : Fix comments #

Total comments: 9

Patch Set 5 : Fix comments #

Patch Set 6 : Fix comments, and make cache_dir a top level setting in .gclient #

Patch Set 7 : Capture not Run #

Total comments: 7

Patch Set 8 : Make cache_dir .gclient only, and move it to GitWrapper #

Unified diffs Side-by-side diffs Delta from patch set Stats (+133 lines, -26 lines) Patch
M gclient.py View 1 2 3 4 5 6 7 6 chunks +13 lines, -4 lines 0 comments Download
M gclient_scm.py View 1 2 3 4 5 6 7 8 chunks +107 lines, -16 lines 0 comments Download
M tests/gclient_scm_test.py View 1 2 3 4 5 6 7 2 chunks +3 lines, -0 lines 0 comments Download
M tests/gclient_smoketest.py View 1 2 3 4 5 6 chunks +10 lines, -6 lines 0 comments Download

Messages

Total messages: 26 (0 generated)
iannucci
7 years, 5 months ago (2013-07-01 11:06:04 UTC) #1
iannucci
On 2013/07/01 11:06:04, iannucci wrote: One thing I'm not sure about is if this should ...
7 years, 5 months ago (2013-07-01 19:54:39 UTC) #2
iannucci
On 2013/07/01 19:54:39, iannucci wrote: > On 2013/07/01 11:06:04, iannucci wrote: > > One thing ...
7 years, 5 months ago (2013-07-01 20:03:53 UTC) #3
iannucci
7 years, 5 months ago (2013-07-01 23:08:27 UTC) #4
iannucci
+mmoss PTAL :)
7 years, 5 months ago (2013-07-01 23:09:06 UTC) #5
szager1
I like the idea, as long as it's billed to users as experimental. https://codereview.chromium.org/18328003/diff/5001/gclient.py File ...
7 years, 5 months ago (2013-07-01 23:38:29 UTC) #6
iannucci
The only users I plan on billing it to are the bots :) The main ...
7 years, 5 months ago (2013-07-01 23:55:51 UTC) #7
szager1
https://codereview.chromium.org/18328003/diff/5001/gclient.py File gclient.py (right): https://codereview.chromium.org/18328003/diff/5001/gclient.py#newcode1559 gclient.py:1559: options.cache_lock = threading.Lock() On 2013/07/01 23:55:52, iannucci wrote: > ...
7 years, 5 months ago (2013-07-02 00:08:05 UTC) #8
iannucci
PTAL :) https://codereview.chromium.org/18328003/diff/5001/gclient_scm.py File gclient_scm.py (right): https://codereview.chromium.org/18328003/diff/5001/gclient_scm.py#newcode179 gclient_scm.py:179: if (now - self.last_time) >= 2: On ...
7 years, 5 months ago (2013-07-02 00:22:27 UTC) #9
iannucci
does this LGTY? I'd like to get this in so I can start playing with ...
7 years, 5 months ago (2013-07-02 00:59:15 UTC) #10
iannucci
On 2013/07/02 00:59:15, iannucci wrote: > does this LGTY? I'd like to get this in ...
7 years, 5 months ago (2013-07-02 18:59:24 UTC) #11
Michael Moss
lgtm https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py File gclient_scm.py (right): https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py#newcode159 gclient_scm.py:159: """Create a new GitFilter. I tend to overdo ...
7 years, 5 months ago (2013-07-02 20:47:52 UTC) #12
szager1
https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py File gclient_scm.py (right): https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py#newcode764 gclient_scm.py:764: # --progress, so use fetch instead. existing_url = self._Run(['config', ...
7 years, 5 months ago (2013-07-02 20:58:21 UTC) #13
iannucci
Fixed stuff. See comment re: exception for not-matching urls :) https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py File gclient_scm.py (right): https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py#newcode159 ...
7 years, 5 months ago (2013-07-02 22:10:07 UTC) #14
szager1
https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py File gclient_scm.py (right): https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py#newcode764 gclient_scm.py:764: # --progress, so use fetch instead. On 2013/07/02 22:10:07, ...
7 years, 5 months ago (2013-07-02 22:14:04 UTC) #15
iannucci
On 2013/07/02 22:14:04, szager1 wrote: > https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py > File gclient_scm.py (right): > > https://chromiumcodereview.appspot.com/18328003/diff/11001/gclient_scm.py#newcode764 > ...
7 years, 5 months ago (2013-07-02 22:33:26 UTC) #16
szager1
lgtm > I'm confused... How /could/ there be a collision? We just generated the folder ...
7 years, 5 months ago (2013-07-02 22:44:50 UTC) #17
iannucci
On 2013/07/02 22:44:50, szager1 wrote: > lgtm > > > I'm confused... How /could/ there ...
7 years, 5 months ago (2013-07-02 22:48:45 UTC) #18
iannucci
On 2013/07/02 22:48:45, iannucci wrote: > On 2013/07/02 22:44:50, szager1 wrote: > > lgtm > ...
7 years, 5 months ago (2013-07-02 22:49:31 UTC) #19
szager1
On 2013/07/02 22:49:31, iannucci wrote: > > Ok, that makes sense to me. I think ...
7 years, 5 months ago (2013-07-02 23:04:40 UTC) #20
iannucci
On 2013/07/02 23:04:40, szager1 wrote: > On 2013/07/02 22:49:31, iannucci wrote: > > > > ...
7 years, 5 months ago (2013-07-03 01:02:05 UTC) #21
Michael Moss
lgtm https://chromiumcodereview.appspot.com/18328003/diff/27002/gclient_scm.py File gclient_scm.py (right): https://chromiumcodereview.appspot.com/18328003/diff/27002/gclient_scm.py#newcode128 gclient_scm.py:128: cache_dir = None Since this is git-only, any ...
7 years, 5 months ago (2013-07-03 14:34:23 UTC) #22
szager1
https://chromiumcodereview.appspot.com/18328003/diff/27002/gclient.py File gclient.py (right): https://chromiumcodereview.appspot.com/18328003/diff/27002/gclient.py#newcode901 gclient.py:901: cache_dir = %(cache_dir)r str(None) == repr(None), so for the ...
7 years, 5 months ago (2013-07-03 16:49:52 UTC) #23
iannucci
fixed :) https://codereview.chromium.org/18328003/diff/27002/gclient.py File gclient.py (right): https://codereview.chromium.org/18328003/diff/27002/gclient.py#newcode901 gclient.py:901: cache_dir = %(cache_dir)r On 2013/07/03 16:49:52, szager1 ...
7 years, 5 months ago (2013-07-03 19:07:32 UTC) #24
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/iannucci@chromium.org/18328003/45001
7 years, 5 months ago (2013-07-03 19:36:23 UTC) #25
commit-bot: I haz the power
7 years, 5 months ago (2013-07-03 19:38:35 UTC) #26
Message was sent while issue was closed.
Change committed as 210024

Powered by Google App Engine
This is Rietveld 408576698