run_test_from_archive: Rewritten to be more parallelized.

run_test_from_archive: Rewritten to be more parallelized.

- Fetch files, create directories and symlinks all simultaneously.
- Opportinistically map the first files arrived instead of processing them in
  order.
- Creates all the needed directories in one pass, reducing the number of system
  calls like os.path.isdir().
- Add support for 'includes' manifest, e.g. fetch all files from another
  manifest in addition to the current one. This permits doing something like:
  1. Create a root manifest that contains all the usual script files.
  2. Create one manifest per command to run, referencing the root manifest in
      the 'includes' section by its hash. Note that includes are recursive.
  3. Optionally overide a file in to root manifest from what was defined in an
      included manifest.

Add more tests.

R=cmp@chromium.org
NOTRY=true
BUG=


Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=153755

[cmp, 2012-08-28 17:49:30]

lgtm with nits

one more nit: let's get csharp to review, too :)

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/data/run...
File tools/isolate/data/run_test_from_archive/check_files.py (right):

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/data/run...
tools/isolate/data/run_test_from_archive/check_files.py:12: 
insert an empty line here (i think!)

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
File tools/isolate/run_test_from_archive.py (right):

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:757: 
let's drop this empty line

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:774: # Creates the links if necessary
append a period

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:791: # Do bookeeping while files are
being downloaded in the background.
bookeeping -> bookkeeping

[csharp, 2012-08-28 19:03:31]

LGTM with a few small nits

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
File tools/isolate/run_test_from_archive.py (right):

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:450: # Items fetched but the users didn't
cache about yet.
Nit: Not sure what you mean here. 
users didn't cache about yet->users didn't care about yet (OR) users didn't
cache yet

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:608: """|obj| is really the sha-1 of the
file."""
nit: obj->obj_hash

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:611: # Set once all the left-side of the
tree is parsed.
What tree?

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:668: """Loads the manifest and all the
included manifest asynchronously.
Nit: manifests

[M-A Ruel, 2012-08-28 21:59:36]

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/data/run...
File tools/isolate/data/run_test_from_archive/check_files.py (right):

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/data/run...
tools/isolate/data/run_test_from_archive/check_files.py:12: 
On 2012/08/28 17:49:30, cmp wrote:
> insert an empty line here (i think!)

done, but I had to update the sha-1 in the manifest.

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
File tools/isolate/run_test_from_archive.py (right):

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:450: # Items fetched but the users didn't
cache about yet.
On 2012/08/28 19:03:31, csharp wrote:
> Nit: Not sure what you mean here. 

Me neither. Bad sign.

> users didn't cache about yet->users didn't care about yet (OR) users didn't
> cache yet

Well, it was not used at all, so I just removed it. Probably missed it during
refactoring.

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:608: """|obj| is really the sha-1 of the
file."""
On 2012/08/28 19:03:31, csharp wrote:
> nit: obj->obj_hash

done

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:611: # Set once all the left-side of the
tree is parsed.
On 2012/08/28 19:03:31, csharp wrote:
> What tree?

Added more info.

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:668: """Loads the manifest and all the
included manifest asynchronously.
On 2012/08/28 19:03:31, csharp wrote:
> Nit: manifests

done

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:757: 
On 2012/08/28 17:49:30, cmp wrote:
> let's drop this empty line

done

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:774: # Creates the links if necessary
On 2012/08/28 17:49:30, cmp wrote:
> append a period

done

https://chromiumcodereview.appspot.com/10876044/diff/1/tools/isolate/run_test...
tools/isolate/run_test_from_archive.py:791: # Do bookeeping while files are
being downloaded in the background.
On 2012/08/28 17:49:30, cmp wrote:
> bookeeping -> bookkeeping

done

[commit-bot: I haz the power, 2012-08-28 22:05:43]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/maruel@chromium.org/10876044/6001

[commit-bot: I haz the power, 2012-08-28 22:07:34]

Change committed as 153755