[Android] Enable using isolate files to get a list of data dependencies to push to the device.

[Android] Enable using isolate files to get a list of data dependencies to push to the device.

Convert base_unittests to isolate.

BUG=249870
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=209283

[frankf, 2013-06-20 22:01:15]

@maruel, for now I'm directly using load_isolated() in run_isolated.py to avoid
duplicating the parer, but I can easily re-implement it if you the dependency is
not desirable.

[craigdh, 2013-06-20 22:58:00]

lgtm, but wait for maruel.

[frankf, 2013-06-21 20:38:19]

maruel is OOO. vadimsh/csharp, ptal.

https://codereview.chromium.org/17267004/diff/10001/base/base_unittests.isolate
File base/base_unittests.isolate (right):

https://codereview.chromium.org/17267004/diff/10001/base/base_unittests.isola...
base/base_unittests.isolate:9:
'<(PRODUCT_DIR)/base_unittests_apk/base_unittests-debug.apk',
It doesn't make much sense to list the apk here: crbug.com/252785

[Vadim Sh., 2013-06-24 11:10:13]

Here's how base_unittests.isolated looks like when I generate it on my machine:

{
    "files": {
        "data/file_version_info_unittest/FileVersionInfoTest1.dll": {
            "h": "f1e62a8a212a3facd8d8357f0ae9a5e130de17f7", 
            "m": 488, 
            "s": 13824
        }, 
        "data/file_version_info_unittest/FileVersionInfoTest2.dll": {
            "h": "8453d4c88a5a14fcda342b65a3f7b390c9e06850", 
            "m": 488, 
            "s": 13824
        }
    }, 
    "includes": [
        "a59fc9f443927e9dd4279540126039b403d21809"
    ], 
    "os": "android", 
    "relative_cwd": "."
}


The code in CL will return
['data/file_version_info_unittest/FileVersionInfoTest1.dll',
'data/file_version_info_unittest/FileVersionInfoTest2.dll'] which is not what we
need :( All interesting parts are in 'includes' section which in fact references
generated 'base_unittests.0.isolated' by its SHA1 hash...

I think isolate.py should be modified to run in a mode 'extract plain list of
dependencies only from *.isolate file'. Currently it'll split off some parts
(/test/data/) into separate *.isolated, and it also calculates SHA1 file hashes
which you probably don't care about (right?). 

I need to talk with M-A about that...

[frankf, 2013-06-24 17:10:21]

Hmm. I actually get all the dependencies including /test/data into a single
isolated file using this command:

tools/swarm_client/isolate.py check --isolate=base/base_unittests.isolate
--isolated=out/Debug/base_unittests.isolated -V PRODUCT_DIR=out/Debug -V
OS=android --outdir=out/Debug

What command did you run?

I'm not concerned about generating the hashes since the time it takes is
insignificant to actually pushing them to the device.


On 2013/06/24 11:10:13, Vadim Sh. wrote:
> Here's how base_unittests.isolated looks like when I generate it on my
machine:
> 
> {
>     "files": {
>         "data/file_version_info_unittest/FileVersionInfoTest1.dll": {
>             "h": "f1e62a8a212a3facd8d8357f0ae9a5e130de17f7", 
>             "m": 488, 
>             "s": 13824
>         }, 
>         "data/file_version_info_unittest/FileVersionInfoTest2.dll": {
>             "h": "8453d4c88a5a14fcda342b65a3f7b390c9e06850", 
>             "m": 488, 
>             "s": 13824
>         }
>     }, 
>     "includes": [
>         "a59fc9f443927e9dd4279540126039b403d21809"
>     ], 
>     "os": "android", 
>     "relative_cwd": "."
> }
> 
> 
> The code in CL will return
> ['data/file_version_info_unittest/FileVersionInfoTest1.dll',
> 'data/file_version_info_unittest/FileVersionInfoTest2.dll'] which is not what
we
> need :( All interesting parts are in 'includes' section which in fact
references
> generated 'base_unittests.0.isolated' by its SHA1 hash...
> 
> I think isolate.py should be modified to run in a mode 'extract plain list of
> dependencies only from *.isolate file'. Currently it'll split off some parts
> (/test/data/) into separate *.isolated, and it also calculates SHA1 file
hashes
> which you probably don't care about (right?). 
> 
> I need to talk with M-A about that...

[Vadim Sh., 2013-06-24 17:16:34]

I used steps I described in crbug.com/252785 bug. In particular I removed *.apk
file and PRODUCT_DIR reference (since isolate.py could not deduce correct root
directory without reference to *.apk in PRODUCT_DIR and complained that
PRODUCT_DIR is outside of root directory). 

I guess it triggered different code path in isolate.py.

[frankf, 2013-06-27 00:41:08]

ptal.

[Vadim Sh., 2013-06-28 09:17:19]

On 2013/06/27 00:41:08, frankf wrote:
> ptal.

lgtm, but I don't know conditions under which *.isolated contains 'includes'
section (which is not handled in this CL). Need M-A or csharp opinion on this.

[csharp, 2013-06-28 13:12:17]

LGTM with nit

I'm not sure what is up the includes section, I know there had been plans to
split them up based on the folders (that way if no files changed in a given
file, the .isolated file doesn't have to get uploaded to the server since it is
already there), but I don't know how far along this got.

https://codereview.chromium.org/17267004/diff/22001/build/android/pylib/gtest...
File build/android/pylib/gtest/test_runner.py (right):

https://codereview.chromium.org/17267004/diff/22001/build/android/pylib/gtest...
build/android/pylib/gtest/test_runner.py:60: '--isolate=%s' % isolate_abs_path,
I don't think you need to specify the .isolate file (I think the .isolated file
already points to it).

[frankf, 2013-06-28 18:32:45]

OK, so I guess we're ignoring the 'includes' section of isolated files for now?
The isolate command run by this CL doesn't generate it, so let's make sure
future changes doesn't break this.

https://codereview.chromium.org/17267004/diff/22001/build/android/pylib/gtest...
File build/android/pylib/gtest/test_runner.py (right):

https://codereview.chromium.org/17267004/diff/22001/build/android/pylib/gtest...
build/android/pylib/gtest/test_runner.py:60: '--isolate=%s' % isolate_abs_path,
There's no isolated file at this point, that's what we're generating. 

On 2013/06/28 13:12:17, csharp wrote:
> I don't think you need to specify the .isolate file (I think the .isolated
file
> already points to it).

[craigdh, 2013-06-28 18:50:14]

lgtm.

[csharp, 2013-06-28 19:01:05]

https://codereview.chromium.org/17267004/diff/22001/build/android/pylib/gtest...
File build/android/pylib/gtest/test_runner.py (right):

https://codereview.chromium.org/17267004/diff/22001/build/android/pylib/gtest...
build/android/pylib/gtest/test_runner.py:60: '--isolate=%s' % isolate_abs_path,
On 2013/06/28 18:32:46, frankf wrote:
> There's no isolated file at this point, that's what we're generating. 
> 
> On 2013/06/28 13:12:17, csharp wrote:
> > I don't think you need to specify the .isolate file (I think the .isolated
> file
> > already points to it).
> 

Ah, right. I thought you were performing an action with the .isolated file, my
bad.

[commit-bot: I haz the power, 2013-06-28 19:07:41]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/frankf@chromium.org/17267004/22001

[commit-bot: I haz the power, 2013-06-28 22:31:48]

Retried try job too often on win_rel for step(s) unit_tests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=win_rel&nu...

[commit-bot: I haz the power, 2013-06-28 23:17:38]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/frankf@chromium.org/17267004/22001

[commit-bot: I haz the power, 2013-06-29 13:00:53]

Change committed as 209283