[telemetry] Script for finding all Telemetry dependencies.

[telemetry] Script for finding all Telemetry dependencies.

Given a list of Python modules, it will find all:
- bootstrap_deps folders, recursively.
- Python dependencies.
- Page sets in those modules' folders.
- Serving dirs of those page sets.
And print a list of all files, or zip them up.

Example bot command: tools/telemetry_tools/find_dependencies -v tools/perf/run_benchmark tools/perf/run_measurement tools/perf/record_wpr content/test/gpu/run_gpu_test -z telemetry.zip


BUG=311380
TEST=Ran it.

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=251298

[nduca, 2014-01-07 23:50:58]

can we move telemetry_tools into telemetry/?

[nduca, 2014-01-07 23:51:10]

also unit tests for this :)

[dtu, 2014-01-29 00:53:07]

On 2014/01/07 23:50:58, nduca wrote:
> can we move telemetry_tools into telemetry/?

Why does telemetry_tools exist?

[achuithb, 2014-01-29 01:01:05]

There's a chance that this CL will break telemetry on cros. 

We still don't have a trybot, but please take a look at this bot once your
change lands and make sure it's still green:
http://chromegw.corp.google.com/i/chromium.fyi/builders/Chromium%20Cros64%20Perf

Thank you!

[tonyg, 2014-01-29 05:10:07]

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
File tools/telemetry_tools/find_dependencies (right):

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:2: # Copyright (c) 2013 The Chromium
Authors. All rights reserved.
2014

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:33: common_prefix =
os.path.commonprefix([subdirectory, directory])
This has bitten me before. It does string comparison instead of path. So for
example:
>>> os.path.commonprefix(['/foo', '/foobar'])
'/foo'

Is that what you are expecting?

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:227: '-p', '--include-page-set-data',
action='store_true', default=False,
Shouldn't this default to true?

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:230: '-c',
'--include-cloud-storage-data', action='store_true', default=False,
This seems dangerous and I can't think why we'd ever set it. Do we need it?

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:257: def Main():
Can this thing generate bootstrap_deps now? Since CrOS depends on that, it would
serve as another form of regression test for this.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
File tools/telemetry_tools/find_dependencies_unittest.py (right):

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies_unittest.py:1: #!/usr/bin/env python
Not necessary, right?

We should definitely get these unittests into tools/telemetry/telemetry so that
tools/telemetry/run_tests will run them. Otherwise, we'll have to go through a
big ordeal to get them running on the bots.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies_unittest.py:2: # Copyright (c) 2013 The
Chromium Authors. All rights reserved.
2014

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies_unittest.py:10: class
FindDependenciesTest(unittest.TestCase):
For this test, why not just create a mini, fake telemetry directory with, say
one Measurement and PageSet, etc. Then run it against that directory.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
File tools/telemetry_tools/path_set.py (right):

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/path_set.py:1: # Copyright (c) 2013 The Chromium Authors.
All rights reserved.
2014

[dtu, 2014-02-13 21:49:35]

PTAL

@achuith, I don't think this will affect ChromeOS. It's a mostly self-contained
script that doesn't affect --print-bootstrap-deps. The only modification to
existing files is to use realpath instead of abspath. But you never know, I
guess :)

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
File tools/telemetry_tools/find_dependencies (right):

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:2: # Copyright (c) 2013 The Chromium
Authors. All rights reserved.
On 2014/01/29 05:10:08, tonyg wrote:
> 2014

Done. Fun fact, Ctrl+A in Vim increments the first number on the current line.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:33: common_prefix =
os.path.commonprefix([subdirectory, directory])
On 2014/01/29 05:10:08, tonyg wrote:
> This has bitten me before. It does string comparison instead of path. So for
> example:
> >>> os.path.commonprefix(['/foo', '/foobar'])
> '/foo'
> 
> Is that what you are expecting?

Yup! It's okay in this particular use.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:227: '-p', '--include-page-set-data',
action='store_true', default=False,
On 2014/01/29 05:10:08, tonyg wrote:
> Shouldn't this default to true?

No, for the bots we're actually going to do False, since we can't yet
differentiate between src and src-internal files. This flag is referring to e.g.
third_party/WebKit/PerformanceTests, not to the page set .json and .sha1 files.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:230: '-c',
'--include-cloud-storage-data', action='store_true', default=False,
On 2014/01/29 05:10:08, tonyg wrote:
> This seems dangerous and I can't think why we'd ever set it. Do we need it?

Guess not. Done.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies:257: def Main():
On 2014/01/29 05:10:08, tonyg wrote:
> Can this thing generate bootstrap_deps now? Since CrOS depends on that, it
would
> serve as another form of regression test for this.

Unfortunately, this script can't find some dependencies which are manual in
bootstrap_deps. E.g., profile data, JS dependencies of
html_page_measurement_results. Also I've had issues modifying the bootstrap_deps
in the past and now I'm scared :)

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
File tools/telemetry_tools/find_dependencies_unittest.py (right):

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies_unittest.py:1: #!/usr/bin/env python
On 2014/01/29 05:10:08, tonyg wrote:
> Not necessary, right?
> 
> We should definitely get these unittests into tools/telemetry/telemetry so
that
> tools/telemetry/run_tests will run them. Otherwise, we'll have to go through a
> big ordeal to get them running on the bots.

I kind of think I would like all unit tests to have a __main__. It's more
natural to people who are familiar with Python, but new to Telemetry.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies_unittest.py:2: # Copyright (c) 2013 The
Chromium Authors. All rights reserved.
On 2014/01/29 05:10:08, tonyg wrote:
> 2014

Done.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/find_dependencies_unittest.py:10: class
FindDependenciesTest(unittest.TestCase):
On 2014/01/29 05:10:08, tonyg wrote:
> For this test, why not just create a mini, fake telemetry directory with, say
> one Measurement and PageSet, etc. Then run it against that directory.

I think I will make unit tests in a followup patch. We need to move these files
and the bootstrap into tools/telemetry/ before it's unit-testable.

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
File tools/telemetry_tools/path_set.py (right):

https://chromiumcodereview.appspot.com/103433002/diff/120001/tools/telemetry_...
tools/telemetry_tools/path_set.py:1: # Copyright (c) 2013 The Chromium Authors.
All rights reserved.
On 2014/01/29 05:10:08, tonyg wrote:
> 2014

Done.

[tonyg, 2014-02-13 21:55:29]

lgtm

https://chromiumcodereview.appspot.com/103433002/diff/170001/tools/telemetry_...
File tools/telemetry_tools/find_dependencies_unittest.py (right):

https://chromiumcodereview.appspot.com/103433002/diff/170001/tools/telemetry_...
tools/telemetry_tools/find_dependencies_unittest.py:12: class
FindDependenciesTest(unittest.TestCase):
We want to implement this or remove it, right?

[dtu, 2014-02-13 22:52:00]

The CQ bit was checked by dtu@chromium.org

[commit-bot: I haz the power, 2014-02-13 22:54:01]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/dtu@chromium.org/103433002/260001

[achuithb, 2014-02-13 23:33:34]

On 2014/02/13 21:49:35, dtu wrote:
> PTAL
> 
> @achuith, I don't think this will affect ChromeOS. It's a mostly
self-contained
> script that doesn't affect --print-bootstrap-deps. The only modification to
> existing files is to use realpath instead of abspath. But you never know, I
> guess :)
> 

Thanks for the heads up.

[commit-bot: I haz the power, 2014-02-14 02:45:41]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-02-14 02:45:42]

Retried try job too often on linux_chromeos for step(s) browser_tests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=linux_chro...

[tonyg, 2014-02-14 02:54:41]

The CQ bit was checked by tonyg@chromium.org

[commit-bot: I haz the power, 2014-02-14 02:58:10]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/dtu@chromium.org/103433002/260001

[commit-bot: I haz the power, 2014-02-14 13:59:09]

Change committed as 251298

[dtu, 2014-02-22 01:36:54]

A revert of this CL has been created in
https://codereview.chromium.org/170313010/ by dtu@chromium.org.

The reason for reverting is: May have caused a performance regression, reverting
to confirm.
https://code.google.com/p/chromium/issues/detail?id=345374.