Breakdown memory usage by source file names in dmprof.

Breakdown memory usage by source file names in dmprof.

dmprof has classified memory usage by pattern-matching with function names.
It adds another classification by pattern-matching with source file names.

BUG=225343
NOTRY=true

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=193022

[Dai Mikurube (NOT FULLTIME), 2013-04-08 08:48:24]

Could you take a look also at this change when you have time?

peria@:
I think it's the first time for you to look into these files.  Please ask me
anything.

[peria, 2013-04-08 10:24:09]

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
File tools/deep_memory_profiler/policy.sourcefile.json (right):

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:105: "stacksourcefile":
".*../../v8/src/.*",
This pattern may match with unexpected paths like
"foo/bar/AA/v8/src/hoge.cc", right?

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:115: "stacksourcefile":
".*../../v8/src/.*",
ditto

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:120: "stacksourcefile":
".*../../third_party/skia/src/.*",
ditto.

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:126:
".*../../third_party/WebKit/Source/WebCore/.*",
ditto.

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:131: "stacksourcefile":
".*../../third_party/WebKit/Source/.*",
ditto

[Dai Mikurube (NOT FULLTIME), 2013-04-08 10:54:37]

Fixed.  Thank you!

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
File tools/deep_memory_profiler/policy.sourcefile.json (right):

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:105: "stacksourcefile":
".*../../v8/src/.*",
On 2013/04/08 10:24:09, peria wrote:
> This pattern may match with unexpected paths like
> "foo/bar/AA/v8/src/hoge.cc", right?
> 

Done.

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:115: "stacksourcefile":
".*../../v8/src/.*",
On 2013/04/08 10:24:09, peria wrote:
> ditto

Done.

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:120: "stacksourcefile":
".*../../third_party/skia/src/.*",
On 2013/04/08 10:24:09, peria wrote:
> ditto.

Done.

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:126:
".*../../third_party/WebKit/Source/WebCore/.*",
On 2013/04/08 10:24:09, peria wrote:
> ditto.

Done.

https://codereview.chromium.org/13514003/diff/13004/tools/deep_memory_profile...
tools/deep_memory_profiler/policy.sourcefile.json:131: "stacksourcefile":
".*../../third_party/WebKit/Source/.*",
On 2013/04/08 10:24:09, peria wrote:
> ditto

Done.

[peria, 2013-04-09 02:26:09]

lgtm

[M-A Ruel, 2013-04-09 03:25:17]

lgtm

https://codereview.chromium.org/13514003/diff/6002/tools/deep_memory_profiler...
File tools/deep_memory_profiler/dmprof.py (right):

https://codereview.chromium.org/13514003/diff/6002/tools/deep_memory_profiler...
tools/deep_memory_profiler/dmprof.py:441: stackfunction = None
stackfunction = rule.get('stackfunction') or rule.get('stracktrace')

https://codereview.chromium.org/13514003/diff/6002/tools/deep_memory_profiler...
tools/deep_memory_profiler/dmprof.py:447: stacksourcefile = None
same

[Dai Mikurube (NOT FULLTIME), 2013-04-09 04:31:52]

Thanks!

https://codereview.chromium.org/13514003/diff/6002/tools/deep_memory_profiler...
File tools/deep_memory_profiler/dmprof.py (right):

https://codereview.chromium.org/13514003/diff/6002/tools/deep_memory_profiler...
tools/deep_memory_profiler/dmprof.py:441: stackfunction = None
On 2013/04/09 03:25:17, Marc-Antoine Ruel wrote:
> stackfunction = rule.get('stackfunction') or rule.get('stracktrace')

Done.

https://codereview.chromium.org/13514003/diff/6002/tools/deep_memory_profiler...
tools/deep_memory_profiler/dmprof.py:447: stacksourcefile = None
On 2013/04/09 03:25:17, Marc-Antoine Ruel wrote:
> same

Done.

[commit-bot: I haz the power, 2013-04-09 04:32:23]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/dmikurube@chromium.org/13514003/26001

[commit-bot: I haz the power, 2013-04-09 04:32:32]

Presubmit check for 13514003-26001 failed and returned exit status 1.

INFO:root:Found 6 file(s).

INFO:PRESUBMIT:Running pylint on 6 files
test_find (__main__.PolicyTest) ... ok
test_load (__main__.PolicyTest) ... ok
test_update (__main__.SymbolMappingCacheTest) ... ERROR
test (__main__.UploadCommandTest) ... ok

======================================================================
ERROR: test_update (__main__.SymbolMappingCacheTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py",
line 91, in test_update
    self.MockSymbolFinder(self._TEST_FUNCTION_DICT), cache_f)
  File "/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/dmprof.py",
line 259, in update
    symbol_dict = symbol_finder.find(unresolved_addresses)
  File
"/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py",
line 37, in find
    result = collections.OrderedDict()
AttributeError: 'module' object has no attribute 'OrderedDict'

----------------------------------------------------------------------
Ran 4 tests in 0.025s

FAILED (errors=1)
INFO:PRESUBMIT:Running pylint on 8 files
test (__main__.ReduceDebuglineTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_constants (__main__.ProcMapsTest) ... ok
test_executable (__main__.ProcMapsTest) ... ok
test_executable_and_constants (__main__.ProcMapsTest) ... ok
test_load (__main__.ProcMapsTest) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.002s

OK
Running presubmit commit checks ...
Running /b/commit-queue/workdir/chromium/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/tools/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/tools/deep_memory_profiler/PRESUBMIT.py
Running pylint on 6 files.
Running
/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py
Running /b/commit-queue/workdir/chromium/tools/find_runtime_symbols/PRESUBMIT.py
Running pylint on 8 files.
Running
/b/commit-queue/workdir/chromium/tools/find_runtime_symbols/tests/reduce_debugline_test.py
Running
/b/commit-queue/workdir/chromium/tools/find_runtime_symbols/tests/proc_maps_test.py

** Presubmit ERRORS **
/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py
failed!
Command
/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py
--verbose returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/deep_memory_profiler

Presubmit checks took 4.7s to calculate.

[M-A Ruel, 2013-04-09 04:35:49]

On 2013/04/09 04:32:32, I haz the power (commit-bot) wrote:
>   File
>
"/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py",
> line 37, in find
>     result = collections.OrderedDict()
> AttributeError: 'module' object has no attribute 'OrderedDict'

It's because the CQ runs on python 2.6. I guess in practice you don't really
mind not supporting 2.6. Pawel is in the process of updating to precise which
will bring python 2.7. I'm not sure what's the best recourse here as you won't
be able to use the CQ in the meantime.

[Dai Mikurube (NOT FULLTIME), 2013-04-09 05:06:02]

On 2013/04/09 04:35:49, Marc-Antoine Ruel wrote:
> On 2013/04/09 04:32:32, I haz the power (commit-bot) wrote:
> >   File
> >
>
"/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py",
> > line 37, in find
> >     result = collections.OrderedDict()
> > AttributeError: 'module' object has no attribute 'OrderedDict'
> 
> It's because the CQ runs on python 2.6. I guess in practice you don't really
> mind not supporting 2.6. Pawel is in the process of updating to precise which
> will bring python 2.7. I'm not sure what's the best recourse here as you won't
> be able to use the CQ in the meantime.

Thanks, Marc-Antoine.  I finally found that we may run this script in Python 2.6
in some environments (out of buildbots).  Falling back to
third_party/simplejson's OrderedDict as tools/json_schema_compiler does.

[Dai Mikurube (NOT FULLTIME), 2013-04-09 05:20:08]

Committing with this change...  Will revert if any problems.

[commit-bot: I haz the power, 2013-04-09 05:20:48]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/dmikurube@chromium.org/13514003/37001

[commit-bot: I haz the power, 2013-04-09 05:20:58]

Presubmit check for 13514003-37001 failed and returned exit status 1.

INFO:root:Found 6 file(s).

INFO:PRESUBMIT:Running pylint on 6 files
************* Module dmprof_test
E0611: 17,2: No name 'OrderedDict' in module 'collections'
Traceback (most recent call last):
  File
"/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py",
line 22, in <module>
    from simplejson import OrderedDict
ImportError: No module named simplejson
INFO:PRESUBMIT:Running pylint on 8 files
************* Module find_runtime_symbols
E0611: 20,2: No name 'OrderedDict' in module 'collections'
test (__main__.ReduceDebuglineTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_constants (__main__.ProcMapsTest) ... ok
test_executable (__main__.ProcMapsTest) ... ok
test_executable_and_constants (__main__.ProcMapsTest) ... ok
test_load (__main__.ProcMapsTest) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK
Running presubmit commit checks ...
Running /b/commit-queue/workdir/chromium/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/tools/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/tools/deep_memory_profiler/PRESUBMIT.py
Running pylint on 6 files.
Running
/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py
Running /b/commit-queue/workdir/chromium/tools/find_runtime_symbols/PRESUBMIT.py
Running pylint on 8 files.
Running
/b/commit-queue/workdir/chromium/tools/find_runtime_symbols/tests/reduce_debugline_test.py
Running
/b/commit-queue/workdir/chromium/tools/find_runtime_symbols/tests/proc_maps_test.py

** Presubmit ERRORS **
Fix pylint errors first.

/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py
failed!
Command
/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py
--verbose returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/deep_memory_profiler

Fix pylint errors first.

Presubmit checks took 4.8s to calculate.

[Dai Mikurube (NOT FULLTIME), 2013-04-09 05:22:24]

Ah... I'll take a look at this.

[Dai Mikurube (NOT FULLTIME), 2013-04-09 05:32:25]

The path added in sys.path was wrong.  I actually tried importing OrderedDict
from simplejson before uploading, but I found it always succeeds since Python
2.7 has simplejson.

I tried the uploaded code in my local Python 2.6 environment.  Committing
again...

[commit-bot: I haz the power, 2013-04-09 05:32:52]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/dmikurube@chromium.org/13514003/43001

[commit-bot: I haz the power, 2013-04-09 05:32:59]

Presubmit check for 13514003-43001 failed and returned exit status 1.

INFO:root:Found 6 file(s).

INFO:PRESUBMIT:Running pylint on 6 files
************* Module dmprof_test
E0611: 17,2: No name 'OrderedDict' in module 'collections'
test_find (__main__.PolicyTest) ... ok
test_load (__main__.PolicyTest) ... ok
test_update (__main__.SymbolMappingCacheTest) ... ok
test (__main__.UploadCommandTest) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.025s

OK
INFO:PRESUBMIT:Running pylint on 8 files
************* Module find_runtime_symbols
E0611: 20,2: No name 'OrderedDict' in module 'collections'
test (__main__.ReduceDebuglineTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_constants (__main__.ProcMapsTest) ... ok
test_executable (__main__.ProcMapsTest) ... ok
test_executable_and_constants (__main__.ProcMapsTest) ... ok
test_load (__main__.ProcMapsTest) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK
Running presubmit commit checks ...
Running /b/commit-queue/workdir/chromium/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/tools/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/tools/deep_memory_profiler/PRESUBMIT.py
Running pylint on 6 files.
Running
/b/commit-queue/workdir/chromium/tools/deep_memory_profiler/tests/dmprof_test.py
Running /b/commit-queue/workdir/chromium/tools/find_runtime_symbols/PRESUBMIT.py
Running pylint on 8 files.
Running
/b/commit-queue/workdir/chromium/tools/find_runtime_symbols/tests/reduce_debugline_test.py
Running
/b/commit-queue/workdir/chromium/tools/find_runtime_symbols/tests/proc_maps_test.py

** Presubmit ERRORS **
Fix pylint errors first.

Fix pylint errors first.

Presubmit checks took 4.7s to calculate.

[Dai Mikurube (NOT FULLTIME), 2013-04-09 05:38:18]

Added pylint: disable, and run git cl presubmit in my local Python 2.6
environment...

[commit-bot: I haz the power, 2013-04-09 05:38:31]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/dmikurube@chromium.org/13514003/50001

[commit-bot: I haz the power, 2013-04-09 05:46:15]

Change committed as 193022