Chromium Code Reviews
|
|
Chromium Code Reviews
Issue
10411047:
Type profiler by intercepting 'new' and 'delete' expressions. (Closed)
|
Created:
8 years, 7 months ago by Dai Mikurube (NOT FULLTIME) Modified:
8 years, 2 months ago Reviewers:
Alexander Potapenko, jar (doing other things), Ryan Sleevi, jam, M-A Ruel, Nico, agl, willchan no longer on Chromium CC:
chromium-reviews Base URL:
svn://svn.chromium.org/chrome/trunk/src Visibility:
Public. |
DescriptionType profiler by intercepting 'new' and 'delete' expressions.
It stores mapping between object's starting addresses and their
allocated types when a build option 'clang_type_profiler=1' is
specified. It enables information like
"an object at 0x37f3c88 is an instance of std::string."
Nothing is changed when the option is not specified.
It depends on a modified version of the LLVM/Clang compiler
introduced at deps/third_party/llvm-allocated-type.
BUG=123758
TEST=build with clang_type_profiler=1 and run type_profiler_unittests and type_profiler_map_unittests manually.
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=158752
Patch Set 1 #Patch Set 2 : refined. #Patch Set 3 : Fixed? #Patch Set 4 : loadable_module #Patch Set 5 : Add logging in operator new. #Patch Set 6 : Rebased. #Patch Set 7 : Include <typeinfo> and declare operator new in Chromium's side. #Patch Set 8 : refined. #Patch Set 9 : Rebased and added a license comment. #Patch Set 10 : Add delete overload and some additional anti-warning flags for the latest Clang trunk. #Patch Set 11 : Store allocated types in AddressMap of TCMalloc. #Patch Set 12 : Print allocated sizes per every types. #Patch Set 13 : Renamed some files. #Patch Set 14 : rebased, and updated. #Patch Set 15 : Updated LookupAllocatedType. #Patch Set 16 : Support operator delete[]. #
Total comments: 1
Patch Set 17 : first support for dumping allocated type #Patch Set 18 : Use -fintercept-allocation-functions. #Patch Set 19 : another summary for comparison #Patch Set 20 : suspend type logging between fork and exec. #Patch Set 21 : separate memory for allocated-type out of dmprof. #Patch Set 22 : Separated out dumper. #Patch Set 23 : modified gyp(i) #Patch Set 24 : remove some comments #
Total comments: 74
Patch Set 25 : reflected the comments. #Patch Set 26 : upload again #
Total comments: 4
Patch Set 27 : reflected the comments #Patch Set 28 : stop interceptors in an entire child process #
Total comments: 19
Patch Set 29 : reflected the comments #17-#19. #
Total comments: 22
Patch Set 30 : reflected comments #21,22 #
Total comments: 72
Patch Set 31 : reflected the comment #24. #Patch Set 32 : updated #Patch Set 33 : commented otu #Patch Set 34 : use RAW_DCHECK in allocated_type_map.cc. #Patch Set 35 : refined include path. #Patch Set 36 : fixed dependencies in gyp #
Total comments: 5
Patch Set 37 : fix gyp #
Total comments: 7
Patch Set 38 : reflected maruel's comment. Ready for review. #
Total comments: 30
Patch Set 39 : reflected Jim's comments. #
Total comments: 5
Patch Set 40 : reflected jar's comments, and added tests for allocated_type_profiler and AllocatedTypeMap. #
Total comments: 2
Patch Set 41 : removed "allocated" from all names. #Patch Set 42 : fixed comments. #
Total comments: 34
Patch Set 43 : reflected the comments. #
Total comments: 15
Patch Set 44 : reflected #Patch Set 45 : fixed #include position #
Total comments: 2
Patch Set 46 : stopped NULL checks #Patch Set 47 : modified a comment. #Patch Set 48 : fix IsAvailable #
Total comments: 25
Patch Set 49 : fixed for jar's comments #67 #
Total comments: 6
Patch Set 50 : removed automatic including "type_profiler.h" and fixed for jar's comments. #
Total comments: 12
Patch Set 51 : fixed for jar's comments #72 #
Total comments: 20
Patch Set 52 : fixed for Ryan's comments #78 #
Total comments: 18
Patch Set 53 : removed unnecessary friend #
Total comments: 8
Patch Set 54 : moved/added/deleted #if and #include #Patch Set 55 : fixed for Ryan's comments #83 #Patch Set 56 : nit fix #Patch Set 57 : nit fix 2 #
Total comments: 4
Patch Set 58 : fixed for Ryan's comments #86. #Patch Set 59 : rebased #Patch Set 60 : rebased #
Total comments: 8
Patch Set 61 : fixed #Messages
Total messages: 99 (0 generated)
http://codereview.chromium.org/10411047/diff/35001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/35001/build/common.gypi#newcode2693 build/common.gypi:2693: Add -Wno-unused-private-field here.
Hi, Could you please take a look at this change? (Or, could you forward it to right persons if you know?) It changes a lower layer (malloc and 'new'). But, the change happens only when a build option 'clang_use_allocated_type=1' is given. Jim, I wonder if you could check especially that it changes nothing for normal builds. Adam, William, I wonder if I could have your opinion especially on process_util_posix.cc about http://crbug.com/36678. Alexander, Nico, This is the Chromium-side change for deps/third_party/llvm-allocated-type at http://crrev.com/149572. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type.h:12: void* __op_delete_intercept__(void*, size_t, const std::type_info &); These functions are inserted around all 'new' and 'delete' expressions by the compiler option '-fintercept-allocation-functions' introduced in http://crrev.com/149572. http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:650: #endif These are to avoid the DANGER of http://crbug.com/36678. This change acquires locks in 'new' and 'delete'. As a result, some STL operations cause lock acquisitions. These SuspendAllocatedTypeIntercept stops lock acquisitions. I know they make the code very unreadable... It is bad. If someone has good ideas to fix it, please tell me that. http://codereview.chromium.org/10411047/diff/71001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode1604 build/common.gypi:1604: ['_target_name=="mksnapshot" or _target_name=="protoc" or _target_name=="ppGoogleNaClPluginChrome"', { Some binaries doesn't use tcmalloc. We use no-op __op_{new|delete|_intercept__ for these binaries since the allocated type mapping is recorded in a tcmalloc-dependent data structure.
http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type.h:12: void* __op_delete_intercept__(void*, size_t, const std::type_info &); If you need to clarify something about the change, please write a comment in the code. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_log.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_log.cc:10: std::cerr << "Allocated " Have you tried to run this already? I'm afraid an allocation may occur while you're logging. http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:785: ResumeAllocatedTypeIntercept(); There's no point in resuming the interception if the child process is known to call exec() later. http://codereview.chromium.org/10411047/diff/71001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode245 build/common.gypi:245: # TODO(dmikurube): Support mac. Note that tcmalloc does not work on Mac within Chromium. http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode1604 build/common.gypi:1604: ['_target_name=="mksnapshot" or _target_name=="protoc" or _target_name=="ppGoogleNaClPluginChrome"', { Please write a comment about this. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:41: static AddressMap<AllocatedObject>* allocated_type_map = NULL; How about keeping a per-thread allocated_type_map. It will be a bit harder to aggregate the data, but you'll have less problems with locking. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:10: #ifndef PERFTOOLS_DLL_DECL Isn't PERFTOOLS_DLL_DECL declared somewhere already? Also note it's impossible to build Chrome using Clang on Windows right now.
http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:650: #endif On 2012/08/02 07:43:55, Dai Mikurube wrote: > These are to avoid the DANGER of http://crbug.com/36678. > > This change acquires locks in 'new' and 'delete'. As a result, some STL > operations cause lock acquisitions. These SuspendAllocatedTypeIntercept stops > lock acquisitions. > > I know they make the code very unreadable... It is bad. If someone has good > ideas to fix it, please tell me that. As far as I understand the bug, any code that does allocation between the fork above and the exec below is wrong. So the allocations should be removed (independent of your change), and then you don't need to change this file (?) http://codereview.chromium.org/10411047/diff/71001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode1617 build/common.gypi:1617: '<!(realpath <(DEPTH)/base/allocator/allocator.gyp)' + ':allocated_type_tcmalloc#target', The realpath bit here shouldn't be needed
Hi Alexander, Nico, Thank you for the comments. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type.h:12: void* __op_delete_intercept__(void*, size_t, const std::type_info &); On 2012/08/02 14:57:12, Alexander Potapenko wrote: > If you need to clarify something about the change, please write a comment in the > code. Ok, I'll do that. (I added these codereview comments just before leaving. I didn't have time to re-upload.) http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_log.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_log.cc:10: std::cerr << "Allocated " On 2012/08/02 14:57:12, Alexander Potapenko wrote: > Have you tried to run this already? > I'm afraid an allocation may occur while you're logging. Actually, it is not called by default common.gypi. I sometimes turn on them instead of allocated_type_ignore.cc for debugging. When I used it, it didn't break anything. I'm planning to fix it when it breaks. http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:650: #endif On 2012/08/02 15:35:03, Nico wrote: > On 2012/08/02 07:43:55, Dai Mikurube wrote: > > These are to avoid the DANGER of http://crbug.com/36678. > > > > This change acquires locks in 'new' and 'delete'. As a result, some STL > > operations cause lock acquisitions. These SuspendAllocatedTypeIntercept stops > > lock acquisitions. > > > > I know they make the code very unreadable... It is bad. If someone has good > > ideas to fix it, please tell me that. > > As far as I understand the bug, any code that does allocation between the fork > above and the exec below is wrong. So the allocations should be removed > (independent of your change), and then you don't need to change this file (?) In my understanding, they don't allocate (malloc) additional memory here. "new" is really called in STL, but they are placement news. My interceptor catches also placement news, and it instruments lock acquisitions. That is the problem here. In fact, I'm not completely sure that additional memory is not allocated. It may sometimes happen by <allocator>? Removing all STL operations may contribute to stability. http://codereview.chromium.org/10411047/diff/71001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode245 build/common.gypi:245: # TODO(dmikurube): Support mac. On 2012/08/02 14:57:12, Alexander Potapenko wrote: > Note that tcmalloc does not work on Mac within Chromium. Ah, that's true. Thanks. I have to find another allocated-type-map storage in Mac... http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode1604 build/common.gypi:1604: ['_target_name=="mksnapshot" or _target_name=="protoc" or _target_name=="ppGoogleNaClPluginChrome"', { On 2012/08/02 14:57:12, Alexander Potapenko wrote: > Please write a comment about this. Yes, I'll do that. Thank you. http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode1617 build/common.gypi:1617: '<!(realpath <(DEPTH)/base/allocator/allocator.gyp)' + ':allocated_type_tcmalloc#target', On 2012/08/02 15:35:03, Nico wrote: > The realpath bit here shouldn't be needed It is actually needed because of http://crrev.com/146172. (I was surprised when it happened.) http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:41: static AddressMap<AllocatedObject>* allocated_type_map = NULL; On 2012/08/02 14:57:12, Alexander Potapenko wrote: > How about keeping a per-thread allocated_type_map. It will be a bit harder to > aggregate the data, but you'll have less problems with locking. I thought about that. But in my thought, the problem is that an object may be deleted by another thread than it is newed. (I think it hardly happens, but not never.) http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:10: #ifndef PERFTOOLS_DLL_DECL On 2012/08/02 14:57:12, Alexander Potapenko wrote: > Isn't PERFTOOLS_DLL_DECL declared somewhere already? > Also note it's impossible to build Chrome using Clang on Windows right now. Ah, right. We may want to remote it. I'll consider it later soon.
http://codereview.chromium.org/10411047/diff/71001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode1617 build/common.gypi:1617: '<!(realpath <(DEPTH)/base/allocator/allocator.gyp)' + ':allocated_type_tcmalloc#target', On 2012/08/02 16:14:35, Dai Mikurube wrote: > On 2012/08/02 15:35:03, Nico wrote: > > The realpath bit here shouldn't be needed > > It is actually needed because of http://crrev.com/146172. > (I was surprised when it happened.) No other dependency in gyp uses realpath though, right? Why is this special? It feels like we should open a bug for this being necessary and probably revert 146172 if this is the consequence :-/
http://codereview.chromium.org/10411047/diff/71001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode1617 build/common.gypi:1617: '<!(realpath <(DEPTH)/base/allocator/allocator.gyp)' + ':allocated_type_tcmalloc#target', On 2012/08/02 16:18:49, Nico wrote: > On 2012/08/02 16:14:35, Dai Mikurube wrote: > > On 2012/08/02 15:35:03, Nico wrote: > > > The realpath bit here shouldn't be needed > > > > It is actually needed because of http://crrev.com/146172. > > (I was surprised when it happened.) > > No other dependency in gyp uses realpath though, right? Why is this special? > > It feels like we should open a bug for this being necessary and probably revert > 146172 if this is the consequence :-/ I guess it happens because it is a very hacky gyp dependency description. This 'dependencies' is lazily evaluated after most paths are translated using 'script_dir' and other parameters. To describe it completely, I have to reread gyp. I remember that it is an effect of 'target_conditions'. Such a hacky dependency is required since we need to add dependency depending on its target. (gyp is complicated...)
http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type.h:11: void* __op_new_intercept__(void*, size_t, const std::type_info&); nit: Declarations should have variable names, matching that seen in the definition. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_ignore.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_ignore.cc:9: void* ptr, size_t size, const std::type_info& type) { nit: one arg per line, indent to match paren. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_log.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_log.cc:9: void* ptr, size_t size, const std::type_info& type) { nit: one arg per line; indent to paren http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:11: volatile Atomic32 do_not_intercept = 0; It is not clear why you are using atomics. When defining logical variables, in is always confusing to have a negative in the name. Suggest using: g_enable_intercept which is probably the negation of what you have. Also note that globals should be prefixed by "g_" http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:14: base::subtle::NoBarrier_AtomicIncrement(&do_not_intercept, 1); Given the name of the function, I suspect you should do an AtomicExchange, and not an increment. Increments would be appropriate if calls nested. Since the variable started out at zero, which I assume means false, and the name was negating, I think you start out with a default of intercepting. As a result, I would guess that the only legal call at that point would this SuspendAllocatedTypeIntercept (which drives the value more positive). It is interesting that with the current incrementing semantics, a call to ResumeAlocatedTypeIntercept would *also* turn off interception :-/. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:22: void* ptr, size_t size, const std::type_info& type) { nit: wrap to match the indent of the paren, and put one arg per line. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:23: if (do_not_intercept == 0) { You should not access an atomic directly. Use NoBarrier_Load(). ...but I have no idea why you're doing this with atomics. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:32: void* ptr, size_t size, const std::type_info &type) { nit: wrap to match paren indent. http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:650: #endif I don't follow this motivation. ...but... if your goal is to make this code more readable by not injecting the ifdefs, consider always calling the functions, but having the functions become no-ops (in the implementation file) when !defined(USE_ALLOCATED_TYPE). The compiler will generally optimize out the no-op function... so the impact is identical on released code (and readability is higher). On 2012/08/02 07:43:55, Dai Mikurube wrote: > These are to avoid the DANGER of http://crbug.com/36678. > > This change acquires locks in 'new' and 'delete'. As a result, some STL > operations cause lock acquisitions. These SuspendAllocatedTypeIntercept stops > lock acquisitions. > > I know they make the code very unreadable... It is bad. If someone has good > ideas to fix it, please tell me that. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:41: static AddressMap<AllocatedObject>* allocated_type_map = NULL; Allocation on one thread, and deletion on another thread, is very common. Task objects follow this model commonly, as tasks are constructed on one thread, then run and deleted on a second thread. On 2012/08/02 16:14:35, Dai Mikurube wrote: > On 2012/08/02 14:57:12, Alexander Potapenko wrote: > > How about keeping a per-thread allocated_type_map. It will be a bit harder to > > aggregate the data, but you'll have less problems with locking. > > I thought about that. But in my thought, the problem is that an object may be > deleted by another thread than it is newed. (I think it hardly happens, but not > never.) http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:61: void* address, unsigned int size, const std::type_info& type) { nit: align parameters, one per line, indented to match the paren. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:62: if (allocated_type_lock.IsHeld()) { I suspect you're trying to use this as recursion protection... but this function is only guaranteed correct when the lock is held by *this* thread. It can be wrong when the lock is held by another thread (indicating true or false, with no correlation to reality(?). Do you, for instance, really mean to exit if some other thread is holding the lock? Do you mind that you can be told "false" in this call, and then spin because the lock *is* held by the time you reach the next line? I'm not clear on the logic here. The same logic exists in all the functions below... and I have the same questions. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:96: void (*callback)(const void*, AllocatedObject*, void*), void* arg) { This class of API is always frightening to me. You are going to callback into arbitrary code, with a lock held. It is possible for this to work, but it is common to have subtle undetected error. IMO, general locking philosophy suggests: you want to only lock during access to data, and not lock to prevent other code from "running." In addition, you want to hold the lock for the briefest possible period, and be assured that you never call another function that might block (or wait on a second lock... leading to a deadlock). Most commonly, when code holding a lock calls into other code, the function name even states that a lock is held, so there is less confusion. With a callback, you have no such guarantees whatever. :-(. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:31: void* address, unsigned int size, const std::type_info& type); nit: one arg per line. Indent matching paren.
http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type.h:5: #ifndef BASE_ALLOCATOR_ALLOCATED_TYPE_INTERCEPT_H_ Please either rename the file or the guard definition. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.h (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.h:4: You'll need an #ifdef... guard here. http://codereview.chromium.org/10411047/diff/71001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode2367 build/common.gypi:2367: ['clang==1 and clang_use_allocated_type==1', { If clang_use_allocated_type=1 implies clang=1, you don't need to check both. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:14: #include "base/spinlock.h" Mind the order of includes, please. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:5: #ifndef BASE_ALLOCATED_TYPE_MAP_H_ The include guard should probably have a different name (compare to other guards in this dir) http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:22: struct AllocatedObject { Use the specified order of declarations within a class: public: before private:, methods before data members (variables), etc. (http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml?showone=Declar...)
Thank you, all. Replied and updated the patch. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type.h:11: void* __op_new_intercept__(void*, size_t, const std::type_info&); On 2012/08/02 23:38:30, jar wrote: > nit: Declarations should have variable names, matching that seen in the > definition. Done. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type.h:12: void* __op_delete_intercept__(void*, size_t, const std::type_info &); On 2012/08/02 16:14:35, Dai Mikurube wrote: > On 2012/08/02 14:57:12, Alexander Potapenko wrote: > > If you need to clarify something about the change, please write a comment in > the > > code. > > Ok, I'll do that. (I added these codereview comments just before leaving. I > didn't have time to re-upload.) Done. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_ignore.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_ignore.cc:9: void* ptr, size_t size, const std::type_info& type) { On 2012/08/02 23:38:30, jar wrote: > nit: one arg per line, indent to match paren. Done. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_log.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_log.cc:9: void* ptr, size_t size, const std::type_info& type) { On 2012/08/02 23:38:30, jar wrote: > nit: one arg per line; indent to paren Done. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:11: volatile Atomic32 do_not_intercept = 0; On 2012/08/02 23:38:30, jar wrote: > It is not clear why you are using atomics. > > When defining logical variables, in is always confusing to have a negative in > the name. > > Suggest using: > > g_enable_intercept > > which is probably the negation of what you have. > > Also note that globals should be prefixed by "g_" The reason why I use Atomic is they may be called from multiple threads. I would like to disable the interceptors whenever this value is not zero (in other words, whenever someone is in a dangerous zone -- between fork and exec). I'm afraid LaunchProcess() (the caller of these functions in process_util_posix.cc) can work in parallel. Or, is it guaranteed not to work in parallel? I chose g_disable_intercept_if_positive since it disables interceptors while it is not zero. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:14: base::subtle::NoBarrier_AtomicIncrement(&do_not_intercept, 1); On 2012/08/02 23:38:30, jar wrote: > Given the name of the function, I suspect you should do an AtomicExchange, and > not an increment. Increments would be appropriate if calls nested. > > Since the variable started out at zero, which I assume means false, and the name > was negating, I think you start out with a default of intercepting. As a > result, I would guess that the only legal call at that point would this > SuspendAllocatedTypeIntercept (which drives the value more positive). It is > interesting that with the current incrementing semantics, a call to > ResumeAlocatedTypeIntercept would *also* turn off interception :-/. I agree with that ResumeAllocatedTypeIntercept before Suspend would disable intercepting. I changed the disabling condition to g_disable_intercept_if_positive > 0. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:22: void* ptr, size_t size, const std::type_info& type) { On 2012/08/02 23:38:30, jar wrote: > nit: wrap to match the indent of the paren, and put one arg per line. Done. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:23: if (do_not_intercept == 0) { On 2012/08/02 23:38:30, jar wrote: > You should not access an atomic directly. Use NoBarrier_Load(). > > ...but I have no idea why you're doing this with atomics. Done. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:32: void* ptr, size_t size, const std::type_info &type) { On 2012/08/02 23:38:30, jar wrote: > nit: wrap to match paren indent. Done. http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:650: #endif On 2012/08/02 23:38:30, jar wrote: > I don't follow this motivation. > > ...but... if your goal is to make this code more readable by not injecting the > ifdefs, consider always calling the functions, but having the functions become > no-ops (in the implementation file) when !defined(USE_ALLOCATED_TYPE). > > The compiler will generally optimize out the no-op function... so the impact is > identical on released code (and readability is higher). > > On 2012/08/02 07:43:55, Dai Mikurube wrote: > > These are to avoid the DANGER of http://crbug.com/36678. > > > > This change acquires locks in 'new' and 'delete'. As a result, some STL > > operations cause lock acquisitions. These SuspendAllocatedTypeIntercept stops > > lock acquisitions. > > > > I know they make the code very unreadable... It is bad. If someone has good > > ideas to fix it, please tell me that. > Thank you. That is a good idea. For now, I keep it as is since we should consider more (including removing STL operations) around this issue. I added a temporary comment on #include "base/allocator/allocated_type_tcmalloc.h". http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:785: ResumeAllocatedTypeIntercept(); On 2012/08/02 14:57:12, Alexander Potapenko wrote: > There's no point in resuming the interception if the child process is known to > call exec() later. Ah, right. For now, I kept it as is, but I added a comment. (I don't change the code before decision.) http://codereview.chromium.org/10411047/diff/71001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode1604 build/common.gypi:1604: ['_target_name=="mksnapshot" or _target_name=="protoc" or _target_name=="ppGoogleNaClPluginChrome"', { On 2012/08/02 16:14:35, Dai Mikurube wrote: > On 2012/08/02 14:57:12, Alexander Potapenko wrote: > > Please write a comment about this. > > Yes, I'll do that. Thank you. Done. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:41: static AddressMap<AllocatedObject>* allocated_type_map = NULL; On 2012/08/02 23:38:30, jar wrote: > Allocation on one thread, and deletion on another thread, is very common. Task > objects follow this model commonly, as tasks are constructed on one thread, then > run and deleted on a second thread. > > On 2012/08/02 16:14:35, Dai Mikurube wrote: > > On 2012/08/02 14:57:12, Alexander Potapenko wrote: > > > How about keeping a per-thread allocated_type_map. It will be a bit harder > to > > > aggregate the data, but you'll have less problems with locking. > > > > I thought about that. But in my thought, the problem is that an object may be > > deleted by another thread than it is newed. (I think it hardly happens, but > not > > never.) > Thanks. We cannot use per-thread storage for this case. :( http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:61: void* address, unsigned int size, const std::type_info& type) { On 2012/08/02 23:38:30, jar wrote: > nit: align parameters, one per line, indented to match the paren. Done. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:62: if (allocated_type_lock.IsHeld()) { On 2012/08/02 23:38:30, jar wrote: > I suspect you're trying to use this as recursion protection... but this function > is only guaranteed correct when the lock is held by *this* thread. It can be > wrong when the lock is held by another thread (indicating true or false, with no > correlation to reality(?). > > Do you, for instance, really mean to exit if some other thread is holding the > lock? Do you mind that you can be told "false" in this call, and then spin > because the lock *is* held by the time you reach the next line? > > I'm not clear on the logic here. > > > The same logic exists in all the functions below... and I have the same > questions. They were just-in-case code when I was doing heavy try-and-error. Removed them since they're actually not required. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:96: void (*callback)(const void*, AllocatedObject*, void*), void* arg) { On 2012/08/02 23:38:30, jar wrote: > This class of API is always frightening to me. You are going to callback into > arbitrary code, with a lock held. > > It is possible for this to work, but it is common to have subtle undetected > error. > > IMO, general locking philosophy suggests: you want to only lock during access to > data, and not lock to prevent other code from "running." In addition, you want > to hold the lock for the briefest possible period, and be assured that you never > call another function that might block (or wait on a second lock... leading to a > deadlock). > > Most commonly, when code holding a lock calls into other code, the function name > even states that a lock is held, so there is less confusion. With a callback, > you have no such guarantees whatever. :-(. In fact, I was a little worried about that. The user of this function is actually only one in another patch, I remove this function for now, and consider providing another limited function to the user. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:10: #ifndef PERFTOOLS_DLL_DECL On 2012/08/02 16:14:35, Dai Mikurube wrote: > On 2012/08/02 14:57:12, Alexander Potapenko wrote: > > Isn't PERFTOOLS_DLL_DECL declared somewhere already? > > Also note it's impossible to build Chrome using Clang on Windows right now. > > Ah, right. We may want to remote it. I'll consider it later soon. Removed. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:31: void* address, unsigned int size, const std::type_info& type); On 2012/08/02 23:38:30, jar wrote: > nit: one arg per line. Indent matching paren. Done.
Thanks for your comments, Alexander. We passed each other. I'll do them later. (maybe next week)
I was uploading a wrong patch... Uploaded as #26.
http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:11: volatile Atomic32 do_not_intercept = 0; Please use a positive name, rather than a negative "disable." In this case, please switch to g_enable_intercept_if_positive Use of a negative complicates reading a fair amount. An example where this is commonly (sadly) seen is NDEBUG. This should IMO have been defined as DEBUG, so that you don't have to do mental gymnastics and cancel out a double negative to figure out what is happening. I learned of this philosophy years ago, when discussing compiler flags, which *should* always be in the positive, and shouldn't have words like "disable" or "no" in their names. On 2012/08/03 10:01:50, Dai Mikurube wrote: > On 2012/08/02 23:38:30, jar wrote: > > It is not clear why you are using atomics. > > > > When defining logical variables, in is always confusing to have a negative in > > the name. > > > > Suggest using: > > > > g_enable_intercept > > > > which is probably the negation of what you have. > > > > Also note that globals should be prefixed by "g_" > > The reason why I use Atomic is they may be called from multiple threads. I > would like to disable the interceptors whenever this value is not zero (in other > words, whenever someone is in a dangerous zone -- between fork and exec). > > I'm afraid LaunchProcess() (the caller of these functions in > process_util_posix.cc) can work in parallel. Or, is it guaranteed not to work > in parallel? > > I chose g_disable_intercept_if_positive since it disables interceptors while it > is not zero. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:11: volatile Atomic32 do_not_intercept = 0; re: atomics If this is accessed on multiple threads, then the results will be pretty unpredictable, as allocations and deallocations will be lost when some thread asynchronously turns this flag on and off. Is that confusion, such as a deallocation not being found in the allocation map, acceptable? Is it acceptable that a deallocation may be for a different type than the allocation, because an intervening deallocation and allocation was missed? On 2012/08/03 10:01:50, Dai Mikurube wrote: > On 2012/08/02 23:38:30, jar wrote: > > It is not clear why you are using atomics. > > > > When defining logical variables, in is always confusing to have a negative in > > the name. > > > > Suggest using: > > > > g_enable_intercept > > > > which is probably the negation of what you have. > > > > Also note that globals should be prefixed by "g_" > > The reason why I use Atomic is they may be called from multiple threads. I > would like to disable the interceptors whenever this value is not zero (in other > words, whenever someone is in a dangerous zone -- between fork and exec). > > I'm afraid LaunchProcess() (the caller of these functions in > process_util_posix.cc) can work in parallel. Or, is it guaranteed not to work > in parallel? > > I chose g_disable_intercept_if_positive since it disables interceptors while it > is not zero. http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:650: #endif Your original comment was "...unreadable. It is bad. ...ideas to fix, please tell me". My suggestion to make it readable was to remove the ifdef here, and put it in the called function. Why not do it? Why leave the code harder to read? On 2012/08/03 10:01:50, Dai Mikurube wrote: > On 2012/08/02 23:38:30, jar wrote: > > I don't follow this motivation. > > > > ...but... if your goal is to make this code more readable by not injecting the > > ifdefs, consider always calling the functions, but having the functions become > > no-ops (in the implementation file) when !defined(USE_ALLOCATED_TYPE). > > > > The compiler will generally optimize out the no-op function... so the impact > is > > identical on released code (and readability is higher). > > > > On 2012/08/02 07:43:55, Dai Mikurube wrote: > > > These are to avoid the DANGER of http://crbug.com/36678. > > > > > > This change acquires locks in 'new' and 'delete'. As a result, some STL > > > operations cause lock acquisitions. These SuspendAllocatedTypeIntercept > stops > > > lock acquisitions. > > > > > > I know they make the code very unreadable... It is bad. If someone has > good > > > ideas to fix it, please tell me that. > > > > Thank you. That is a good idea. > > For now, I keep it as is since we should consider more (including removing STL > operations) around this issue. I added a temporary comment on #include > "base/allocator/allocated_type_tcmalloc.h". http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:785: ResumeAllocatedTypeIntercept(); It strikes me that when you fork, you don't want to get copies of duplicate data (from the two sides of the fork). As a result, I'd expect you to permanently stop collecting in the child (that will exec), and abandoning your data (that was collected pre-fork) from the child. What am I missing? On 2012/08/03 10:01:50, Dai Mikurube wrote: > On 2012/08/02 14:57:12, Alexander Potapenko wrote: > > There's no point in resuming the interception if the child process is known to > > call exec() later. > > Ah, right. For now, I kept it as is, but I added a comment. (I don't change > the code before decision.) http://codereview.chromium.org/10411047/diff/66015/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/66015/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:15: #include "base/low_level_alloc.h" As per comment by Ptapenko: Please alphabetize header names. http://codereview.chromium.org/10411047/diff/66015/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:78: SpinLockHolder l(&allocated_type_lock); Doesn't even your specially allocator call the new notification? ...and as a result, does this reenter and deadlock? Or does your notification only take place when a standard TCMalloc based allocation takes place? I thought you noted that you even got notifications when a placement new was called... so it is a bit surprising that a new with your allocator would not be notified.
Thanks for the comments. The uploaded patchset doesn't reflect all comments yet, but at least I added comments. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type.h:5: #ifndef BASE_ALLOCATOR_ALLOCATED_TYPE_INTERCEPT_H_ On 2012/08/03 09:57:07, Alexander Potapenko wrote: > Please either rename the file or the guard definition. Done. http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:11: volatile Atomic32 do_not_intercept = 0; re: atomics Yes, it is acceptable for now. The allocated_type_map is mainly designed as a helper for a long-running profiler (DMP). DMP focuses on trend of memory usage... some error is acceptable. The "disabled" time is usually very short compared to the entire runtime. (LaunchProcess and GetAppOutputInternal called rarely.) Ot doesn't work well for tests of LaunchProcess and GetAppOutputInternal, of course. We give up it for now. Ok, now I know it is confusing behavior. I add a comment on it. On 2012/08/04 01:13:47, jar wrote: > re: atomics > > If this is accessed on multiple threads, then the results will be pretty > unpredictable, as allocations and deallocations will be lost when some thread > asynchronously turns this flag on and off. > > Is that confusion, such as a deallocation not being found in the allocation map, > acceptable? Is it acceptable that a deallocation may be for a different type > than the allocation, because an intervening deallocation and allocation was > missed? > > On 2012/08/03 10:01:50, Dai Mikurube wrote: > > On 2012/08/02 23:38:30, jar wrote: > > > It is not clear why you are using atomics. > > > > > > When defining logical variables, in is always confusing to have a negative > in > > > the name. > > > > > > Suggest using: > > > > > > g_enable_intercept > > > > > > which is probably the negation of what you have. > > > > > > Also note that globals should be prefixed by "g_" > > > > The reason why I use Atomic is they may be called from multiple threads. I > > would like to disable the interceptors whenever this value is not zero (in > other > > words, whenever someone is in a dangerous zone -- between fork and exec). > > > > I'm afraid LaunchProcess() (the caller of these functions in > > process_util_posix.cc) can work in parallel. Or, is it guaranteed not to work > > in parallel? > > > > I chose g_disable_intercept_if_positive since it disables interceptors while > it > > is not zero. > http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:11: volatile Atomic32 do_not_intercept = 0; re: name Ok, your suggestion is so reasonable. But in this case, "g_enable_intercept_if_zero(_or_negative)"? "g_enable_intercept_if_positive" tells a lie. Or, another way may be to make the default value "1", and decrement in Suspend...(). Which do you prefer? On 2012/08/04 01:13:47, jar wrote: > Please use a positive name, rather than a negative "disable." > > In this case, please switch to g_enable_intercept_if_positive > > Use of a negative complicates reading a fair amount. An example where this is > commonly (sadly) seen is NDEBUG. This should IMO have been defined as DEBUG, so > that you don't have to do mental gymnastics and cancel out a double negative to > figure out what is happening. > > I learned of this philosophy years ago, when discussing compiler flags, which > *should* always be in the positive, and shouldn't have words like "disable" or > "no" in their names. > > On 2012/08/03 10:01:50, Dai Mikurube wrote: > > On 2012/08/02 23:38:30, jar wrote: > > > It is not clear why you are using atomics. > > > > > > When defining logical variables, in is always confusing to have a negative > in > > > the name. > > > > > > Suggest using: > > > > > > g_enable_intercept > > > > > > which is probably the negation of what you have. > > > > > > Also note that globals should be prefixed by "g_" > > > > The reason why I use Atomic is they may be called from multiple threads. I > > would like to disable the interceptors whenever this value is not zero (in > other > > words, whenever someone is in a dangerous zone -- between fork and exec). > > > > I'm afraid LaunchProcess() (the caller of these functions in > > process_util_posix.cc) can work in parallel. Or, is it guaranteed not to work > > in parallel? > > > > I chose g_disable_intercept_if_positive since it disables interceptors while > it > > is not zero. > http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.h (right): http://codereview.chromium.org/10411047/diff/71001/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.h:4: On 2012/08/03 09:57:07, Alexander Potapenko wrote: > You'll need an #ifdef... guard here. Done. http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:650: #endif I'm still thinking also about removing all STL operations from this function. With that, we don't need to call Suspend... and Resume... anywhere. Since Suspend... and Resume... are the only changes in this file, I decided to change this file after the final decision. On 2012/08/04 01:13:47, jar wrote: > Your original comment was "...unreadable. It is bad. ...ideas to fix, please > tell me". > > My suggestion to make it readable was to remove the ifdef here, and put it in > the called function. > > Why not do it? Why leave the code harder to read? > > > On 2012/08/03 10:01:50, Dai Mikurube wrote: > > On 2012/08/02 23:38:30, jar wrote: > > > I don't follow this motivation. > > > > > > ...but... if your goal is to make this code more readable by not injecting > the > > > ifdefs, consider always calling the functions, but having the functions > become > > > no-ops (in the implementation file) when !defined(USE_ALLOCATED_TYPE). > > > > > > The compiler will generally optimize out the no-op function... so the impact > > is > > > identical on released code (and readability is higher). > > > > > > On 2012/08/02 07:43:55, Dai Mikurube wrote: > > > > These are to avoid the DANGER of http://crbug.com/36678. > > > > > > > > This change acquires locks in 'new' and 'delete'. As a result, some STL > > > > operations cause lock acquisitions. These SuspendAllocatedTypeIntercept > > stops > > > > lock acquisitions. > > > > > > > > I know they make the code very unreadable... It is bad. If someone has > > good > > > > ideas to fix it, please tell me that. > > > > > > > Thank you. That is a good idea. > > > > For now, I keep it as is since we should consider more (including removing STL > > operations) around this issue. I added a temporary comment on #include > > "base/allocator/allocated_type_tcmalloc.h". > http://codereview.chromium.org/10411047/diff/71001/base/process_util_posix.cc... base/process_util_posix.cc:785: ResumeAllocatedTypeIntercept(); Ahhh, maybe, you are right. guessing we can suspend in the beginning of a child process, and no resume... I'll try it just in case. (It's not fixed in the uploaded patch, but I'm trying it locally. I'll upload it later soon.) On 2012/08/04 01:13:47, jar wrote: > It strikes me that when you fork, you don't want to get copies of duplicate data > (from the two sides of the fork). As a result, I'd expect you to permanently > stop collecting in the child (that will exec), and abandoning your data (that > was collected pre-fork) from the child. > > What am I missing? > > On 2012/08/03 10:01:50, Dai Mikurube wrote: > > On 2012/08/02 14:57:12, Alexander Potapenko wrote: > > > There's no point in resuming the interception if the child process is known > to > > > call exec() later. > > > > Ah, right. For now, I kept it as is, but I added a comment. (I don't change > > the code before decision.) > http://codereview.chromium.org/10411047/diff/71001/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/71001/build/common.gypi#newcode2367 build/common.gypi:2367: ['clang==1 and clang_use_allocated_type==1', { On 2012/08/03 09:57:07, Alexander Potapenko wrote: > If clang_use_allocated_type=1 implies clang=1, you don't need to check both. Done. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:14: #include "base/spinlock.h" On 2012/08/03 09:57:07, Alexander Potapenko wrote: > Mind the order of includes, please. Done. http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:5: #ifndef BASE_ALLOCATED_TYPE_MAP_H_ On 2012/08/03 09:57:07, Alexander Potapenko wrote: > The include guard should probably have a different name (compare to other guards > in this dir) tcmalloc, malloc_hook and stacktrace have a different style, but heap-checker, heap-profiler, malloc_extension, profiler have this style. I think it's good. Another candidate is ALLOCATED_TYPE_MAP_H_? http://codereview.chromium.org/10411047/diff/71001/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:22: struct AllocatedObject { On 2012/08/03 09:57:07, Alexander Potapenko wrote: > Use the specified order of declarations within a class: public: before private:, > methods before data members (variables), etc. > (http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml?showone=Declar...) Done: added "public:" and re-ordered constructors and data members. http://codereview.chromium.org/10411047/diff/66015/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/66015/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:15: #include "base/low_level_alloc.h" On 2012/08/04 01:13:47, jar wrote: > As per comment by Ptapenko: Please alphabetize header names. Done. http://codereview.chromium.org/10411047/diff/66015/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:78: SpinLockHolder l(&allocated_type_lock); On 2012/08/04 01:13:47, jar wrote: > Doesn't even your specially allocator call the new notification? ...and as a > result, does this reenter and deadlock? > > Or does your notification only take place when a standard TCMalloc based > allocation takes place? I thought you noted that you even got notifications > when a placement new was called... so it is a bit surprising that a new with > your allocator would not be notified. Due to that and some other problems, "allocator"-related packages are out of target by the interceptors. http://codereview.chromium.org/10411047/diff/66015/build/common.gypi (line: 2374)
Finally, LaunchProcess and GetAppOutputInternal stops interceptors for an entire child process. Renamed the function name and variable name.
http://codereview.chromium.org/10411047/diff/78015/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/process_util_posix.cc... base/process_util_posix.cc:649: #endif Note: I kept #ifdef here since the file has just two StopAllocatedTypeIntercept() calls. For only two #ifdefs, they make the code easier to read since readers can skip the #ifdef blocks easily.
http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type.h:21: const std::type_info & type); Please fix the space. http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type_ignore.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type_ignore.cc:16: const std::type_info &type) { Should be "type_info& type" http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type_log.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type_log.cc:24: const std::type_info &type) { Please fix the space. http://codereview.chromium.org/10411047/diff/78015/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/78015/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:5: #ifndef BASE_ALLOCATED_TYPE_MAP_H_ Is this "BASE"? Sorry, I don't know how it's called in the headers in the same directory.
http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type.h:20: size_t size, BTW why do you need |size| and |type| in __op_delete_intercept__? http://codereview.chromium.org/10411047/diff/78015/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/78015/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:32: } Please insert a newline after the function.
http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:25: if (base::subtle::NoBarrier_Load(&g_enable_intercept) != 0) { You are using atomics as though they are locks.... but they are not. Note that a thread may sit idle for an arbitrary period of time after executing line 25. During that unbounded period, a caller make call StopAllocatedTypeIntercept(). Sadly, after that point, this thread can awaken at any time, and execute lines 26 and 27. I don't see what you are getting from using atomics. As the atomics.h header file notes, there are very few reasons to use them (unless you're writing very special code), and this does not seem to fit that need. Can you justify why you are using atomics? Can you explain what this code is intended to do? http://codereview.chromium.org/10411047/diff/78015/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/process_util_posix.cc... base/process_util_posix.cc:649: #endif Readers can't skip the ifdef without trying to figure out what it is, and what it does. As noted, these ifdefs are details about your implementation, and probably should not be in this file. On 2012/08/07 11:35:55, Dai Mikurube wrote: > Note: I kept #ifdef here since the file has just two > StopAllocatedTypeIntercept() calls. > > For only two #ifdefs, they make the code easier to read since readers can skip > the #ifdef blocks easily.
Thank you for the comments, Alexander and Jim. Updated the patch. http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type.h:20: size_t size, On 2012/08/07 12:05:21, Alexander Potapenko wrote: > BTW why do you need |size| and |type| in __op_delete_intercept__? I implemented it with these arguments in the modified Clang. They make it easy to manage a simple map from type to total size. All we should do is only "map[type] -= size" for object deletion. We use another map from address to (size, type) in Chromium. But, they are reasonable in the Clang compiler. http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type.h:21: const std::type_info & type); On 2012/08/07 11:56:56, Alexander Potapenko wrote: > Please fix the space. Done. http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type_ignore.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type_ignore.cc:16: const std::type_info &type) { On 2012/08/07 11:56:56, Alexander Potapenko wrote: > Should be "type_info& type" Done. http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type_log.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type_log.cc:24: const std::type_info &type) { On 2012/08/07 11:56:56, Alexander Potapenko wrote: > Please fix the space. Done. http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:25: if (base::subtle::NoBarrier_Load(&g_enable_intercept) != 0) { I was worried about a rare case that two parallel non-atomic "x = 0;" may result in "x != 0". If I remember correctly, it can happen in a few of NUMA CPU memory and cache consistency models... but as you imagine, I am not sure. I'll check it again. My purpose is not locking. I would like to guarantee "Interceptors are stopped in a thread when the thread may do new/delete between fork and exec." It is just by-effect that interceptors stop in other threads. On 2012/08/07 17:07:34, jar wrote: > You are using atomics as though they are locks.... but they are not. > > Note that a thread may sit idle for an arbitrary period of time after executing > line 25. During that unbounded period, a caller make call > StopAllocatedTypeIntercept(). Sadly, after that point, this thread can awaken > at any time, and execute lines 26 and 27. > > I don't see what you are getting from using atomics. As the atomics.h header > file notes, there are very few reasons to use them (unless you're writing very > special code), and this does not seem to fit that need. > > Can you justify why you are using atomics? Can you explain what this code is > intended to do? http://codereview.chromium.org/10411047/diff/78015/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/process_util_posix.cc... base/process_util_posix.cc:649: #endif Ok, done in base/allocator/allocated_type_tcmalloc.cc. On 2012/08/07 17:07:34, jar wrote: > Readers can't skip the ifdef without trying to figure out what it is, and what > it does. As noted, these ifdefs are details about your implementation, and > probably should not be in this file. > > On 2012/08/07 11:35:55, Dai Mikurube wrote: > > Note: I kept #ifdef here since the file has just two > > StopAllocatedTypeIntercept() calls. > > > > For only two #ifdefs, they make the code easier to read since readers can skip > > the #ifdef blocks easily. > http://codereview.chromium.org/10411047/diff/78015/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/78015/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:32: } On 2012/08/07 12:05:21, Alexander Potapenko wrote: > Please insert a newline after the function. Done. (It followed other files like heap-profiler.cc.) http://codereview.chromium.org/10411047/diff/78015/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/78015/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:5: #ifndef BASE_ALLOCATED_TYPE_MAP_H_ On 2012/08/07 11:56:56, Alexander Potapenko wrote: > Is this "BASE"? Sorry, I don't know how it's called in the headers in the same > directory. E.g. heap-checker is not in base/BASE, but it uses BASE_HEAP_CHECKER_H_. It might come from its original code. Ok, there's no consistency in this directory. I use just "ALLOCATED_TYPE_MAP_H_" here. What do you think?
Two nits. The change looks good in general. http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_ignore.cc (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_ignore.cc:6: #include <iostream> You don't need <iostream> here. http://codereview.chromium.org/10411047/diff/72024/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/72024/build/common.gypi#newcode258 build/common.gypi:258: # modified clang, but clang_use_allocated_type=1 implies clang=1 Nit: period at the end of the line.
http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:25: if (base::subtle::NoBarrier_Load(&g_enable_intercept) != 0) { Please cite a reference, or remove the use of atomics here. I was unable to find mention of this issue when researching NUMA CPU. On 2012/08/08 06:16:23, Dai Mikurube wrote: > I was worried about a rare case that two parallel non-atomic "x = 0;" may result > in "x != 0". > > If I remember correctly, it can happen in a few of NUMA CPU memory and cache > consistency models... but as you imagine, I am not sure. I'll check it again. > > My purpose is not locking. I would like to guarantee "Interceptors are stopped > in a thread when the thread may do new/delete between fork and exec." It is > just by-effect that interceptors stop in other threads. > > > On 2012/08/07 17:07:34, jar wrote: > > You are using atomics as though they are locks.... but they are not. > > > > Note that a thread may sit idle for an arbitrary period of time after > executing > > line 25. During that unbounded period, a caller make call > > StopAllocatedTypeIntercept(). Sadly, after that point, this thread can awaken > > at any time, and execute lines 26 and 27. > > > > I don't see what you are getting from using atomics. As the atomics.h header > > file notes, there are very few reasons to use them (unless you're writing very > > special code), and this does not seem to fit that need. > > > > Can you justify why you are using atomics? Can you explain what this code is > > intended to do? > http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type.h:11: // These functions are inserted around all 'new' and 'delete' expressions by Can you indicate that this is only done or Clang? (assuming I correctly understood that you only hacked that compiler). Is it possible to clarify this further by either naming the file for the platform (android?), or by surrounding the declarations in an ifdef for that platform? http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_control.cc (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_control.cc:7: #ifndef USE_ALLOCATED_TYPE This name seems difficult to parse as you would hope. It says to "use allocated type," which doesn't seem meaningful. Suggest something like: PROFILING_TYPE_ALLOCATION You might similarly rename functions to: StopProfilingTypeAllocation() IsProfilingTypeAllocation() Perhaps you should rename the unified file: type_allocation_profiler.cc http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_control.cc:21: volatile static base::subtle::Atomic32 g_enable_intercept = 1; As noted... please remove the use of atomics, or provide a justification in a comment, including a reference. http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_control.h (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_control.h:8: void StopAllocatedTypeIntercept(); Please put these in a namespace to avoid global pollution. Given that the allocated_type.h only has two functions, and this file only has two functions (that control the latter), I think you should put both in a single header file, with the corresponding code in a single CC file. http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_log.cc (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_log.cc:8: void* __op_new_intercept__(void* ptr, Is there a reason why you don't put all viable implementations in one file, and have an ifdef flag select between them? Do you need to have all code available for run-time switching?? http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:13: if (IsAllocatedTypeInterceptEnabled()) { Perhaps an alternative name, better mirroring the style of line 14, is: IsTypeProfilerRunning() http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:49: if (allocated_type_map_memory == NULL) { Given that both of these should be NULL, or both should be non-NULL, why not just test one each time we enter: if (allocated_type_map_memory != NULL) { DCHECK_NE(allocated_type_map, NULL); return; } It was sad that this lazy initialization is on the path for every malloc and free, so you may as well not go out of your way to do extra work. Also note that IF the allocation fails for either call, you're doomed to crash RSN anyway... so both had better succeed (with calling code as written). http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:61: extern "C" void InsertAllocatedType(void* address, Why do these need to be "C"? Who is calling them? http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:18: AllocatedObject(): size(0), type(NULL) {} What does it mean to have an extern "C" around a declaration of methods?
Thank you for reviewing. Updated the patch. http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/78015/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:25: if (base::subtle::NoBarrier_Load(&g_enable_intercept) != 0) { On 2012/08/08 23:21:04, jar wrote: > Please cite a reference, or remove the use of atomics here. > > I was unable to find mention of this issue when researching NUMA CPU. > > On 2012/08/08 06:16:23, Dai Mikurube wrote: > > I was worried about a rare case that two parallel non-atomic "x = 0;" may > result > > in "x != 0". > > > > If I remember correctly, it can happen in a few of NUMA CPU memory and cache > > consistency models... but as you imagine, I am not sure. I'll check it again. > > > > My purpose is not locking. I would like to guarantee "Interceptors are > stopped > > in a thread when the thread may do new/delete between fork and exec." It is > > just by-effect that interceptors stop in other threads. > > > > > > On 2012/08/07 17:07:34, jar wrote: > > > You are using atomics as though they are locks.... but they are not. > > > > > > Note that a thread may sit idle for an arbitrary period of time after > > executing > > > line 25. During that unbounded period, a caller make call > > > StopAllocatedTypeIntercept(). Sadly, after that point, this thread can > awaken > > > at any time, and execute lines 26 and 27. > > > > > > I don't see what you are getting from using atomics. As the atomics.h > header > > > file notes, there are very few reasons to use them (unless you're writing > very > > > special code), and this does not seem to fit that need. > > > > > > Can you justify why you are using atomics? Can you explain what this code > is > > > intended to do? > > > It might be my mistake. Removed Atomic. http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type.h (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type.h:11: // These functions are inserted around all 'new' and 'delete' expressions by On 2012/08/08 23:21:04, jar wrote: > Can you indicate that this is only done or Clang? (assuming I correctly > understood that you only hacked that compiler). > > Is it possible to clarify this further by either naming the file for the > platform (android?), or by surrounding the declarations in an ifdef for that > platform? Added a comment and #if defined(__linux__) && defined(__clang__). I chose #if (not filename) since we can extend it easily to other platforms, e.g. Mac, though I haven't tested on other platforms. http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_control.cc (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_control.cc:7: #ifndef USE_ALLOCATED_TYPE On 2012/08/08 23:21:04, jar wrote: > This name seems difficult to parse as you would hope. It says to "use allocated > type," which doesn't seem meaningful. > > Suggest something like: > > PROFILING_TYPE_ALLOCATION > > You might similarly rename functions to: > StopProfilingTypeAllocation() > > IsProfilingTypeAllocation() > > Perhaps you should rename the unified file: > type_allocation_profiler.cc I chose * PROFILING_ALLOCATED_TYPE * {Stop|Is}ProfilingAllocatedType * allocated_type_profiler*.{cc|h} for those. I keep to use "allocated type" since "allocated type" is a common phrase to describe "a type of an allocated object". * http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3376.pdf (p.106) * http://clang.llvm.org/doxygen/CGCXXABI_8h_source.html#l00219 * http://www.almostinfinite.com/memtrack.html http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_control.cc:21: volatile static base::subtle::Atomic32 g_enable_intercept = 1; On 2012/08/08 23:21:04, jar wrote: > As noted... please remove the use of atomics, or provide a justification in a > comment, including a reference. Done. http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_control.h (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_control.h:8: void StopAllocatedTypeIntercept(); On 2012/08/08 23:21:04, jar wrote: > Please put these in a namespace to avoid global pollution. > > Given that the allocated_type.h only has two functions, and this file only has > two functions (that control the latter), I think you should put both in a single > header file, with the corresponding code in a single CC file. Put them in a namespace. I separated them since it is a little confusing in turning on/off this allocated_type feature. * These two functions should be declared in "a very part of files" of "all build cases". It is good to declare them by "#include <...>" in some necessary files -- e.g. process_util_posix.cc. * __op_{new|delete}_intercept__ should be declared in "almost all the files" of "a part of build cases with clang_use_allocated_type=1". They should be declared by the compiler option "-include ...". If we merge these four functions into one file, the way to switch the allocated_type feature gets complex. The merged file is included by "-include" in case of clang_use_allocated_type=1, but process_util_posix.cc should always include it "#include". It is hard to write a good gyp(i) files... It is reasonable to add a guard for OS and compilers like did it in allocated_type.h. But, switching should be done in one place -- gyp. http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_ignore.cc (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_ignore.cc:6: #include <iostream> On 2012/08/08 11:27:14, Alexander Potapenko wrote: > You don't need <iostream> here. Done. http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_log.cc (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_log.cc:8: void* __op_new_intercept__(void* ptr, On 2012/08/08 23:21:04, jar wrote: > Is there a reason why you don't put all viable implementations in one file, and > have an ifdef flag select between them? Do you need to have all code available > for run-time switching?? The implementations must be switched in common.gypi for each target binary at the linking phase. Multiple implementations are required in one build sequence: _ignore for protoc, _tcmalloc for chrome. But, only one viable implementation in one target binary. We may be able to generate multiple lib* files from a single cc file by #ifdef and gyp settings, but separated files are easier to understand the situation. http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... File base/allocator/allocated_type_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/72024/base/allocator/allocated_t... base/allocator/allocated_type_tcmalloc.cc:13: if (IsAllocatedTypeInterceptEnabled()) { On 2012/08/08 23:21:04, jar wrote: > Perhaps an alternative name, better mirroring the style of line 14, is: > IsTypeProfilerRunning() Done. http://codereview.chromium.org/10411047/diff/72024/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/72024/build/common.gypi#newcode258 build/common.gypi:258: # modified clang, but clang_use_allocated_type=1 implies clang=1 On 2012/08/08 11:27:14, Alexander Potapenko wrote: > Nit: period at the end of the line. Done. http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:49: if (allocated_type_map_memory == NULL) { On 2012/08/08 23:21:04, jar wrote: > Given that both of these should be NULL, or both should be non-NULL, why not > just test one each time we enter: > if (allocated_type_map_memory != NULL) { > DCHECK_NE(allocated_type_map, NULL); > return; > } > > It was sad that this lazy initialization is on the path for every malloc and > free, so you may as well not go out of your way to do extra work. > > Also note that IF the allocation fails for either call, you're doomed to crash > RSN anyway... so both had better succeed (with calling code as written). Agreed. Done. http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:61: extern "C" void InsertAllocatedType(void* address, On 2012/08/08 23:21:04, jar wrote: > Why do these need to be "C"? Who is calling them? Callers must be C++. Ok, removed them. http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:18: AllocatedObject(): size(0), type(NULL) {} On 2012/08/08 23:21:04, jar wrote: > What does it mean to have an extern "C" around a declaration of methods? Done.
drive by review; This is mostly style nits, but I am concerned about the GYP changes and suggest some alternative approaches below. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.h (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler.h:3: // found in the LICENSE file. style nit: See chromium-dev thread about new copyright boilerplate http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler.h:9: // option '-include' before build/build_config.h. Can you not include build/build_config.h? Alternatively, can/should you use some build/common.gypi-specified #define, rather than these two? That certainly seems more appropriate, since as written, it seems to be intended for "any clang build" and not your custom clang build. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler.h:17: // modified version of Clang at http://crrev.com/149572. nit: This comment can easily get out of date; it's not uncommon for arguments to be renamed during the Clang upstreaming process. It would be better to describe what these functions do (wrap new and delete), rather than how they do it. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:7: #ifdef PROFILING_ALLOCATED_TYPE style nit: In general, Chromium code tends more to follow the #if defined() notation rather than #ifdef This is codified in http://www.chromium.org/developers/coding-style#TOC-Platform-specific-code , about platform-specific code, although in practice this has been extended from "platform == OS" to "platform == tooling" (such as X vs Views vs Cocoa or NSS vs CryptoAPI vs CDSA or Clang vs GCC). http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:30: #else // PROFILING_ALLOCATED_TYPE nit: Having a broad swath of file between #ifdefs like this makes it hard to read and hard to maintain. Since the function signatures are identical, you should consider the #ifdefs being within the funciton itself; eg: bool IsProfilingAllocatedType() { #if defined(PROFILING_ALLOCATED_TYPE) return g_... #else return true; #endif } http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:40: return true; Is this really true? Seems like you should be returning false instead http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_ignore.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_ignore.cc:17: } Seems like you shouldn't even be wrapping these functions if you're not going to do anything. I'm not even sure if Clang/GCC will be able to optimize these out - even though they're *effectively* no-op, without Whole Program optimization, it won't be able to no-op the calls out at all the callsites. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_log.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_log.cc:18: << std::endl; style nit: Please see the comments on http://www.chromium.org/developers/coding-style#TOC-Code-formatting about code formatting operator<<() calls. (Personal nit) This line breaking format seems to hinder readability more than help it. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_log.cc:27: << "bytes of an instance of \"" " bytes" (note the extra space) same on 13 http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_log.cc:32: << std::endl; perf nit: std::endl is going to force a sync flush every call here. Should you be using \n instead, and letting the system flush as appropriate? http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_tcmalloc.cc:8: #include <gperftools/heap-profiler.h> style nit: Please see http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Names_and_Orde... and order & space appropriately. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocator.gyp File base/allocator/allocator.gyp (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocator.g... base/allocator/allocator.gyp:547: 'allocated_type_profiler_tcmalloc.cc', nit: sort http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocator.g... base/allocator/allocator.gyp:567: 'target_name': 'allocated_type_profiler_log', Can you explain why you're using three separate targets, rather than simply one target that conditionally compiles the appropriate sources (tcmalloc, ignore, or log) ? http://codereview.chromium.org/10411047/diff/69004/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode258 build/common.gypi:258: # modified clang, but clang_profiling_allocated_type=1 implies clang=1. I'd remove the entire second sentence - it does not seem to add any help here http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode260 build/common.gypi:260: # TODO(dmikurube): Support mac with another AllocatedTypeMap store. TODOs like this without an http://crbug.com/ link are quickly lost into the nether. Please consider filing a bug to track this effort http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1199 build/common.gypi:1199: ['clang_profiling_allocated_type==1', { nit: Since you don't support all Clang-supporting OSes yet, please add an appropriate OS guard http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1615 build/common.gypi:1615: ['_target_name=="mksnapshot" or _target_name=="protoc" or _target_name=="ppGoogleNaClPluginChrome"', { This is a nasty layering violation that we try very much to avoid. Please use a target_defaults: target_conditions to appropriately gate this and allow individual targets to override this behaviour as appropriate, and then use a target-specific variable to gate this behaviour. An example of this is the chromium_code variable. More importantly, rather than injecting the ignore stub, you should instead not even be passing whatever command-line args you're passing *for these targets*. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1620 build/common.gypi:1620: '<!(realpath <(DEPTH)/base/allocator/allocator.gyp)' + ':allocated_type_profiler_ignore#target', 1) The use of realpath here is troubling. It definitely seems that GYP's existing path handling should be appropriate here. 2) The specification of #target is incorrect here - this is specifically for cross-compiling, and is an internal GYP detail not (generally) meant to be exposed. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2385 build/common.gypi:2385: ['_target_name!="allocator" and _target_name!="allocator_extension_thunks" and _target_name!="libcmt" and _target_name!="allocator_extension_thunks_win64" and _target_name!="allocated_type_profiler_log" and _target_name!="allocated_type_profiler_ignore" and _target_name!="allocated_type_profiler_tcmalloc"', { See comments above about layering violation. These should be refactored. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2386 build/common.gypi:2386: 'cflags_cc+': [ style nit: The indents here should all be GYP indents, which are two-space indents. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2389 build/common.gypi:2389: '-I../../base/allocator', BUG: You should not be specifying relative paths via -I, as they'll ignore GYP pathname canonicalization and thus be generator dependent. Instead, you should consider using the "include_dirs" GYP variable, which is the correct way to pass such variables. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2391 build/common.gypi:2391: 'allocated_type_profiler.h', However, more generally, ISTM that instead of forcing base/allocator into all targets' include_dirs, you should instead be specifying the canonical path for allocated_type_profiler and using -include with the full path. 'variables': { 'some_var_name_path': '<(DEPTH)/base/allocator/allocated_type_profiler.h' }, 'cflags_cc+': [ '-include', '<(some_var_name_path)', ] Note the suffix of _path, which ensures the proper path relative to where/how the files are compiled is yielded ( see http://code.google.com/p/gyp/wiki/InputFormatReference#Pathname_Relativization ) http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2399 build/common.gypi:2399: ['_toolset=="target"', { note: Like ASAN, ISTM like you may need to gate on _toolset - please confirm. http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:23: static SpinLock allocated_type_lock(SpinLock::LINKER_INITIALIZED); Please consider the guidance from Google's Open Source C++ Style Guide, specifically http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml?showone=Variab... "There are no special requirements for global variables, which should be rare in any case, but if you use one, consider prefixing it with g_ or some other marker to easily distinguish it from local variables." I'll defer to TCMalloc style here, but the current naming definitely makes it subtle. http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:51: DCHECK_NE(allocated_type_map, NULL); BUG? Your use of std::cerr in the _logger wrappers suggests to me that you're trying to avoid any form of heap allocation during error logging, in the event of re-entrancy. Thus, the use of DCHECK_NE here seems counter-productive to that, since DCHECK_NE may invoke the DLOG() operators and/or re-enter new/delete. It seems to me that you should either: 1) CHECK 2) Remove this http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:60: AddressMap<AllocatedObject>(AllocatedTypeMalloc, AllocatedTypeFree); BUG? Do you not need to add any barriers here? Does the SpinLockHolder properly introduce memory barriers? I'm just wondering whether it's possible for the spinlock to be released by InsertAllocatedType before other cache lines/CPUs have updated allocated_type_map_memory + allocated_type_map, which might cause the DCHECK to fire on 51 http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:29: #endif /* ALLOCATED_TYPE_MAP_H_ */ nit: Mixing comment styles between C++ (//) and C (/* */) between here and line 10. Based on the C++ Style guide, and that this is necessarily C++ code, it would seem like this should be // ALLOCATED_TYPE_MAP_H_ , however, I defer to TCMalloc convention.
Thanks for the comments, Ryan. I was wondering if someone could help about the dirty gyp file. Your comments are helpful. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.h (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler.h:3: // found in the LICENSE file. On 2012/08/10 04:16:09, Ryan Sleevi wrote: > style nit: See chromium-dev thread about new copyright boilerplate Done for all new files. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler.h:9: // option '-include' before build/build_config.h. On 2012/08/10 04:16:09, Ryan Sleevi wrote: > Can you not include build/build_config.h? > > Alternatively, can/should you use some build/common.gypi-specified #define, > rather than these two? That certainly seems more appropriate, since as written, > it seems to be intended for "any clang build" and not your custom clang build. Used #if defined(PROFILING_ALLOCATED_TYPE). http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler.h:17: // modified version of Clang at http://crrev.com/149572. On 2012/08/10 04:16:09, Ryan Sleevi wrote: > nit: This comment can easily get out of date; it's not uncommon for arguments to > be renamed during the Clang upstreaming process. > > It would be better to describe what these functions do (wrap new and delete), > rather than how they do it. Ok, but I guess we should have a description that it depends on the modified compiler. I changed it to // These functions wrap all 'new' and 'delete' expressions. // It depends on a modified version of Clang checked-in at // deps/third_party/llvm-allocated-type. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:7: #ifdef PROFILING_ALLOCATED_TYPE On 2012/08/10 04:16:09, Ryan Sleevi wrote: > style nit: In general, Chromium code tends more to follow the #if defined() > notation rather than #ifdef > > This is codified in > http://www.chromium.org/developers/coding-style#TOC-Platform-specific-code , > about platform-specific code, although in practice this has been extended from > "platform == OS" to "platform == tooling" (such as X vs Views vs Cocoa or NSS vs > CryptoAPI vs CDSA or Clang vs GCC). Done. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:30: #else // PROFILING_ALLOCATED_TYPE On 2012/08/10 04:16:09, Ryan Sleevi wrote: > nit: Having a broad swath of file between #ifdefs like this makes it hard to > read and hard to maintain. > > Since the function signatures are identical, you should consider the #ifdefs > being within the funciton itself; eg: > > bool IsProfilingAllocatedType() { > #if defined(PROFILING_ALLOCATED_TYPE) > return g_... > #else > return true; > #endif > } Done. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:40: return true; On 2012/08/10 04:16:09, Ryan Sleevi wrote: > Is this really true? > > Seems like you should be returning false instead Thanks for the good catch. Right. It affects nothing in fact, but false is better. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_ignore.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_ignore.cc:17: } On 2012/08/10 04:16:09, Ryan Sleevi wrote: > Seems like you shouldn't even be wrapping these functions if you're not going to > do anything. > > I'm not even sure if Clang/GCC will be able to optimize these out - even though > they're *effectively* no-op, without Whole Program optimization, it won't be > able to no-op the calls out at all the callsites. These no-op interceptors are used for non-tcmalloc binaries. It is because the storage for allocated types depends on tcmalloc's lock. We need to switch it by target binaries: e.g. turn off for mksnapshot and turn on for chrome. Almost all .cc files should have prototype declarations of these functions when this feature is enabled. A .cc file may be used for two kinds of binaries: e.g. mksnapshot and chrome. (I am not sure, but I cannot guarantee... or, can we assume that?) It is the reason why I switch them at linking phase by preparing multiple build targets: allocated_type_profiler_tcmalloc and allocated_type_profiler_ignore. I agree with that they might not be "effective" no-ops. But, this feature is enabled only for profiling, and these _ignore interceptors are used only for some commands: e.g. mksnapshot and protoc. I think it is acceptable and reasonable. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_log.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_log.cc:18: << std::endl; On 2012/08/10 04:16:09, Ryan Sleevi wrote: > style nit: Please see the comments on > http://www.chromium.org/developers/coding-style#TOC-Code-formatting about code > formatting operator<<() calls. (Personal nit) This line breaking format seems to > hinder readability more than help it. Do you mean I should do the following? std::cerr << "Allocated " << size << " bytes as an instance of \"" << type.name() << "\" at " << ptr << "." << std::endl; The _log is used rarely in fact -- it is sometimes turned on when we want to check this profiler is working correctly for non-tcmalloc commands such as mksnapshot and protoc. Such format is good for the purpose. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_log.cc:27: << "bytes of an instance of \"" On 2012/08/10 04:16:09, Ryan Sleevi wrote: > " bytes" (note the extra space) > > same on 13 Done. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_log.cc:32: << std::endl; On 2012/08/10 04:16:09, Ryan Sleevi wrote: > perf nit: std::endl is going to force a sync flush every call here. Should you > be using \n instead, and letting the system flush as appropriate? Flushing is better for the purpose written in the reply above. We don't need performance here. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_tcmalloc.cc:8: #include <gperftools/heap-profiler.h> On 2012/08/10 04:16:09, Ryan Sleevi wrote: > style nit: Please see > http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Names_and_Orde... > and order & space appropriately. Done. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocator.gyp File base/allocator/allocator.gyp (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocator.g... base/allocator/allocator.gyp:547: 'allocated_type_profiler_tcmalloc.cc', On 2012/08/10 04:16:09, Ryan Sleevi wrote: > nit: sort Done. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocator.g... base/allocator/allocator.gyp:567: 'target_name': 'allocated_type_profiler_log', On 2012/08/10 04:16:09, Ryan Sleevi wrote: > Can you explain why you're using three separate targets, rather than simply one > target that conditionally compiles the appropriate sources (tcmalloc, ignore, or > log) ? Please read my reply at http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t.... http://codereview.chromium.org/10411047/diff/69004/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode258 build/common.gypi:258: # modified clang, but clang_profiling_allocated_type=1 implies clang=1. On 2012/08/10 04:16:09, Ryan Sleevi wrote: > I'd remove the entire second sentence - it does not seem to add any help here Removed "-fintercept-allocated-type only works with the modified clang, but". http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode260 build/common.gypi:260: # TODO(dmikurube): Support mac with another AllocatedTypeMap store. On 2012/08/10 04:16:09, Ryan Sleevi wrote: > TODOs like this without an http://crbug.com/ link are quickly lost into the > nether. Please consider filing a bug to track this effort Added a link to description page, and crbug.com. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1199 build/common.gypi:1199: ['clang_profiling_allocated_type==1', { On 2012/08/10 04:16:09, Ryan Sleevi wrote: > nit: Since you don't support all Clang-supporting OSes yet, please add an > appropriate OS guard Done. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1615 build/common.gypi:1615: ['_target_name=="mksnapshot" or _target_name=="protoc" or _target_name=="ppGoogleNaClPluginChrome"', { On 2012/08/10 04:16:09, Ryan Sleevi wrote: > This is a nasty layering violation that we try very much to avoid. > > Please use a target_defaults: target_conditions to appropriately gate this and > allow individual targets to override this behaviour as appropriate, and then use > a target-specific variable to gate this behaviour. > > An example of this is the chromium_code variable. > > More importantly, rather than injecting the ignore stub, you should instead not > even be passing whatever command-line args you're passing *for these targets*. Thank you for the information. Yes, I knew it is a bad hacky way, but I didn't know how to fix it. It was a result of tries-and-errors with reading gyp code. GYP is complex and undocumented... What I would like is to inject a dependency to a different linked library by target binaries as replied at http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t.... But, modifying each gyp file was not a good idea since we would like to control this profiling feature for a whole build in one place -- common.gypi. But, I still don't understand target_defaults well. How can I do that? http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1620 build/common.gypi:1620: '<!(realpath <(DEPTH)/base/allocator/allocator.gyp)' + ':allocated_type_profiler_ignore#target', On 2012/08/10 04:16:09, Ryan Sleevi wrote: > 1) The use of realpath here is troubling. It definitely seems that GYP's > existing path handling should be appropriate here. > 2) The specification of #target is incorrect here - this is specifically for > cross-compiling, and is an internal GYP detail not (generally) meant to be > exposed. realpath got required from http://crrev.com/146172. From this change, realpath is a key of a hashtable in gyp. Both look like because this part is evaluated lazily after some path and target operations. It's a large part of this change's hacky points. But, I have no idea to fix them. :( http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2385 build/common.gypi:2385: ['_target_name!="allocator" and _target_name!="allocator_extension_thunks" and _target_name!="libcmt" and _target_name!="allocator_extension_thunks_win64" and _target_name!="allocated_type_profiler_log" and _target_name!="allocated_type_profiler_ignore" and _target_name!="allocated_type_profiler_tcmalloc"', { On 2012/08/10 04:16:09, Ryan Sleevi wrote: > See comments above about layering violation. These should be refactored. I agree with that, but I have no idea to refactor it... Here, I would like to change cflags_cc for almost all the .cc files except for some files related to specific targets. Can I do that? http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2386 build/common.gypi:2386: 'cflags_cc+': [ On 2012/08/10 04:16:09, Ryan Sleevi wrote: > style nit: The indents here should all be GYP indents, which are two-space > indents. Done. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2389 build/common.gypi:2389: '-I../../base/allocator', On 2012/08/10 04:16:09, Ryan Sleevi wrote: > BUG: You should not be specifying relative paths via -I, as they'll ignore GYP > pathname canonicalization and thus be generator dependent. > > Instead, you should consider using the "include_dirs" GYP variable, which is the > correct way to pass such variables. Trying below. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2391 build/common.gypi:2391: 'allocated_type_profiler.h', On 2012/08/10 04:16:09, Ryan Sleevi wrote: > However, more generally, ISTM that instead of forcing base/allocator into all > targets' include_dirs, you should instead be specifying the canonical path for > allocated_type_profiler and using -include with the full path. > > 'variables': { > 'some_var_name_path': '<(DEPTH)/base/allocator/allocated_type_profiler.h' > }, > 'cflags_cc+': [ > '-include', '<(some_var_name_path)', > ] > > Note the suffix of _path, which ensures the proper path relative to where/how > the files are compiled is yielded ( see > http://code.google.com/p/gyp/wiki/InputFormatReference#Pathname_Relativization ) It was impossible maybe because it is lazily evaluated (in target_conditions) like above. The defined variable was not found here. "KeyError: 'Undefined variable clang_profiling_allocated_type_header_path in .../build/all.gyp while trying to load .../build/all.gyp'" We may have to stop target_conditions before fix it. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2399 build/common.gypi:2399: ['_toolset=="target"', { On 2012/08/10 04:16:09, Ryan Sleevi wrote: > note: Like ASAN, ISTM like you may need to gate on _toolset - please confirm. Ah... sorry, I have no idea about "_toolset". Do you know any documents about that? http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:23: static SpinLock allocated_type_lock(SpinLock::LINKER_INITIALIZED); On 2012/08/10 04:16:09, Ryan Sleevi wrote: > Please consider the guidance from Google's Open Source C++ Style Guide, > specifically > http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml?showone=Variab... > > "There are no special requirements for global variables, which should be rare in > any case, but if you use one, consider prefixing it with g_ or some other marker > to easily distinguish it from local variables." > > I'll defer to TCMalloc style here, but the current naming definitely makes it > subtle. It was following other tcmalloc files. I renamed the variables in this file. http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:51: DCHECK_NE(allocated_type_map, NULL); On 2012/08/10 04:16:09, Ryan Sleevi wrote: > BUG? Your use of std::cerr in the _logger wrappers suggests to me that you're > trying to avoid any form of heap allocation during error logging, in the event > of re-entrancy. Thus, the use of DCHECK_NE here seems counter-productive to > that, since DCHECK_NE may invoke the DLOG() operators and/or re-enter > new/delete. > > It seems to me that you should either: > 1) CHECK > 2) Remove this _log doesn't call this function. Functions in this file are called only from _tcmalloc. (_log is rarely used as I replied, that can be removed if that's not good to commit.) This checks "g_allocated_type_map and g_allocated_type_map_memory get non-NULL value at the same time. Both are NULL, or both are non-NULL." Usually, it doesn't happen because *AllocatedType functions have locks. http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:60: AddressMap<AllocatedObject>(AllocatedTypeMalloc, AllocatedTypeFree); On 2012/08/10 04:16:09, Ryan Sleevi wrote: > BUG? > > Do you not need to add any barriers here? Does the SpinLockHolder properly > introduce memory barriers? > > I'm just wondering whether it's possible for the spinlock to be released by > InsertAllocatedType before other cache lines/CPUs have updated > allocated_type_map_memory + allocated_type_map, which might cause the DCHECK to > fire on 51 Ah, I understand your question. I was thinking it doesn't happen. atomicops (e.g. Acquire_CompareAndSwap) works with lock-prefixes. http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:29: #endif /* ALLOCATED_TYPE_MAP_H_ */ On 2012/08/10 04:16:09, Ryan Sleevi wrote: > nit: Mixing comment styles between C++ (//) and C (/* */) between here and line > 10. > > Based on the C++ Style guide, and that this is necessarily C++ code, it would > seem like this should be // ALLOCATED_TYPE_MAP_H_ , however, I defer to TCMalloc > convention. Good catch. Thanks. (It might come from heap-profiler.h. But, we should follow Chromium's guideline in completely new files.)
Regarding the _toolset: I don't know if there are any docs, but you should probably instrument only the target binaries (i.e. Chrome and the tests) leaving the host binaries (various tools mostly used at build time) alone. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.h (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.h:17: #endif // BASE_ALLOCATOR_ALLOCATED_TYPE_PROFILER_CONTROL_H_ Nit: spare space between #endif and the comment
+cc jamesr, for point #1 in this "summary" Hi Dai, I took a second pass and try to respond to your GYP comments. Since it may be a bit hard to follow each response, I think the high-level change should be as follows: 1) If you really only care about tcmalloc (which seems to be the case), move the 'profile-or-log' handling directly into allocator.gyp, as a split between the 'sources' for the 'allocator' target (the interposing parts) and as the allocator_thunks (for the enable/disable parts). Do not try to "inject" the dependency via common.gypi - this is something GYP correctly chokes on. willchan or jamesr are probably most knowledgeable about the GYP+tcmalloc dependency fun on linux - willchan is OOO, hence +cc jamesr. 2) Only inject (cflags, ccflags, include dirs, defines) in a target_conditions for targets where 'linux_use_tcmalloc=1 and clang_profiling_allocated_type=1 and your_new_variable_here=1'. Individual targets can set the 'your_new_variable_here' on a per-target basis to "opt-out" of having the whole interposing. You can probably combine 'your_new_variable_here' and 'clang_profiling_allocated_type' into a single variable with the right structuring, but using two variables (one set via command-line, one set in common.gypi and the individual excluded targets) will help you debug easier. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_ignore.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_ignore.cc:17: } On 2012/08/10 07:30:03, Dai Mikurube wrote: > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > Seems like you shouldn't even be wrapping these functions if you're not going > to > > do anything. > > > > I'm not even sure if Clang/GCC will be able to optimize these out - even > though > > they're *effectively* no-op, without Whole Program optimization, it won't be > > able to no-op the calls out at all the callsites. > > These no-op interceptors are used for non-tcmalloc binaries. It is because the > storage for allocated types depends on tcmalloc's lock. We need to switch it by > target binaries: e.g. turn off for mksnapshot and turn on for chrome. > > Almost all .cc files should have prototype declarations of these functions when > this feature is enabled. A .cc file may be used for two kinds of binaries: e.g. > mksnapshot and chrome. (I am not sure, but I cannot guarantee... or, can we > assume that?) > > It is the reason why I switch them at linking phase by preparing multiple build > targets: allocated_type_profiler_tcmalloc and allocated_type_profiler_ignore. > > I agree with that they might not be "effective" no-ops. But, this feature is > enabled only for profiling, and these _ignore interceptors are used only for > some commands: e.g. mksnapshot and protoc. I think it is acceptable and > reasonable. In that case, wouldn't it just be better not to interpose at all for non-audited targets? (see GYP comments) http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_log.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_log.cc:18: << std::endl; On 2012/08/10 07:30:03, Dai Mikurube wrote: > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > style nit: Please see the comments on > > http://www.chromium.org/developers/coding-style#TOC-Code-formatting about code > > formatting operator<<() calls. (Personal nit) This line breaking format seems > to > > hinder readability more than help it. > > Do you mean I should do the following? > > std::cerr << "Allocated " << size << " bytes as an instance of \"" > << type.name() << "\" at " << ptr << "." << std::endl; Yes, I believe that to be more readable. > > The _log is used rarely in fact -- it is sometimes turned on when we want to > check this profiler is working correctly for non-tcmalloc commands such as > mksnapshot and protoc. Such format is good for the purpose. I understood your response to the no-op file being that you explicitly don't want to interpose for such targets. I had thus assumed logging was for when you *are* using tcmalloc and you just want to log, rather than actually wire it up into tcmalloc. So is this logging file for: 1) no tcmalloc 2) yes tcmalloc If #1, I'm having trouble understanding the utility of this log? if #2, could this be incorporated into the tcmalloc file under an #if defined(), rather than a separate file and target? http://codereview.chromium.org/10411047/diff/69004/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1615 build/common.gypi:1615: ['_target_name=="mksnapshot" or _target_name=="protoc" or _target_name=="ppGoogleNaClPluginChrome"', { On 2012/08/10 07:30:03, Dai Mikurube wrote: > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > This is a nasty layering violation that we try very much to avoid. > > > > Please use a target_defaults: target_conditions to appropriately gate this and > > allow individual targets to override this behaviour as appropriate, and then > use > > a target-specific variable to gate this behaviour. > > > > An example of this is the chromium_code variable. > > > > More importantly, rather than injecting the ignore stub, you should instead > not > > even be passing whatever command-line args you're passing *for these targets*. > > Thank you for the information. Yes, I knew it is a bad hacky way, but I didn't > know how to fix it. It was a result of tries-and-errors with reading gyp code. > GYP is complex and undocumented... > > What I would like is to inject a dependency to a different linked library by > target binaries as replied at > http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t.... I'm sorry, I'm having trouble understanding your response and the connection between these two statements. It seems that for these targets, you shouldn't be intercepting at all then? > But, modifying each gyp file was not a good idea since we would like to control > this profiling feature for a whole build in one place -- common.gypi. I don't think these two things are in opposition. My point of highlighting how chromium_code is that you can, on a per-target basis, in the target's gyp file, include or exclude specific behaviours as appropriate. For example, you might have some: 'variables': { 'interpose_new%': 1, }, 'target_defaults': { 'variables': { # Dai, see the comment on http://code.google.com/searchframe#OAMlx_jo-ck/src/build/common.gypi&exact_pa... # to understand why this is this way 'interpose_new%': '<(interpose_new)', }, 'target_conditions': [ ['clang_profiling_allocated_type==1 and interpose_new==1', { 'cflags_cc': [ ], 'include_dirs': [ ], }], ], }, > > But, I still don't understand target_defaults well. How can I do that? http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1620 build/common.gypi:1620: '<!(realpath <(DEPTH)/base/allocator/allocator.gyp)' + ':allocated_type_profiler_ignore#target', On 2012/08/10 07:30:03, Dai Mikurube wrote: > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > 1) The use of realpath here is troubling. It definitely seems that GYP's > > existing path handling should be appropriate here. > > 2) The specification of #target is incorrect here - this is specifically for > > cross-compiling, and is an internal GYP detail not (generally) meant to be > > exposed. > > realpath got required from http://crrev.com/146172. From this change, realpath > is a key of a hashtable in gyp. > > Both look like because this part is evaluated lazily after some path and target > operations. It's a large part of this change's hacky points. But, I have no > idea to fix them. :( Oh, right, that's because GYP dependency resolution happens *between* the pre- and the post- phase, but you're decidedly in the post- phase here (target_conditions is post phase) See http://code.google.com/p/gyp/wiki/InputFormatReference#Timing_of_Variable_Exp... to understand what I mean by pre- and post- phases This is controlled by http://code.google.com/searchframe#OAMlx_jo-ck/tools/gyp/pylib/gyp/input.py&e... And it's "intentional" to make sure dependencies are properly specified and not willy-nilly cross-injected. Can you not just specify the correct dependencies in allocator.gyp:allocator, which will cause all linkable types (eg: executables, shared_libraries, loadable modules) to link in these targets as appropriate? http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1626 build/common.gypi:1626: ['clang_profiling_allocated_type==1', { BUG? This double nested condition seems wrong. Shouldn't it be? 'conditions': [ ['clang_profiling_allocated_type == 1', { 'conditions': [ ['linux_use_tcmalloc == 1', { 'dependencies': [ '<(DEPTH)/base/allocator/allocator.gyp:allocated_type_profiler_tcmalloc', ], }, { 'dependencies': [ '<(DEPTH)/base/allocator/allocator.gyp:allocated_type_profiler_ignore', ], }], ], }], ], http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2385 build/common.gypi:2385: ['_target_name!="allocator" and _target_name!="allocator_extension_thunks" and _target_name!="libcmt" and _target_name!="allocator_extension_thunks_win64" and _target_name!="allocated_type_profiler_log" and _target_name!="allocated_type_profiler_ignore" and _target_name!="allocated_type_profiler_tcmalloc"', { On 2012/08/10 07:30:03, Dai Mikurube wrote: > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > See comments above about layering violation. These should be refactored. > > I agree with that, but I have no idea to refactor it... Here, I would like to > change cflags_cc for almost all the .cc files except for some files related to > specific targets. Can I do that? In those specific targets that you wish to exclude, manually set the variable (eg: override the default being inherited from this file) to exclude them. This is what I meant by comparing with the 'chromium_code' usage, which selectively enables or disables certain flags on a per-target basis, with a well-defined default for all targets, and then certain targets opting in. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2391 build/common.gypi:2391: 'allocated_type_profiler.h', On 2012/08/10 07:30:03, Dai Mikurube wrote: > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > However, more generally, ISTM that instead of forcing base/allocator into all > > targets' include_dirs, you should instead be specifying the canonical path for > > allocated_type_profiler and using -include with the full path. > > > > 'variables': { > > 'some_var_name_path': '<(DEPTH)/base/allocator/allocated_type_profiler.h' > > }, > > 'cflags_cc+': [ > > '-include', '<(some_var_name_path)', > > ] > > > > Note the suffix of _path, which ensures the proper path relative to where/how > > the files are compiled is yielded ( see > > http://code.google.com/p/gyp/wiki/InputFormatReference#Pathname_Relativization > ) > > It was impossible maybe because it is lazily evaluated (in target_conditions) > like above. The defined variable was not found here. > > "KeyError: 'Undefined variable clang_profiling_allocated_type_header_path in > .../build/all.gyp while trying to load .../build/all.gyp'" > > We may have to stop target_conditions before fix it. Oh, that's probably just due to GYP order-of-evaluation. Use >(some_var_name_path) rather than <(some_var_name_path) See http://code.google.com/p/gyp/wiki/InputFormatReference#Timing_of_Variable_Exp... for more about the ordering of evaluations http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:51: DCHECK_NE(allocated_type_map, NULL); On 2012/08/10 07:30:03, Dai Mikurube wrote: > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > BUG? Your use of std::cerr in the _logger wrappers suggests to me that you're > > trying to avoid any form of heap allocation during error logging, in the event > > of re-entrancy. Thus, the use of DCHECK_NE here seems counter-productive to > > that, since DCHECK_NE may invoke the DLOG() operators and/or re-enter > > new/delete. > > > > It seems to me that you should either: > > 1) CHECK > > 2) Remove this > > _log doesn't call this function. Functions in this file are called only from > _tcmalloc. (_log is rarely used as I replied, that can be removed if that's not > good to commit.) > > This checks "g_allocated_type_map and g_allocated_type_map_memory get non-NULL > value at the same time. Both are NULL, or both are non-NULL." Usually, it > doesn't happen because *AllocatedType functions have locks. I think you may have missed my point. I'm aware of what it is checking, but my concern is you may be setting yourself up for a recursion overflow. Consider when this situation happens (how/when it can happen was described in the original comment for line 60). Your initial callstack will look something like: 0: InitializeAllocatedTypeMemory() 1: InsertAllocatedType() 2: __op_new_intercept__() 3: operator new(...) This DCHECK_NE fires, which will call DLOG() with the DCHECK_NE's statement, which will call logging::LogMessage(). If logging::LogMessage() calls new, your stack will look like 0: InitializeAllocatedTypeMemory() 1: InsertAllocatedType() 2: __op_new_intercept__() 3: operator new(...) 4: logging::LogMessage() [expanded from DCHECK_NE] 5: InitializeAllocatedTypeMemory() 6: InsertAllocatedType() 7: __op_new_intercept__() 8: operator new(...) and thus you'll end up re-entering yourself and recursing repeatedly. The recursion "should" be bounded by the time that it takes the cache lines to update, so it shouldn't run off forever. While this file's operator new calls are not interposed (because you have it excluded at the .gypi level), thus making line 58-60 safe, logging::LogMessage is part of the base library, which is interposed, and thus does seem risky. However, the point being is that any time you hit this DCHECK, you reasonably know that any further allocations (with new) will also hit this DCHECK, which means you *could* end up looping indefinitely. It would be safer thus to not have this DCHECK or to forcibly crash at this point, since it's hard to get right. This is probably entirely contingent upon how you've got Clang doing the interposing, but it's very subtle that you're "inside new". It makes me think any new is at risk of recursion.
Mainly I'll reply and show my patch update next Monday, but I added one comment. http://codereview.chromium.org/10411047/diff/69004/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1626 build/common.gypi:1626: ['clang_profiling_allocated_type==1', { On 2012/08/10 09:38:50, Ryan Sleevi wrote: > BUG? This double nested condition seems wrong. Shouldn't it be? > > 'conditions': [ > ['clang_profiling_allocated_type == 1', { > 'conditions': [ > ['linux_use_tcmalloc == 1', { > 'dependencies': [ > '<(DEPTH)/base/allocator/allocator.gyp:allocated_type_profiler_tcmalloc', > ], > }, { > 'dependencies': [ > '<(DEPTH)/base/allocator/allocator.gyp:allocated_type_profiler_ignore', > ], > }], > ], > }], > ], That's the point I wanted to discuss. tcmalloc is not used for mksnapshot and protoc even if linux_use_tcmalloc==1. mksnapshot doesn't run with allocated_type_profiler_tcmalloc.
Thank you for the comments. Ok, I summarize my requirements : 1) In compiling *ALL* .cc files which are used in *ANY* binary linked with tcmalloc, except for some .cc files for allocator*, 1-a) add compiler flags -fintercept-allocation-functions and -frtti, and 1-b) declare prototypes of __op_new_intercept__ and __op_delete_intercept__ . 2) In all binaries linked with tcmalloc, enable profiling with AllocatedTypeMap. 3) In all binaries without tcmalloc, disable profiling with AllocatedTypeMap. Important points are : A) linux_use_tcmalloc doesn't mean that tcmalloc is actually used or not. For example, even if linux_use_tcmalloc==1, mksnapshot doesn't use tcmalloc. mksnapshot is built on the way building chrome with linux_use_tcmalloc==1. B) I am not sure that all .cc files which are used in any binary linked with tcmalloc are never used for other binaires without tcmalloc. If foo.cc is used in both binaries with/without tcmalloc, foo.cc should be compiled both with/without interceptors to switch it when compiling. C) (small point) v8/tools/gyp/v8.gyp is out of Chromium. I wanted to control it from Chromium-side. For the reasons, I chose to disable profiling ( == (3)) by linking empty interceptors for the reasons above. This change is to inject profiling feature sideaway. (I think it's a very natural way for stuff like profiling. Isn't?) Generally, injecting can be performed in compiling-phase or linking-phase. But, I gave up doing ALL in compiling-phase because of (A) and (B). It is our current status. Do you have good ideas for the situation? It is a kind of irregular change for federated profiling among many components (including externals: v8, WebKit, ...). The profiling feature is well isolated in gyp... no impact for normal builds. I guess a little "strong-arm" way is unavoidable and acceptable. What do you think? http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.h (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.h:17: #endif // BASE_ALLOCATOR_ALLOCATED_TYPE_PROFILER_CONTROL_H_ On 2012/08/10 08:43:19, Alexander Potapenko wrote: > Nit: spare space between #endif and the comment Done. http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_log.cc (right): http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t... base/allocator/allocated_type_profiler_log.cc:18: << std::endl; On 2012/08/10 09:38:50, Ryan Sleevi wrote: > On 2012/08/10 07:30:03, Dai Mikurube wrote: > > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > > style nit: Please see the comments on > > > http://www.chromium.org/developers/coding-style#TOC-Code-formatting about > code > > > formatting operator<<() calls. (Personal nit) This line breaking format > seems > > to > > > hinder readability more than help it. > > > > Do you mean I should do the following? > > > > std::cerr << "Allocated " << size << " bytes as an instance of \"" > > << type.name() << "\" at " << ptr << "." << std::endl; > > Yes, I believe that to be more readable. > > > > > The _log is used rarely in fact -- it is sometimes turned on when we want to > > check this profiler is working correctly for non-tcmalloc commands such as > > mksnapshot and protoc. Such format is good for the purpose. > > I understood your response to the no-op file being that you explicitly don't > want to interpose for such targets. I had thus assumed logging was for when you > *are* using tcmalloc and you just want to log, rather than actually wire it up > into tcmalloc. > > So is this logging file for: > 1) no tcmalloc > 2) yes tcmalloc > > If #1, I'm having trouble understanding the utility of this log? > if #2, could this be incorporated into the tcmalloc file under an #if defined(), > rather than a separate file and target? (1). Non-tcmalloc binaries are sometimes switched locally from _ignore to _log. I don't usually need it. When something is wrong with intercepting, I turn on _log and check if intercepting is working as expected. (See the top "summary" for the reason why I need intercepting for non-tcmalloc... I cannot turn it off completely. So I sometimes wanted to check it's correctly intercepted and ignored.) http://codereview.chromium.org/10411047/diff/69004/build/common.gypi File build/common.gypi (right): http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1615 build/common.gypi:1615: ['_target_name=="mksnapshot" or _target_name=="protoc" or _target_name=="ppGoogleNaClPluginChrome"', { On 2012/08/10 09:38:50, Ryan Sleevi wrote: > On 2012/08/10 07:30:03, Dai Mikurube wrote: > > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > > This is a nasty layering violation that we try very much to avoid. > > > > > > Please use a target_defaults: target_conditions to appropriately gate this > and > > > allow individual targets to override this behaviour as appropriate, and then > > use > > > a target-specific variable to gate this behaviour. > > > > > > An example of this is the chromium_code variable. > > > > > > More importantly, rather than injecting the ignore stub, you should instead > > not > > > even be passing whatever command-line args you're passing *for these > targets*. > > > > Thank you for the information. Yes, I knew it is a bad hacky way, but I > didn't > > know how to fix it. It was a result of tries-and-errors with reading gyp > code. > > GYP is complex and undocumented... > > > > What I would like is to inject a dependency to a different linked library by > > target binaries as replied at > > > http://codereview.chromium.org/10411047/diff/69004/base/allocator/allocated_t.... > > I'm sorry, I'm having trouble understanding your response and the connection > between these two statements. It seems that for these targets, you shouldn't be > intercepting at all then? > > > But, modifying each gyp file was not a good idea since we would like to > control > > this profiling feature for a whole build in one place -- common.gypi. > > I don't think these two things are in opposition. My point of highlighting how > chromium_code is that you can, on a per-target basis, in the target's gyp file, > include or exclude specific behaviours as appropriate. > > For example, you might have some: > 'variables': { > 'interpose_new%': 1, > }, > 'target_defaults': { > 'variables': { > # Dai, see the comment on > http://code.google.com/searchframe#OAMlx_jo-ck/src/build/common.gypi&exact_pa... > # to understand why this is this way > 'interpose_new%': '<(interpose_new)', > }, > 'target_conditions': [ > ['clang_profiling_allocated_type==1 and interpose_new==1', { > 'cflags_cc': [ > ], > 'include_dirs': [ > ], > }], > ], > }, > > > > > But, I still don't understand target_defaults well. How can I do that? > I looks requiring all things done in compiling-phase. It is hard for the reason written in the top "summary". http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode1620 build/common.gypi:1620: '<!(realpath <(DEPTH)/base/allocator/allocator.gyp)' + ':allocated_type_profiler_ignore#target', On 2012/08/10 09:38:50, Ryan Sleevi wrote: > On 2012/08/10 07:30:03, Dai Mikurube wrote: > > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > > 1) The use of realpath here is troubling. It definitely seems that GYP's > > > existing path handling should be appropriate here. > > > 2) The specification of #target is incorrect here - this is specifically for > > > cross-compiling, and is an internal GYP detail not (generally) meant to be > > > exposed. > > > > realpath got required from http://crrev.com/146172. From this change, > realpath > > is a key of a hashtable in gyp. > > > > Both look like because this part is evaluated lazily after some path and > target > > operations. It's a large part of this change's hacky points. But, I have no > > idea to fix them. :( > > Oh, right, that's because GYP dependency resolution happens *between* the pre- > and the post- phase, but you're decidedly in the post- phase here > (target_conditions is post phase) > > See > http://code.google.com/p/gyp/wiki/InputFormatReference#Timing_of_Variable_Exp... > to understand what I mean by pre- and post- phases > > This is controlled by > > http://code.google.com/searchframe#OAMlx_jo-ck/tools/gyp/pylib/gyp/input.py&e... > > And it's "intentional" to make sure dependencies are properly specified and not > willy-nilly cross-injected. > > Can you not just specify the correct dependencies in allocator.gyp:allocator, > which will cause all linkable types (eg: executables, shared_libraries, loadable > modules) to link in these targets as appropriate? For the reason written in the "top" summary, I have to do something for non-tcmalloc binaries. How to do it? http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2385 build/common.gypi:2385: ['_target_name!="allocator" and _target_name!="allocator_extension_thunks" and _target_name!="libcmt" and _target_name!="allocator_extension_thunks_win64" and _target_name!="allocated_type_profiler_log" and _target_name!="allocated_type_profiler_ignore" and _target_name!="allocated_type_profiler_tcmalloc"', { On 2012/08/10 09:38:50, Ryan Sleevi wrote: > On 2012/08/10 07:30:03, Dai Mikurube wrote: > > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > > See comments above about layering violation. These should be refactored. > > > > I agree with that, but I have no idea to refactor it... Here, I would like to > > change cflags_cc for almost all the .cc files except for some files related to > > specific targets. Can I do that? > > In those specific targets that you wish to exclude, manually set the variable > (eg: override the default being inherited from this file) to exclude them. > > This is what I meant by comparing with the 'chromium_code' usage, which > selectively enables or disables certain flags on a per-target basis, with a > well-defined default for all targets, and then certain targets opting in. I tried it for the compiler flags, and it looks working. Could you take a look at this? http://codereview.chromium.org/10411047/diff/69004/build/common.gypi#newcode2391 build/common.gypi:2391: 'allocated_type_profiler.h', On 2012/08/10 09:38:50, Ryan Sleevi wrote: > On 2012/08/10 07:30:03, Dai Mikurube wrote: > > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > > However, more generally, ISTM that instead of forcing base/allocator into > all > > > targets' include_dirs, you should instead be specifying the canonical path > for > > > allocated_type_profiler and using -include with the full path. > > > > > > 'variables': { > > > 'some_var_name_path': '<(DEPTH)/base/allocator/allocated_type_profiler.h' > > > }, > > > 'cflags_cc+': [ > > > '-include', '<(some_var_name_path)', > > > ] > > > > > > Note the suffix of _path, which ensures the proper path relative to > where/how > > > the files are compiled is yielded ( see > > > > http://code.google.com/p/gyp/wiki/InputFormatReference#Pathname_Relativization > > ) > > > > It was impossible maybe because it is lazily evaluated (in target_conditions) > > like above. The defined variable was not found here. > > > > "KeyError: 'Undefined variable clang_profiling_allocated_type_header_path in > > .../build/all.gyp while trying to load .../build/all.gyp'" > > > > We may have to stop target_conditions before fix it. > > Oh, that's probably just due to GYP order-of-evaluation. Use > >(some_var_name_path) rather than <(some_var_name_path) > > See > http://code.google.com/p/gyp/wiki/InputFormatReference#Timing_of_Variable_Exp... > for more about the ordering of evaluations It doesn't work, too. KeyError: 'Undefined variable clang_profiling_allocated_type_header_path in .../src/third_party/WebKit/Source/WebCore/WebCore.gyp/WebCore.gyp' Actually, I added copying the variable at line 601, and tried '<' again. It worked on gyp_chromium, but the result was wrong. I added a comment here.
http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:51: DCHECK_NE(allocated_type_map, NULL); Sorry, I overlooked it. I'll handle it tomorrow. (It's introduced at http://codereview.chromium.org/10411047/diff/72024/third_party/tcmalloc/chrom.... Reverting may be one idea.)
Updated the patch. Replaced DCHECK_NE with RAW_DCHECK. http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/69004/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:51: DCHECK_NE(allocated_type_map, NULL); On 2012/08/10 09:38:50, Ryan Sleevi wrote: > On 2012/08/10 07:30:03, Dai Mikurube wrote: > > On 2012/08/10 04:16:09, Ryan Sleevi wrote: > > > BUG? Your use of std::cerr in the _logger wrappers suggests to me that > you're > > > trying to avoid any form of heap allocation during error logging, in the > event > > > of re-entrancy. Thus, the use of DCHECK_NE here seems counter-productive to > > > that, since DCHECK_NE may invoke the DLOG() operators and/or re-enter > > > new/delete. > > > > > > It seems to me that you should either: > > > 1) CHECK > > > 2) Remove this > > > > _log doesn't call this function. Functions in this file are called only from > > _tcmalloc. (_log is rarely used as I replied, that can be removed if that's > not > > good to commit.) > > > > This checks "g_allocated_type_map and g_allocated_type_map_memory get non-NULL > > value at the same time. Both are NULL, or both are non-NULL." Usually, it > > doesn't happen because *AllocatedType functions have locks. > > I think you may have missed my point. I'm aware of what it is checking, but my > concern is you may be setting yourself up for a recursion overflow. > > Consider when this situation happens (how/when it can happen was described in > the original comment for line 60). Your initial callstack will look something > like: > > 0: InitializeAllocatedTypeMemory() > 1: InsertAllocatedType() > 2: __op_new_intercept__() > 3: operator new(...) > > This DCHECK_NE fires, which will call DLOG() with the DCHECK_NE's statement, > which will call logging::LogMessage(). If logging::LogMessage() calls new, your > stack will look like > > 0: InitializeAllocatedTypeMemory() > 1: InsertAllocatedType() > 2: __op_new_intercept__() > 3: operator new(...) > 4: logging::LogMessage() [expanded from DCHECK_NE] > 5: InitializeAllocatedTypeMemory() > 6: InsertAllocatedType() > 7: __op_new_intercept__() > 8: operator new(...) > > and thus you'll end up re-entering yourself and recursing repeatedly. The > recursion "should" be bounded by the time that it takes the cache lines to > update, so it shouldn't run off forever. > > While this file's operator new calls are not interposed (because you have it > excluded at the .gypi level), thus making line 58-60 safe, logging::LogMessage > is part of the base library, which is interposed, and thus does seem risky. > > However, the point being is that any time you hit this DCHECK, you reasonably > know that any further allocations (with new) will also hit this DCHECK, which > means you *could* end up looping indefinitely. It would be safer thus to not > have this DCHECK or to forcibly crash at this point, since it's hard to get > right. > > This is probably entirely contingent upon how you've got Clang doing the > interposing, but it's very subtle that you're "inside new". It makes me think > any new is at risk of recursion. Actually, this DCHECK_NE is in third_party/tcmalloc/chromium/src/base/logging.h. LogMessage() is not used. But, for safety, I replaced it to RAW_DCHECK, which is defined only in third_party/tcmalloc/chromium/src/base/logging.h.
This change got ready for review again. Could you PTAL this? (Though common.gypi (line 1613-1637) remains. Do you have any ideas for that? See http://codereview.chromium.org/10411047/#msg29 for requirements.)
On 2012/08/13 08:32:14, Dai Mikurube wrote: > Thank you for the comments. > > Ok, I summarize my requirements : > > 1) In compiling *ALL* .cc files which are used in *ANY* binary linked > with tcmalloc, except for some .cc files for allocator*, > 1-a) add compiler flags -fintercept-allocation-functions and -frtti, and > 1-b) declare prototypes of __op_new_intercept__ and __op_delete_intercept__ . Is 1-b strictly necessary? It would seem like -fintercept-allocation-functions should be providing the prototypes directly, and that they should be extern'd symbols that don't need to be resolved until the link phase. > > 2) In all binaries linked with tcmalloc, enable profiling with AllocatedTypeMap. > 3) In all binaries without tcmalloc, disable profiling with AllocatedTypeMap. > > > Important points are : > > A) linux_use_tcmalloc doesn't mean that tcmalloc is actually used or not. > For example, even if linux_use_tcmalloc==1, mksnapshot doesn't use tcmalloc. > mksnapshot is built on the way building chrome with linux_use_tcmalloc==1. linux_use_tcmalloc=0 is a perfect and reliable sign that TCMalloc will not be used. linux_use_tcmalloc=1 is not a *perfect* sign that TCMalloc will be used in a target. However, linux_use_tcmalloc=1 AND the binary linking to allocator.gyp:allocator IS a perfect sign - any binary (On Linux) that wishes to use TCMalloc *must* end up linking in that target. > > B) I am not sure that all .cc files which are used in any binary linked > with tcmalloc are never used for other binaires without tcmalloc. > If foo.cc is used in both binaries with/without tcmalloc, foo.cc should > be compiled both with/without interceptors to switch it when compiling. This is essentially "build variants", which are not supported in GYP and, in general, have been found to significantly add complexity to the build system. I am concerned because it sounds like what you're asking for, whether explicitly (compile the file twice) or implicitly (injecting the stubs and then conditionally adding dummy stubs to the final binaries). I think both approaches are problematic, which is perhaps why I'm having so much trouble with the approach being done here. > > C) (small point) v8/tools/gyp/v8.gyp is out of Chromium. > I wanted to control it from Chromium-side. And in the past, this has been handled by things such as 'chromium_code'. All code which is meant to be affected by some sort of special behaviour should 'opt-in', while the default should be all other code is implicitly opting out. > > For the reasons, I chose to disable profiling ( == (3)) by linking empty > interceptors for the reasons above. > > This change is to inject profiling feature sideaway. (I think it's a very > natural way for stuff like profiling. Isn't?) Generally, injecting can > be performed in compiling-phase or linking-phase. But, I gave up doing ALL > in compiling-phase because of (A) and (B). As part of GYP's design, conditionally injecting dependencies during the linking phase isn't supported. The closest approach is to indicate via link_settings a (pre-compiled) library dependency, but that's not at all the same thing here. The approach in Chromium so far has been to inject always at compile time. > > It is our current status. Do you have good ideas for the situation? > > > It is a kind of irregular change for federated profiling among many > components (including externals: v8, WebKit, ...). The profiling > feature is well isolated in gyp... no impact for normal builds. > I guess a little "strong-arm" way is unavoidable and acceptable. > > What do you think? > I think of this similar to how I would look at C++ code. If I saw code in net/ that was depending on a particular implementation detail deep in the bowels of WebKit, I think I'd raise a concern. I've already explained why your 'realpath' hack is bypassing an intentional behaviour in GYP, one which is not only not supported but which now adds maintenance cost to both GYP and Chromium, and why I think that hacks such as _target_name are out-and-out layering violations. GYP's design is that dependencies cannot and should not be added or removed during 'target_conditions'. If that's preventing forward progress on this, then I think the correct fix is to fix it in GYP so that it doesn't become a burden on all Chromium developers to maintain the hack. It seems to me that this is a similar problem to allocator_thunks, which is why I pointed it out. You're making all targets (effectively) depend on the allocator choice, which is something we've tried to carefully control and avoid. The way this has been accomplished in the past is to introduce a layer in between - base/ depends on thunks, which provide a set of function pointers. The allocator also depends on the thunks, and is responsible for setting those function pointers. This is a common and well-documented technique to avoid strict coupling between two components. To bring all of these things together: 1) Have a single implementation for __op_new_intercept__/__op_delete_intercept__. 2) This target (a static library) will be injected into ALL targets (eg: in build/common.gypi, a section in 'target_defaults' that adds 'my_target' to the 'dependencies'). This is a supported form of DI 3) The implementation of __op_new_intercept__ will rely on either a) Stub function pointers that can be set at run-time when enabling TCmalloc (ala allocator_thunks) b) Weak symbols that are resolved at linking (ala __malloc_initialize_hook , which tcmalloc overrides). 4) When linking in allocator.gyp:allocator (which only tcmalloc-using targets should be doing), override the weak symbol / set the run time function pointers to the 'real' auditing functions This should avoid all of the current complexity in GYP, and guarantee the 'right' behaviour for all targets. Only targets which have opted in to tcmalloc (via allocator.gyp dependency) will be set. The RTTI & special c-flags can be handled by a target_conditions block, so you should be safe there. You can also dependencies! within those specific targets (the allocator targets) to 'undo' the main target_defaults from build/common.gypi. WDYT?
Thank you, Ryan. I learned many things about gyp. :) First of all, I could remove "_target_name" from common.gyp in the Patch Set #36. It is a good progress. But, it still has "realpath" because it is in "target_conditions". "target_conditions" is still required for "_type" to avoid cyclic dependencies. Please look at the paragraphs at last, and comments in common.gypi line 1613-. On 2012/08/14 21:21:04, Ryan Sleevi wrote: > On 2012/08/13 08:32:14, Dai Mikurube wrote: > > Thank you for the comments. > > > > Ok, I summarize my requirements : > > > > 1) In compiling *ALL* .cc files which are used in *ANY* binary linked > > with tcmalloc, except for some .cc files for allocator*, > > 1-a) add compiler flags -fintercept-allocation-functions and -frtti, and > > 1-b) declare prototypes of __op_new_intercept__ and __op_delete_intercept__ . > > Is 1-b strictly necessary? It would seem like -fintercept-allocation-functions > should be providing the prototypes directly, and that they should be extern'd > symbols that don't need to be resolved until the link phase. > Actually, the modified Clang doesn't declare prototypes. But even if prototypes are provided by -fintercept-allocation-functions, what we have to do is to provide compiler options to almost all .cc. files. The difference is only '-include'. > > > > 2) In all binaries linked with tcmalloc, enable profiling with > AllocatedTypeMap. > > 3) In all binaries without tcmalloc, disable profiling with AllocatedTypeMap. > > > > > > Important points are : > > > > A) linux_use_tcmalloc doesn't mean that tcmalloc is actually used or not. > > For example, even if linux_use_tcmalloc==1, mksnapshot doesn't use tcmalloc. > > mksnapshot is built on the way building chrome with linux_use_tcmalloc==1. > > linux_use_tcmalloc=0 is a perfect and reliable sign that TCMalloc will not be > used. > > linux_use_tcmalloc=1 is not a *perfect* sign that TCMalloc will be used in a > target. However, linux_use_tcmalloc=1 AND the binary linking to > allocator.gyp:allocator IS a perfect sign - any binary (On Linux) that wishes to > use TCMalloc *must* end up linking in that target. > Got it. We can add dependencies to allocator.gyp:allocator when linux_use_tcmalloc==1. (If linux_use_tcmalloc==1, allocator.gyp:allocator is added to dependencies.) But, to add compiler flags to almost all the .cc files, I guess we have to know "Is the target binary linking to allocator.gyp:allocator?" in compiling-phase for each .cc file. Isn't it? > > > > > B) I am not sure that all .cc files which are used in any binary linked > > with tcmalloc are never used for other binaires without tcmalloc. > > If foo.cc is used in both binaries with/without tcmalloc, foo.cc should > > be compiled both with/without interceptors to switch it when compiling. > > This is essentially "build variants", which are not supported in GYP and, in > general, have been found to significantly add complexity to the build system. I > am concerned because it sounds like what you're asking for, whether explicitly > (compile the file twice) or implicitly (injecting the stubs and then > conditionally adding dummy stubs to the final binaries). > > I think both approaches are problematic, which is perhaps why I'm having so much > trouble with the approach being done here. > That is a sad news. But it is reasonable that we avoid complexity in gyp. > > > > C) (small point) v8/tools/gyp/v8.gyp is out of Chromium. > > I wanted to control it from Chromium-side. > > And in the past, this has been handled by things such as 'chromium_code'. All > code which is meant to be affected by some sort of special behaviour should > 'opt-in', while the default should be all other code is implicitly opting out. > Make sense. I understood the motivation of 'chromium_code' at last. > > > > For the reasons, I chose to disable profiling ( == (3)) by linking empty > > interceptors for the reasons above. > > > > This change is to inject profiling feature sideaway. (I think it's a very > > natural way for stuff like profiling. Isn't?) Generally, injecting can > > be performed in compiling-phase or linking-phase. But, I gave up doing ALL > > in compiling-phase because of (A) and (B). > > As part of GYP's design, conditionally injecting dependencies during the linking > phase isn't supported. The closest approach is to indicate via link_settings a > (pre-compiled) library dependency, but that's not at all the same thing here. > > The approach in Chromium so far has been to inject always at compile time. > > > > > It is our current status. Do you have good ideas for the situation? > > > > > > It is a kind of irregular change for federated profiling among many > > components (including externals: v8, WebKit, ...). The profiling > > feature is well isolated in gyp... no impact for normal builds. > > I guess a little "strong-arm" way is unavoidable and acceptable. > > > > What do you think? > > > > I think of this similar to how I would look at C++ code. If I saw code in net/ > that was depending on a particular implementation detail deep in the bowels of > WebKit, I think I'd raise a concern. > > I've already explained why your 'realpath' hack is bypassing an intentional > behaviour in GYP, one which is not only not supported but which now adds > maintenance cost to both GYP and Chromium, and why I think that hacks such as > _target_name are out-and-out layering violations. > Agreed. I also believe removing such violations are the best if it is reasonably possible. We could remove one thanks to you. gyp sometimes requires delicate work-arounds here and there... > GYP's design is that dependencies cannot and should not be added or removed > during 'target_conditions'. If that's preventing forward progress on this, then > I think the correct fix is to fix it in GYP so that it doesn't become a burden > on all Chromium developers to maintain the hack. > > It seems to me that this is a similar problem to allocator_thunks, which is why > I pointed it out. You're making all targets (effectively) depend on the > allocator choice, which is something we've tried to carefully control and avoid. > The way this has been accomplished in the past is to introduce a layer in > between - base/ depends on thunks, which provide a set of function pointers. The > allocator also depends on the thunks, and is responsible for setting those > function pointers. > > This is a common and well-documented technique to avoid strict coupling between > two components. > > To bring all of these things together: > 1) Have a single implementation for > __op_new_intercept__/__op_delete_intercept__. > 2) This target (a static library) will be injected into ALL targets (eg: in > build/common.gypi, a section in 'target_defaults' that adds 'my_target' to the > 'dependencies'). This is a supported form of DI > 3) The implementation of __op_new_intercept__ will rely on either > a) Stub function pointers that can be set at run-time when enabling TCmalloc > (ala allocator_thunks) > b) Weak symbols that are resolved at linking (ala __malloc_initialize_hook , > which tcmalloc overrides). > 4) When linking in allocator.gyp:allocator (which only tcmalloc-using targets > should be doing), override the weak symbol / set the run time function pointers > to the 'real' auditing functions > > This should avoid all of the current complexity in GYP, and guarantee the > 'right' behaviour for all targets. Only targets which have opted in to tcmalloc > (via allocator.gyp dependency) will be set. > This could remove "_target_name". Thanks! As wrote above, common.gypi still has "realpath" because it is in "target_conditions" for "_type". "_type" is still alive to avoid cyclic dependencies. If we add "dependencies" to all targets as you say in (2), "allocated_type_profiler" is added to "allocated_type_profiler". (Or, it may have another cyclic dependencies.) Limiting DI to "executables" and "loadable_modules" is an idea to deal with the problem. I'd like to remove "realpath", too, if possible. Do you have other ideas? If we don't have http://crrev.com/146172, realpath was not required (though "#target" is required). I think 146172 was not a good fix... > The RTTI & special c-flags can be handled by a target_conditions block, so you > should be safe there. You can also dependencies! within those specific targets > (the allocator targets) to 'undo' the main target_defaults from > build/common.gypi. > > WDYT?
On Wed, Aug 15, 2012 at 2:12 AM, <dmikurube@chromium.org> wrote: > Thank you, Ryan. I learned many things about gyp. :) > > First of all, I could remove "_target_name" from common.gyp in the Patch Set > #36. It is a good progress. > > But, it still has "realpath" because it is in "target_conditions". > "target_conditions" is still required for "_type" to avoid cyclic > dependencies. > Please look at the paragraphs at last, and comments in common.gypi line > 1613-. > > > > On 2012/08/14 21:21:04, Ryan Sleevi wrote: >> >> On 2012/08/13 08:32:14, Dai Mikurube wrote: >> > Thank you for the comments. >> > >> > Ok, I summarize my requirements : >> > >> > 1) In compiling *ALL* .cc files which are used in *ANY* binary linked >> > with tcmalloc, except for some .cc files for allocator*, >> > 1-a) add compiler flags -fintercept-allocation-functions and -frtti, and >> > 1-b) declare prototypes of __op_new_intercept__ and >> > __op_delete_intercept__ > > . > >> Is 1-b strictly necessary? It would seem like >> -fintercept-allocation-functions >> should be providing the prototypes directly, and that they should be >> extern'd >> symbols that don't need to be resolved until the link phase. > > > Actually, the modified Clang doesn't declare prototypes. > But even if prototypes are provided by -fintercept-allocation-functions, > what we have to do is to provide compiler options to almost all .cc. files. > The difference is only '-include'. > > >> > >> > 2) In all binaries linked with tcmalloc, enable profiling with >> AllocatedTypeMap. >> > 3) In all binaries without tcmalloc, disable profiling with > > AllocatedTypeMap. >> >> > >> > >> > Important points are : >> > >> > A) linux_use_tcmalloc doesn't mean that tcmalloc is actually used or >> > not. >> > For example, even if linux_use_tcmalloc==1, mksnapshot doesn't use >> > tcmalloc. >> > mksnapshot is built on the way building chrome with >> > linux_use_tcmalloc==1. > > >> linux_use_tcmalloc=0 is a perfect and reliable sign that TCMalloc will not >> be >> used. > > >> linux_use_tcmalloc=1 is not a *perfect* sign that TCMalloc will be used in >> a >> target. However, linux_use_tcmalloc=1 AND the binary linking to >> allocator.gyp:allocator IS a perfect sign - any binary (On Linux) that >> wishes > > to >> >> use TCMalloc *must* end up linking in that target. > > > Got it. We can add dependencies to allocator.gyp:allocator when > linux_use_tcmalloc==1. > (If linux_use_tcmalloc==1, allocator.gyp:allocator is added to > dependencies.) > > But, to add compiler flags to almost all the .cc files, I guess we have to > know "Is the target binary linking to allocator.gyp:allocator?" in > compiling-phase > for each .cc file. Isn't it? > > > >> > >> > B) I am not sure that all .cc files which are used in any binary linked >> > with tcmalloc are never used for other binaires without tcmalloc. >> > If foo.cc is used in both binaries with/without tcmalloc, foo.cc should >> > be compiled both with/without interceptors to switch it when compiling. > > >> This is essentially "build variants", which are not supported in GYP and, >> in >> general, have been found to significantly add complexity to the build >> system. > > I >> >> am concerned because it sounds like what you're asking for, whether >> explicitly >> (compile the file twice) or implicitly (injecting the stubs and then >> conditionally adding dummy stubs to the final binaries). > > >> I think both approaches are problematic, which is perhaps why I'm having >> so > > much >> >> trouble with the approach being done here. > > > That is a sad news. But it is reasonable that we avoid complexity in gyp. > > >> > >> > C) (small point) v8/tools/gyp/v8.gyp is out of Chromium. >> > I wanted to control it from Chromium-side. > > >> And in the past, this has been handled by things such as 'chromium_code'. >> All >> code which is meant to be affected by some sort of special behaviour >> should >> 'opt-in', while the default should be all other code is implicitly opting >> out. > > > Make sense. I understood the motivation of 'chromium_code' at last. > > >> > >> > For the reasons, I chose to disable profiling ( == (3)) by linking empty >> > interceptors for the reasons above. >> > >> > This change is to inject profiling feature sideaway. (I think it's a >> > very >> > natural way for stuff like profiling. Isn't?) Generally, injecting can >> > be performed in compiling-phase or linking-phase. But, I gave up doing >> > ALL >> > in compiling-phase because of (A) and (B). > > >> As part of GYP's design, conditionally injecting dependencies during the > > linking >> >> phase isn't supported. The closest approach is to indicate via >> link_settings a >> (pre-compiled) library dependency, but that's not at all the same thing >> here. > > >> The approach in Chromium so far has been to inject always at compile time. > > >> > >> > It is our current status. Do you have good ideas for the situation? >> > >> > >> > It is a kind of irregular change for federated profiling among many >> > components (including externals: v8, WebKit, ...). The profiling >> > feature is well isolated in gyp... no impact for normal builds. >> > I guess a little "strong-arm" way is unavoidable and acceptable. >> > >> > What do you think? >> > > > >> I think of this similar to how I would look at C++ code. If I saw code in >> net/ >> that was depending on a particular implementation detail deep in the >> bowels of >> WebKit, I think I'd raise a concern. > > >> I've already explained why your 'realpath' hack is bypassing an >> intentional >> behaviour in GYP, one which is not only not supported but which now adds >> maintenance cost to both GYP and Chromium, and why I think that hacks such >> as >> _target_name are out-and-out layering violations. > > > Agreed. I also believe removing such violations are the best if it is > reasonably possible. > We could remove one thanks to you. gyp sometimes requires delicate > work-arounds > here and there... > > >> GYP's design is that dependencies cannot and should not be added or >> removed >> during 'target_conditions'. If that's preventing forward progress on this, > > then >> >> I think the correct fix is to fix it in GYP so that it doesn't become a >> burden >> on all Chromium developers to maintain the hack. > > >> It seems to me that this is a similar problem to allocator_thunks, which >> is > > why >> >> I pointed it out. You're making all targets (effectively) depend on the >> allocator choice, which is something we've tried to carefully control and > > avoid. >> >> The way this has been accomplished in the past is to introduce a layer in >> between - base/ depends on thunks, which provide a set of function >> pointers. > > The >> >> allocator also depends on the thunks, and is responsible for setting those >> function pointers. > > >> This is a common and well-documented technique to avoid strict coupling > > between >> >> two components. > > >> To bring all of these things together: >> 1) Have a single implementation for >> __op_new_intercept__/__op_delete_intercept__. >> 2) This target (a static library) will be injected into ALL targets (eg: >> in >> build/common.gypi, a section in 'target_defaults' that adds 'my_target' to >> the >> 'dependencies'). This is a supported form of DI >> 3) The implementation of __op_new_intercept__ will rely on either >> a) Stub function pointers that can be set at run-time when enabling >> TCmalloc >> (ala allocator_thunks) >> b) Weak symbols that are resolved at linking (ala >> __malloc_initialize_hook , >> which tcmalloc overrides). >> 4) When linking in allocator.gyp:allocator (which only tcmalloc-using >> targets >> should be doing), override the weak symbol / set the run time function > > pointers >> >> to the 'real' auditing functions > > >> This should avoid all of the current complexity in GYP, and guarantee the >> 'right' behaviour for all targets. Only targets which have opted in to > > tcmalloc >> >> (via allocator.gyp dependency) will be set. > > > This could remove "_target_name". Thanks! > > As wrote above, common.gypi still has "realpath" because it is in > "target_conditions" for "_type". "_type" is still alive to avoid cyclic > dependencies. If we add "dependencies" to all targets as you say in (2), > "allocated_type_profiler" is added to "allocated_type_profiler". (Or, it > may > have another cyclic dependencies.) > > Limiting DI to "executables" and "loadable_modules" is an idea to deal with > the > problem. I'd like to remove "realpath", too, if possible. Do you have > other > ideas? > > If we don't have http://crrev.com/146172, realpath was not required (though > "#target" is required). I think 146172 was not a good fix... As mentioned above, if it's causing problems, mention that on https://chromiumcodereview.appspot.com/10692086 . Maybe the author of that CL can think of a good fix, or agrees to revert. > > >> The RTTI & special c-flags can be handled by a target_conditions block, so >> you >> should be safe there. You can also dependencies! within those specific >> targets >> (the allocator targets) to 'undo' the main target_defaults from >> build/common.gypi. > > >> WDYT? > > > http://codereview.chromium.org/10411047/
On Wed, Aug 15, 2012 at 2:12 AM, <dmikurube@chromium.org> wrote: > Thank you, Ryan. I learned many things about gyp. :) > > First of all, I could remove "_target_name" from common.gyp in the Patch Set > #36. It is a good progress. > > But, it still has "realpath" because it is in "target_conditions". > "target_conditions" is still required for "_type" to avoid cyclic > dependencies. > Please look at the paragraphs at last, and comments in common.gypi line > 1613-. The use of realpath is a hack that is relying on a bug/non-supported functionality in GYP. The dependency you're trying to synthesize is not being processed according to GYP's dependency handling, which is why you're having to do things like "realpath" and "#target" (the latter being an internal GYP variable). While http://crrev.com/146172 may be related, again, fundamentally you're relying on undefined/unsupported behaviour. That's why it's so troubling. It's like casting from a void pointer to a function pointer - it may work for your compiler, but it's not at all a good idea. Like I said, you can get around this by, within the targets inside allocator.gyp (eg: all of these targets are in the chromium tree), adding sections that undo the target_defaults from build/common.gypi build/common.gypi 'target_defaults': { 'conditions' [ ['your_condition==1', { 'dependencies': [ 'allocator.gyp:my_intercept_target', ], }], ], } base/allocator/allocator.gyp 'target_defaults': { 'conditions': [ ['your_condition==1', { 'dependencies!': [ 'my_intercept_target' ], 'cflags_cc!': [ '-fyour-flag-here' ], }], ], } If the 'dependencies!' gives you trouble, try with 'dependencies/': [ [ 'exclude', 'foo' ] ], but I don't think you'll run into that here (has to do with the order in which GYP processes includes relative to target_defaults) > Actually, the modified Clang doesn't declare prototypes. > But even if prototypes are provided by -fintercept-allocation-functions, > what we have to do is to provide compiler options to almost all .cc. files. > The difference is only '-include'. Right, which I think *is* valuable here. Having to -include to force a declaration that the compiler needs seems very counter-intuitive. Much like glibc and friends declare functions that an application can implement to override functionality, it would seem like the compiler should be providing implicit declarations that are flagged as extern, so you can avoid the -include. After all, it's not like you can actually change the decls within your .h file - as then the compiler will start complaining about the missing functions. > Got it. We can add dependencies to allocator.gyp:allocator when > linux_use_tcmalloc==1. > (If linux_use_tcmalloc==1, allocator.gyp:allocator is added to > dependencies.) > > But, to add compiler flags to almost all the .cc files, I guess we have to > know "Is the target binary linking to allocator.gyp:allocator?" in > compiling-phase > for each .cc file. Isn't it? Adding the compiler flag to all cc files is fine. Further, if you go with a weak symbol approach like I suggested, then you don't have to worry about whether a target will or will not link to allocator.gyp:allocator. If it does, then allocator.gyp will provide that symbol (turning it into a strong symbol, like it does for __malloc_initialize_hook), and thus all your recording functionality will be enabled. If you want to enable logging for a target that *isn't* using tcmalloc, then you just cause that target binary to link against allocator.gyp:my_logging_target, which provides a *different* implementation of that weak symbol. Again, at link time, it will be bound into a strong symbol, and all your logging functionality will now work. For any target that fails to link against allocator.gyp:allocator or allocator.gyp:my_logging_target, then your function pointer(s) will be left as weak, which is to say they'll be NULL-initialized, and thus you won't have to add any special hooking. > This could remove "_target_name". Thanks! > > As wrote above, common.gypi still has "realpath" because it is in > "target_conditions" for "_type". "_type" is still alive to avoid cyclic > dependencies. If we add "dependencies" to all targets as you say in (2), > "allocated_type_profiler" is added to "allocated_type_profiler". (Or, it > may > have another cyclic dependencies.) That's only half of the solution. As mentioned above, the other half is that within allocator.gyp (eg: *not* build/common.gypi), you exclude the specific targets from allocator that shouldn't have the thunks injected, nor should you have cyclic dependencies,
Style nit. Looks like more experienced reviewers have been pulled into the review, so I'm not sure I can find anything else they haven't pointed to. http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.h (right): http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.h:14: } please add "// namespace <name>" here and in other files.
http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:30: return base::allocated_type::g_new_intercept(ptr, size, type); Please consider using a thread safe construction. FWIU, it's not guaranteed that the value g_new_intercept will stay constant between the check and the function call.
Thanks for your comments, Nico, Ryan, Alexander and Marc. Updated the patch set, and replied for some comments. For the 'realpath' issue : Thanks! It is not fixed with Ryan's 'target_defaults' example code. I didn't understand 'target_defaults' well... My only concern is that targets have many dependency items for liballocated_type_profiler.a though it doesn't have any impact in building. For example, out/Debug/obj/chrome/chrome.ninja has 33 liballocated_type_profiler.a in "build". I decided to do nothing about http://crrev.com/146172 for now. I am not sure if it will have problems with "currect" gyp usage. For the '-include' compiler option : Make sense. But, I'd like to use this -include continuously, since I am planning to make the modified Clang switchable its behavior by declared prototypes per interceptor functions. Our previous Clang prototype had that, but the current uploaded one doesn't have. I am trying/testing it locally, and I'd like to decide the final design of the compiler option's while operating and testing practically. Having -include makes us very easy to test it. (And, another problem is that implicit declaration is a little complex for the current modified Clang. We should spend time to consider the design.) For 'linux_use_tcmalloc' : Added a check for linux_use_tcmalloc. When required later, we should consider adding another logging stuff for targets *without* tcmalloc. (For now, we removed it.) http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:30: return base::allocated_type::g_new_intercept(ptr, size, type); On 2012/08/19 02:23:26, Marc-Antoine Ruel wrote: > Please consider using a thread safe construction. FWIU, it's not guaranteed that > the value g_new_intercept will stay constant between the check and the function > call. Do you have a good idea to avoid speed overhead? It is called very frequently, I would like to take a reasonable way. It is now doing the same way with base/allocator/allocator_extension. If this variable changes only from NULL to some value, it is safe. (Ah, allocator_extension's Set* do DCHECK to set functions only when g_* is NULL though some problems still may happen when Set* is called from multiple threads.) I think adding DCHECK and some comments to call it only from one place/thread are enough for that like allocator_extension. What do you think? If ok, I'll add them. http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.h (right): http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.h:14: } On 2012/08/17 11:53:46, Alexander Potapenko wrote: > please add "// namespace <name>" here and in other files. Done.
http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:30: return base::allocated_type::g_new_intercept(ptr, size, type); On 2012/08/20 09:43:56, Dai Mikurube wrote: > On 2012/08/19 02:23:26, Marc-Antoine Ruel wrote: > > Please consider using a thread safe construction. FWIU, it's not guaranteed > that > > the value g_new_intercept will stay constant between the check and the > function > > call. > > Do you have a good idea to avoid speed overhead? It is called very frequently, > I would like to take a reasonable way. > > It is now doing the same way with base/allocator/allocator_extension. If this > variable changes only from NULL to some value, it is safe. (Ah, > allocator_extension's Set* do DCHECK to set functions only when g_* is NULL > though some problems still may happen when Set* is called from multiple > threads.) You have 2 ways; 1. Ignore ARM and just read the value in a temporary variable and use it after. Not a great idea. 2. Use a proper atomic operation. See atomicops.h.
On 2012/08/20 18:09:04, Marc-Antoine Ruel wrote: > http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... > File base/allocator/allocated_type_profiler.cc (right): > > http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... > base/allocator/allocated_type_profiler.cc:30: return > base::allocated_type::g_new_intercept(ptr, size, type); > On 2012/08/20 09:43:56, Dai Mikurube wrote: > > On 2012/08/19 02:23:26, Marc-Antoine Ruel wrote: > > > Please consider using a thread safe construction. FWIU, it's not guaranteed > > that > > > the value g_new_intercept will stay constant between the check and the > > function > > > call. > > > > Do you have a good idea to avoid speed overhead? It is called very > frequently, > > I would like to take a reasonable way. > > > > It is now doing the same way with base/allocator/allocator_extension. If this > > variable changes only from NULL to some value, it is safe. (Ah, > > allocator_extension's Set* do DCHECK to set functions only when g_* is NULL > > though some problems still may happen when Set* is called from multiple > > threads.) > > You have 2 ways; > 1. Ignore ARM and just read the value in a temporary variable and use it after. > Not a great idea. > 2. Use a proper atomic operation. See atomicops.h. What is the reason why you think DCHECK is not enough? If we guarantee g_* changes only from NULL, 1 doesn't look like fixing any problem. Jim (jar@) says not to use atomicops so casually. What do you think, Jim?
This change is ready for review (maybe good) again except for Marc's point. Could you take a look at this?
Added John (jam@) to reviewers for content/app/ change. John, could you take a look at my change in content/app/content_main_runner.cc? Please see the discussion by Ryan and me for the reason of this change. It is doing a similar thing to base/allocator/allocator_extension.*. (base::allocator::SetGetStatsFunction and SetReleaseFreeMemoryFunction just below my change.)
On 2012/08/21 09:01:09, Dai Mikurube wrote: > Added John (jam@) to reviewers for content/app/ change. > > John, could you take a look at my change in content/app/content_main_runner.cc? > > Please see the discussion by Ryan and me for the reason of this change. It is > doing a similar thing to base/allocator/allocator_extension.*. > (base::allocator::SetGetStatsFunction and SetReleaseFreeMemoryFunction just > below my change.) there are a lot of messages. can you point me to which one, or give a summary?
GYP changes are massively cleaner, so they LGTM. Thanks for addressing the concerns.
John,
On 2012/08/21 16:07:30, John Abd-El-Malek wrote:
> On 2012/08/21 09:01:09, Dai Mikurube wrote:
> > Added John (jam@) to reviewers for content/app/ change.
> >
> > John, could you take a look at my change in
> content/app/content_main_runner.cc?
> >
> > Please see the discussion by Ryan and me for the reason of this change. It
is
> > doing a similar thing to base/allocator/allocator_extension.*.
> > (base::allocator::SetGetStatsFunction and SetReleaseFreeMemoryFunction just
> > below my change.)
> there are a lot of messages. can you point me to which one, or give a summary?
It is introduced to turn on new/delete hooks dynamically.
Due to compilation/linking dependency issues in GYP, we needed to use weak
__op_{new|delete}_intercept__ references and enable them dynamically. The 4
lines in content_main_runner (line 455-458) enables them in the beginning of
chrome processes.
In addition, it's s similar way to existing base/allocator/allocator_extension, and its SetGetStatsFunction and SetReleaseFreeMemoryFunction.
On 2012/08/21 23:06:13, Dai Mikurube wrote: > In addition, it's s similar way to existing base/allocator/allocator_extension, > and its SetGetStatsFunction and SetReleaseFreeMemoryFunction. Two comments: 1.) Your initialization in content/app/content_main_runner.cc is in an #else of an #if defined(OS_WIN) block. There's a comment next to the SetGetsStats.. block indicating why - those thunks are initialized on windows inside _heap_init() inside allocator_shim.cc. You haven't patched allocator_shim.cc, so I suspect this doesn't work on windows. Is this intentional? 2.) The SetGetStats.. and SetReleaseFree.. thunks can be initialized somewhat lazily in content_main_runner because it's only important that they be initialized before we start up any chrome threads. Specifically it's OK for these to be initialized after we've done any number of heap allocations or started up any non-chrome code that may spin up threads. I somewhat doubt that's true for new/delete interceptors - is it OK to new an object before you set up the thunks and delete it after? Is it OK if threads not under our direct control spin up and start attempting to new/delete while you are trying to set up your thunks? If not you need to initialize way earlier.
On 2012/08/21 23:43:23, jamesr wrote: > On 2012/08/21 23:06:13, Dai Mikurube wrote: > > In addition, it's s similar way to existing > base/allocator/allocator_extension, > > and its SetGetStatsFunction and SetReleaseFreeMemoryFunction. > > Two comments: > > 1.) Your initialization in content/app/content_main_runner.cc is in an #else of > an #if defined(OS_WIN) block. There's a comment next to the SetGetsStats.. block > indicating why - those thunks are initialized on windows inside _heap_init() > inside allocator_shim.cc. You haven't patched allocator_shim.cc, so I suspect > this doesn't work on windows. Is this intentional? > Yes, you are correct. It is only for Linux and Clang. (As written in common.gypi: ['OS=="linux" and clang_profiling_allocated_type==1']) > 2.) The SetGetStats.. and SetReleaseFree.. thunks can be initialized somewhat > lazily in content_main_runner because it's only important that they be > initialized before we start up any chrome threads. Specifically it's OK for > these to be initialized after we've done any number of heap allocations or > started up any non-chrome code that may spin up threads. I somewhat doubt > that's true for new/delete interceptors - is it OK to new an object before you > set up the thunks and delete it after? Is it OK if threads not under our direct > control spin up and start attempting to new/delete while you are trying to set > up your thunks? If not you need to initialize way earlier. It is acceptable for now. It obtains good enough data for memory profiling as we think in our local experiment compared to earlier initialization. Almost all memory issues are after start-up. When we have to investigate start-up memory problems in future, we may want to initialize them earlier.
lgtm with nit http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:29: if (base::allocated_type::g_new_intercept) { As I said before, it's still be simpler and safer to do: InterceptFunction* new_intercept = g_new_intercept; if (new_intercept) { return new_intercept(ptr, size, type); }
On 2012/08/22 00:21:12, Dai Mikurube wrote: > On 2012/08/21 23:43:23, jamesr wrote: > > On 2012/08/21 23:06:13, Dai Mikurube wrote: > > > In addition, it's s similar way to existing > > base/allocator/allocator_extension, > > > and its SetGetStatsFunction and SetReleaseFreeMemoryFunction. > > > > Two comments: > > > > 1.) Your initialization in content/app/content_main_runner.cc is in an #else > of > > an #if defined(OS_WIN) block. There's a comment next to the SetGetsStats.. > block > > indicating why - those thunks are initialized on windows inside _heap_init() > > inside allocator_shim.cc. You haven't patched allocator_shim.cc, so I suspect > > this doesn't work on windows. Is this intentional? > > > Yes, you are correct. It is only for Linux and Clang. > (As written in common.gypi: ['OS=="linux" and > clang_profiling_allocated_type==1']) > > > 2.) The SetGetStats.. and SetReleaseFree.. thunks can be initialized somewhat > > lazily in content_main_runner because it's only important that they be > > initialized before we start up any chrome threads. Specifically it's OK for > > these to be initialized after we've done any number of heap allocations or > > started up any non-chrome code that may spin up threads. I somewhat doubt > > that's true for new/delete interceptors - is it OK to new an object before you > > set up the thunks and delete it after? Is it OK if threads not under our > direct > > control spin up and start attempting to new/delete while you are trying to set > > up your thunks? If not you need to initialize way earlier. > > It is acceptable for now. It obtains good enough data for memory profiling as > we think in our local experiment compared to earlier initialization. Almost all > memory issues are after start-up. > > When we have to investigate start-up memory problems in future, we may want to > initialize them earlier. Note: If you do use the weak symbols like I suggested, you would have them initialized at the earliest possible point, and would avoid needing to touch these files.
Thanks Marc. Updated the patch. Now, ready for review again. I wonder if I could have OWNER's review for base/ and content/app/. > Note: If you do use the weak symbols like I suggested, you would have them > initialized at the earliest possible point, and would avoid needing to touch > these files. Sorry, I did not understand well, but do you know a better earlier point for that? http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:29: if (base::allocated_type::g_new_intercept) { On 2012/08/22 13:38:17, Marc-Antoine Ruel wrote: > As I said before, it's still be simpler and safer to do: > > InterceptFunction* new_intercept = g_new_intercept; > if (new_intercept) { > return new_intercept(ptr, size, type); > } Done.
http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:26: void* __op_new_intercept__(void* ptr, nit: functions should appear in the same order as they do in the header file. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:30: base::allocated_type::g_new_intercept; nit: If you prefixed this definition code with "::", then you could list them all inside the namespaces, and not have to repeat the base::allocated_type qualifier repeatedly here. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.h (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler.h:14: // It depends on a modified version of Clang checked-in at nit: "It depends on" --> "The are called by" http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:12: namespace allocated_type { nit: we don't "allocate types," so this namespace seems to drop the interesting word "profiler" and keep the less significant word "allocated." Please consider: type_profiler http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:17: // See http://crbug.com/36678. nit: This comment should probably only be placed at the call site. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_tcmalloc.cc:20: if (base::allocated_type::IsProfilingAllocatedType()) { nit: no need to qualify the namespace here. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_tcmalloc.cc:21: if (IsHeapProfilerRunning()) { nit: consider use of && to combine if statements. (and you'll have no reason to have any curlies). Same below. http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... File content/app/content_main_runner.cc (right): http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... content/app/content_main_runner.cc:455: base::allocated_type::SetNewInterceptFunction( nit: Do you ever set these separately? IF you always set them as a pair, why not have one function: SetInterceptFunctions(...) http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:66: SpinLockHolder l(&g_allocated_type_lock); nit: don't use one letter variable names like "l". http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:90: static const TCMallocGuard tcmalloc_initializer; Please move to the top of the file, per suggestion in header file. Please add comment: // Ensure that TCMalloc has been initialized before static constructors are run. http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:15: AllocatedObject(unsigned int a, const std::type_info* b): size(a), type(b) {} nit: don't use one letter variable names.
Thanks Jim. Updated the patch set. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:26: void* __op_new_intercept__(void* ptr, On 2012/08/23 19:55:25, jar wrote: > nit: functions should appear in the same order as they do in the header file. > Done. Reordered .h. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:30: base::allocated_type::g_new_intercept; On 2012/08/23 19:55:25, jar wrote: > nit: If you prefixed this definition code with "::", then you could list them > all inside the namespaces, and not have to repeat the base::allocated_type > qualifier repeatedly here. Sorry, I didn't understand what you meant... If you meant moving __op_*_intercept__ to base::allocated_type, it is impossible. They must be in the global scope for the modified compiler. I tried moving them to base::allocated_type and prefixing with "::" (void* ::__op_new_intercept), it didn't build... I guess I'm misunderstanding what you say. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.h (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler.h:14: // It depends on a modified version of Clang checked-in at On 2012/08/23 19:55:25, jar wrote: > nit: "It depends on" --> "The are called by" Done. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:12: namespace allocated_type { On 2012/08/23 19:55:25, jar wrote: > nit: we don't "allocate types," so this namespace seems to drop the interesting > word "profiler" and keep the less significant word "allocated." Please > consider: > > type_profiler Renamed it to allocated_type_profiler. As I said before, "allocated type" is a very common phrase to describe "a type of an allocated object". So keeping "allocated" is reasonable. * http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3376.pdf (p.106) * http://clang.llvm.org/doxygen/CGCXXABI_8h_source.html#l00219 * http://www.almostinfinite.com/memtrack.html http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:17: // See http://crbug.com/36678. On 2012/08/23 19:55:25, jar wrote: > nit: This comment should probably only be placed at the call site. Done. Moved it to the callsite, base/process_util_posix.cc. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_tcmalloc.cc:20: if (base::allocated_type::IsProfilingAllocatedType()) { On 2012/08/23 19:55:25, jar wrote: > nit: no need to qualify the namespace here. Done. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_tcmalloc.cc:21: if (IsHeapProfilerRunning()) { On 2012/08/23 19:55:25, jar wrote: > nit: consider use of && to combine if statements. (and you'll have no reason to > have any curlies). Same below. Done. http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... File content/app/content_main_runner.cc (right): http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... content/app/content_main_runner.cc:455: base::allocated_type::SetNewInterceptFunction( On 2012/08/23 19:55:25, jar wrote: > nit: Do you ever set these separately? IF you always set them as a pair, why not > have one function: > > SetInterceptFunctions(...) Done. I kept individual functions since I often use them locally. http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:66: SpinLockHolder l(&g_allocated_type_lock); On 2012/08/23 19:55:25, jar wrote: > nit: don't use one letter variable names like "l". Done. (It's very common in tcmalloc...) http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:90: static const TCMallocGuard tcmalloc_initializer; On 2012/08/23 19:55:25, jar wrote: > Please move to the top of the file, per suggestion in header file. Please add > comment: > // Ensure that TCMalloc has been initialized before static constructors are run. Done. http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h (right): http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/gperftools/allocated_type_map.h:15: AllocatedObject(unsigned int a, const std::type_info* b): size(a), type(b) {} On 2012/08/23 19:55:25, jar wrote: > nit: don't use one letter variable names. Done.
http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:12: namespace allocated_type { I will remove the courtesy of calling this a nit, which I have used up to this point. I don't believe it is a nit, as it is damaging readability. The namespace name must be small enough to not dominate the line, and consistently cause lines to wrap. The commonly seen: base::allocated_type_profiler is too long, and uncessarily so. The shortened version: base::allocated_type maintains a meaningless qualifier, and discards critical information (that this is about profiling). When a reader see "allocated_type" the presumption is that both words are critical, and hence there is probably an "unallocated_type" as well somewhere. There is no such thing, and that it why the refinement to say "allocated" is pretty useless, but the refinement to say "*_profiler" is extremely valuable. A much better name is "type_profiler". The references to technical publications, which are not under pressure to be concise, is not significant readability in C++. Please change this throughout to "type_profile," as I have asked repeatedly. On 2012/08/24 02:47:27, Dai Mikurube wrote: > On 2012/08/23 19:55:25, jar wrote: > > nit: we don't "allocate types," so this namespace seems to drop the > interesting > > word "profiler" and keep the less significant word "allocated." Please > > consider: > > > > type_profiler > > Renamed it to allocated_type_profiler. > > As I said before, "allocated type" is a very common phrase to describe "a type > of an allocated object". So keeping "allocated" is reasonable. > > * http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3376.pdf (p.106) > * http://clang.llvm.org/doxygen/CGCXXABI_8h_source.html#l00219 > * http://www.almostinfinite.com/memtrack.html http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... File content/app/content_main_runner.cc (right): http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... content/app/content_main_runner.cc:455: base::allocated_type::SetNewInterceptFunction( It was not clear which of these should be set first, and it was not clear if it mattered. Using a single function allows you to encapsulate the ordering of the sets in your module. The individual functions should not be public, but you are free to use them as private methods to assist in your implementation. On 2012/08/24 02:47:27, Dai Mikurube wrote: > On 2012/08/23 19:55:25, jar wrote: > > nit: Do you ever set these separately? IF you always set them as a pair, why > not > > have one function: > > > > SetInterceptFunctions(...) > > Done. I kept individual functions since I often use them locally. http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:66: SpinLockHolder l(&g_allocated_type_lock); TCMalloc, and other third party code, is generally not complaint with Google readability, nor Chromium readability. Please try to comply with Chromium readability when writing chromium code. This file is not a branched version of any TCMalloc code, so there is no reason to deviate from our standards. On 2012/08/24 02:47:27, Dai Mikurube wrote: > On 2012/08/23 19:55:25, jar wrote: > > nit: don't use one letter variable names like "l". > > Done. (It's very common in tcmalloc...) http://codereview.chromium.org/10411047/diff/101003/base/allocator/allocated_... File base/allocator/allocated_type_profiler_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/101003/base/allocator/allocated_... base/allocator/allocated_type_profiler_tcmalloc.cc:22: } nit: curlies not needed here, and on line 31. http://codereview.chromium.org/10411047/diff/101003/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/101003/base/process_util_posix.c... base/process_util_posix.cc:646: base::allocated_type_profiler::StopProfilingAllocatedType(); nit: please shorten this method name to StopProfiling(); The addition wording is redundant relative to the namespace. Please be sure to provide tests that verify that the profiler works well with deliberate loss of new and/or delete intercepts.
Hi Jim, Thanks for your review. I just replied your comments, and will update soon. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:12: namespace allocated_type { On 2012/08/27 17:17:12, jar wrote: > I will remove the courtesy of calling this a nit, which I have used up to this > point. > > I don't believe it is a nit, as it is damaging readability. > > The namespace name must be small enough to not dominate the line, and > consistently cause lines to wrap. The commonly seen: > > base::allocated_type_profiler > > is too long, and uncessarily so. The shortened version: > > base::allocated_type > > maintains a meaningless qualifier, and discards critical information (that this > is about profiling). > > When a reader see "allocated_type" the presumption is that both words are > critical, and hence there is probably an "unallocated_type" as well somewhere. > There is no such thing, and that it why the refinement to say "allocated" is > pretty useless, but the refinement to say "*_profiler" is extremely valuable. > > A much better name is "type_profiler". > > The references to technical publications, which are not under pressure to be > concise, is not significant readability in C++. > > Please change this throughout to "type_profile," as I have asked repeatedly. > > On 2012/08/24 02:47:27, Dai Mikurube wrote: > > On 2012/08/23 19:55:25, jar wrote: > > > nit: we don't "allocate types," so this namespace seems to drop the > > interesting > > > word "profiler" and keep the less significant word "allocated." Please > > > consider: > > > > > > type_profiler > > > > Renamed it to allocated_type_profiler. > > > > As I said before, "allocated type" is a very common phrase to describe "a type > > of an allocated object". So keeping "allocated" is reasonable. > > > > * http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3376.pdf (p.106) > > * http://clang.llvm.org/doxygen/CGCXXABI_8h_source.html#l00219 > > * http://www.almostinfinite.com/memtrack.html > Ok, I'll rename the namespace to type_profiler. Is that enough? http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... File content/app/content_main_runner.cc (right): http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... content/app/content_main_runner.cc:455: base::allocated_type::SetNewInterceptFunction( On 2012/08/27 17:17:12, jar wrote: > It was not clear which of these should be set first, and it was not clear if it > mattered. Using a single function allows you to encapsulate the ordering of the > sets in your module. > > The individual functions should not be public, but you are free to use them as > private methods to assist in your implementation. > > On 2012/08/24 02:47:27, Dai Mikurube wrote: > > On 2012/08/23 19:55:25, jar wrote: > > > nit: Do you ever set these separately? IF you always set them as a pair, why > > not > > > have one function: > > > > > > SetInterceptFunctions(...) > > > > Done. I kept individual functions since I often use them locally. > Ok, I'll remove individual functions. When I need individual functions locally, I add them locally. http://codereview.chromium.org/10411047/diff/101003/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/101003/base/process_util_posix.c... base/process_util_posix.cc:646: base::allocated_type_profiler::StopProfilingAllocatedType(); On 2012/08/27 17:17:13, jar wrote: > nit: please shorten this method name to StopProfiling(); > > The addition wording is redundant relative to the namespace. > > Please be sure to provide tests that verify that the profiler works well with > deliberate loss of new and/or delete intercepts. Ok, I'll rename it. What kind of tests do you want? It would be a manual test, or a #if test in unit_tests.
Hi Jim, I fixed the patch as you suggested. And, added tests for allocated_type_profiler and third_party/tcmalloc/chromium/src/allocated_type_map.cc. These tests are independent from other tests and executed manually. The reason is described in the source files. http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... File base/allocator/allocated_type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/98002/base/allocator/allocated_t... base/allocator/allocated_type_profiler_control.cc:12: namespace allocated_type { On 2012/08/27 17:17:12, jar wrote: > I will remove the courtesy of calling this a nit, which I have used up to this > point. > > I don't believe it is a nit, as it is damaging readability. > > The namespace name must be small enough to not dominate the line, and > consistently cause lines to wrap. The commonly seen: > > base::allocated_type_profiler > > is too long, and uncessarily so. The shortened version: > > base::allocated_type > > maintains a meaningless qualifier, and discards critical information (that this > is about profiling). > > When a reader see "allocated_type" the presumption is that both words are > critical, and hence there is probably an "unallocated_type" as well somewhere. > There is no such thing, and that it why the refinement to say "allocated" is > pretty useless, but the refinement to say "*_profiler" is extremely valuable. > > A much better name is "type_profiler". > > The references to technical publications, which are not under pressure to be > concise, is not significant readability in C++. > > Please change this throughout to "type_profile," as I have asked repeatedly. > > On 2012/08/24 02:47:27, Dai Mikurube wrote: > > On 2012/08/23 19:55:25, jar wrote: > > > nit: we don't "allocate types," so this namespace seems to drop the > > interesting > > > word "profiler" and keep the less significant word "allocated." Please > > > consider: > > > > > > type_profiler > > > > Renamed it to allocated_type_profiler. > > > > As I said before, "allocated type" is a very common phrase to describe "a type > > of an allocated object". So keeping "allocated" is reasonable. > > > > * http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3376.pdf (p.106) > > * http://clang.llvm.org/doxygen/CGCXXABI_8h_source.html#l00219 > > * http://www.almostinfinite.com/memtrack.html > Done. http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... File content/app/content_main_runner.cc (right): http://codereview.chromium.org/10411047/diff/98002/content/app/content_main_r... content/app/content_main_runner.cc:455: base::allocated_type::SetNewInterceptFunction( On 2012/08/27 17:17:12, jar wrote: > It was not clear which of these should be set first, and it was not clear if it > mattered. Using a single function allows you to encapsulate the ordering of the > sets in your module. > > The individual functions should not be public, but you are free to use them as > private methods to assist in your implementation. > > On 2012/08/24 02:47:27, Dai Mikurube wrote: > > On 2012/08/23 19:55:25, jar wrote: > > > nit: Do you ever set these separately? IF you always set them as a pair, why > > not > > > have one function: > > > > > > SetInterceptFunctions(...) > > > > Done. I kept individual functions since I often use them locally. > Done. http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... File third_party/tcmalloc/chromium/src/allocated_type_map.cc (right): http://codereview.chromium.org/10411047/diff/98002/third_party/tcmalloc/chrom... third_party/tcmalloc/chromium/src/allocated_type_map.cc:66: SpinLockHolder l(&g_allocated_type_lock); On 2012/08/27 17:17:12, jar wrote: > TCMalloc, and other third party code, is generally not complaint with Google > readability, nor Chromium readability. > > Please try to comply with Chromium readability when writing chromium code. This > file is not a branched version of any TCMalloc code, so there is no reason to > deviate from our standards. > > On 2012/08/24 02:47:27, Dai Mikurube wrote: > > On 2012/08/23 19:55:25, jar wrote: > > > nit: don't use one letter variable names like "l". > > > > Done. (It's very common in tcmalloc...) > Ok. http://codereview.chromium.org/10411047/diff/101003/base/allocator/allocated_... File base/allocator/allocated_type_profiler_tcmalloc.cc (right): http://codereview.chromium.org/10411047/diff/101003/base/allocator/allocated_... base/allocator/allocated_type_profiler_tcmalloc.cc:22: } On 2012/08/27 17:17:13, jar wrote: > nit: curlies not needed here, and on line 31. Done. http://codereview.chromium.org/10411047/diff/101003/base/process_util_posix.cc File base/process_util_posix.cc (right): http://codereview.chromium.org/10411047/diff/101003/base/process_util_posix.c... base/process_util_posix.cc:646: base::allocated_type_profiler::StopProfilingAllocatedType(); On 2012/08/27 17:17:13, jar wrote: > nit: please shorten this method name to StopProfiling(); > > The addition wording is redundant relative to the namespace. > > Please be sure to provide tests that verify that the profiler works well with > deliberate loss of new and/or delete intercepts. Done.
I just started to review your newest file addition (unittests). It appears I was unclear in my previous reviews, where I tried to stress the following: Please stop using the phrase AllocatedType. The prefix "Allocated" is wastefully verbose, and adds no value (other than text) to the phrase, in pretty much every context you have chosen to use it. It makes it harder to read your text in all cases, and at best, adds confusion. I could go through and find the extensive list, but below are two such examples (in your newest addition of code). http://codereview.chromium.org/10411047/diff/113001/base/allocator/allocated_... File base/allocator/allocated_type_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/113001/base/allocator/allocated_... base/allocator/allocated_type_map_unittests.cc:1: // Copyright (c) 2012 The Chromium Authors. All rights reserved. Please change the names of all these files that have "allocator" in their path to not have the word "allocated" in their file name. For instance, this file name should be: base/allocator/type_profiler_unittests.cc All files of the form: base/allocator/allocated_type_profiler*.* should be: base/allocator/type_profiler_*.* http://codereview.chromium.org/10411047/diff/113001/base/allocator/allocated_... base/allocator/allocated_type_map_unittests.cc:19: TEST(AllocatedTypeMapTest, NormalOperation) { Please avoid the use of the phrase "AllocatedType." I have tried to stress that there are no "UnallocatedType" instances, so the phrase "AllocatedType" should generally be avoided. In this case, there is similarly no test focusing on TypeMap and how it is Allocated, so it is IMO similarly confusing to a reader to use it here. Please change these tests group name to: TypeProfilerTest here and on line 41.
Hi Jim, Thanks for your comments. I removed "allocated" from all filenames, identifiers and macros. What do you think?
http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler.cc:22: return g_new_intercept != NULL || g_delete_intercept != NULL; Note that this is not very thread safe, relative to the setting of the intercept functions. The values were not atomically set, and on platforms such as ARM, we're run the risk of getting a corrupted pointer if we depended on this check to decide if we want to call the intercept. Can you guarantee that SetInterceptFunctions() was called while we were single threaded? At a minimum, there should be a comment in the header that the setter should ONLY be called while single threaded. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler.cc:28: void* __op_new_intercept__(void* ptr, can we say: void* ::__op_new_intercept(...) { ... } and put that inside your namespace block (between lines 10 and 25? http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_control.cc:7: #if defined(PROFILING_TYPE) Ifdefs make the code hard to read. You only need one ifdef block in this file. Use an ifdef on only this declaration, and set it to false if the macro value suggests that. Don't use an ifdef to change StopProfiling (since g_enable_intercept will never be true!). Don't use an ifdef in IsProfiling(), since the global will indeed have the correct value. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_control.cc:8: static int g_enable_intercept = 1; nit: suggest using bool. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:15: #if defined(PROFILING_TYPE) Please change this macro to: TYPE_PROFILING As currently written, it confusingly sounds like there is a special "TYPE" which is the PROFILING_TYPE. In this context it could be deduced what you are trying to say, but in other contexts (e.g., what is this macro define used for?) it is harder to understand. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:16: static const void* const g_const_null = static_cast<const void*>(NULL); Is this cast needed? http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:43: int* dummy = new int(48); use a scoped ref ptr, so you don't have to worry about delete (and early exits). Same in line 57. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:54: TEST(TypeProfilerMapTest, InsertThenMultipleErase) { Please also test multiple inserts of a single address, with varying sizes, without corresponding deletes. This is the sort of corruption that will result from dropping intermediary delete() calls (due to asynchronous toggling of enables and disables, and setting of intercepts), and is what JamesR and I were concerned about. You can probably achieve this result (with genuine allocated data) by making several consecutive large allocations (over 32K). TCMalloc will (usually) reuse the same blocks immediately after they are freed (as spans). IF your repeat allocation differ by less than one page, they *should* reuse the same areas of memory (assuming you do a delete between). You could do teh same thing with "small" allocations, but you'd need to be careful to have two classes that were close enough in sizes that they both were part of the same "TCMalloc class size." Using large allocations (over 32K currently in TCMalloc) makes this easy. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:25: #define DUMMY_ARRAY_SIZE 10 nit: In C++, we prefer using of static const size_t. http://codereview.chromium.org/10411047/diff/115005/content/app/content_main_... File content/app/content_main_runner.cc (right): http://codereview.chromium.org/10411047/diff/115005/content/app/content_main_... content/app/content_main_runner.cc:446: // NOTE(willchan): One might ask why this call is done here rather than in Please either edit this comment so that "this" is elaborated, or move it next to the function call that "this" is refering to. http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h (right): http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h:12: void TypeProfilerInsertType(void* address, Why isn't this in a namespace? It seems like you're building up a complex name to incorporate something akin to a namespace. Do we have to be at global scope? Can we use the namespace you've established for this code (and show commonality)? http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:76: new(TypeProfilerMalloc(sizeof(AddressMap<ObjectInfo>))) better is: sizeof(*g_type_profiler_map) http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:81: unsigned int size, nit: indent http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:94: g_type_profiler_map->FindAndRemove(address, &obj); Is this the only users of FindAndRemove? Why not stick with std::map style interfaces, such as finding, and then deleting? This code, though surely working, was a bit jarring to see the declaration of an object during an attempt at an erase.
Thank you, Jim. Updated the patch. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler.cc:22: return g_new_intercept != NULL || g_delete_intercept != NULL; On 2012/08/30 16:02:10, jar wrote: > Note that this is not very thread safe, relative to the setting of the intercept > functions. The values were not atomically set, and on platforms such as ARM, > we're run the risk of getting a corrupted pointer if we depended on this check > to decide if we want to call the intercept. > > Can you guarantee that SetInterceptFunctions() was called while we were single > threaded? At a minimum, there should be a comment in the header that the setter > should ONLY be called while single threaded. Yeah, I know... In fact, it is not called in the main code, but required only in type_profiler_unittests. Added a comment on *THIS* function not to call SetInterceptFunctions() if this function is used. What do you think about it? http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler.cc:28: void* __op_new_intercept__(void* ptr, On 2012/08/30 16:02:10, jar wrote: > can we say: > > void* ::__op_new_intercept(...) { > ... > } > > and put that inside your namespace block (between lines 10 and 25? It is impossible. Clang++ says : ../../base/allocator/type_profiler.cc:25:9: error: definition or redeclaration of '__op_new_intercept__' cannot name the global scope../../base/allocator/type_profiler.cc:25:9: error: definition or redeclaration of '__op_new_intercept__' cannot name the global scope void* ::__op_new_intercept__(void* ptr, ~~^ ../../base/allocator/type_profiler.cc:36:9: error: definition or redeclaration of '__op_delete_intercept__' cannot name the global scope void* ::__op_delete_intercept__(void* ptr, ~~^ You may be correct in terms of c++ standards (I don't know), but we should submit clang-available code. We have the linux_clang bot. fyi, similar error even if we put prototype declarations to the namespace in type_profiler.h. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_control.cc:7: #if defined(PROFILING_TYPE) On 2012/08/30 16:02:10, jar wrote: > Ifdefs make the code hard to read. You only need one ifdef block in this file. > > Use an ifdef on only this declaration, and set it to false if the macro value > suggests that. > > Don't use an ifdef to change StopProfiling (since g_enable_intercept will never > be true!). > > > Don't use an ifdef in IsProfiling(), since the global will indeed have the > correct value. Done. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_control.cc:8: static int g_enable_intercept = 1; On 2012/08/30 16:02:10, jar wrote: > nit: suggest using bool. Done. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:15: #if defined(PROFILING_TYPE) On 2012/08/30 16:02:10, jar wrote: > Please change this macro to: > TYPE_PROFILING > > As currently written, it confusingly sounds like there is a special "TYPE" which > is the PROFILING_TYPE. In this context it could be deduced what you are trying > to say, but in other contexts (e.g., what is this macro define used for?) it is > harder to understand. Done. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:16: static const void* const g_const_null = static_cast<const void*>(NULL); On 2012/08/30 16:02:10, jar wrote: > Is this cast needed? Required. Using NULL directly, clang++ says: ../../testing/gtest/include/gtest/gtest.h:1532:28: error: comparison between pointer and integer ('const std::type_info *' and 'long') GTEST_IMPL_CMP_HELPER_(NE, !=); ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~ http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:43: int* dummy = new int(48); On 2012/08/30 16:02:10, jar wrote: > use a scoped ref ptr, so you don't have to worry about delete (and early exits). > Same in line 57. scoped_ptr? Done. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:54: TEST(TypeProfilerMapTest, InsertThenMultipleErase) { On 2012/08/30 16:02:10, jar wrote: > Please also test multiple inserts of a single address, with varying sizes, > without corresponding deletes. This is the sort of corruption that will result > from dropping intermediary delete() calls (due to asynchronous toggling of > enables and disables, and setting of intercepts), and is what JamesR and I were > concerned about. > > You can probably achieve this result (with genuine allocated data) by making > several consecutive large allocations (over 32K). TCMalloc will (usually) reuse > the same blocks immediately after they are freed (as spans). IF your repeat > allocation differ by less than one page, they *should* reuse the same areas of > memory (assuming you do a delete between). > > You could do teh same thing with "small" allocations, but you'd need to be > careful to have two classes that were close enough in sizes that they both were > part of the same "TCMalloc class size." Using large allocations (over 32K > currently in TCMalloc) makes this easy. Done for TypeProfilerMapTest for Insert/EraseType. A new test "MultipleInsertWithoutErase" does it. I think it is enough to test this issue. It is impossible to reproduce it with the intercepted new/delete in TypeProfilerTest. The type profiler's status can change only as follows: 1) Non-initialized (disabled) 2) => Initialized (enabled) 3) => Stopped (disabled) Once the profiler is stopped, it never gets re-enabled. The type profiler provides an initializer and a stopper. If new <1> is recorded (== called after initialized), and delete <1> is dropped (== called after stopped), next new <2> must not be recorded (it cannot be re-enabled). But, similar behavior has been already happening, and it doesn't fail. It is because of placement new. Our compiler modification catches also placement new. But, objects allocated by placement new are sometimes removed by calling the destructor and "operator delete" directly... not a 'delete' expression. It can cause double new for a single address. (We're giving up such cases.) I added a new test "MultipleNewWithDroppingDelete" in TypeProfilerTest for this case. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:25: #define DUMMY_ARRAY_SIZE 10 On 2012/08/30 16:02:10, jar wrote: > nit: In C++, we prefer using of static const size_t. Done. http://codereview.chromium.org/10411047/diff/115005/content/app/content_main_... File content/app/content_main_runner.cc (right): http://codereview.chromium.org/10411047/diff/115005/content/app/content_main_... content/app/content_main_runner.cc:446: // NOTE(willchan): One might ask why this call is done here rather than in On 2012/08/30 16:02:10, jar wrote: > Please either edit this comment so that "this" is elaborated, or move it next to > the function call that "this" is refering to. Changed it to : // ... One might ask why these TCMalloc-related calls are done // here rather than in process_util_linux.cc My understanding is that this comment is for calls in "#if !defined(OS_MACOSX) && defined(USE_TCMALLOC)". http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h (right): http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h:12: void TypeProfilerInsertType(void* address, On 2012/08/30 16:02:10, jar wrote: > Why isn't this in a namespace? It seems like you're building up a complex name > to incorporate something akin to a namespace. Do we have to be at global scope? > Can we use the namespace you've established for this code (and show > commonality)? I chose the same way with HeapProfiler to consider upstreaming this change. We (some Clang guys) were discussing upstreaming such type profile storage in tcmalloc when we succeed to upstream the Clang modification in future. But at this time, moving them into the namespace sounds reasonable. Done. http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:76: new(TypeProfilerMalloc(sizeof(AddressMap<ObjectInfo>))) On 2012/08/30 16:02:10, jar wrote: > better is: > sizeof(*g_type_profiler_map) Done. http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:81: unsigned int size, On 2012/08/30 16:02:10, jar wrote: > nit: indent Done. http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:94: g_type_profiler_map->FindAndRemove(address, &obj); On 2012/08/30 16:02:10, jar wrote: > Is this the only users of FindAndRemove? > Why not stick with std::map style interfaces, such as finding, and then > deleting? > > This code, though surely working, was a bit jarring to see the declaration of an > object during an attempt at an erase. heap-profile-table.cc uses it, and the only one removing function in AddressMap since very old days. Latest: https://code.google.com/p/gperftools/source/browse/trunk/src/heap-profile-tab... r99: https://code.google.com/p/gperftools/source/browse/trunk/src/heap-profile-tab...
Basic readability has gotten much better. Most of the comments are now starting to focus on more stubstantive issues and questions. Thanks! http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler.cc:22: return g_new_intercept != NULL || g_delete_intercept != NULL; The most common pattern for enforcing this is to have it as a private member of a class, and have that class friend your tests as needed. This avoids exposing it as a public API, and allows you to actually restrict the callers. On 2012/08/31 06:10:30, Dai Mikurube wrote: > On 2012/08/30 16:02:10, jar wrote: > > Note that this is not very thread safe, relative to the setting of the > intercept > > functions. The values were not atomically set, and on platforms such as ARM, > > we're run the risk of getting a corrupted pointer if we depended on this check > > to decide if we want to call the intercept. > > > > Can you guarantee that SetInterceptFunctions() was called while we were single > > threaded? At a minimum, there should be a comment in the header that the > setter > > should ONLY be called while single threaded. > > Yeah, I know... In fact, it is not called in the main code, but required only > in type_profiler_unittests. > > Added a comment on *THIS* function not to call SetInterceptFunctions() if this > function is used. What do you think about it? http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:54: TEST(TypeProfilerMapTest, InsertThenMultipleErase) { Good answer: (re: this CL has changed so that now you are no longer turning this on and off repeatedly). What does it mean when you say your are "giving up such cases?" On 2012/08/31 06:10:30, Dai Mikurube wrote: > On 2012/08/30 16:02:10, jar wrote: > > Please also test multiple inserts of a single address, with varying sizes, > > without corresponding deletes. This is the sort of corruption that will > result > > from dropping intermediary delete() calls (due to asynchronous toggling of > > enables and disables, and setting of intercepts), and is what JamesR and I > were > > concerned about. > > > > You can probably achieve this result (with genuine allocated data) by making > > several consecutive large allocations (over 32K). TCMalloc will (usually) > reuse > > the same blocks immediately after they are freed (as spans). IF your repeat > > allocation differ by less than one page, they *should* reuse the same areas of > > memory (assuming you do a delete between). > > > > You could do teh same thing with "small" allocations, but you'd need to be > > careful to have two classes that were close enough in sizes that they both > were > > part of the same "TCMalloc class size." Using large allocations (over 32K > > currently in TCMalloc) makes this easy. > > Done for TypeProfilerMapTest for Insert/EraseType. A new test > "MultipleInsertWithoutErase" does it. I think it is enough to test this issue. > > > It is impossible to reproduce it with the intercepted new/delete in > TypeProfilerTest. The type profiler's status can change only as follows: > 1) Non-initialized (disabled) > 2) => Initialized (enabled) > 3) => Stopped (disabled) > > Once the profiler is stopped, it never gets re-enabled. The type profiler > provides an initializer and a stopper. > > If new <1> is recorded (== called after initialized), and > delete <1> is dropped (== called after stopped), > next new <2> must not be recorded (it cannot be re-enabled). > > > But, similar behavior has been already happening, and it doesn't fail. It is > because of placement new. Our compiler modification catches also placement new. > But, objects allocated by placement new are sometimes removed by calling the > destructor and "operator delete" directly... not a 'delete' expression. It can > cause double new for a single address. (We're giving up such cases.) > > I added a new test "MultipleNewWithDroppingDelete" in TypeProfilerTest for this > case. http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:94: g_type_profiler_map->FindAndRemove(address, &obj); You'll notice in the examples you cited, that the return value of FindAndReplace() is checked, and then those sites use the returned |obj| in its adjustments to global tallies. That seems to be the justification for the API. If you're trying to follow that pattern, why aren't you maintaining tallies? Note that you even have a lock held at this call site, so you could maintain global tallies quite efficiently (and mirror the code you seem to be looking at). On 2012/08/31 06:10:30, Dai Mikurube wrote: > On 2012/08/30 16:02:10, jar wrote: > > Is this the only users of FindAndRemove? > > Why not stick with std::map style interfaces, such as finding, and then > > deleting? > > > > This code, though surely working, was a bit jarring to see the declaration of > an > > object during an attempt at an erase. > > heap-profile-table.cc uses it, and the only one removing function in AddressMap > since very old days. > > Latest: > https://code.google.com/p/gperftools/source/browse/trunk/src/heap-profile-tab... > r99: > https://code.google.com/p/gperftools/source/browse/trunk/src/heap-profile-tab... http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler.cc:15: void SetInterceptFunctions(InterceptFunction* new_intercept, You added a comment on IsInterceptFunctionAvailable(), when I asked you a question about the thread safety of SetInterceptFunction(), and whether it is guaranteed to only be called while Chrome is single threaded. When is this function called? The header file should state that it must not be called in a multi-threaded context. http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler.cc:22: // is called in a single-threaded status such as type_profiler_unittests. This comment should appear in the header, rather than the implementation. As noted, it would be better to have a private member function, which only friend (tests) can call. http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler.cc:33: base::type_profiler::InterceptFunction* new_intercept = As noted above, this is a racy assignment, relative to SetInterceptFunctions(). It is critical that this only be called after a SetInterceptFunctions() call has been made. Why are you pulling the value into a local, rather than accessing the global? Is this a performance optimization attempt, or were you hoping to resolve the race? ...or is there another reason? http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:25: #include <iostream> nit: include order http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:35: base::type_profiler::SetInterceptFunctions( Please put this entire test suite into the base::type_profiler namespace, so that you don't need to ever put this prefix on any calls, etc. in these tests. http://codereview.chromium.org/10411047/diff/127003/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/127003/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:24: struct ObjectInfo { Please surround this in an anonymous namespace, including all the local structures, and what you have as static data and methods. As it currently stands, ObjectInfo has global linkage appearing at global scope, and can interfere with other code all across Chrome. Best practice is to use an anonymous namespace for all the local stuff (and I don't mind seeing the "static" keyword as well... but that is personal preference, since it then has no impact on linkage once you're in an anonymous namespace.). The nice pattern here should be: // No code above this point typically (but some folks put // statics here.) namespace Whatever_namespace_most_code_lives_in { namespace { // All your global static stuff (can reference // Whatever_namespace_most_code_lives_in stuff). } // anonymous // All your real code for this CC file with external linkage. } // Whatever_namespace_most_code_lives_in
Thank you, Jim. Updated the patch. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler.cc:22: return g_new_intercept != NULL || g_delete_intercept != NULL; On 2012/09/02 01:37:11, jar wrote: > The most common pattern for enforcing this is to have it as a private member of > a class, and have that class friend your tests as needed. This avoids exposing > it as a public API, and allows you to actually restrict the callers. > > On 2012/08/31 06:10:30, Dai Mikurube wrote: > > On 2012/08/30 16:02:10, jar wrote: > > > Note that this is not very thread safe, relative to the setting of the > > intercept > > > functions. The values were not atomically set, and on platforms such as > ARM, > > > we're run the risk of getting a corrupted pointer if we depended on this > check > > > to decide if we want to call the intercept. > > > > > > Can you guarantee that SetInterceptFunctions() was called while we were > single > > > threaded? At a minimum, there should be a comment in the header that the > > setter > > > should ONLY be called while single threaded. > > > > Yeah, I know... In fact, it is not called in the main code, but required only > > in type_profiler_unittests. > > > > Added a comment on *THIS* function not to call SetInterceptFunctions() if this > > function is used. What do you think about it? > Done. http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... File base/allocator/type_profiler_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/115005/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:54: TEST(TypeProfilerMapTest, InsertThenMultipleErase) { On 2012/09/02 01:37:11, jar wrote: > Good answer: (re: this CL has changed so that now you are no longer turning this > on and off repeatedly). > > What does it mean when you say your are "giving up such cases?" > What I'd like to say was: we accept this kind of wrong data when "operator delete" is called directly unless it causes a significant runtime failure. > On 2012/08/31 06:10:30, Dai Mikurube wrote: > > On 2012/08/30 16:02:10, jar wrote: > > > Please also test multiple inserts of a single address, with varying sizes, > > > without corresponding deletes. This is the sort of corruption that will > > result > > > from dropping intermediary delete() calls (due to asynchronous toggling of > > > enables and disables, and setting of intercepts), and is what JamesR and I > > were > > > concerned about. > > > > > > You can probably achieve this result (with genuine allocated data) by making > > > several consecutive large allocations (over 32K). TCMalloc will (usually) > > reuse > > > the same blocks immediately after they are freed (as spans). IF your repeat > > > allocation differ by less than one page, they *should* reuse the same areas > of > > > memory (assuming you do a delete between). > > > > > > You could do teh same thing with "small" allocations, but you'd need to be > > > careful to have two classes that were close enough in sizes that they both > > were > > > part of the same "TCMalloc class size." Using large allocations (over 32K > > > currently in TCMalloc) makes this easy. > > > > Done for TypeProfilerMapTest for Insert/EraseType. A new test > > "MultipleInsertWithoutErase" does it. I think it is enough to test this > issue. > > > > > > It is impossible to reproduce it with the intercepted new/delete in > > TypeProfilerTest. The type profiler's status can change only as follows: > > 1) Non-initialized (disabled) > > 2) => Initialized (enabled) > > 3) => Stopped (disabled) > > > > Once the profiler is stopped, it never gets re-enabled. The type profiler > > provides an initializer and a stopper. > > > > If new <1> is recorded (== called after initialized), and > > delete <1> is dropped (== called after stopped), > > next new <2> must not be recorded (it cannot be re-enabled). > > > > > > But, similar behavior has been already happening, and it doesn't fail. It is > > because of placement new. Our compiler modification catches also placement > new. > > But, objects allocated by placement new are sometimes removed by calling the > > destructor and "operator delete" directly... not a 'delete' expression. It > can > > cause double new for a single address. (We're giving up such cases.) > > > > I added a new test "MultipleNewWithDroppingDelete" in TypeProfilerTest for > this > > case. > http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:94: g_type_profiler_map->FindAndRemove(address, &obj); On 2012/09/02 01:37:11, jar wrote: > You'll notice in the examples you cited, that the return value of > FindAndReplace() is checked, and then those sites use the returned |obj| in its > adjustments to global tallies. That seems to be the justification for the API. > > If you're trying to follow that pattern, why aren't you maintaining tallies? > > Note that you even have a lock held at this call site, so you could maintain > global tallies quite efficiently (and mirror the code you seem to be looking > at). > That is because we don't think accumulating 'deleted' objects is not so important in our type profiler. The purpose of the type profiler is simplified: to map addresses to types. Size accumulation is already done by heap-profiler. We use heap-profiler for such data like http://codereview.chromium.org/10830113/ does. > On 2012/08/31 06:10:30, Dai Mikurube wrote: > > On 2012/08/30 16:02:10, jar wrote: > > > Is this the only users of FindAndRemove? > > > Why not stick with std::map style interfaces, such as finding, and then > > > deleting? > > > > > > This code, though surely working, was a bit jarring to see the declaration > of > > an > > > object during an attempt at an erase. > > > > heap-profile-table.cc uses it, and the only one removing function in > AddressMap > > since very old days. > > > > Latest: > > > https://code.google.com/p/gperftools/source/browse/trunk/src/heap-profile-tab... > > r99: > > > https://code.google.com/p/gperftools/source/browse/trunk/src/heap-profile-tab... > http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler.cc:15: void SetInterceptFunctions(InterceptFunction* new_intercept, On 2012/09/02 01:37:11, jar wrote: > You added a comment on IsInterceptFunctionAvailable(), when I asked you a > question about the thread safety of SetInterceptFunction(), and whether it is > guaranteed to only be called while Chrome is single threaded. > > When is this function called? > > The header file should state that it must not be called in a multi-threaded > context. > ContentMainRunnerImpl::Initialize (content/app/content_main_runner.cc) is the only caller except for unit tests. My original thought was it should be called only ONCE in a process. I forgot to describe it. (Many people gave me suggestions around here. I was a little confused like http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... .) I added checks in this function to show it (a similar idea with the long-living base/allocator/allocator_extensions), and added comments on the header. It doesn't work perfectly for multi-thread context, but locking sounds too much for this (allocator_extensions, too). Making ContentMainRunnerImpl a 'friend' may be an additional idea... http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler.cc:22: // is called in a single-threaded status such as type_profiler_unittests. On 2012/09/02 01:37:11, jar wrote: > This comment should appear in the header, rather than the implementation. As > noted, it would be better to have a private member function, which only friend > (tests) can call. Done. http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler.cc:33: base::type_profiler::InterceptFunction* new_intercept = On 2012/09/02 01:37:11, jar wrote: > As noted above, this is a racy assignment, relative to SetInterceptFunctions(). > It is critical that this only be called after a SetInterceptFunctions() call has > been made. > > Why are you pulling the value into a local, rather than accessing the global? > Is this a performance optimization attempt, or were you hoping to resolve the > race? ...or is there another reason? It is because of maruel's suggestion... http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:25: #include <iostream> On 2012/09/02 01:37:11, jar wrote: > nit: include order Removed it. (I overlooked.) http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:35: base::type_profiler::SetInterceptFunctions( On 2012/09/02 01:37:11, jar wrote: > Please put this entire test suite into the base::type_profiler namespace, so > that you don't need to ever put this prefix on any calls, etc. in these tests. Done. http://codereview.chromium.org/10411047/diff/127003/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/127003/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:24: struct ObjectInfo { On 2012/09/02 01:37:11, jar wrote: > Please surround this in an anonymous namespace, including all the local > structures, and what you have as static data and methods. As it currently > stands, ObjectInfo has global linkage appearing at global scope, and can > interfere with other code all across Chrome. > > Best practice is to use an anonymous namespace for all the local stuff (and I > don't mind seeing the "static" keyword as well... but that is personal > preference, since it then has no impact on linkage once you're in an anonymous > namespace.). > > The nice pattern here should be: > > // No code above this point typically (but some folks put > // statics here.) > namespace Whatever_namespace_most_code_lives_in { > namespace { > // All your global static stuff (can reference > // Whatever_namespace_most_code_lives_in stuff). > } // anonymous > // All your real code for this CC file with external linkage. > } // Whatever_namespace_most_code_lives_in Done.
http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:29: if (base::allocated_type::g_new_intercept) { I don't think this is simpler, or safer. The loading of the local can be done piecemeal (especially on ARM), so loading it into a local does not help. The use of a local does not guarantee that the value will only be loaded once (i.e., there is not even a typically useless volatile semantic guarantee on the number of access to the global). IMO, if you want performance, you should set this value while you are single threaded, and NEVER change it. If you want the ability to asynchronously change it, you should use a lock. It is plausible to use atomics, but it is generally recomended against such use. If you note that the function you are about to call can grow and shrink a map, then you can also see that it will take time, and will sometimes use locks. Looking closer at your implementation, you use a lock all the time. As a result, the use of a lock here will not seemingly change the performance characteristics significantly, and will seem to provide correct code. On 2012/08/23 10:30:50, Dai Mikurube wrote: > On 2012/08/22 13:38:17, Marc-Antoine Ruel wrote: > > As I said before, it's still be simpler and safer to do: > > > > InterceptFunction* new_intercept = g_new_intercept; > > if (new_intercept) { > > return new_intercept(ptr, size, type); > > } > > Done. http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/127003/base/allocator/type_profi... base/allocator/type_profiler.cc:33: base::type_profiler::InterceptFunction* new_intercept = I put comments into the thread you suggested. My bottom line summary: You should be using locks here. On 2012/09/03 08:16:30, Dai Mikurube wrote: > On 2012/09/02 01:37:11, jar wrote: > > As noted above, this is a racy assignment, relative to > SetInterceptFunctions(). > > It is critical that this only be called after a SetInterceptFunctions() call > has > > been made. > > > > Why are you pulling the value into a local, rather than accessing the global? > > Is this a performance optimization attempt, or were you hoping to resolve the > > race? ...or is there another reason? > > It is because of maruel's suggestion... > > http://codereview.chromium.org/10411047/diff/71025/base/allocator/allocated_t... > http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... http://codereview.chromium.org/10411047/diff/134004/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/134004/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:56: DeleteInterceptForTCMalloc)); Note that you perform this at the start every single test. Please move it into the SetUp() code which is automatically executed before each test.
Thanks Jim. Updated. http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:29: if (base::allocated_type::g_new_intercept) { On 2012/09/04 17:40:38, jar wrote: > I don't think this is simpler, or safer. > > The loading of the local can be done piecemeal (especially on ARM), so loading > it into a local does not help. > > The use of a local does not guarantee that the value will only be loaded once > (i.e., there is not even a typically useless volatile semantic guarantee on the > number of access to the global). > > IMO, if you want performance, you should set this value while you are single > threaded, and NEVER change it. > > If you want the ability to asynchronously change it, you should use a lock. It > is plausible to use atomics, but it is generally recomended against such use. > If you note that the function you are about to call can grow and shrink a map, > then you can also see that it will take time, and will sometimes use locks. > Looking closer at your implementation, you use a lock all the time. As a > result, the use of a lock here will not seemingly change the performance > characteristics significantly, and will seem to provide correct code. > > On 2012/08/23 10:30:50, Dai Mikurube wrote: > > On 2012/08/22 13:38:17, Marc-Antoine Ruel wrote: > > > As I said before, it's still be simpler and safer to do: > > > > > > InterceptFunction* new_intercept = g_new_intercept; > > > if (new_intercept) { > > > return new_intercept(ptr, size, type); > > > } > > > > Done. > As written in comments in type_profiler.h, these values should be changed only ONCE. Ok, they should be set while single-threaded (modified the comment). Looking at the startup code (chrome/app/chrome_exe_main_*, chrome_main.cc, content/app/content_main.cc), ContentMainRunnerImpl::Initialize looks in single-thread. Is my understanding wrong? If wrong, could you tell me where is the good single-thread place to set them? Using locks is difficult for another reason (not performance reason). This file is linked into many executables. Some may not have any locking function (library). Some may have a locking function A. Another may have B. Only primitives should be used in this file. In addition, I removed any check (NULL-checks) in __op_{new|delete}_intercept__ by introducing NopIntercept. http://codereview.chromium.org/10411047/diff/134004/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/134004/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:56: DeleteInterceptForTCMalloc)); On 2012/09/04 17:40:38, jar wrote: > Note that you perform this at the start every single test. > > Please move it into the SetUp() code which is automatically executed before each > test. "TestProfileDeleteWithoutProfiledNew" has it at "non"-top. The order is important.
Added one comment just fyi. http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:29: if (base::allocated_type::g_new_intercept) { On 2012/09/05 04:53:23, Dai Mikurube wrote: > On 2012/09/04 17:40:38, jar wrote: > > I don't think this is simpler, or safer. > > > > The loading of the local can be done piecemeal (especially on ARM), so loading > > it into a local does not help. > > > > The use of a local does not guarantee that the value will only be loaded once > > (i.e., there is not even a typically useless volatile semantic guarantee on > the > > number of access to the global). > > > > IMO, if you want performance, you should set this value while you are single > > threaded, and NEVER change it. > > > > If you want the ability to asynchronously change it, you should use a lock. > It > > is plausible to use atomics, but it is generally recomended against such use. > > If you note that the function you are about to call can grow and shrink a map, > > then you can also see that it will take time, and will sometimes use locks. > > Looking closer at your implementation, you use a lock all the time. As a > > result, the use of a lock here will not seemingly change the performance > > characteristics significantly, and will seem to provide correct code. > > > > On 2012/08/23 10:30:50, Dai Mikurube wrote: > > > On 2012/08/22 13:38:17, Marc-Antoine Ruel wrote: > > > > As I said before, it's still be simpler and safer to do: > > > > > > > > InterceptFunction* new_intercept = g_new_intercept; > > > > if (new_intercept) { > > > > return new_intercept(ptr, size, type); > > > > } > > > > > > Done. > > > > As written in comments in type_profiler.h, these values should be changed only > ONCE. Ok, they should be set while single-threaded (modified the comment). > > Looking at the startup code (chrome/app/chrome_exe_main_*, chrome_main.cc, > content/app/content_main.cc), ContentMainRunnerImpl::Initialize looks in > single-thread. > > Is my understanding wrong? If wrong, could you tell me where is the good > single-thread place to set them? > > > Using locks is difficult for another reason (not performance reason). This file > is linked into many executables. Some may not have any locking function > (library). Some may have a locking function A. Another may have B. Only > primitives should be used in this file. > > In addition, I removed any check (NULL-checks) in __op_{new|delete}_intercept__ > by introducing NopIntercept. Just fyi, crashing is the only problem to be considered here . A little broken data (address->type mapping) doesn't matter.
http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:29: if (base::allocated_type::g_new_intercept) { Crashing is indeed the problem we're concerned about. When you fetch a multi-byte item (g_delete_intercept) that is asynchronously modified without the protection of a lock (or an atomic load), you risk getting only the part of the pointer intact. Using that corrupt pointer to call a function will surely crash the code. You've been jumping back and forth between using complex code that is insufficient to protect against async access, and saying there is no async modification. The 48th rev of the CL seems to have removed the wasteful code, and promises to not do any async changing of the data. I don't understand your fear of using a lock. This binary has internal values that can only appear once, so they would be handled identically to a lock. This is not a case where code is inlined all over creation. Can you clarify what your concern is when you say "This file is linked in many files. Some may have locking function A. Another may have B." Can you cite instances where this happens in Chrome? If there was intent to asynchronously change (shut down?) the profiler, I think you should use a lock such as the spin lock you used in the other file, as it can rely on linker initialization. If you are really never modifying the data when multithreaded, then you don't need the confused gyrations. I think this revision 48 of this CL is consistent about not changing state (while multithreaded). Assuming it stays that way... this won't be a problem. On 2012/09/05 15:50:42, Dai Mikurube wrote: > On 2012/09/05 04:53:23, Dai Mikurube wrote: > > On 2012/09/04 17:40:38, jar wrote: > > > I don't think this is simpler, or safer. > > > > > > The loading of the local can be done piecemeal (especially on ARM), so > loading > > > it into a local does not help. > > > > > > The use of a local does not guarantee that the value will only be loaded > once > > > (i.e., there is not even a typically useless volatile semantic guarantee on > > the > > > number of access to the global). > > > > > > IMO, if you want performance, you should set this value while you are single > > > threaded, and NEVER change it. > > > > > > If you want the ability to asynchronously change it, you should use a lock. > > > It > > > is plausible to use atomics, but it is generally recomended against such > use. > > > If you note that the function you are about to call can grow and shrink a > map, > > > then you can also see that it will take time, and will sometimes use locks. > > > Looking closer at your implementation, you use a lock all the time. As a > > > result, the use of a lock here will not seemingly change the performance > > > characteristics significantly, and will seem to provide correct code. > > > > > > On 2012/08/23 10:30:50, Dai Mikurube wrote: > > > > On 2012/08/22 13:38:17, Marc-Antoine Ruel wrote: > > > > > As I said before, it's still be simpler and safer to do: > > > > > > > > > > InterceptFunction* new_intercept = g_new_intercept; > > > > > if (new_intercept) { > > > > > return new_intercept(ptr, size, type); > > > > > } > > > > > > > > Done. > > > > > > > As written in comments in type_profiler.h, these values should be changed only > > ONCE. Ok, they should be set while single-threaded (modified the comment). > > > > Looking at the startup code (chrome/app/chrome_exe_main_*, chrome_main.cc, > > content/app/content_main.cc), ContentMainRunnerImpl::Initialize looks in > > single-thread. > > > > Is my understanding wrong? If wrong, could you tell me where is the good > > single-thread place to set them? > > > > > > Using locks is difficult for another reason (not performance reason). This > file > > is linked into many executables. Some may not have any locking function > > (library). Some may have a locking function A. Another may have B. Only > > primitives should be used in this file. > > > > In addition, I removed any check (NULL-checks) in > __op_{new|delete}_intercept__ > > by introducing NopIntercept. > > Just fyi, crashing is the only problem to be considered here . A little broken > data (address->type mapping) doesn't matter. http://codereview.chromium.org/10411047/diff/127003/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/127003/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:24: struct ObjectInfo { Please use best practice, which is anonymous namespace, per previous review comment. I had previously said that the anonymous namespace was needed, and that static keyword was optional. I asked around, and since global static is deprecated in C++, contrary to my previous comment about "optional," the "static" keyword should be removed once you use the required anonymous namespace. On 2012/09/03 08:16:30, Dai Mikurube wrote: > On 2012/09/02 01:37:11, jar wrote: > > Please surround this in an anonymous namespace, including all the local > > structures, and what you have as static data and methods. As it currently > > stands, ObjectInfo has global linkage appearing at global scope, and can > > interfere with other code all across Chrome. > > > > Best practice is to use an anonymous namespace for all the local stuff (and I > > don't mind seeing the "static" keyword as well... but that is personal > > preference, since it then has no impact on linkage once you're in an anonymous > > namespace.). > > > > The nice pattern here should be: > > > > // No code above this point typically (but some folks put > > // statics here.) > > namespace Whatever_namespace_most_code_lives_in { > > namespace { > > // All your global static stuff (can reference > > // Whatever_namespace_most_code_lives_in stuff). > > } // anonymous > > // All your real code for this CC file with external linkage. > > } // Whatever_namespace_most_code_lives_in > > Done. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.cc:12: // These variables are not in the InterceptFunctios class so that: nit: InterceptFunctios --> InterceptFunctions http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.cc:14: // * They are invisible from any other functions. I don't understand why you didn't just make them private, and give friend function access to the __op* functions. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.cc:17: // since __op_new_intercept__ and __op_delete_intercept__ must be global. Global function can be friends. Nice example in http://en.wikipedia.org/wiki/Friend_function http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.cc:25: // DCHECKs are wanted for these checks, but base/logging is not acceptable. It is confusing to have code in base/allocator/*, and then claim you can't use code in base. The comment doesn't make sense to me. If you want to insist on not using code from base, call standard code, such as: #include <assert.h> assert(...) It automatically handles the NDEBUG tests etc. TCMalloc code often takes this a step further, defining their own raw assert, but that seems unnecessary here. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.h:18: void* NopIntercept(void* ptr, size_t size, const std::type_info& type); Why isn't this private to type_profiler.cc? Is it used by a test? Note that this probably shouldn't be used by external code, as it seems to have special significance to your implementation. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.h:43: void* __op_new_intercept__(void* ptr, If these are not called by any code (test code?), they shouldn't be part of a public interface (and shouldn't be advertised in this header). Are they called directly from tests? http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:14: // NOTE: Each test in TypeProfilerTest must be executed separately with using How is this done via the unit test harness? http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:17: // because type_profiler provides only an initializer and a stopper for safety. All your code below is single threaded. As a result, you could call private methods to reset the state (unset the intercept functions, and clear the map). That is a common approach taken in testing, rather than run completely separate runs of the test harness. The methods should be private, and labeled for testing only. Access can be granted via the standard FRIEND_TEST* macro. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:123: ::operator delete(dummy_char); Is it not well defined to new up an array, and call operator delete on the resulting pointer. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:130: int* dummy_int = new int[large_size / sizeof(int) - 1]; nit: sizeof(*dummy_int) For clarity, you might also have the same pattern on line 115 http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:141: delete dummy_int; delete[] dummy_int;
Thanks, Jim. Updated the patch. http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... File base/allocator/allocated_type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/80024/base/allocator/allocated_t... base/allocator/allocated_type_profiler.cc:29: if (base::allocated_type::g_new_intercept) { As chatted, I kept these functions as r48 is. On 2012/09/06 19:31:42, jar wrote: > Crashing is indeed the problem we're concerned about. When you fetch a > multi-byte item (g_delete_intercept) that is asynchronously modified without the > protection of a lock (or an atomic load), you risk getting only the part of the > pointer intact. Using that corrupt pointer to call a function will surely crash > the code. > > You've been jumping back and forth between using complex code that is > insufficient to protect against async access, and saying there is no async > modification. The 48th rev of the CL seems to have removed the wasteful code, > and promises to not do any async changing of the data. > > I don't understand your fear of using a lock. This binary has internal values > that can only appear once, so they would be handled identically to a lock. This > is not a case where code is inlined all over creation. Can you clarify what > your concern is when you say "This file is linked in many files. Some may have > locking function A. Another may have B." Can you cite instances where this > happens in Chrome? > > If there was intent to asynchronously change (shut down?) the profiler, I think > you should use a lock such as the spin lock you used in the other file, as it > can rely on linker initialization. > > If you are really never modifying the data when multithreaded, then you don't > need the confused gyrations. > > I think this revision 48 of this CL is consistent about not changing state > (while multithreaded). Assuming it stays that way... this won't be a problem. > > > > On 2012/09/05 15:50:42, Dai Mikurube wrote: > > On 2012/09/05 04:53:23, Dai Mikurube wrote: > > > On 2012/09/04 17:40:38, jar wrote: > > > > I don't think this is simpler, or safer. > > > > > > > > The loading of the local can be done piecemeal (especially on ARM), so > > loading > > > > it into a local does not help. > > > > > > > > The use of a local does not guarantee that the value will only be loaded > > once > > > > (i.e., there is not even a typically useless volatile semantic guarantee > on > > > the > > > > number of access to the global). > > > > > > > > IMO, if you want performance, you should set this value while you are > single > > > > threaded, and NEVER change it. > > > > > > > > If you want the ability to asynchronously change it, you should use a > lock. > > > > > It > > > > is plausible to use atomics, but it is generally recomended against such > > use. > > > > If you note that the function you are about to call can grow and shrink a > > map, > > > > then you can also see that it will take time, and will sometimes use > locks. > > > > Looking closer at your implementation, you use a lock all the time. As a > > > > result, the use of a lock here will not seemingly change the performance > > > > characteristics significantly, and will seem to provide correct code. > > > > > > > > On 2012/08/23 10:30:50, Dai Mikurube wrote: > > > > > On 2012/08/22 13:38:17, Marc-Antoine Ruel wrote: > > > > > > As I said before, it's still be simpler and safer to do: > > > > > > > > > > > > InterceptFunction* new_intercept = g_new_intercept; > > > > > > if (new_intercept) { > > > > > > return new_intercept(ptr, size, type); > > > > > > } > > > > > > > > > > Done. > > > > > > > > > > As written in comments in type_profiler.h, these values should be changed > only > > > ONCE. Ok, they should be set while single-threaded (modified the comment). > > > > > > Looking at the startup code (chrome/app/chrome_exe_main_*, chrome_main.cc, > > > content/app/content_main.cc), ContentMainRunnerImpl::Initialize looks in > > > single-thread. > > > > > > Is my understanding wrong? If wrong, could you tell me where is the good > > > single-thread place to set them? > > > > > > > > > Using locks is difficult for another reason (not performance reason). This > > file > > > is linked into many executables. Some may not have any locking function > > > (library). Some may have a locking function A. Another may have B. Only > > > primitives should be used in this file. > > > > > > In addition, I removed any check (NULL-checks) in > > __op_{new|delete}_intercept__ > > > by introducing NopIntercept. > > > > Just fyi, crashing is the only problem to be considered here . A little > broken > > data (address->type mapping) doesn't matter. > http://codereview.chromium.org/10411047/diff/127003/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/127003/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:24: struct ObjectInfo { On 2012/09/06 19:31:43, jar wrote: > Please use best practice, which is anonymous namespace, per previous review > comment. > > I had previously said that the anonymous namespace was needed, and that static > keyword was optional. I asked around, and since global static is deprecated in > C++, contrary to my previous comment about "optional," the "static" keyword > should be removed once you use the required anonymous namespace. > > > On 2012/09/03 08:16:30, Dai Mikurube wrote: > > On 2012/09/02 01:37:11, jar wrote: > > > Please surround this in an anonymous namespace, including all the local > > > structures, and what you have as static data and methods. As it currently > > > stands, ObjectInfo has global linkage appearing at global scope, and can > > > interfere with other code all across Chrome. > > > > > > Best practice is to use an anonymous namespace for all the local stuff (and > I > > > don't mind seeing the "static" keyword as well... but that is personal > > > preference, since it then has no impact on linkage once you're in an > anonymous > > > namespace.). > > > > > > The nice pattern here should be: > > > > > > // No code above this point typically (but some folks put > > > // statics here.) > > > namespace Whatever_namespace_most_code_lives_in { > > > namespace { > > > // All your global static stuff (can reference > > > // Whatever_namespace_most_code_lives_in stuff). > > > } // anonymous > > > // All your real code for this CC file with external linkage. > > > } // Whatever_namespace_most_code_lives_in > > > > Done. > Removed "static"s. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.cc:12: // These variables are not in the InterceptFunctios class so that: On 2012/09/06 19:31:43, jar wrote: > nit: InterceptFunctios --> InterceptFunctions Finally, removed the comment. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.cc:14: // * They are invisible from any other functions. On 2012/09/06 19:31:43, jar wrote: > I don't understand why you didn't just make them private, and give friend > function access to the __op* functions. Done. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.cc:17: // since __op_new_intercept__ and __op_delete_intercept__ must be global. On 2012/09/06 19:31:43, jar wrote: > Global function can be friends. > Nice example in http://en.wikipedia.org/wiki/Friend_function Done. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.cc:25: // DCHECKs are wanted for these checks, but base/logging is not acceptable. On 2012/09/06 19:31:43, jar wrote: > It is confusing to have code in base/allocator/*, and then claim you can't use > code in base. The comment doesn't make sense to me. > > If you want to insist on not using code from base, call standard code, such as: > > #include <assert.h> > > assert(...) > > It automatically handles the NDEBUG tests etc. > > TCMalloc code often takes this a step further, defining their own raw assert, > but that seems unnecessary here. Done. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.h:18: void* NopIntercept(void* ptr, size_t size, const std::type_info& type); On 2012/09/06 19:31:43, jar wrote: > Why isn't this private to type_profiler.cc? Is it used by a test? > > Note that this probably shouldn't be used by external code, as it seems to have > special significance to your implementation. Removed from type_profiler.h, and moved it to anonymous namespace. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.h:43: void* __op_new_intercept__(void* ptr, On 2012/09/06 19:31:43, jar wrote: > If these are not called by any code (test code?), they shouldn't be part of a > public interface (and shouldn't be advertised in this header). > > Are they called directly from tests? > They don't look "called" lexically in source files, but they are actually called by inserted code at every new/delete expression by our modified Clang compiler. (It's the essential point of the compiler extension.) These prototype declarations are required. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:14: // NOTE: Each test in TypeProfilerTest must be executed separately with using On 2012/09/06 19:31:43, jar wrote: > How is this done via the unit test harness? Removed this limitation. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:17: // because type_profiler provides only an initializer and a stopper for safety. On 2012/09/06 19:31:43, jar wrote: > All your code below is single threaded. As a result, you could call private > methods to reset the state (unset the intercept functions, and clear the map). > That is a common approach taken in testing, rather than run completely separate > runs of the test harness. > > The methods should be private, and labeled for testing only. Access can be > granted via the standard FRIEND_TEST* macro. Done. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:123: ::operator delete(dummy_char); On 2012/09/06 19:31:43, jar wrote: > Is it not well defined to new up an array, and call operator delete on the > resulting pointer. Thanks for the good catch. I overlooked it when I changed the implementation of this test a few times (e.g. array <=> non-array). Fixed to call "::operator delete[]". http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:130: int* dummy_int = new int[large_size / sizeof(int) - 1]; On 2012/09/06 19:31:43, jar wrote: > nit: sizeof(*dummy_int) > > > For clarity, you might also have the same pattern on line 115 Done. http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:141: delete dummy_int; On 2012/09/06 19:31:43, jar wrote: > delete[] dummy_int; Done.
http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.h:43: void* __op_new_intercept__(void* ptr, Does the compiler trigger on the *declaration* of these functions, and do something different? I would have expected it to trigger on the *definition* of the function, or the compiler flag, to try to call the function. Header declarations are generally *only* useful for other code we have. Even if the *declaration* is the trigger, there seems no reason to put it in in a header. Can you point me to documentation that says you must have this declaration in a header file (which is conceptually unrelated to the source file containing the definition)? On 2012/09/07 08:06:45, Dai Mikurube wrote: > On 2012/09/06 19:31:43, jar wrote: > > If these are not called by any code (test code?), they shouldn't be part of a > > public interface (and shouldn't be advertised in this header). > > > > Are they called directly from tests? > > > > They don't look "called" lexically in source files, but they are actually called > by inserted code at every new/delete expression by our modified Clang compiler. > (It's the essential point of the compiler extension.) > These prototype declarations are required. http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:13: // NOTE: This test must be run with an environment variable "HEAPPROFILE". Are these tests now being run as part of the try-bot, and build bot tests? http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:169: ResetInterceptFunctions(); I think you still want StopProfiling() here, as that is actual public functionality that you are testing. You need to probably do a ResetInterceptFunctions() at the end of each test, and hence it should be in your cleanup for the test class. Am I missing something?
Thanks, Jim. I replied for 2 of 3 comments, and will update soon for http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi.... http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.h:43: void* __op_new_intercept__(void* ptr, *Declaration" triggers. Calling "__op*" is inserted at the compiling phase naturally, not at the linking phase. *Definition" is usually in the other source files. The modified Clang is designed so that developers should declare these functions by themselves since : 1) We may be able to declare them automatically if the compiler flag is specified. But... 2) These declaration needs another header file (<typeinfo>) before declaration. 3) We found that it makes our Clang modification more complicated to control the order of "automatically" inserted <typeinfo> and declaration. It is design decision at the compiler side. If the modification gets more complicated, rolling up its base Clang gets difficult. It easily brings us some terrible "compiler-side bugs". We'd like to avoid it. When we upstream the Clang modification, it would be improved. There is no public document about that, but I'll add a description at /deps/third_party/llvm-allocated-type/README.chromium. What do you think about this? On 2012/09/10 16:33:33, jar wrote: > Does the compiler trigger on the *declaration* of these functions, and do > something different? I would have expected it to trigger on the *definition* of > the function, or the compiler flag, to try to call the function. Header > declarations are generally *only* useful for other code we have. > > Even if the *declaration* is the trigger, there seems no reason to put it in in > a header. > > Can you point me to documentation that says you must have this declaration in a > header file (which is conceptually unrelated to the source file containing the > definition)? > > > On 2012/09/07 08:06:45, Dai Mikurube wrote: > > On 2012/09/06 19:31:43, jar wrote: > > > If these are not called by any code (test code?), they shouldn't be part of > a > > > public interface (and shouldn't be advertised in this header). > > > > > > Are they called directly from tests? > > > > > > > They don't look "called" lexically in source files, but they are actually > called > > by inserted code at every new/delete expression by our modified Clang > compiler. > > (It's the essential point of the compiler extension.) > > These prototype declarations are required. > http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:13: // NOTE: This test must be run with an environment variable "HEAPPROFILE". On 2012/09/10 16:33:33, jar wrote: > Are these tests now being run as part of the try-bot, and build bot tests? I think these tests are good enough as manual tests like allocator_unittests. They're small, and the type profiler doesn't change so frequently http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:169: ResetInterceptFunctions(); On 2012/09/10 16:33:33, jar wrote: > I think you still want StopProfiling() here, as that is actual public > functionality that you are testing. > > You need to probably do a ResetInterceptFunctions() at the end of each test, and > hence it should be in your cleanup for the test class. > > Am I missing something? Ok, I'll prepare a "RetartProfiling()" in type_profiler_control.cc only for tests.
Thanks Jim. Succeeded to fixed the change for all your comments! http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/133026/base/allocator/type_profi... base/allocator/type_profiler.h:43: void* __op_new_intercept__(void* ptr, On 2012/09/11 00:57:58, Dai Mikurube wrote: > *Declaration" triggers. Calling "__op*" is inserted at the compiling phase > naturally, not at the linking phase. *Definition" is usually in the other > source files. > > The modified Clang is designed so that developers should declare these functions > by themselves since : > 1) We may be able to declare them automatically if the compiler flag is > specified. But... > 2) These declaration needs another header file (<typeinfo>) before declaration. > 3) We found that it makes our Clang modification more complicated to control the > order of "automatically" inserted <typeinfo> and declaration. > > It is design decision at the compiler side. If the modification gets more > complicated, rolling up its base Clang gets difficult. It easily brings us some > terrible "compiler-side bugs". We'd like to avoid it. > > When we upstream the Clang modification, it would be improved. > > There is no public document about that, but I'll add a description at > /deps/third_party/llvm-allocated-type/README.chromium. What do you think about > this? > > > On 2012/09/10 16:33:33, jar wrote: > > Does the compiler trigger on the *declaration* of these functions, and do > > something different? I would have expected it to trigger on the *definition* > of > > the function, or the compiler flag, to try to call the function. Header > > declarations are generally *only* useful for other code we have. > > > > Even if the *declaration* is the trigger, there seems no reason to put it in > in > > a header. > > > > Can you point me to documentation that says you must have this declaration in > a > > header file (which is conceptually unrelated to the source file containing the > > definition)? > > > > > > On 2012/09/07 08:06:45, Dai Mikurube wrote: > > > On 2012/09/06 19:31:43, jar wrote: > > > > If these are not called by any code (test code?), they shouldn't be part > of > > a > > > > public interface (and shouldn't be advertised in this header). > > > > > > > > Are they called directly from tests? > > > > > > > > > > They don't look "called" lexically in source files, but they are actually > > called > > > by inserted code at every new/delete expression by our modified Clang > > compiler. > > > (It's the essential point of the compiler extension.) > > > These prototype declarations are required. > > > Finally, we could modify Clang to declare required prototypes automatically. It allowed us to stop including type_profiler.h in all source files by the "-include" option ("-include" is removed from gyp/gypi). But, I found that these declarations are still required in type_profiler.h to make them friends of InterceptFunctions (as you suggested). To declare "friend ...", prototype declaration is required before "friend ...". I added a comment to describe this. http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:13: // NOTE: This test must be run with an environment variable "HEAPPROFILE". On 2012/09/11 00:57:58, Dai Mikurube wrote: > On 2012/09/10 16:33:33, jar wrote: > > Are these tests now being run as part of the try-bot, and build bot tests? > > I think these tests are good enough as manual tests like allocator_unittests. > They're small, and the type profiler doesn't change so frequently Finally, I found that this condition is not required. Removed this comment. It was required since we check IsHeapProfilerRunning() in type_profiler_tcmalloc.cc, but it is now not mandatory (previously requried). I removed these checks. http://codereview.chromium.org/10411047/diff/128007/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:169: ResetInterceptFunctions(); On 2012/09/11 00:57:58, Dai Mikurube wrote: > On 2012/09/10 16:33:33, jar wrote: > > I think you still want StopProfiling() here, as that is actual public > > functionality that you are testing. > > > > You need to probably do a ResetInterceptFunctions() at the end of each test, > and > > hence it should be in your cleanup for the test class. > > > > Am I missing something? > > Ok, I'll prepare a "RetartProfiling()" in type_profiler_control.cc only for > tests. Done.
Ryan, Can you also comment on this, since you looked at it earlier to help with the gyp issues. Thanks, Jim http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler.h:43: friend void* ::__op_delete_intercept__(void*, size_t, const std::type_info&); Why does this require a pre-declaration of the functions? Is the code snippet in http://en.wikipedia.org/wiki/Friend_function incorrect? It does not pre-declare friends. Do we have a compiler option enabled that requires a predeclaration? http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... File base/allocator/type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_control.cc:23: return enable_intercept_ != 0; Don't compare a bool to zero. Just use the bool. http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_control.cc:28: enable_intercept_ = true; The creation of this function will make it harder for the compiler to realize this is all dead code when the TYPE_PROFILING macro is undefined. Add the following to clarify: namespace { Instead of having the ifdef around line 10, please use: #if defined(TYPE_PROFILING) const bool kTypeProfilingEnabled = true; #else const bool kTypeProfilingEnabled = false; #endif } // anonymous namespace. The above should be place around line 9. You can then avoid the ifdef on line 10, with: bool ProfilingController::enable_intercept_ = kTypeProfilingEnabled ; You can hten modify IsProfiling to be: kTypeProfilingEnabled && enable_intercept_; http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... File base/allocator/type_profiler_control.h (right): http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_control.h:11: class ProfilingController { Change name to Controller. It was bad enough that the "type_profiler" had a function called "StopProfiling" rather than merely "Stop," but now we have the word "Profile" three times in: base::typeprofiler::ProfilingController::StopProfiling() You should also change: StopProfiling() to Stop() RestartProfiling() to Restart() http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_control.h:20: // is not allowed to simplify the profiler (except for TypeProfilerTest). Typo in first sentence "Is must be used..." The second sentence is awkward, and ambiguous. For instance: Why is this "not allowed to simplify the profiler?" Why don't we allow it to simplify the profiler?? Please change to: The following is only allowed for use in unit tests. Profiling should never be restarted in regular use. http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:48: } The fact that you realized here that these names were too long really highlights that the calls should be: Controller::Stop() Controller::Restart() Controller::IsProfiling() You then wouldn't need these short-cut forwarding functions.
Thanks Jim. Fixed for all your comments (except for the forward declaration issue -- see the reply). And, could you take a look, Ryan? Thank you for your advice. :) http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler.h:43: friend void* ::__op_delete_intercept__(void*, size_t, const std::type_info&); On 2012/09/12 00:34:09, jar wrote: > Why does this require a pre-declaration of the functions? > > Is the code snippet in http://en.wikipedia.org/wiki/Friend_function > incorrect? It does not pre-declare friends. > > Do we have a compiler option enabled that requires a predeclaration? Clang without any compiler option says as follows for the Wikipedia example: ---------- friend.cc:12:19: error: no function named 'show' with type 'void (A &, B &)' was found in the specified scope friend void ::show(A& x, B& y); // declaration of global friend ^ friend.cc:22:20: error: no function named 'show' with type 'void (A &, B &)' was found in the specified scope friend void ::show(A& x, B& y); // declaration of global friend ^ friend.cc:38:26: error: 'a' is a private member of 'A' cout << "A::a = " << x.a << endl; ^ friend.cc:8:9: note: declared private here int a; ^ friend.cc:39:26: error: 'b' is a private member of 'B' cout << "B::b = " << y.b << endl; ^ friend.cc:18:9: note: declared private here int b; ^ 4 errors generated. ---------- It looks that gcc is just relaxing the spec. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3337.pdf 7.3.1.2p3 looks saying it should have forward declaration. (This reference is thanks to Nico.) In other words, the Wikipedia page may be wrong. In addition as far as I know, we, the Chromium project, use Clang as a more accurate C++ syntax checker than gcc. (The linux_clang bot does it.) http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... File base/allocator/type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_control.cc:23: return enable_intercept_ != 0; On 2012/09/12 00:34:09, jar wrote: > Don't compare a bool to zero. Just use the bool. Done. http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_control.cc:28: enable_intercept_ = true; On 2012/09/12 00:34:09, jar wrote: > The creation of this function will make it harder for the compiler to realize > this is all dead code when the TYPE_PROFILING macro is undefined. > > Add the following to clarify: > > namespace { > Instead of having the ifdef around line 10, please use: > #if defined(TYPE_PROFILING) > const bool kTypeProfilingEnabled = true; > #else > const bool kTypeProfilingEnabled = false; > #endif > } // anonymous namespace. > > The above should be place around line 9. > > You can then avoid the ifdef on line 10, with: > bool ProfilingController::enable_intercept_ = kTypeProfilingEnabled ; > > You can hten modify IsProfiling to be: > > kTypeProfilingEnabled && enable_intercept_; Done. http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... File base/allocator/type_profiler_control.h (right): http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_control.h:11: class ProfilingController { On 2012/09/12 00:34:09, jar wrote: > Change name to Controller. > > It was bad enough that the "type_profiler" had a function called "StopProfiling" > rather than merely "Stop," but now we have the word "Profile" three times in: > > base::typeprofiler::ProfilingController::StopProfiling() > > You should also change: > StopProfiling() to Stop() > RestartProfiling() to Restart() Done. http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_control.h:20: // is not allowed to simplify the profiler (except for TypeProfilerTest). On 2012/09/12 00:34:09, jar wrote: > Typo in first sentence "Is must be used..." > > The second sentence is awkward, and ambiguous. For instance: Why is this "not > allowed to simplify the profiler?" Why don't we allow it to simplify the > profiler?? > > Please change to: > The following is only allowed for use in unit tests. Profiling should never be > restarted in regular use. Done. http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/140027/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:48: } On 2012/09/12 00:34:09, jar wrote: > The fact that you realized here that these names were too long really highlights > that the calls should be: > > Controller::Stop() > Controller::Restart() > Controller::IsProfiling() > > You then wouldn't need these short-cut forwarding functions. Renamed them.
I did not examine the files under tools/ or third_party/. I assume Jim has reviewed them already. I will follow up with Jim in person to make sure we're on the same page here. I'm still concerned that the compiler is not providing the definitions for the 'magic' functions automatically. The fact that it's necessary to force such definitions on callers at all suggests to me that the Clang plugin may not be properly encapsulating it's interposing details. From a cursory examination of the Clang code (and I'll readily defer to Nico and glider here on the correctness of the statement), it would seem like you're simply making a naieve lookup of any (global? naming context dependent?) function called "__op_new_intercept__", and then arranging the call to directly place and arrange function arguments for that call. This is based on http://src.chromium.org/viewvc/chrome/trunk/deps/third_party/llvm-allocated-t... . What happens if a caller defines __op_new_intercept__ to take a different set of arguments? Will it cause a compiler crash? A warning message to be emitted? A linker warning? Or will it perhaps corrupt the (application) stack due to mismatched calling convention/function arrangements? Also, a comment with respect to directly calling "::operator delete" leaves me concerned whether the plugin is correctly reporting what it intends to report. http://codereview.chromium.org/10411047/diff/142005/base/allocator/allocator.gyp File base/allocator/allocator.gyp (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/allocator.... base/allocator/allocator.gyp:573: 'targets': [ I think it would be helpful to have a comment above here explaining why the repeated pattern: 'dependencies!': [ 'type_profiler' ], 'cflags_cc!': [ '-fintercept-allocation-functions' ], exists (namely, that it's undoing what build/common.gypi injects into all targets) http://codereview.chromium.org/10411047/diff/142005/base/allocator/allocator.... base/allocator/allocator.gyp:587: '../..', I don't believe any of the include_dirs are necessary/correct for this target, since its purpose is to act as a shim between the layers. http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.cc:41: // They don't link base/ files. nit: wording // Don't use DCHECK, as this file is injected into targets // that do not and should not depend on base/base.gyp:base http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.h:26: I think I may have to disagree with Jim here, unfortunately, in that I think these two functions above are logically and functionally different than the remaining functions below. I still strongly believe that the declarations for __op_new_intercept__/__op_delete_intercept__ should be provided by the compiler. At that point, you would not/do not need to inject this header into all targets, and so the only dependency on 'type_profiler' would be 'allocator_extension_thunks' (eg: you would remove the global injection from build/common.gypi) http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.h:38: InterceptFunction* delete_intercept); It's unclear to me why this is a static class, given that it only has a single static function, and it's already within a namespace. If I were to wager a guess (without having read all of the review feedback with Jim), I suspect it's to accomodate the friend definitions from lines 41-43. A different way to accomodate this would be to structure the headers as such: type_profiler.h - Has a single function, SetFunctions(...) type_profiler_internal.h - Has a function ResetFunctions (for unit tests) - Has a function IsAvailable (for the implementation of _new/_delete_intercept__) type_profiler.cc - Actually implements all three functions. The static variables (eg: line 50, 51) are then kept entirely opaque/internal to that .cc file http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler_control.cc:32: enable_intercept_ = true; BUG? Shouldn't this be = kTypeProfilingEnabled instead? http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:113: // Call "::operator delete" directly to drop __op_delete_intercept__. This comment actually makes me nervous, from the implementation site, to understand how __op_new_intercept__ / _op__delete_intercept__ are injected by the Clang plugin. While not 'pretty', it's not uncommon to see direct calls to the function (eg: the default STL allocator for GCC is new_allocator, which directly wraps ::operator new and ::operator delete). This suggests that there may be some bug or under-representation of data going on.
http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.h:26: On 2012/09/12 19:13:34, Ryan Sleevi wrote: > I think I may have to disagree with Jim here, unfortunately, in that I think > these two functions above are logically and functionally different than the > remaining functions below. > > I still strongly believe that the declarations for > __op_new_intercept__/__op_delete_intercept__ should be provided by the compiler. > At that point, you would not/do not need to inject this header into all targets, > and so the only dependency on 'type_profiler' would be > 'allocator_extension_thunks' (eg: you would remove the global injection from > build/common.gypi) Have you checked in the latest version of your clang plugin? I didn't see the .gypi forcing a -include, but I also didn't see your (current) clang plugin providing a declaration for these functions. So I'm not sure where the names are coming from. If the checked in copy is the latest, then I fear it may be relying on Clang's inherited-from-GCC ability to implicitly assume the definitions of functions and then let the linker resolve them, which would be bad. http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.h:38: InterceptFunction* delete_intercept); On 2012/09/12 19:13:34, Ryan Sleevi wrote: > It's unclear to me why this is a static class, given that it only has a single > static function, and it's already within a namespace. > > If I were to wager a guess (without having read all of the review feedback with > Jim), I suspect it's to accomodate the friend definitions from lines 41-43. > > A different way to accomodate this would be to structure the headers as such: > > type_profiler.h > - Has a single function, SetFunctions(...) > type_profiler_internal.h > - Has a function ResetFunctions (for unit tests) > - Has a function IsAvailable (for the implementation of > _new/_delete_intercept__) > > type_profiler.cc > - Actually implements all three functions. The static variables (eg: line 50, > 51) are then kept entirely opaque/internal to that .cc file I spoke with Jim, and while we agree both solutions would work, let's keep this as is for now (eg: do not implement the separation). Despite the style guide discouraging classes-as-static-method-containers (as opposed to using namespaces), it does provide a useful check here, in that it *forces* the compiler to check the access rules, as opposed to _internal.h, which anybody "could" include. http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.h:51: static InterceptFunction* delete_intercept_; However, please remove all static member variables from these classes, as they're unnecessary. The 'leak' implementation details to everyone who includes the .h, and, because there are concrete friends defined, it may allow them to inadvertently modify these variables. Instead, move the definition and declaration of these variables to an unnamed namespace within the .cc file that backs this header, and have SetFunctions/ResetFunctions modify them. They should follow the global naming scheme (g_new_intercept, g_delete_intercept) Please do this for all static members of classes within this CL.
Thanks Ryan. At first, I replied to your comments that don't need fixes. http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.h:26: On 2012/09/12 22:04:18, Ryan Sleevi wrote: > On 2012/09/12 19:13:34, Ryan Sleevi wrote: > > I think I may have to disagree with Jim here, unfortunately, in that I think > > these two functions above are logically and functionally different than the > > remaining functions below. > > > > I still strongly believe that the declarations for > > __op_new_intercept__/__op_delete_intercept__ should be provided by the > compiler. > > At that point, you would not/do not need to inject this header into all > targets, > > and so the only dependency on 'type_profiler' would be > > 'allocator_extension_thunks' (eg: you would remove the global injection from > > build/common.gypi) > > Have you checked in the latest version of your clang plugin? > > I didn't see the .gypi forcing a -include, but I also didn't see your (current) > clang plugin providing a declaration for these functions. So I'm not sure where > the names are coming from. If the checked in copy is the latest, then I fear it > may be relying on Clang's inherited-from-GCC ability to implicitly assume the > definitions of functions and then let the linker resolve them, which would be > bad. The latest modified compiler has been checked-in. My extension is implemented by modifying Clang code. Declaration of these functions are now instrumented by the modified Clang binary. The compiler option is not provided for this file (now, it is not included automatically). So, these functions are not declared automatically in this file. http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.h:38: InterceptFunction* delete_intercept); On 2012/09/12 22:04:18, Ryan Sleevi wrote: > On 2012/09/12 19:13:34, Ryan Sleevi wrote: > > It's unclear to me why this is a static class, given that it only has a single > > static function, and it's already within a namespace. > > > > If I were to wager a guess (without having read all of the review feedback > with > > Jim), I suspect it's to accomodate the friend definitions from lines 41-43. > > > > A different way to accomodate this would be to structure the headers as such: > > > > type_profiler.h > > - Has a single function, SetFunctions(...) > > type_profiler_internal.h > > - Has a function ResetFunctions (for unit tests) > > - Has a function IsAvailable (for the implementation of > > _new/_delete_intercept__) > > > > type_profiler.cc > > - Actually implements all three functions. The static variables (eg: line > 50, > > 51) are then kept entirely opaque/internal to that .cc file > > I spoke with Jim, and while we agree both solutions would work, let's keep this > as is for now (eg: do not implement the separation). > > Despite the style guide discouraging classes-as-static-method-containers (as > opposed to using namespaces), it does provide a useful check here, in that it > *forces* the compiler to check the access rules, as opposed to _internal.h, > which anybody "could" include. Ok, I keep it as is. http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:113: // Call "::operator delete" directly to drop __op_delete_intercept__. On 2012/09/12 19:13:34, Ryan Sleevi wrote: > This comment actually makes me nervous, from the implementation site, to > understand how __op_new_intercept__ / _op__delete_intercept__ are injected by > the Clang plugin. > > While not 'pretty', it's not uncommon to see direct calls to the function (eg: > the default STL allocator for GCC is new_allocator, which directly wraps > ::operator new and ::operator delete). This suggests that there may be some bug > or under-representation of data going on. When ::operator delete is called directly, our extension neglects it. It sometimes happens, but such data error is intentionally accepted for now. At first, we find that it is difficult to collect perfectly accurate data. The purpose of this profiler is, for now, to understand "trend", not perfect numbers. Second, recorded mapping (address => type) will be used with malloc profiles (like http://codereview.chromium.org/10830113/). It means that we will look up "type" only for every malloc'ed blocks. If a malloc'ed block is already free'd, its type won't be counted. It provides a practically good data for us. We'll be improving it, of course. But, it will need more study with experiments.
http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:113: // Call "::operator delete" directly to drop __op_delete_intercept__. On 2012/09/13 01:12:40, Dai Mikurube wrote: > On 2012/09/12 19:13:34, Ryan Sleevi wrote: > > This comment actually makes me nervous, from the implementation site, to > > understand how __op_new_intercept__ / _op__delete_intercept__ are injected by > > the Clang plugin. > > > > While not 'pretty', it's not uncommon to see direct calls to the function (eg: > > the default STL allocator for GCC is new_allocator, which directly wraps > > ::operator new and ::operator delete). This suggests that there may be some > bug > > or under-representation of data going on. > > When ::operator delete is called directly, our extension neglects it. It > sometimes happens, but such data error is intentionally accepted for now. > > At first, we find that it is difficult to collect perfectly accurate data. The > purpose of this profiler is, for now, to understand "trend", not perfect > numbers. > > Second, recorded mapping (address => type) will be used with malloc profiles > (like http://codereview.chromium.org/10830113/). It means that we will look up > "type" only for every malloc'ed blocks. If a malloc'ed block is already free'd, > its type won't be counted. It provides a practically good data for us. > > We'll be improving it, of course. But, it will need more study with > experiments. fyi, we already know STL <allocator> is the largest missing part. Now, memory allocated via <allocator> is dropped. We need additional different approach for <allocator>. But actually, we also know STL memory is not so large in Chromium. When we find a case that STL memory occupies large memory, we'll tackle it to grow the coverage.
http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.h:26: On 2012/09/13 01:12:40, Dai Mikurube wrote: > On 2012/09/12 22:04:18, Ryan Sleevi wrote: > > On 2012/09/12 19:13:34, Ryan Sleevi wrote: > > > I think I may have to disagree with Jim here, unfortunately, in that I think > > > these two functions above are logically and functionally different than the > > > remaining functions below. > > > > > > I still strongly believe that the declarations for > > > __op_new_intercept__/__op_delete_intercept__ should be provided by the > > compiler. > > > At that point, you would not/do not need to inject this header into all > > targets, > > > and so the only dependency on 'type_profiler' would be > > > 'allocator_extension_thunks' (eg: you would remove the global injection from > > > build/common.gypi) > > > > Have you checked in the latest version of your clang plugin? > > > > I didn't see the .gypi forcing a -include, but I also didn't see your > (current) > > clang plugin providing a declaration for these functions. So I'm not sure > where > > the names are coming from. If the checked in copy is the latest, then I fear > it > > may be relying on Clang's inherited-from-GCC ability to implicitly assume the > > definitions of functions and then let the linker resolve them, which would be > > bad. > > The latest modified compiler has been checked-in. My extension is implemented > by modifying Clang code. Declaration of these functions are now instrumented by > the modified Clang binary. > > The compiler option is not provided for this file (now, it is not included > automatically). So, these functions are not declared automatically in this > file. Have you not updated http://src.chromium.org/viewvc/chrome/trunk/deps/third_party/llvm-allocated-t... then? The last update to this file was before we requested you to implement the declarations, and the current implementation is still reliant on the identifier table having a declaration existing in the translation unit: + IdentifierInfo *NewInterceptInfo = + &PP.getIdentifierTable().get("__op_new_intercept__"); + LookupResult R(*this, NewInterceptInfo, SourceLocation(), + LookupOrdinaryName); + LookupQualifiedName(R, Context.getTranslationUnitDecl());
On 2012/09/13 01:34:59, Ryan Sleevi wrote: > http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... > File base/allocator/type_profiler.h (right): > > http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... > base/allocator/type_profiler.h:26: > On 2012/09/13 01:12:40, Dai Mikurube wrote: > > On 2012/09/12 22:04:18, Ryan Sleevi wrote: > > > On 2012/09/12 19:13:34, Ryan Sleevi wrote: > > > > I think I may have to disagree with Jim here, unfortunately, in that I > think > > > > these two functions above are logically and functionally different than > the > > > > remaining functions below. > > > > > > > > I still strongly believe that the declarations for > > > > __op_new_intercept__/__op_delete_intercept__ should be provided by the > > > compiler. > > > > At that point, you would not/do not need to inject this header into all > > > targets, > > > > and so the only dependency on 'type_profiler' would be > > > > 'allocator_extension_thunks' (eg: you would remove the global injection > from > > > > build/common.gypi) > > > > > > Have you checked in the latest version of your clang plugin? > > > > > > I didn't see the .gypi forcing a -include, but I also didn't see your > > (current) > > > clang plugin providing a declaration for these functions. So I'm not sure > > where > > > the names are coming from. If the checked in copy is the latest, then I fear > > it > > > may be relying on Clang's inherited-from-GCC ability to implicitly assume > the > > > definitions of functions and then let the linker resolve them, which would > be > > > bad. > > > > The latest modified compiler has been checked-in. My extension is implemented > > by modifying Clang code. Declaration of these functions are now instrumented > by > > the modified Clang binary. > > > > The compiler option is not provided for this file (now, it is not included > > automatically). So, these functions are not declared automatically in this > > file. > > Have you not updated > http://src.chromium.org/viewvc/chrome/trunk/deps/third_party/llvm-allocated-t... > then? > > The last update to this file was before we requested you to implement the > declarations, and the current implementation is still reliant on the identifier > table having a declaration existing in the translation unit: > > + IdentifierInfo *NewInterceptInfo = > + &PP.getIdentifierTable().get("__op_new_intercept__"); > + LookupResult R(*this, NewInterceptInfo, SourceLocation(), > + LookupOrdinaryName); > + LookupQualifiedName(R, Context.getTranslationUnitDecl()); hmm, strange. I checked-in at https://src.chromium.org/viewvc/chrome?view=rev&revision=155949
On 2012/09/13 02:02:39, Dai Mikurube wrote: > On 2012/09/13 01:34:59, Ryan Sleevi wrote: > > > http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... > > File base/allocator/type_profiler.h (right): > > > > > http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... > > base/allocator/type_profiler.h:26: > > On 2012/09/13 01:12:40, Dai Mikurube wrote: > > > On 2012/09/12 22:04:18, Ryan Sleevi wrote: > > > > On 2012/09/12 19:13:34, Ryan Sleevi wrote: > > > > > I think I may have to disagree with Jim here, unfortunately, in that I > > think > > > > > these two functions above are logically and functionally different than > > the > > > > > remaining functions below. > > > > > > > > > > I still strongly believe that the declarations for > > > > > __op_new_intercept__/__op_delete_intercept__ should be provided by the > > > > compiler. > > > > > At that point, you would not/do not need to inject this header into all > > > > targets, > > > > > and so the only dependency on 'type_profiler' would be > > > > > 'allocator_extension_thunks' (eg: you would remove the global injection > > from > > > > > build/common.gypi) > > > > > > > > Have you checked in the latest version of your clang plugin? > > > > > > > > I didn't see the .gypi forcing a -include, but I also didn't see your > > > (current) > > > > clang plugin providing a declaration for these functions. So I'm not sure > > > where > > > > the names are coming from. If the checked in copy is the latest, then I > fear > > > it > > > > may be relying on Clang's inherited-from-GCC ability to implicitly assume > > the > > > > definitions of functions and then let the linker resolve them, which would > > be > > > > bad. > > > > > > The latest modified compiler has been checked-in. My extension is > implemented > > > by modifying Clang code. Declaration of these functions are now > instrumented > > by > > > the modified Clang binary. > > > > > > The compiler option is not provided for this file (now, it is not included > > > automatically). So, these functions are not declared automatically in this > > > file. > > > > Have you not updated > > > http://src.chromium.org/viewvc/chrome/trunk/deps/third_party/llvm-allocated-t... > > then? > > > > The last update to this file was before we requested you to implement the > > declarations, and the current implementation is still reliant on the > identifier > > table having a declaration existing in the translation unit: > > > > + IdentifierInfo *NewInterceptInfo = > > + &PP.getIdentifierTable().get("__op_new_intercept__"); > > + LookupResult R(*this, NewInterceptInfo, SourceLocation(), > > + LookupOrdinaryName); > > + LookupQualifiedName(R, Context.getTranslationUnitDecl()); > > hmm, strange. I checked-in at > https://src.chromium.org/viewvc/chrome?view=rev&revision=155949 ah, sorry, i forgot to update the patch file. will update soon. (bin is updated)
On 2012/09/13 02:06:37, Dai Mikurube wrote: > On 2012/09/13 02:02:39, Dai Mikurube wrote: > > On 2012/09/13 01:34:59, Ryan Sleevi wrote: > > > > > > http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... > > > File base/allocator/type_profiler.h (right): > > > > > > > > > http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... > > > base/allocator/type_profiler.h:26: > > > On 2012/09/13 01:12:40, Dai Mikurube wrote: > > > > On 2012/09/12 22:04:18, Ryan Sleevi wrote: > > > > > On 2012/09/12 19:13:34, Ryan Sleevi wrote: > > > > > > I think I may have to disagree with Jim here, unfortunately, in that I > > > think > > > > > > these two functions above are logically and functionally different > than > > > the > > > > > > remaining functions below. > > > > > > > > > > > > I still strongly believe that the declarations for > > > > > > __op_new_intercept__/__op_delete_intercept__ should be provided by the > > > > > compiler. > > > > > > At that point, you would not/do not need to inject this header into > all > > > > > targets, > > > > > > and so the only dependency on 'type_profiler' would be > > > > > > 'allocator_extension_thunks' (eg: you would remove the global > injection > > > from > > > > > > build/common.gypi) > > > > > > > > > > Have you checked in the latest version of your clang plugin? > > > > > > > > > > I didn't see the .gypi forcing a -include, but I also didn't see your > > > > (current) > > > > > clang plugin providing a declaration for these functions. So I'm not > sure > > > > where > > > > > the names are coming from. If the checked in copy is the latest, then I > > fear > > > > it > > > > > may be relying on Clang's inherited-from-GCC ability to implicitly > assume > > > the > > > > > definitions of functions and then let the linker resolve them, which > would > > > be > > > > > bad. > > > > > > > > The latest modified compiler has been checked-in. My extension is > > implemented > > > > by modifying Clang code. Declaration of these functions are now > > instrumented > > > by > > > > the modified Clang binary. > > > > > > > > The compiler option is not provided for this file (now, it is not included > > > > automatically). So, these functions are not declared automatically in > this > > > > file. > > > > > > Have you not updated > > > > > > http://src.chromium.org/viewvc/chrome/trunk/deps/third_party/llvm-allocated-t... > > > then? > > > > > > The last update to this file was before we requested you to implement the > > > declarations, and the current implementation is still reliant on the > > identifier > > > table having a declaration existing in the translation unit: > > > > > > + IdentifierInfo *NewInterceptInfo = > > > + &PP.getIdentifierTable().get("__op_new_intercept__"); > > > + LookupResult R(*this, NewInterceptInfo, SourceLocation(), > > > + LookupOrdinaryName); > > > + LookupQualifiedName(R, Context.getTranslationUnitDecl()); > > > > hmm, strange. I checked-in at > > https://src.chromium.org/viewvc/chrome?view=rev&revision=155949 > > ah, sorry, i forgot to update the patch file. will update soon. (bin is updated) It's now committed. Will be available at src.chromium.org soon. (r156467)
Hi Ryan, Fixed the patch as you suggested. Thanks. http://codereview.chromium.org/10411047/diff/142005/base/allocator/allocator.gyp File base/allocator/allocator.gyp (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/allocator.... base/allocator/allocator.gyp:573: 'targets': [ On 2012/09/12 19:13:34, Ryan Sleevi wrote: > I think it would be helpful to have a comment above here explaining why the > repeated pattern: > > 'dependencies!': [ 'type_profiler' ], > 'cflags_cc!': [ '-fintercept-allocation-functions' ], > > exists (namely, that it's undoing what build/common.gypi injects into all > targets) Done. http://codereview.chromium.org/10411047/diff/142005/base/allocator/allocator.... base/allocator/allocator.gyp:587: '../..', On 2012/09/12 19:13:34, Ryan Sleevi wrote: > I don't believe any of the include_dirs are necessary/correct for this target, > since its purpose is to act as a shim between the layers. Good catch. Removed unnecessary dirs. http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.cc:41: // They don't link base/ files. On 2012/09/12 19:13:34, Ryan Sleevi wrote: > nit: wording > > // Don't use DCHECK, as this file is injected into targets > // that do not and should not depend on base/base.gyp:base Done. http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler.h:51: static InterceptFunction* delete_intercept_; On 2012/09/12 22:04:18, Ryan Sleevi wrote: > However, please remove all static member variables from these classes, as > they're unnecessary. > > The 'leak' implementation details to everyone who includes the .h, and, because > there are concrete friends defined, it may allow them to inadvertently modify > these variables. > > Instead, move the definition and declaration of these variables to an unnamed > namespace within the .cc file that backs this header, and have > SetFunctions/ResetFunctions modify them. > > They should follow the global naming scheme (g_new_intercept, > g_delete_intercept) > > Please do this for all static members of classes within this CL. Done. As a side effect, we could remove forward declarations and friend declaration for __op_*. http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... File base/allocator/type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/142005/base/allocator/type_profi... base/allocator/type_profiler_control.cc:32: enable_intercept_ = true; On 2012/09/12 19:13:34, Ryan Sleevi wrote: > BUG? Shouldn't this be = kTypeProfilingEnabled instead? Done.
It would seem like that your changes to tools/deep_memory are unnecessary at this time, or perhaps incorrect. Unless this code is actively hooked up (and it doesn't seem to be, since this is the first landing of it), I would remove those files from this CL, since they're untest(ed/able) http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.gyp File base/allocator/allocator.gyp (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.... base/allocator/allocator.gyp:609: '../..', Why the ../../ include (which is effectively <(DEPTH)) ? What's being included here that needs the Chromium scope? Seems like it can/should be removed http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.... base/allocator/allocator.gyp:615: '<(tcmalloc_dir)/src/gperftools/type_profiler_map.h', nit: sort http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.... base/allocator/allocator.gyp:657: '<(tcmalloc_dir)/src/gperftools/type_profiler_map.h', nit: sort http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler.cc:20: } // anonymous namespace For unnamed namespaces (nit-peeve: not anonymous, unnamed ;-) ), it's not necessary to include a trailing comment, so remove it. http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler.h:10: #include <cstddef> // for size_t Change this to <stddef.h> <cstddef> is only required to declare std::size_t, which you're not using here http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... File base/allocator/type_profiler_control.h (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler_control.h:20: FRIEND_TEST_ALL_PREFIXES(TypeProfilerTest, If you've friended TypeProfilerTest, you do not need to FRIEND_TEST_ALL_PREFIXES The two are (generally) mutually exclusive. If you friend a class (really, a test fixture), then you can access private/protected members. However, note that all unit tests will be subclasses of that fixture, ergo they will not inherit the friend-edness. To work around that, you have to use protected helper method *on the fixture* to allow individual tests to access. However, when you only have one or two tests, it's often easier and cleaner to directly friend the test, rather than go through all that. Since you only have a single test, remove the "friend class TypeProfilerTest", and the right thing will happen (only TestProfileNewWithoutProfiledDelete will be friended) http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... File base/allocator/type_profiler_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:10: #include <cstddef> stddef.h http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:30: EXPECT_EQ(type, g_const_null); Please re-order all of the test comparisons to match the correct form of EXPECT_EQ(expected, actual) https://code.google.com/p/googletest/wiki/Primer#Binary_Comparison Alternatively, you can just use the implicit bool conversions, which can avoid the GCC 4.6 issues related to NULL handling EXPECT_FALSE(type) ASSERT_TRUE(type) etc http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h (right): http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h:15: void InsertType(void* address, unsigned int size, const std::type_info& type); BUG: Your interposed __op_delete_intercept__ function takes its parameters as a size_t ( see http://codereview.chromium.org/10411047/diff/133036/base/allocator/type_profi... ) You're potentially downcasting here to an unsigned int here. On an LP64 system, you're converting a 64-bit size to a 32-bit size. While I doubt we'll be allocating more than 4 gigs in a Chromium process any time soon (well, unless it's a gmail tab...), I think you should try to maintain accuracy here with relation to your types. If you are going to shear off 32 bits, please make this explicit in every call, as this will be a subtlety that will affect every developer who has to read/maintain this code (which is why just using size_t is likely better) http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:9: missing stddef.h (<new> only pulls in std::size_t) http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:45: SpinLock g_type_profiler_lock(SpinLock::LINKER_INITIALIZED); It is unfortunate, and perhaps meritous of future work, that this effectively serializes all allocation/deallocations to block on the spin lock, given that TCmalloc is so good at keeping track of thread-local storage. I don't think it's a blocker for this change, but if you're trying to avoid negatively impacting perf while collecting data, I believe it would be good to consider the use of thread locals, rather than system globals, in the future. http://codereview.chromium.org/10411047/diff/133036/tools/deep_memory_profile... File tools/deep_memory_profiler/policy.l0.txt (right): http://codereview.chromium.org/10411047/diff/133036/tools/deep_memory_profile... tools/deep_memory_profiler/policy.l0.txt:5: mmap-type-profiler mmap .*(TypeProfilerMalloc).* It's unclear why these changes are necessary, as I do not see you dumping any data yet (unlike src/third_party/tcmalloc/chromium/src/heap-profiler.cc, which does/can dump) Unfortunately, I was unable to find the policy.README these files refer to either.
Thanks Ryan. Ok, I won't commit the deep_memory_profiler changes until this is landed. http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.gyp File base/allocator/allocator.gyp (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.... base/allocator/allocator.gyp:609: '../..', On 2012/09/13 09:13:25, Ryan Sleevi wrote: > Why the ../../ include (which is effectively <(DEPTH)) ? > > What's being included here that needs the Chromium scope? > > Seems like it can/should be removed Without "../..", I cannot include base/allocator/type_profiler_tcmalloc.h... Isn't it??? I used ../.. (instead of DEPTH) for consistency with other targets in this file. When removed, I was said: [2/12] CXX obj/base/allocator/type_profiler_tcmalloc.type_profiler_tcmalloc.o ... ../../base/allocator/type_profiler_tcmalloc.cc:7:10: fatal error: 'base/allocator/type_profiler_tcmalloc.h' file not found #include "base/allocator/type_profiler_tcmalloc.h" ^ 1 error generated. [2/12] CXX obj/third_party/tcmalloc/chromium/src/type_profiler_tcmalloc.type_profiler_map.o ... In file included from ../../third_party/tcmalloc/chromium/src/type_profiler_map.cc:7: ../../third_party/tcmalloc/chromium/src/config.h:7:10: fatal error: 'build/build_config.h' file not found #include "build/build_config.h" ^ 1 error generated. http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.... base/allocator/allocator.gyp:615: '<(tcmalloc_dir)/src/gperftools/type_profiler_map.h', On 2012/09/13 09:13:25, Ryan Sleevi wrote: > nit: sort Done. http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.... base/allocator/allocator.gyp:657: '<(tcmalloc_dir)/src/gperftools/type_profiler_map.h', On 2012/09/13 09:13:25, Ryan Sleevi wrote: > nit: sort Done. http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... File base/allocator/type_profiler.cc (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler.cc:20: } // anonymous namespace On 2012/09/13 09:13:25, Ryan Sleevi wrote: > For unnamed namespaces (nit-peeve: not anonymous, unnamed ;-) ), it's not > necessary to include a trailing comment, so remove it. > Done. (also for type_profiler_control.cc) http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... File base/allocator/type_profiler.h (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler.h:10: #include <cstddef> // for size_t On 2012/09/13 09:13:25, Ryan Sleevi wrote: > Change this to <stddef.h> > > <cstddef> is only required to declare std::size_t, which you're not using here Done. http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... File base/allocator/type_profiler_control.h (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler_control.h:20: FRIEND_TEST_ALL_PREFIXES(TypeProfilerTest, On 2012/09/13 09:13:25, Ryan Sleevi wrote: > If you've friended TypeProfilerTest, you do not need to FRIEND_TEST_ALL_PREFIXES > > The two are (generally) mutually exclusive. If you friend a class (really, a > test fixture), then you can access private/protected members. However, note that > all unit tests will be subclasses of that fixture, ergo they will not inherit > the friend-edness. To work around that, you have to use protected helper method > *on the fixture* to allow individual tests to access. > > However, when you only have one or two tests, it's often easier and cleaner to > directly friend the test, rather than go through all that. Since you only have a > single test, remove the "friend class TypeProfilerTest", and the right thing > will happen (only TestProfileNewWithoutProfiledDelete will be friended) Removed "friend class TypeProfilerTest;" in the next patch. http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... File base/allocator/type_profiler_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:10: #include <cstddef> On 2012/09/13 09:13:25, Ryan Sleevi wrote: > stddef.h Actually, is has been removed. http://codereview.chromium.org/10411047/diff/131013/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:30: EXPECT_EQ(type, g_const_null); On 2012/09/13 09:13:25, Ryan Sleevi wrote: > Please re-order all of the test comparisons to match the correct form of > > EXPECT_EQ(expected, actual) > > https://code.google.com/p/googletest/wiki/Primer#Binary_Comparison > > Alternatively, you can just use the implicit bool conversions, which can avoid > the GCC 4.6 issues related to NULL handling > > EXPECT_FALSE(type) > > ASSERT_TRUE(type) > > etc I chose just re-ordering. http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h (right): http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h:15: void InsertType(void* address, unsigned int size, const std::type_info& type); On 2012/09/13 09:13:25, Ryan Sleevi wrote: > BUG: Your interposed __op_delete_intercept__ function takes its parameters as a > size_t ( see > http://codereview.chromium.org/10411047/diff/133036/base/allocator/type_profi... > ) > > You're potentially downcasting here to an unsigned int here. On an LP64 system, > you're converting a 64-bit size to a 32-bit size. > > While I doubt we'll be allocating more than 4 gigs in a Chromium process any > time soon (well, unless it's a gmail tab...), I think you should try to maintain > accuracy here with relation to your types. > > If you are going to shear off 32 bits, please make this explicit in every call, > as this will be a subtlety that will affect every developer who has to > read/maintain this code (which is why just using size_t is likely better) Good catch. Switched to size_t. http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/type_profiler_map.cc (right): http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:9: On 2012/09/13 09:13:25, Ryan Sleevi wrote: > missing stddef.h (<new> only pulls in std::size_t) Done. http://codereview.chromium.org/10411047/diff/133036/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/type_profiler_map.cc:45: SpinLock g_type_profiler_lock(SpinLock::LINKER_INITIALIZED); On 2012/09/13 09:13:25, Ryan Sleevi wrote: > It is unfortunate, and perhaps meritous of future work, that this effectively > serializes all allocation/deallocations to block on the spin lock, given that > TCmalloc is so good at keeping track of thread-local storage. > > I don't think it's a blocker for this change, but if you're trying to avoid > negatively impacting perf while collecting data, I believe it would be good to > consider the use of thread locals, rather than system globals, in the future. Yeah, I know. I'll try that in future, but TCMalloc's heap-profiler also uses such lock: RecordAlloc and RecordFree in heap-profiler.cc. I should also fix them at that time. ;) http://codereview.chromium.org/10411047/diff/133036/tools/deep_memory_profile... File tools/deep_memory_profiler/policy.l0.txt (right): http://codereview.chromium.org/10411047/diff/133036/tools/deep_memory_profile... tools/deep_memory_profiler/policy.l0.txt:5: mmap-type-profiler mmap .*(TypeProfilerMalloc).* On 2012/09/13 09:13:25, Ryan Sleevi wrote: > It's unclear why these changes are necessary, as I do not see you dumping any > data yet (unlike src/third_party/tcmalloc/chromium/src/heap-profiler.cc, which > does/can dump) > > Unfortunately, I was unable to find the policy.README these files refer to > either. It is required to omit type_profiler memory from profiling results for "actually used by Chrome". Even unless type information is dumped, existing profiler should ignore type_profiler memory. And, sorry, I mis-described the filename. I fixed it to "README.policy".
> It would seem like that your changes to tools/deep_memory are unnecessary at > this time, or perhaps incorrect. Ah, you said about tools/deep_memory_profiler changes in this CL? It is required for this CL. See my reply.
Dai, Thanks for your patience in dealing with our feedback. I believe we're in the home stretch. This change LGTM, but I'm punting one last thing back to Jim for his confirmation on (the comment "However, in light of") Please wait for his feedback and confirmation on change/no change before committing. http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.gyp File base/allocator/allocator.gyp (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.... base/allocator/allocator.gyp:609: '../..', On 2012/09/13 10:45:04, Dai Mikurube wrote: > On 2012/09/13 09:13:25, Ryan Sleevi wrote: > > Why the ../../ include (which is effectively <(DEPTH)) ? > > > > What's being included here that needs the Chromium scope? > > > > Seems like it can/should be removed > > Without "../..", I cannot include base/allocator/type_profiler_tcmalloc.h... > Isn't it??? > > I used ../.. (instead of DEPTH) for consistency with other targets in this file. Thank you for your clarification. It seems that most targets omit this, as they depend directly on base, which has a block that does 'direct_dependent_settings': { 'include_dirs': [ '..' ] } That ensures the equivalent of this statement. While it's the dominant pattern to omit it when depending on base/ (such as 'type_profiler_[map_]unittests' do), I think it's better to keep what you have. Just something to consider in future CLs in other areas. http://codereview.chromium.org/10411047/diff/137052/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h (right): http://codereview.chromium.org/10411047/diff/137052/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h:12: // We don't use PERFTOOLS_DLL_DECL since TypeProfilerMap works only on Linux. nit: Please don't use "we" in comments. // PERFTOOLS_DLL_DECL is unnecessary, as it is Windows specific. Note this also removes the note about "works only on Linux", as it's clear you intend to support OS X "at some point". http://codereview.chromium.org/10411047/diff/137052/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h:15: namespace type_profiler { However, in light of the above comment, it makes me think that perhaps these functions do not belong in the base::type_profiler namespace, as you're treating them as part of TCMalloc (by virtue of being in third_party/tcmalloc), and **NOT** as part of base/ or "Chromium proper". If that's the case, I'd suggest these files do not belong in this namespace. I'm going to have to defer to Jim's preferences here, since I think he feels more strongly about it.
Thank you for reviewing, Ryan. Updated the patch, and replied. http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.gyp File base/allocator/allocator.gyp (right): http://codereview.chromium.org/10411047/diff/131013/base/allocator/allocator.... base/allocator/allocator.gyp:609: '../..', On 2012/09/14 23:11:05, Ryan Sleevi wrote: > On 2012/09/13 10:45:04, Dai Mikurube wrote: > > On 2012/09/13 09:13:25, Ryan Sleevi wrote: > > > Why the ../../ include (which is effectively <(DEPTH)) ? > > > > > > What's being included here that needs the Chromium scope? > > > > > > Seems like it can/should be removed > > > > Without "../..", I cannot include base/allocator/type_profiler_tcmalloc.h... > > Isn't it??? > > > > I used ../.. (instead of DEPTH) for consistency with other targets in this > file. > > Thank you for your clarification. > > It seems that most targets omit this, as they depend directly on base, which has > a block that does > 'direct_dependent_settings': { > 'include_dirs': [ '..' ] > } > > That ensures the equivalent of this statement. > > While it's the dominant pattern to omit it when depending on base/ (such as > 'type_profiler_[map_]unittests' do), I think it's better to keep what you have. > Just something to consider in future CLs in other areas. Thanks for the information. I keep it as is. http://codereview.chromium.org/10411047/diff/137052/third_party/tcmalloc/chro... File third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h (right): http://codereview.chromium.org/10411047/diff/137052/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h:12: // We don't use PERFTOOLS_DLL_DECL since TypeProfilerMap works only on Linux. On 2012/09/14 23:11:06, Ryan Sleevi wrote: > nit: Please don't use "we" in comments. > > // PERFTOOLS_DLL_DECL is unnecessary, as it is Windows specific. > > Note this also removes the note about "works only on Linux", as it's clear you > intend to support OS X "at some point". Thanks. Done. http://codereview.chromium.org/10411047/diff/137052/third_party/tcmalloc/chro... third_party/tcmalloc/chromium/src/gperftools/type_profiler_map.h:15: namespace type_profiler { On 2012/09/14 23:11:06, Ryan Sleevi wrote: > However, in light of the above comment, it makes me think that perhaps these > functions do not belong in the base::type_profiler namespace, as you're treating > them as part of TCMalloc (by virtue of being in third_party/tcmalloc), and > **NOT** as part of base/ or "Chromium proper". > > If that's the case, I'd suggest these files do not belong in this namespace. > > I'm going to have to defer to Jim's preferences here, since I think he feels > more strongly about it. I agree with Ryan. These functions have been moved into the namespace due to the following discussion: http://codereview.chromium.org/10411047/diff/115005/third_party/tcmalloc/chro... What do you think, Jim? In my latest patch, I just moved them to the global namespace temporarily. 1) Reverting (move them into base::type_profiler again) is fine based on Jim'm preference. 2) Or, they may want to have another function names like TypeProfilerInsertType (like HeapProfilerStart in heap-profiler.cc). Just "InsertType" may be too short for the global namespace.
WDYT? ping..
Swamped by perf and other task... will look RSN. Jim On Thu, Sep 20, 2012 at 3:38 PM, <dmikurube@chromium.org> wrote: > WDYT? ping.. > > http://codereview.chromium.**org/10411047/<http://codereview.chromium.org/104... >
Thanks. I was just worried about forgotten... On 2012/09/21 00:28:25, jar wrote: > Swamped by perf and other task... will look RSN. > > Jim > > On Thu, Sep 20, 2012 at 3:38 PM, <mailto:dmikurube@chromium.org> wrote: > > > WDYT? ping.. > > > > > http://codereview.chromium.**org/10411047/%3Chttp://codereview.chromium.org/1...> > >
ping, again. just a reminder.
btw the content part lgtm, I defer to jar on it
Just about there.... http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... File base/allocator/type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... base/allocator/type_profiler_control.cc:20: } nit: // anonymous http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... File base/allocator/type_profiler_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:96: int main(int argc, char** argv) { Why is a separate main() needed here, rather than running as part of the standard test harness? http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:186: int main(int argc, char** argv) { Why does this need a main(), rather than running out of a standard harness?
http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... File base/allocator/type_profiler_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:96: int main(int argc, char** argv) { On 2012/09/25 22:09:31, jar wrote: > Why is a separate main() needed here, rather than running as part of the > standard test harness? I'll save the time lag for Dai, since I had the same question, and looked it up. This follows the existing TCMalloc tests. Using the existing test harnesses, such as run_all_unittests.cc, forces a dependency on base/, which we specifically want to avoid. So I agree with Dai's approach here and think this is the correct way to handle these specific tests (note: the dependency on base/scoped_ptr.h is a little bit of a cheat, but it's a fully inlined template)
http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... File base/allocator/type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... base/allocator/type_profiler_control.cc:20: } On 2012/09/25 22:09:31, jar wrote: > nit: // anonymous slight tweak: // namespace as per http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml?showone=Namesp...
OK... with the nit correction made by Ryan (comment // anonymous).... LGTM
Thank you all reviewers for the long discussion to improve this change! Fixed the comment. I'll be committing this patch. http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... File base/allocator/type_profiler_control.cc (right): http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... base/allocator/type_profiler_control.cc:20: } On 2012/09/25 22:18:41, Ryan Sleevi wrote: > On 2012/09/25 22:09:31, jar wrote: > > nit: // anonymous > > slight tweak: // namespace > > as per > http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml?showone=Namesp... Done. http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... File base/allocator/type_profiler_map_unittests.cc (right): http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... base/allocator/type_profiler_map_unittests.cc:96: int main(int argc, char** argv) { On 2012/09/25 22:15:06, Ryan Sleevi wrote: > On 2012/09/25 22:09:31, jar wrote: > > Why is a separate main() needed here, rather than running as part of the > > standard test harness? > > I'll save the time lag for Dai, since I had the same question, and looked it up. > > This follows the existing TCMalloc tests. Using the existing test harnesses, > such as run_all_unittests.cc, forces a dependency on base/, which we > specifically want to avoid. > > So I agree with Dai's approach here and think this is the correct way to handle > these specific tests (note: the dependency on base/scoped_ptr.h is a little bit > of a cheat, but it's a fully inlined template) Thanks for the answer, Ryan. http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... File base/allocator/type_profiler_unittests.cc (right): http://codereview.chromium.org/10411047/diff/151001/base/allocator/type_profi... base/allocator/type_profiler_unittests.cc:186: int main(int argc, char** argv) { On 2012/09/25 22:09:31, jar wrote: > Why does this need a main(), rather than running out of a standard harness? The same as type_profiler_map_unittests.
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/dmikurube@chromium.org/10411047/153003
Change committed as 158752 |
