In my quest of making MySQL usable, I managed to hit contention that wasn’t spotted by performance masters before. Meet most useless mutex ever (this is actual contention event, not just a hold):
Count nsec Lock
1451 511364 mysqld`ut_list_mutex
nsec ---- Distribution --- count Stack
2048 |@@ | 132 libc.so.1`mutex_lock_impl
4096 |@@@ | 186 libc.so.1`mutex_lock
8192 | | 3 ut_malloc_low
16384 |@@ | 137 mem_heap_create_block
32768 |@ | 105 row_sel_store_mysql_rec
65536 |@ | 89 row_search_for_mysql
131072 |@ | 99 ha_innobase::general_fetch
262144 |@@ | 131 ha_innobase::rnd_next
524288 |@@@ | 195 rr_sequential
1048576 |@@@ | 223
2097152 |@@ | 137
4194304 | | 14
----------------------------------------------------------------
ut_list_mutex guards a memory structure which has all memory blocks allocated by InnoDB (via ut_malloc/ut_free) in it.
It has two uses:
- Printing “Total memory allocated” in SHOW INNODB STATUS (though this can still be implemented lock-free)
- Deallocating all memory on shutdown (though, all modern operating systems do that anyway, so this is purely just to shut up valgrind)
If you have any BLOB/TEXT data in your tables, you’re definitely hitting this contention spot (it is #1 contention in such cases).
Fix? Kill the eyecandy, replace ut_malloc and ut_free with direct calls to malloc() and free(), oh and of course, use scalable allocators like tcmalloc or Hoard.
domas mituzas 
Domas,
Looking at the code for 5.0.67 and based on the call stack that you provide, the call chain is mem_heap_create_block -> mem_area_alloc -> ut_malloc -> ut_malloc_low. There are a couple of things to note:
* mem_area_alloc/mem_area_free is protected by a separate mutex that can be hot
* mem_area_alloc only calls ut_malloc when the mem_pool_t runs out of memory
* most of us don’t get to use dtrace. Were you using it?
* I have not had see the ut_malloc mutex get hot. What workload did you use?
* the Google patch changes mem_area_malloc/mem_area_free to call malloc/free directly. Even without tcmalloc, most system libraries are probably better than what is provided by mem_area_alloc/mem_area_free/mem_pool_t for multithreaded usage
damnit, to figure out where the problem was happening I rebuilt 5.1.30 with -fno-inline and now I’m hitting mem_pool mutexes, rather than ut_malloc one. Debugging inlined functions has always been pain :) Maybe I had some specific situation/edge case, where it was being hit, but I reproduced it on both 4.0 and 5.1 (I’m not sure I even have anything else on this Solaris box).
Will try to understand what is happening in code path more.
Yes, I was using dtrace plockstat thing here.
The workload was BLOB reading en-masse. :)
We avoid blobs as much as possibe. I think that Innodb has special code paths for large allocations which blobs might use.