Eyecandy mutexes!

In my quest of making MySQL usable, I managed to hit contention that wasn’t spotted by performance masters before. Meet most useless mutex ever (this is actual contention event, not just a hold):

Count     nsec Lock
 1451   511364 mysqld`ut_list_mutex

      nsec ---- Distribution --- count    Stack
      2048 |@@                 |   132    libc.so.1`mutex_lock_impl
      4096 |@@@                |   186    libc.so.1`mutex_lock
      8192 |                   |     3    ut_malloc_low
     16384 |@@                 |   137    mem_heap_create_block
     32768 |@                  |   105    row_sel_store_mysql_rec
     65536 |@                  |    89    row_search_for_mysql
    131072 |@                  |    99    ha_innobase::general_fetch
    262144 |@@                 |   131    ha_innobase::rnd_next
    524288 |@@@                |   195    rr_sequential
   1048576 |@@@                |   223
   2097152 |@@                 |   137
   4194304 |                   |    14

ut_list_mutex guards a memory structure which has all memory blocks allocated by InnoDB (via ut_malloc/ut_free) in it.
It has two uses:

  1. Printing “Total memory allocated” in SHOW INNODB STATUS (though this can still be implemented lock-free)
  2. Deallocating all memory on shutdown (though, all modern operating systems do that anyway, so this is purely just to shut up valgrind)

If you have any BLOB/TEXT data in your tables, you’re definitely hitting this contention spot (it is #1 contention in such cases).

Fix? Kill the eyecandy, replace ut_malloc and ut_free with direct calls to malloc() and free(), oh and of course, use scalable allocators like tcmalloc or Hoard.

3 thoughts on “Eyecandy mutexes!”

  1. Domas,
    Looking at the code for 5.0.67 and based on the call stack that you provide, the call chain is mem_heap_create_block -> mem_area_alloc -> ut_malloc -> ut_malloc_low. There are a couple of things to note:
    * mem_area_alloc/mem_area_free is protected by a separate mutex that can be hot
    * mem_area_alloc only calls ut_malloc when the mem_pool_t runs out of memory
    * most of us don’t get to use dtrace. Were you using it?
    * I have not had see the ut_malloc mutex get hot. What workload did you use?
    * the Google patch changes mem_area_malloc/mem_area_free to call malloc/free directly. Even without tcmalloc, most system libraries are probably better than what is provided by mem_area_alloc/mem_area_free/mem_pool_t for multithreaded usage

  2. damnit, to figure out where the problem was happening I rebuilt 5.1.30 with -fno-inline and now I’m hitting mem_pool mutexes, rather than ut_malloc one. Debugging inlined functions has always been pain :) Maybe I had some specific situation/edge case, where it was being hit, but I reproduced it on both 4.0 and 5.1 (I’m not sure I even have anything else on this Solaris box).

    Will try to understand what is happening in code path more.

    Yes, I was using dtrace plockstat thing here.

    The workload was BLOB reading en-masse. :)

Comments are closed.