How to Tune and Optimize Performance of MySQL 5.7 on an Ubuntu 18.04 VPS or Dedicated Server

Introduction

Before we start you’ve got to understand the factors affecting MySQL performance, so that you can learn to tune and optimize MySQL server in the right way and get the right results. The main factors include:

Volume of data being retrieved
Resources available i.e. CPU, Memory (RAM)
Load run by MySQL server>

Causes of poor MySQL server performance include:

Poor database design
Hardware bottlenecks
Poor coding
Poor indexing
Inefficient configuration

Special Note: as you can see, several of these factors can be addressed by upgrading to a better web hosting service. HostAdvice brings you the best MySQL hosting providers based on expert and user reviews.

Installing mysqltuner and tuning-primer

Install mysqltuner

Mysqltuner is a high-performance MySQL tuning Perl script that gives a snapshot of the MySQL server’s health status and gives specific recommendations for improved, increased performance, stability and efficiency.

Run the commands below to install mysqltuner

$ cd /tmp
$ sudo wget https://raw.github.com/major/MySQLTuner-perl/master/mysqltuner.pl --quiet --no-check-certificate --output-document=/usr/local/bin/mysqltuner
$ sudo chmod u+x /usr/local/bin/mysqltuner

Install tuning-primer

Tuning Primer is a shell script that obtains information from MySQL server internals and makes recommendations on adjustments of the server variables.

Run the commands below to install tuning primer:

$ sudo apt install bc -y
$ sudo wget http://www.day32.com/MySQL/tuning-primer.sh --output-document=/usr/local/bin/tuning-primer
$ sudo chmod u+x /usr/local/bin/tuning-primer

Tuning and optimizing mysql server

The tuning of MySQL server is done on MySQL configuration file.

/etc/mysql/mysql.conf.d/mysqld.cnf

Run mysqltuner.

$ sudo mysqltuner

$ sudo mysqltuner
 >>  MySQLTuner 1.7.9 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from debian maintenance account.
[OK] Currently running supported MySQL version 5.7.22-0ubuntu18.04.1
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: /var/log/mysql/error.log(18K)
[OK] Log file /var/log/mysql/error.log exists
[OK] Log file /var/log/mysql/error.log is readable.
[OK] Log file /var/log/mysql/error.log is not empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[!!] /var/log/mysql/error.log contains 14 warning(s).
[!!] /var/log/mysql/error.log contains 3 error(s).
[--] 3 start(s) detected in /var/log/mysql/error.log
[--] 1) 2018-06-06T17:55:47.928847Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 2) 2018-06-06T17:55:44.798684Z 0 [Note] mysqld: ready for connections.
[--] 3) 2018-06-06T17:55:41.931105Z 0 [Note] mysqld: ready for connections.
[--] 2 shutdown(s) detected in /var/log/mysql/error.log
[--] 1) 2018-06-06T17:55:46.410548Z 0 [Note] mysqld: Shutdown complete
[--] 2) 2018-06-06T17:55:43.758366Z 0 [Note] mysqld: Shutdown complete

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in InnoDB tables: 16K (Tables: 1)
[OK] Total fragmented tables: 0

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[--] Bug #80860 MySQL 5.7: Avoid testing password when validate_password is activated

-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 9m 8s (110 q [0.201 qps], 38 conn, TX: 187K, RX: 10K)
[--] Reads / Writes: 98% / 2%
[--] Binary logging is disabled
[--] Physical Memory : 3.6G
[--] Max MySQL memory: 352.4M
[--] Other process memory: 175.4M
[--] Total buffers: 192.0M global + 1.1M per thread (151 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 194.1M (5.26% of installed RAM)
[OK] Maximum possible memory usage: 352.4M (9.54% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/110)
[OK] Highest usage of available connections: 1% (2/151)
[OK] Aborted connections: 0.00%  (0/38)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 49 selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 2 sorts)
[OK] No joins without indexes
[OK] Temporary tables created on disk: 4% (36 on disk / 846 total)
[OK] Thread cache hit rate: 94% (2 created / 38 connections)
[OK] Table cache hit rate: 95% (591 open / 620 opened)
[OK] Open file limit used: 1% (50/5K)
[OK] Table locks acquired immediately: 100% (158 immediate / 158 locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 72B
[--] Sys schema is installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.3% (3M used / 16M cache)
[OK] Key buffer size / total MyISAM indexes: 16.0M/43.0K
[!!] Read Key buffer hit rate: 94.1% (118 cached / 7 reads)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 128.0M/16.0K
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[!!] InnoDB Read buffer efficiency: 88.39% (1933 hits/ 2187 total)
[!!] InnoDB Write Log efficiency: 0% (5 hits/ 0 total)
[OK] InnoDB log waits: 0.00% (0 waits / 5 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- RocksDB Metrics ---------------------------------------------------------------------------
[--] RocksDB is disabled.

-------- Spider Metrics ----------------------------------------------------------------------------
[--] Spider is disabled.

-------- Connect Metrics ---------------------------------------------------------------------------
[--] Connect is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
  Control warning line(s) into /var/log/mysql/error.log file
  Control error line(s) into /var/log/mysql/error.log file
  MySQL was started within the last 24 hours - recommendations may be inaccurate
  Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
  Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: http://bit.ly/2wgkDvS
Variables to adjust:
  query_cache_size (=0)
  query_cache_type (=0)
  query_cache_limit (> 1M, or use smaller result sets)
  innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

Focus on the last part of the output in section “Recommendations” and check the variables that need to be increased.

Create a backup of the mysql configuration file and edit the actual configuration file.

$ sudo cp /etc/mysql/mysql.conf.d/mysqld.cnf /etc/mysql/mysql.conf.d/mysqld.cnf.bak
$ sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

Adjust the file’s variable according to the recommendations. You can comment out the default variable values and increase the variable values to twice the default value. Actually, it may take a few days to come up with the best values for your server.

After making the changes, restart mysql server.

$ sudo systemctl restart mysql

Special Note: If it fails to restart, go back and comment out the new values and start adjusting one by one and see which variable adjustment causes the server failur

If server successfully reloads, rerun mysqltuner and see if there are any more recommendations concerning the variable values adjustment.

Do similar process for tuning primer.

Run tuning primer

$ sudo tuning-primer

$ sudo tuning-primer

    -- MYSQL PERFORMANCE TUNING PRIMER --
         - By: Matthew Montgomery -

MySQL Version 5.7.22-0ubuntu18.04.1 x86_64

Uptime = 0 days 0 hrs 1 min 49 sec
Avg. qps = 0
Total Questions = 15
Threads Connected = 1

Warning: Server has not been running for at least 48hrs.
It may not be safe to use these recommendations

To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service

SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 10.000000 sec.
You have 0 out of 36 that take longer than 10.000000 sec. to complete
Your long_query_time seems to be fine

BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/5.7/en/point-in-time-recovery.html

WORKER THREADS
Current thread_cache_size = 8
Current threads_cached = 0
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MAX CONNECTIONS
Current max_connections = 151
Current threads_connected = 1
Historic max_used_connections = 1
The number of used connections is 0% of the configured maximum.
You are using less than 10% of your configured max_connections.
Lowering max_connections could help to avoid an over-allocation of memory
See "MEMORY USAGE" section to make sure you are not over-allocating

No InnoDB Support Enabled!

MEMORY USAGE
Max Memory Ever Allocated : 177 M
Configured Max Per-thread Buffers : 160 M
Configured Max Global Buffers : 176 M
Configured Max Memory Limit : 336 M
Physical Memory : 3.60 G
Max memory limit seem to be within acceptable norms

KEY BUFFER
Current MyISAM index space = 43 K
Current key_buffer_size = 16 M
Key cache miss rate is 1 : 2
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine

QUERY CACHE
Query cache is enabled
Current query_cache_size = 16 M
Current query_cache_used = 16 K
Current query_cache_limit = 1 M
Current Query cache Memory fill ratio = .10 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size

SORT OPERATIONS
Current sort_buffer_size = 256 K
Current read_rnd_buffer_size = 256 K
No sort operations have been performed
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 260.00 K
You have had 0 queries where a join could not use an index properly
Your joins seem to be using indexes properly

OPEN FILES LIMIT
Current open_files_limit = 5000 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine

TABLE CACHE
Current table_open_cache = 2000 tables
Current table_definition_cache = 1400 tables
You have a total of 119 tables
You have 249 open tables.
The table_cache value seems to be fine

TEMP TABLES
Current max_heap_table_size = 16 M
Current tmp_table_size = 16 M
Of 361 temp tables, 6% were created on disk
Created disk tmp tables ratio seems fine

TABLE SCANS
Current read_buffer_size = 128 K
Current table scan ratio = 673 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 0 : 219
Your table locking seems to be fine

Special Note: Focus more on the red and yellow highlighted text as it gives recommendations on how to optimize and improve mysql performance.

Adjust the file’s variable according to the recommendations. You can comment out the default variable values and adjust the variable values as suggested in the recommendations.

After making the changes, restart MySQL server.

$ sudo systemctl restart mysql

Special Note: If it fails to restart, go back and comment out the new values and start adjusting one by one and see which variable adjustment causes the server failure.

If server successfully restarts, rerun tuneprimer and see if there are any more recommendations concerning the variable values adjustment.

Conclusion

It is very important to note that the tuning tools mysqltuner or tuning-primer are not the solutions to poorly performing MySQL server, but rather scripts to guide you to fine tune the MySQL server.

For best performance,

Do a thorough review of queries sent to the server and optimize database structure/design and SQL query
Evaluate the factors affecting MySQL performance as explained in the introduction.
Consider upgrading hardware of the physical server for better performance and optimize MySQL configurations based on the hardware specifications.
Do more research on how to tune MySQL configurations based on applications utilizing MySQL server.

It is highly recommended to regularly run mysqltuner or tuningprimer tool over 48 hr span after restarting MySQL server, so as to obtain meaningful usage statistic.