Apache Performance Tuning
This article will or should provide enough information in order to diagnose, troubleshoot, and resolve issues encountered regarding Apache performance on Debian based Linux machines. Usually the easiest fix is to simply beef the box up, but it’s not always the recommended solution. If after trying to tune Apache using mpm_prefork, switch to mpm_worker or mpm_event. Since you’re reading this because you most likely have performance issues, think about putting a service like Cloudflare in front of your website to take the bulk/lion’s share of the load off your front-end web servers. Cloudflare’s free plan is being utilized as well to function as front-line global AnyCast caching reverse proxy.
I would like to highly recommend Cloudflare to everyone and anyone on a scale from a single small website owner such as myself, all the way to large-size companies which do not already have internal staff with specializations in CDN/WAF/CCIE-level networking/Linux Engineers for their customized WAF, network capacity, and global infrastructure footprint. And no, I’m not being paid to advertise them nor do I work for them. This small website has been on their free plan for a long time, using a small part of their global infrastructure at absolutely no cost to me. Very cool. I have dealt with their sales/technical team members in the past while working at another company and have always had a pleasant experience.
I have moved the backend web server from Apache to NGINX, and the database from Amazon RDS to MariaDB. The entire site now resides on my Intel i5 desktop w/ 16GB memory over multiple Linux virtual machines running on ESXi, which actually communicates over a wired to wireless bridge I built. Since I am caching FastCGI data directly to tmpfs (memory-backed temporary file system) in the /var/run directory structure, the site usually performs very well once both CloudFlare and my vmware server has built up its own cache:
root@nginx02:~# df -h Filesystem Size Used Avail Use% Mounted on udev 468M 0 468M 0% /dev tmpfs 98M 40M 59M 41% /run /dev/mapper/vg--group-root 15G 6.2G 7.3G 46% / tmpfs 488M 0 488M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 488M 0 488M 0% /sys/fs/cgroup /dev/sda1 472M 106M 342M 24% /boot tmpfs 98M 0 98M 0% /run/user/0 //win2016-3/DFS 2.0T 1.6T 429G 79% /mnt/windows/dfs //win2016-3/DFS 2.0T 1.6T 429G 79% /mnt/windows/dfs-rw tmpfs 98M 0 98M 0% /run/user/65534 root@nginx02:~# du -sh /var/run/nginx-cache/ 29M /var/run/nginx-cache/
The reason for the migration is because AWS was costing more to run than my little site generates in ad revenue. So please, help out and share to social media so I can continue to run this sole operation.
Check memory usage of the Apache process\
apache2ctl -V
Determine the Server MPM used by apache (should be Prefork) and version data by running the command:
watch -n 1 "echo -n 'Apache Processes: ' && ps -C apache2 --no-headers | wc -l && free -m"
Pay attention to the number of Apache processes and the amount of memory in the free column of
-/+ buffers/cache
If there are a large number of Apache processes and a low number in the free column , this should immediately signify that the MaxClients directive in the global Apache config is set too high for the amount of memory installed on the machine. Either adjust the MaxClients or increase the memory on the server.
Run the following ApacheBuddy perl script to review the current setup and it will even make tuning suggestions (≤ Apache 2.2). This particular script does not work in versions higher than 2.3.13 because MaxClients was superseded by MaxRequestWorkers (and there are advances in Cipher suite support)
Instead try wget this modified version of ApacheBuddy which I’ve had limited success with.
For Apache version ≤ 2.2 use this
For Apache version ≥ 2.4 use this
Version 2.2:
curl -L http://blog.travisrunyard.us/downloads/apachebuddy-2.2.pl | perl
Version 2.4:
curl -L http://blog.travisrunyard.us/downloads/apachebuddy-2.4.7.pl | perl
Analyze Apache log files for unusual activity
- Download and install goaccess with apt-get install goaccess
- Running goaccess will allow you to analyze the Apache access.log file in real-time which may aid in troubleshooting an issue
goaccess -b -s -a f /var/log/apache2/{website-name}_access.log
- Find top memory consumers with any of the following shell scripts:
ps -e -orss=,args=,command,user,pid | sort -b -k1n | awk '{ hr=($1/1024)/1024 ; printf("%13.2f GB ",hr) } { for ( x=2 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' 2>&1ps -ewwwo pid,size,command --sort -size | head -100 | awk '{ pid=$1 ; printf("%7s ", pid) }{ hr=$2/1024 ; printf("%8.2f Mb ", hr) } { for ( x=3 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' 2>&1 - I have been testing switching from mpm_prefork to mpm_worker but be warned, the configuration definition is a little more advanced and may introduce additional complexities. Though mpm_worker did perform better for me, I would recommend switching out for NGINX or using NGINX in front as a reverse proxy/load balancer.
The main consideration is PHP being compiled thread-safe.
Prefork vs Worker vs Event
There are a number of MPM modules (Multi-Processing Modules), but by far the most widely used (at least on *nix platforms) are the three main ones: prefork, worker, and event. Essentially, they represent the evolution of the Apache web server, and the different ways that the server has been built to handle HTTP requests within the computing constraints of the time over its long (in software terms) history.
prefork
mpm_prefork is.. well.. it’s compatible with everything. It spins of a number of child processes for serving requests, and the child processes only serve one request at a time. Because it’s got the server process sitting there, ready for action, and not needing to deal with thread marshaling, it’s actually faster than the more modern threaded MPMs when you’re only dealing with a single request at a time – but concurrent requests suffer, since they’re made to wait in line until a server process is free. Additionally, attempting to scale up in the count of prefork child processes, you’ll easily suck down some serious RAM.
It’s probably not advisable to use prefork unless you need a module that’s not thread safe.
Use if: You need modules that break when threads are used, like mod_php. Even then, consider using FastCGI and php-fpm.
Don’t use if: Your modules won’t break in threading.
worker
mpm_worker uses threading – which is a big help for concurrency. Worker spins off some child processes, which in turn spin off child threads; similar to prefork, some spare threads are kept ready if possible, to service incoming connections. This approach is much kinder on RAM, since the thread count doesn’t have a direct bearing on memory use like the server count does in prefork. It also handles concurrency much more easily, since the connections just need to wait for a free thread (which is usually available) instead of a spare server in prefork.
Use if: You’re on Apache 2.2, or 2.4 and you’re running primarily SSL.
Don’t use if: You really can’t go wrong, unless you need prefork for compatibility.
However, note that the treads are attached to connections and not requests – which means that a keep-alive connection always keeps ahold of a thread until it’s closed (which can be a long time, depending on your configuration). Which is why we have..
event
mpm_event is very similar to worker, structurally; it’s just been moved from ‘experimental’ to ‘stable’ status in Apache 2.4. The big difference is that it uses a dedicated thread to deal with the kept-alive connections, and hands requests down to child threads only when a request has actually been made (allowing those threads to free back up immediately after the request is completed). This is great for concurrency of clients that aren’t necessarily all active at a time, but make occasional requests, and when the clients might have a long keep-alive timeout.
The exception here is with SSL connections; in that case, it behaves identically to worker (gluing a given connection to a given thread until the connection closes).
Use if: You’re on Apache 2.4 and like threads, but you don’t like having threads waiting for idle connections. Everyone likes threads!
Don’t use if: You’re not on Apache 2.4, or you need prefork for compatibility.
In today’s world of slowloris, AJAX, and browsers that like to multiplex 6 TCP connections (with keep-alive, of course) to your server, concurrency is an important factor in making your server scale and scale well. Apache’s history has tied it down in this regard, and while it’s really still not up to par with the likes of nginx or lighttpd in terms of resource usage or scale, it’s clear that the development team is working toward building a web server that’s still relevant in today’s high-request-concurrency world.
What is the Difference between Prefork and Worker modules?
Prefork MPM uses multiple child processes with one thread each and each process handles one connection at a time.
Worker MPM uses multiple child processes with many threads each. Each thread handles one connection at a time.
On most of the systems, speed of both the MPMs is comparable but prefork uses more memory than worker.
Which one to use?
On high traffic websites worker is preferable because of low memory usage as comparison to prefork MPM but prefork is more safe if you are using libraries which are not thread safe.
For example you cannot use mod_php(not thread safe) with worker MPM but can use with prefork.
So if you are using all thread safe libraries then go with worker and if you are not sure then use default prefork MPM, you may have to increase your RAM in case of high traffic.
If you are on linux then run following command to check which MPM is on your machine
On Debian/Ubuntu:
apachectl -V | grep -i 'mpm'
apache2 -V | grep -i 'mpm'
If you receive gibberish output then Apache was started with init (which causes environment variable issues) instead of using apachectl start
Which should output:
Server MPM: prefork
On RHEL/Amazon AMI Linux:6
apachectl -V | grep -i mpm
httpd -V | grep -i 'mpm'
If you receive gibberish output then Apache was started with init (which causes environment variable issues) instead of using apachectl start
Which should output:
Server MPM: prefork
