Frequent crashes of Apache2 with HestiaCP? Monit automated monitoring and troubleshooting guide (with complete configuration)

Frequent Apache2 crashes or Monit auto-restart failures in HestiaCP environments? This article provides a practical guide to avoiding common pitfalls when monitoring Apache2 with Monit, deeply analyzing common issues such as PID path misalignment and permission blocking, and offering production-grade Monit automation configuration files. Master high-availability server maintenance techniques now and achieve second-level automatic recovery from failures!

The pitfalls I encountered while using Monit to monitor Apache2

Last Friday, the server gave me a Monit alert in the middle of the night.

I glanced at the panel in a daze, and in the apache2 status column, there was a red Timeout.

I thought about it for a bit. I just added Monit monitoring to the server during the day, and I copied and pasted the configuration from an online tutorial. There shouldn't be any problems, right?

The next morning, it timed out again. After the third time, Monitor simply gave up, and the panel displayed "Not monitored".

I. . .

I admit, I didn't take it seriously at first. Apache2 monitoring? You can find tons of template configurations online, just copy and paste. But that pasting process really made me furious.

The root cause of the conflict between HestiaCP's default architecture and Monit ports

Let me first show you the configuration that caused me so much trouble, so you can see if it's exactly the same as the version you've seen.

check process apache2 with pidfile /var/run/apache2/apache2.pid
    start program = "/usr/sbin/service apache2 start"
    stop program  = "/usr/sbin/service apache2 stop"
    if failed host 127.0.0.1 port 80 protocol http then restart
    if 5 restarts within 5 cycles then timeout

It seems fine, right? It checks port 80, and if it crashes, it restarts. If it still crashes after 5 restarts, it times out.

The problem is, your Apache2 isn't even running on port 80.

This is a pitfall of HestiaCP, and the root cause of many people falling into it. HestiaCP's default architecture is a reverse proxy of Nginx + Apache2, with Nginx occupying ports 80 and 443 in front, and Apache2 running on the local port 8081 in the back.

If you ask Monit to probe the liveness of Apache2 on port 80, it's like going to McDonald's to find KFC. The server looks at you blankly, and you two stare at each other. In the end, Monit determines that you're down and starts frantically restarting.

After restarting, the port is still 8081. Monit then tries to probe port 80, which also fails, so it restarts again. This cycle repeats until Monit decides it's beyond repair and times out.

When I first encountered this, I was genuinely stunned. Nine out of ten tutorials I found online used port 80. If you followed them, the problem wasn't with you, but with the source of the information itself.

A corrupted Apache2 PID file caused Monit to mistakenly identify the process as nonexistent.

After changing the port from 80 to 8081, Monit should theoretically be able to detect it, right?

However, in reality, it still occasionally reports "Execution failed ".

After struggling for a long time, I finally discovered the reason was simple: the PID file was corrupted.

Think about it, Monit was frantically restarting Apache2, each time forcibly killing and restarting it, going back and forth several times. During this process, the file /var/run/apache2/apache2.pid might become 0 bytes.

In other words, the file is still there, but it's empty.

When Monit reads this file, it finds nothing. It doesn't recognize your Apache2, even if your Apache2 is running perfectly fine in the background; Monit doesn't think the process exists.

When I saw this, I was speechless for a moment.

This is a deadlock. Monit fails to detect the Apache 2 instance, restarts Apache2, corrupts the PID file during the restart process, fails the next detection, and restarts again. This cycle continues until the timeout occurs.

Troubleshooting and Repair Steps for Apache2 Monitoring in HestiaCP Environment

To be honest, the investigation process isn't complicated, but you need to know which direction to investigate.

The first step is to determine which port your Apache2 is listening on. Simply type a command in the terminal.

netstat -tulpn | grep apache2

Alternatively, you can use the `ss` command; the effect is the same.

ss -tulpn | grep apache2

You will see output similar to this.

tcp  0  0 127.0.0.1:8081       0.0.0.0:*  LISTEN  2942372/apache2

It's confirmed to be 8081, not 80. That's the root of the problem.

The second step is to repair the corrupted PID file. This is simpler.

monit unmonitor apache2
systemctl restart apache2
cat /var/run/apache2/apache2.pid

First, pause Monit monitoring to prevent it from interfering while you're fixing things. Then, restart Apache2 to allow it to rewrite a clean PID. Finally, use `cat` to check the file content; it should contain a string of numbers, not an empty string.

Once this step is completed, the problem is basically solved.

Comparative Analysis of Monit Traditional Adaptive and Aggressive Protective Configurations

Online tutorials on configuring Apache2 with Monit generally fall into two categories.

One type is the "traditional adaptation type," which uses the `service` command to manage services and check local ports without adding too many complicated restrictions. This configuration can be used on HestiaCP by simply changing the port, and it is relatively stable.

Another approach is the "aggressive protection" method, which uses systemctl to manage services, adds child process restrictions, and employs stricter detection logic. It looks great, but it has a fatal flaw: the stop command used is `killall -9`.

What does `killall -9` mean? It means to forcibly kill the device regardless of what it's doing. This brute-force operation can easily leave behind corrupted PID files, which is the problem I just mentioned.

My personal experience is that limiting the number of child processes in an aggressive configuration is indeed useful. When your Apache2 is overwhelmed by a CC attack, limiting the number of child processes can prevent the server from running out of memory. However, the `killall -9` approach is truly unusable.

So in the end I compromised and combined the advantages of both configurations.

HestiaCP Apache2 Monit Best Practice Configuration

Modify the file /etc/monit/conf.d/apache2 with the following content.

check process apache2 with pidfile /var/run/apache2/apache2.pid
    start program = "/bin/systemctl start apache2"
    stop program  = "/bin/systemctl stop apache2"
    if children > 120 for 2 cycles then restart
    if failed host 127.0.0.1 port 8081 protocol http for 2 cycles then restart
    if 5 restarts within 10 cycles then timeout

Let me briefly explain the logic behind these few lines of configuration.

Write port 8081 to precisely match HestiaCP's reverse proxy architecture; stop foolishly writing port 80.

Use the `systemctl stop` command instead of `killall -9` to stop the PID file, so as not to corrupt it.

A child process limit has been added: if the number of children exceeds 120, the process will restart after two consecutive cycles to prevent CC attacks, but it's not too aggressive.

The logic for detecting failures has been modified to use a "for 2 cycles" approach, meaning a restart is only triggered after two consecutive failures, reducing false positives. The previous configuration, which restarted after just one detection, was frankly a bit overly sensitive.

The final timeout threshold is relaxed to 5 restarts within 10 cycles, leaving sufficient fault tolerance.

HestiaCP Monit monitoringConfiguration Troubleshooting Summary and Experience Sharing

After making the configuration changes, I monitored apache2, and the panel finally showed a green "OK" indicator.

How to describe my feelings at the time? It was like spending two days struggling with a bug, only to find out the cause was a single line of configuration that was wrong. It was both frustrating and laughable.

Monit is a good thing in itself, and monitoring daemons is something every server should do. But the problem is that many online tutorials are based on the assumption that "Apache2 exclusively uses port 80," while HestiaCP uses a reverse proxy, which means this assumption doesn't hold true.

If you follow the instructions, the problem isn't you; it's that the tutorial is applicable to a different scenario than yours.

So if you're also using HestiaCP and messing around with Monit to monitor Apache2, just remember two things: Change the port to 8081, and use the `systemctl` command to stop it, not `killall -9`. If you do these two things, you should be able to avoid any more problems.

Since you've read this far, if you found it helpful, please like and share it. If you want to receive updates first, you can also follow me!

Thank you for reading my article. See you next time.