Vault is unresponsive with SQL Error: 50000 shown in the log
Description
The Vault becomes unresponsive, and agents are no longer able to backup.
How to identify the problem
On the EPC Cloud Portal, under the Protection tab, multiple devices are showing Agent Errors.
On the Vault tab, the Vault alerts shows a warning on the Vault.
The Vault log file shows the following entry:
WARN 3812 --- [HttpClient@12496bd7-436] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 50000, SQLState: HY000
or
WARN 15884 --- [HttpClient@7c5de257-38770] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 50000, SQLState: HY000
Log phrase to look for: "SQL Error: 50000, SQLState: HY000"
What caused this problem?
This error happens because the Vault server is overloaded, and the operating system can not free up resources as it should. The problem is usually due to memory or disk performance issues, which can be caused by other applications on the server, back-end services, external services (like virtual machine backups), or the hosting environment being under stress.
How do I resolve this problem?
- Identify performance issue with Windows Resource Monitor, taking note of CPU activity, Memory in use and disk time and response.
- Investigate if other applications are running that could be competing for resources.
- Find out if there are any scheduled tasks that would use resources,
- Investigate the back-end hosting performance where the Vault server is running.
- Restart the Cibecs Endpoint Cloud Private Cloud Vault service in Windows.
Check if the problem has been fixed
If the problem has been fixed successfully, then the Vault will show online on the EPC Cloud Portal, and the Agent errors will drop over time as backups are completed.