Last week our customer Florida Institute of CPAs had an issue with a production iSCSI VMware datastore going offline. Rescanning storage brought the volume back online, but we needed to find the cause so we could prevent a recurrence.
Unfortunately, by the time we got VMware support on the case, the VMware logs had already been overwritten.
VMware ESXi automatically rotates and overwrites logs on internal storage. This is by design, so as not to fill up the limited space on the default volume containing the log files.
They needed a better way to collect and retain ESXi system logs.
Collect system logs with syslog
Syslog is a standard method of log collection used by a variety of systems and devices. It allows for collection of logs into a centralized repository for retention and analysis.
VMware ESXi has a built-in syslog feature. This allows you to capture logs from ESXi hosts to a syslog server for longer-term retention and analysis.
If you need to capture logs from VMware ESXi host servers for longer-term retention and analysis, you can enable remote logging via the built-in syslog feature. The method for doing this on VMware ESXi hosts is revealed in VMware knowledge base article 1016621, “Enabling syslog on ESXi”.
To locate the article manually (in case the above link doesn’t work), browse to kb.vmware.com and enter KB article ID 1016621 in the field labeled “View by Article ID”.
Simple VMware ESXi syslog configuration
While several configuration options are offered by the knowledge base article, I found the simplest method was to specify the syslog server for each host using the VI client. (This works best for one or a few hosts.)
You will need to know the hostname or IP address of your syslog server, and its listening port (if other than the default port 514). Follow these steps to configure ESXi to log to your syslog server.
- Log in to your VI client and select the host you wish to configure from the inventory in the left pane.
- With the host selected, click the Configuration tab in the right pane.
- From the configuration menu on the left side, under Software, select Advanced Settings.
- In the Advanced Settings dialog, select Syslog on the left side.
- Enter the hostname or IP address of your syslog server in the field labeled Syslog.Remote.Hostname, e.g.
logserver.mydomain.comor192.168.1.105. - If your syslog server listens on a port other than the default of 514, enter the port number in the field labeled Syslog.Remote.Port.
- Click OK to save your changes.
VMware ESXi setting Syslog.Remote.Hostname
After enabling the remote syslog feature in VMware ESXi, make sure to check your syslog server to verify the logs are being collected. You may also want to set your retention period for logs to ensure that you have sufficient history to trace longer-running or intermittent issues.
Other configuration options
The VMware knowledge base article also details methods of enabling remote syslog via vCLI and PowerCLI. If you need to enable this feature on many hosts, using one of these scriptable methods might prove a better option.
The command for enabling remote syslog using the vCLI will look something like this:
vicfg-syslog.pl --server esxhost1 --username root --password MyPassword --setserver logserver.mydomain.com --setport 514For PowerCLI, you would use something like this:
Connect-VIServer HostnameOrIP
Get-VMHost esxhost1 | Set-VMHostSysLogServer -SysLogServer logserver.mydomain.com -SysLogServerPort 514In each case, the command should be all on one line. Adjust the values to suit your environment, and verify success by checking for logs on your syslog server.
Syslog servers
There are many syslog server options, both open source and commercial. Any standards-compliant syslog server should be able to serve as a repository for log collection from remote devices.
If you run an all-Windows shop, you could use something like the free Kiwi Syslog Server from SolarWinds. Paid versions are available with additional features.
Linux and Unix machines typically come with syslogd for local logging installed by default. Generally you will just need to enable remote logging in your syslog config, permit incoming traffic on port 514, and restart syslogd.
A quick online search for e.g. enable remote syslog on [distribution] should yield helpful tips for configuring your distribution.
Shane Brooks, network admin for FICPA, uses OSSEC for log collection and analysis. In addition to log analysis, OSSEC offers security features such as file integrity checking, rootkit detection, and other features that can be helpful for organizations needing to maintain PCI compliance.
Whatever server option you you use, you now know how to configure your VMware ESXi hosts to send system logs to your syslog server.