UEC: Troubleshooting
Eucalyptus Log files
The following log files help in debugging issues encountered while working with Eucalyptus:
Eucalyptus Component Log file(s) CC cc.log, httpd-cc_error_log, cc-registration.log NC nc.log, httpd-nc_error_log, euca_test_nc.log CLC cloud-debug.log, cloud-error.log, cloud-output.log, axis2c.log SC sc-stats.log, sc-registration.log WS3 walrus-stats.log, walrus-registration.log
The logging level is controlled by the LOGLEVEL macro in eucalyptus.conf of the respective component. The log levels are DEBUG, INFO, WARN, ERROR, and FATAL (in descending order of verbosity). The default level is DEBUG (everything).
‘tail -f ‘ on the log files is a good way to understand what is happening with the components of Eucalyptus.
Linux commands
Linux comamnds like ps, top etc. run on CC and NC are helpful to understand the state of the processes related to Eucalyptus.
Libvirt tools
NC launches the instances by making libvirt calls to launch a VM. You can use libvirt tools like virsh for troubleshooting any hypervisor related issues.
Launching an instance with VNC attached(HACK):
If the output of euca-describe-instances shows the status of an instance as ‘running’, but if you are not able to SSH into it or RDP into it, it is is tough to find out where the failure is. In such cases, a VNC interface to the instance is generally of great help to troubleshoot further. But Eucalyptus does not provide a way of connecting to the instance through VNC. The following hack can help in setting up a VNC interface to the instance. Windows Instances
On the NC, edit /usr/share/eucalyptus/gen_kvm_libvirt_xml (if KVM is the hypervisor) to change
<devices> <disk type='file'> <source file='BASEPATH/disk'/> <target dev='sda'/> </disk> <interface type='bridge'> <source bridge='BRIDGEDEV'/> <mac address='PRIVMACADDR'/> <model type='e1000'/> </interface> <serial type="file"> <source path='BASEPATH/console.log'/> <target port='1'/> </serial> </devices>
to
<devices> <disk type='file'> <source file='BASEPATH/disk'/> <target dev='sda'/> </disk> <interface type='bridge'> <source bridge='BRIDGEDEV'/> <mac address='PRIVMACADDR'/> <model type='e1000'/> </interface> <serial type="file"> <source path='BASEPATH/console.log'/> <target port='1'/> </serial> <graphics type='vnc' listen='A.B.C.D' port='5904'/> </devices>
where A.B.C.D is the IP address of the interface on NC that is reachable from the client machine where you want to run the vnc client. We assume that your NC has 2 interfaces and you are able to use the second interface to reach it without going through CC. Again, this is only for troubleshooting. In a production setup, you may have the NC connected to the outside world only through CC.
With these changes, KVM adds the “-vnc” option to the command with the specified NC’s ip address on display number 4.
To connect to the instance through VNC client from a different machine, use the NC’s ip address and display number 4. Ex: A.B.C.D :4
In the case of Xen, you would modify the relevant sections of /usr/share/eucalyptus/gen_libvirt_xml. We have not tested with Xen, though.
Linux Instances
You can use the same approach for Linux instances as well, with the caveat that you will not see any output with euca-get-console-output as console output does not get written to /var/lib/eucalyptus/instances/<user>/i-xxxxxxxx/console.log. The output will, of course, be displayed though VNC.
NOTE: This works only if we are debugging one instance at a time. If you attempt to start another instance while one instance is already running, KVM will try to attach the same VNC display 4 to the second instance and will fail with the following error in /var/log/libvirtd/qemu/i-XXXXXXXX.log
inet_listen_opts: bind(ipv4,A.B.C.D,5904): Address already in use inet_listen_opts:FAILED