vmware’s vma 4.1 and splunk working together

So I stumbled across a post by Chris Chadwell on setting up vmware syslog and splunk on vMA (vmware managemnet appliance) and thought hell yeah..

So here is my twist on this gem of an idea.

This should not take long at all provided a few things are done first.

  1. Know a spare IP and your hostname, Subnet mask and Gateway that can access your ESX(i) hosts, or Vcenter.
  2. Have vma 4.1 dowloaded and extracted from the iso so you have the ovf available.
  3. Grab a redbull, mother, monster, power juice, boost to get yourself full throttle.

The end goal is to have the dashboard below for your environment.

– without the errors of course.

A good environment… still some work can be done to clean the errors, mainly ha errors.

splunkdashboard-good

A bad environment… – keeping in mind that in the following screen we would get some alarms every now and then but other than that it seemed to be working… the people who administered this environment thought it was fine until I showed them this dashboard.

splunkdashboard-bad

The two methods…

Automated path…

    • Download vma from vmware
    • Unzip with your weapon of choice, 7zip is my fav at the moment.
    • Logon to your vcenter
    • Choose a suitable location – if you are a junior admin or it is not your environment get permission!
    • File, deploy OVF template
    • Browse to the extracted folder from step 2 and select vMA**.ovf
    • Next
    • Next
    • Accept – after reading of course
    • Next
    • Enter hostname
    • Choose Folder
    • Next
    • Choose Resource Pool
    • Next
    • Choose Datastore
    • Next
    • Thick or Thin your call
    • Next
    • Choose network vlan
    • Next
    • Finish
    • Right click and upgrade hardware – this allows you to specify the below setting and get vmxnet3
    • Edit virtual machine, Options, change Other Linux to Other 2.6x Linux (32-bit)
    • Remove network
    • Increase Memory – I allocated 2Gb
    • Add extra disk 25 Gb – can be same scsi adapter or separate.
    • Add extra disk 25 Gb – can be same scsi adapter or separate.
    • Close
    • Edit virtual machine
    • Add network device vmxnet3 and select right vlan
    • Power on.
    • Set IP
    • Set password
    • open putty session
    • open winscp session and copy the following three files to /tmp on your vma server
    • vmasplunk.sh
    • vmasplunk.zip
    • VMTools – of course if you get your vmtools you can copy that in, the scripts should pick it up
    • on your putty session enter

sudo -s

chmod a+x /tmp/vmasplunk.sh

./vmasplunk.sh

  • Now sit back and watch the disks get setup and formated, vmtools-open removed and vmtools installed, splunk installed, conf files uploaded to spunk for views and searches
  • You will be asked to say Y to the license agreement, so space space space and then y, enter
  • Whilst in your putty sesssion enter “vifp addserver ESXHOST.FQDN –username root –password bunny” for each host you want to monitor
  • If you have a large number of hosts I suggest downloading this excel spreadsheet I created here
  • Then enter “vilogger enable –server ESXHOST.FQDN –numrotation 10 –maxfilesize 10 –collectionperiod 30” to start the logs flowing
  • Open your browser and enter the vma ip address or dns
  • logon with Admin and whackdiddy
  • Change the password
  • Click App in the top right corner and search
  • Then click dashboards and select “vSphere View”
  • The only thing left is to shutdown vma and change the scsi device to paravirtual and you are done.

The manual path

  1. Download vma from vmware
  2. Unzip with your weapon of choice, 7zip is my fav at the moment.
  3. Logon to your vcenter
  4. Choose a suitable location – if you are a junior admin or it is not your environment get permission!
  5. File, deploy OVF template
  6. Browse to the extracted folder from step 2 and select vMA**.ovf
  7. Next
  8. Next
  9. Accept – after reading of course
  10. Next
  11. Enter hostname
  12. Choose Folder
  13. Next
  14. Choose Resource Pool
  15. Next
  16. Choose Datastore
  17. Next
  18. Thick or Thin your call
  19. Next
  20. Choose network vlan
  21. Next
  22. Finish
  23. Upgrade hardware
  24. Edit virtual machine, Options, change Other Linux to Other 2.6x Linux (32-bit)
  25. Remove network
  26. Increase Memory – I allocated 2Gb
  27. Add extra disk 25 Gb – can be same scsi adapter or separate.
  28. Add extra disk 25 Gb – can be same scsi adapter or separate.
  29. Close
  30. Edit virtual machine
  31. Add network device vmxnet3 and select right vlan
  32. Power on.
  33. Set IP
  34. 10.1.213.112 255.255.255.0 10.1.213.1
  35. cp /etc/DIR_COLORS ~/.dir_colors
  36. Vi ~/.dir_colors
  37. #Change
  38. DIR 01;34 #directory
  39. #To
  40. DIR 01;33 #directory
  41. Logoff
  42. Logon
  43. Cd /
  44. ls
  45. # you should now see yellow directories and not dark bloody blue
  46. yum remove vm-
  47. Winscp linux vmtools from linux iso to vma03
  48. Tar -zxf VM
  49. Cd vmware-tools-distrib
  50. Answer defaults… enter enter enter
  51. Shutdown -h now
  52. Change SCSI controller to paravirtual
  53. Power on
  54. Fdisk -l
  55. Fdisk /dev/sdb
  56. N
  57. P
  58. 3
  59. Mkfs -t /dev/sdb1
  60. Mkdir /var/vmlogs
  61. Mount /dev/sdb1 /var/logs/vmware
  62. Vi /etc/init.d/fstab
  63. Add “/dev/sdb1 /var/vmlogs ext3 default 0 1”
  64. For each host you wish to monitor
  65. vifp addserver <hostname or ip of ESX/i server>
  66. For each host you wish to capture logs from
  67. Vilogger enable –server hostname or ip of ESX/i server –numrotation 10 –maxfilesize 10 –collectionperiod 10
  68. Edit vilogger for location of files.
  69. Vi /etc/vmware/vMA/vMA.conf
  70. Download the latest Splunk here. Make sure you pick the Linux distribution (32 bit)
  71. Copy the file to the vMA, I used WinSCP as I was using my Windows 7 machine.
  72. Get yourself a root bash prompt
  73. Sudo bash
  74. rpm -i splunk-xxxxx-.rpm
  75. /opt/splunk/bin/splunk start
  76. /opt/splunk/bin/splunk set web-port 80
  77. /opt/splunk/bin/splunk restart
  78. /opt/splunk/bin/splunk enable boot-start
  79. /opt/splunk/bin/splunk edit user admin -password whackdiddy -roles admin -auth admin:changeme
  80. open ie, firefox, chrome (your weapon of choice) http:\\ip-address or dns\
  81. Logon with admin and [passoword set at 40]
  82. Click on manager
  83. Data inputs
  84. Files and directories
  85. New
  86. Skip preview
  87. Enter path
  88. /var/vmlogs/vmware/*/hostd.log
  89. More settings
  90. Change Set Host to Segment in path
  91. Segment number = 4
  92. Whitelist = blank unless you choose not to specify hostd.log
  93. vma/ vma
  94. In the top right hand corner you will see App with a drop down arrow.
  95. Click the drop down and press search or on the left had side press search
  96. Paste this into the search bar
  97. error OR failed OR severe OR ( sourcetype=access_* ( 404 OR 500 OR 503 ) ) | timechart count by host usenull=f useother=f
  98. Click on the chart view (like mobile reception indicator on ya phone) and you should have a nice chart. Change the formating options to line combined connect.
  99. And we are done… Congratulations

20130123-175949.jpg