ESXi 6 hangs/freezes up

Troubleshooting is so fun when you made tens of changes at once haha.  So my ole' trusty ESXi 6 HPz800 machine started to hang up after a day or so of operation.  Looking into the ESXi logs, they are not much help as you find what you would think are red alarms, but after lots of googling they seem to be normal logs.  But skipping over to vmdkwarning logs and I got a lot of (copied and pasted from another forum member's issues, too lazy to SSH into the system again):

Lost access to volume

4bcce772-3bfe7a35-dceb-001b21541d90 (1_5WD

1) due to connectivity issues. Recovery attempt

is in progress and outcome will be reported

shortly.

And then:

Successfully restored access to volume 4bcce772-

3bfe7a35-dceb-001b21541d90 (1_5WD_1_)

following connectivity issues.

info

4/21/2010 1:16:16 PM

Then according to a few forum posts, it might be HDD failing, or overheating, or get in and tweak heart beat settings (IMO the latter a band-aid to symptoms of something needing to get fixed).

In denial that it is already time to replace my WD Black 1TB I have the datastore on, I thought about what things have changed on my system:

- put some buffer material along my HDDs in software RAID to reduce noise, but this could be reducing their airflow too

- pulled out a Hauppauge HVR card from a defunct MythTV build

- pulled out 24 gigs of 'original' RAM and put in 48 gigs of Ebay ECC RAM

Software side

- upgraded to FreeNAS 10

- Ubuntu server with Plex media server, fstab FreeNAS10 CIF of Movies share

- New Splunk server on CentOS7

Oh joy, plenty to look into.  But after lots of forums and poking around I was finding the Ubuntu Plex server to be getting hung up, particularly the kswapd process eating all of the resources- typical Linux forums leads one on a chase is it the kernel?  Kind of known yet unknownish bug? You the installer is just dumb etc.  What I settled on is I didn't give the Ubuntu server enough RAM to run plex effectively (though I watched two movies without issue, it spins out of control randomly at idle)- hopefully that is the fix, or else I will just have to start over on a CentOS server build (Ubuntu is showing it's desktop user bias as it has not been good at 'services' jobs like running splunk server or now in this case Plex).  This thread doesn't give me much optimism:

http://serverfault.com/questions/316560/how-do-i-tell-what-process-is-causing-kswapd-to-be-in-use/316636

Mounting FreeNAS share to Ubuntu and installing Plex on Ubuntu

The notes are for the goal of installing Plex into the headless Ubuntu server repository so 1) its easy to install and b) updates are easy- then install Plex, then mount the FreeNAS SMB share to the Plex server running in Ubuntu.  Updates to this thread should be made over time after getting the setup to 'work' to then work towards security.  For now, its just within the NAT.

Then something I find a lot harder to do (and scary from past instances of bricking machines), editing fstab to mount drives or shares in Linux.  Why Linux do you make this so hard?

So to get Plex going on Ubuntu server-

Add Plex to your repo for updates (I thought this would be of use to install as well, but I guess not)

echo deb https://downloads.plex.tv/repo/deb/ public main | sudo tee /etc/apt/sources.list.d/plexmediaserver.list

curl https://downloads.plex.tv/plex-keys/PlexSign.key | sudo apt-key add -

After that, it's just a matter of running the normal sudo apt-get update and the Plex Media Server repo will be enabled on the OS.

Then SSH into your server (ensure you have installed SSH) and in my case type: $ wget https://downloads.plex.tv/plex-media-server/1.4.4.3495-edef59192/plexmediaserver_1.4.4.3495-edef59192_amd64.deb

Then sudo dpkg -i plexmediaserver_1.4.4.3495-edef59192_amd64.deb

To prep Ubuntu to mount the samba/CIF share:

sudo apt-get install rpcbind nfs-common

sudo apt-get install cifs-utils

______

Now the fun part, mounting my FreeNAS share of movies to the Plex server running on Ubuntu server before logging into the Plex webUI to configure the library (this is all on a single box running ESXi).

Getting the path to the FreeNAS share-

my FreeNAS IP is 192.168.*.***, target is /mnt/ZFS/movies, but come to find out that path FreeNAS shows is not how you should go about this, I used smbclient

to poke around with random path tries until seeing its simply FreeNAS IP / share name.  With that info, nano fstab time following this format:

//<IP address of NAS box 192.168.x.y>/<shared folder> /<mount point> cifs guest,_netdev,uid=<your user name on Linux box> 0 0

More info from the site that helped- the end goal is to make it persistent with reboots and secure.  I am running this within a NAT and do not have the IPs locked down as static, so I will need to put in work to remedy this.
Helpful site:  https://wiki.samba.org/index.php/Mounting_samba_shares_from_a_unix_client

CentOS7 Server, Splunk, Homemonitor

After installing CentOS7 Live with GNOME, i finalized the install, then
yum update (as root).

After the update, installed Splunk 6.5 (download the RPM package on Splunk's website), install:
Downloads$ rpm -i splunk_package_name.rpm

Then needed to open up the firewall for some ports

firewall-cmd --permanent --zone=public --add-port=8000/tcp (this is for the webUI access)

firewall-cmd --permanent --zone=public --add-port=8089/tcp (this is for managing forwarders)

firewall-cmd --permanent --zone=public --add-port=9997/tcp (this is for data input from forwarders)

firewall-cmd --permanent --zone=public --add-port=514/udp (this is for the router's syslog to feed into splunk as an input)

Install homemonitor after installing Splunk via the app manager.  Follow the prompts for the setup, you may have to go into data inputs, add, port 514 and in my case ensure source is set to Asus.

Profit.