ESXi 6 hangs/freezes up

Troubleshooting is so fun when you made tens of changes at once haha.  So my ole' trusty ESXi 6 HPz800 machine started to hang up after a day or so of operation.  Looking into the ESXi logs, they are not much help as you find what you would think are red alarms, but after lots of googling they seem to be normal logs.  But skipping over to vmdkwarning logs and I got a lot of (copied and pasted from another forum member's issues, too lazy to SSH into the system again):

Lost access to volume

4bcce772-3bfe7a35-dceb-001b21541d90 (1_5WD

1) due to connectivity issues. Recovery attempt

is in progress and outcome will be reported

shortly.

And then:

Successfully restored access to volume 4bcce772-

3bfe7a35-dceb-001b21541d90 (1_5WD_1_)

following connectivity issues.

info

4/21/2010 1:16:16 PM

Then according to a few forum posts, it might be HDD failing, or overheating, or get in and tweak heart beat settings (IMO the latter a band-aid to symptoms of something needing to get fixed).

In denial that it is already time to replace my WD Black 1TB I have the datastore on, I thought about what things have changed on my system:

- put some buffer material along my HDDs in software RAID to reduce noise, but this could be reducing their airflow too

- pulled out a Hauppauge HVR card from a defunct MythTV build

- pulled out 24 gigs of 'original' RAM and put in 48 gigs of Ebay ECC RAM

Software side

- upgraded to FreeNAS 10

- Ubuntu server with Plex media server, fstab FreeNAS10 CIF of Movies share

- New Splunk server on CentOS7

Oh joy, plenty to look into.  But after lots of forums and poking around I was finding the Ubuntu Plex server to be getting hung up, particularly the kswapd process eating all of the resources- typical Linux forums leads one on a chase is it the kernel?  Kind of known yet unknownish bug? You the installer is just dumb etc.  What I settled on is I didn't give the Ubuntu server enough RAM to run plex effectively (though I watched two movies without issue, it spins out of control randomly at idle)- hopefully that is the fix, or else I will just have to start over on a CentOS server build (Ubuntu is showing it's desktop user bias as it has not been good at 'services' jobs like running splunk server or now in this case Plex).  This thread doesn't give me much optimism:

http://serverfault.com/questions/316560/how-do-i-tell-what-process-is-causing-kswapd-to-be-in-use/316636

Mounting FreeNAS share to Ubuntu and installing Plex on Ubuntu

The notes are for the goal of installing Plex into the headless Ubuntu server repository so 1) its easy to install and b) updates are easy- then install Plex, then mount the FreeNAS SMB share to the Plex server running in Ubuntu.  Updates to this thread should be made over time after getting the setup to 'work' to then work towards security.  For now, its just within the NAT.

Then something I find a lot harder to do (and scary from past instances of bricking machines), editing fstab to mount drives or shares in Linux.  Why Linux do you make this so hard?

So to get Plex going on Ubuntu server-

Add Plex to your repo for updates (I thought this would be of use to install as well, but I guess not)

echo deb https://downloads.plex.tv/repo/deb/ public main | sudo tee /etc/apt/sources.list.d/plexmediaserver.list

curl https://downloads.plex.tv/plex-keys/PlexSign.key | sudo apt-key add -

After that, it's just a matter of running the normal sudo apt-get update and the Plex Media Server repo will be enabled on the OS.

Then SSH into your server (ensure you have installed SSH) and in my case type: $ wget https://downloads.plex.tv/plex-media-server/1.4.4.3495-edef59192/plexmediaserver_1.4.4.3495-edef59192_amd64.deb

Then sudo dpkg -i plexmediaserver_1.4.4.3495-edef59192_amd64.deb

To prep Ubuntu to mount the samba/CIF share:

sudo apt-get install rpcbind nfs-common

sudo apt-get install cifs-utils

______

Now the fun part, mounting my FreeNAS share of movies to the Plex server running on Ubuntu server before logging into the Plex webUI to configure the library (this is all on a single box running ESXi).

Getting the path to the FreeNAS share-

my FreeNAS IP is 192.168.*.***, target is /mnt/ZFS/movies, but come to find out that path FreeNAS shows is not how you should go about this, I used smbclient

to poke around with random path tries until seeing its simply FreeNAS IP / share name.  With that info, nano fstab time following this format:

//<IP address of NAS box 192.168.x.y>/<shared folder> /<mount point> cifs guest,_netdev,uid=<your user name on Linux box> 0 0

More info from the site that helped- the end goal is to make it persistent with reboots and secure.  I am running this within a NAT and do not have the IPs locked down as static, so I will need to put in work to remedy this.
Helpful site:  https://wiki.samba.org/index.php/Mounting_samba_shares_from_a_unix_client

CentOS7 Server, Splunk, Homemonitor

After installing CentOS7 Live with GNOME, i finalized the install, then
yum update (as root).

After the update, installed Splunk 6.5 (download the RPM package on Splunk's website), install:
Downloads$ rpm -i splunk_package_name.rpm

Then needed to open up the firewall for some ports

firewall-cmd --permanent --zone=public --add-port=8000/tcp (this is for the webUI access)

firewall-cmd --permanent --zone=public --add-port=8089/tcp (this is for managing forwarders)

firewall-cmd --permanent --zone=public --add-port=9997/tcp (this is for data input from forwarders)

firewall-cmd --permanent --zone=public --add-port=514/udp (this is for the router's syslog to feed into splunk as an input)

Install homemonitor after installing Splunk via the app manager.  Follow the prompts for the setup, you may have to go into data inputs, add, port 514 and in my case ensure source is set to Asus.

Profit.

Install Conky in Ubuntu

So I failed miserably at following Kantoolin instructions to intall Kali tools in Ubuntu, so I figured to get something done today, to install conky (system monitor), get rid of teh flickering it had, and create a file so that it could be displayed as if its part of your wallpaper.

First I followed the instructions found in Ubuntu's resources:  https://help.ubuntu.com/community/SettingUpConky

Then ran into the fluttering issue, so started to follow instructions on this other site: http://conky.sourceforge.net/faq.html

And for a little salt and pepper, read over some of Arch Linux's resources as well:  http://conky.sourceforge.net/faq.html

Greddy EMU in Tacoma Truck (5vz-fe with TRD Supercharger)

I have a 2001 TRD 4x4 ext. cab Tacoma with TRD supercharger.  I bought it used with 200K on the odometer, the original owner put the S/C at 30K miles, up to this point it had no fuel mods-- yes, the 5vzfe is a fricken tank of a motor that seems to laugh at ping- but I don't like ping.

I didn't want to run the off-the-shelf solution that was the standard at the time, URD kits based off of split second piggybacks.  I had been looking at AEM FICs and Greddy EMUs and decided the AEM FIC has the features the 5vz-fe needed, namely the o2 sensor scewing feature set as the 5vz has a lot of closed loop boost.

Long story short, the FIC was a real pain to early adopt.  AEM themselves needed to make an updated firmware for the o2 sensor scewing circuit to have the ranges to work on the Toyota sensors, and the cam and crank circuit seems to not have much noise filtering, many Toyota installs require bridge resistors of around 2.2K to ground to pull down the noise so the FIC doesn't give false signals to the ECU.

Two things made me finally move away from the FIC though:
- A tuner in Canada that posted dyno tested results of the FIC not having the same timing retard accuracy as the Greddy EMU.  The FIC simply intercepts and offsets the cam and crank signal, the EMU intercepts the cam and crank, but also the IGT signals themselves.
- the FIC did not have a auto shifting timing retard feature-- I was getting ping when the tranny shifted.  The EMU does have a feature to retard timing during shifts (as the OE ECU advances timing during shifts, thus the ping).
- I use E85 from time to time.  The FIC has a map switching feature, and the EMU can be modded to do the same, but the EMU can advance timing, the FIC can only retard timing- score one more for the EMU to get more power out of E85, more MPGs out of 91 octane in vacuum (OE engine built to run 87 and the OE knock sensor is not used under 3K RPM so the OEM ECU does not optimize vacuum timing).

But the reason I didn't go with the EMU to begin with is that it DOES NOT have any o2 correction features/wiring.  But I wanted the piggy that didn't have finicky cam/crank circuits, more power, and accurate timing retard.  So I installed the EMU but to shore up the crucial o2 feedback requirement, also wired in a Split Second enricher.

So far its a great success, I have great start-up (the AEM FIC seemed to have to sync first before allowing cam/crank to pass through), no more little electrical gremlins the FIC seemed to have, and overall less ping with the same timing retard settings so I feel it really is more effectively retarding timing in boost.  I will update when I run E85 and further refine the tune.

Raspberry Pi3 self hosting

I had been following the Pi hype for a while, since the first models hit the market, but I did not have a use case for buying one, until I wanted to stop using an ecommerce that was charging large sums every month to host a very modest sales site of mine-- bam, Pi use case found.

Searching around the interwebz I found this amazing tutorial/write-up to setup a Pi to be an internet facing server- seriously, just an amazing a-z guide to not only getting a Pi spun up to be a webserver and email server, but with good security best practices and additions thrown into it-- but I ran into some issues-

- do not get rid of root as a login option, there are installs that need root despite using 'sudo' with the account you create and having added all known groups to said account.

- ran into this for the iptables section

libkmod: ERROR ../libkmod/libkmod.c:554 kmod_search_moddep: could not open moddep file '/lib/modules/3.6.11+/modules.dep.bin'
iptables v1.4.14: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

fix that issue with updating rpi with 'rpi-update' (did not have to type sudo, I su into root)

I was able to SSH login via username and password in Putty despite the mentioned instruction to install keys and supposedly disable password login- so I followed the linked guide to take a few more steps to make it so only a key method can be used to login.

  -- Towards the "make your own website" portion, I started to deviate from the guide as I wanted to make a wordpress site, so I started to follow THIS guide with some differences to the directories mentioned.

After all of this, I just could not get nginx to work, I could not get a 'hello world' html to show up with my pi's IP address, try as I did.  I gave up, sudo apt-get purge nginx to go with the more established (meaning= lots of tutorials) apache.  Installed Apache using the Raspberry official documentation and finally had a website to view in my NAT.

To setup my router to forward the necessary ports- WAN > Virtual Server / Port Forwarding

Last but not least, once you have your wordpress instance up and running, you will quickly find your pi part of a xml-rpc bot net-- when in your wordpress admin page, go to plugins, install "disable xml-rpc".

When your free Splunk runs out

So tisk tisk, I didn't log into splunk for a long time, didn't convert from the Enterprise trail to free license before the cutoff and got locked out of doing any searches in splunk.

So I had to do what many Splunk plebes must do- backup your database, nuke splunk, start over, re-import database and settings.

Thankfully this guy Nick had an awesome tutorial in his blog on how to do so: URL https://techblog.jeppson.org/2015/03/fix-splunk-lockout-after-exceeded-quota/