Auditing FreeNAS CIF/SMB activity

Having setup Splunk to ingest pfSense router/firewall/snort IDS/IPS logs and my windows laptop logs, I have a lot of goals, so little skills:
- build upon already great splunk apps for the above logs
- One of those apps is for FreeNAS, and I want to also catch smb share activity
- Ingest Foxhound Raspi BRO logs

For FreeNAS, first off before even tackling the Splunk setup, I began searching how to setup auditing on FreeNAS to even create the SMB activity logs, and found this thread:

https://forums.freenas.org/index.php?threads/tutorial-add-full-logging-on-samba-shares-full_audit-freenas-9-3.13840/

Basically adding in the SMB edit part of the GUI:

full_audit:prefix = %u|%I|%m|%S
full_audit:failure = connect
full_audit:success = mkdir rename unlink rmdir pwrite
full_audit:facility = LOCAL5
full_audit:priority = NOTICE

A bit noisy, another config is:


full_audit:prefix = %u|%I|%m|%S
full_audit:failure = connect
full_audit:success = mkdir rename unlink rmdir
full_audit:facility = LOCAL5
full_audit:priority = NOTICE

More stuff about CIF/SMB logging:
http://a32.me/2009/10/samba-audit-trail/

I can't help but feel its outdated for v11 though.  I did the first part of the tutorial (to wait to see results before then worrying about log retention) and the results were not there.  Then when simply using the GUI to go to services > logging "normal", then I started to see user activity in var/log/samba4/log.smbd--- I'm not sure if both the tutorial and my last bit both have to be done, or if with v11 all one needs to do is set logging to "normal".

* note, freebsd freenas requires `clog -f` instead of `tail -f`, but in FreeNAS `tail -f` is used...

Never the less, the security auditor nerd in me rejoiced when seeing detailed activity of videos and pdf docs accessed from a linux laptop user.

Next is to get the Splunk FreeNAS TA working, and adding to it's dashboard panels that highlight smb file activity.

--- 4.17.2020 the sega continues ---

Getting data into Splunk:

There is a two-prong approach here- Splunk has an app for FreeNAS but I believe it uses a REST API to gain data, and the data is about the FreeNAS itself such as drive use, temps etc.

Having those logs are nice, but I want the SMB logs as well.  The issue in the earlier post about filling up internal drive space with log data might become a non-issue if the data gets spirited away and FreeNAS overwrites effectively (without having to be told to do so, we will see if some crash on audit fail happens or the data just dries up).

You can setup a syslog output from FreeNAS.  System > General > at the bottom will be a Syslog Server line.  I put my splunk host IP and went with port 9083.  Respectively, put this as a data input in Splunk and ensure the firewall allows 9083 traffice.

Success:

So next is to tell FreeNAS to grab those SMB logs and spirit them away via syslog as well, ontop of the generic general default syslog data (that maybe we want to blacklist later?)

This person's thread is a good start:

https://www.ixsystems.com/community/threads/samba-audit-logs-to-centralise-log-servers.79133/

Read how he/she fixed in a few posts later, and try to implement this.

```destination m_samba_audit { file("/mnt/ie/logs/smb/smb.log"); };
log { source(src); filter(f_local5); destination(m_samba_audit); flags(final); $
destination loghost { udp("192.168.3.42" port(5514) localport(514)); };
log { source(src); filter(f_info); destination(loghost); };```
But the mistake was putting "flags(final);"     Remove that part, and tailor the rest of the lines to match my environment =

"destination m_samba_audit { file("/var/log/samba4/log.smbd"); };

log { source(src); filter(f_local5); destination(m_samba_audit);

Syslog.conf if found in etc/

Looking at syslog.conf (copy below):

# $FreeBSD$
#
#       Spaces ARE valid field separators in this file. However,
#       other *nix-like systems still insist on using tabs as field
#       separators. If you are sharing this file between systems, you
#       may want to use only tabs as field separators here.
#       Consult the syslog.conf(5) manpage.
*.err;kern.warning;auth.notice;mail.crit                /dev/console
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err   /var/log/messages
security.*                                      /var/log/security
auth.info;authpriv.info                         /var/log/auth.log
mail.info                                       /var/log/maillog
lpr.info                                        /var/log/lpd-errs
ftp.info                                        /var/log/xferlog
cron.*                                          /var/log/cron
!-devd
*.=debug                                        /var/log/debug.log
*.emerg                                         *
# uncomment this to log all writes to /dev/console to /var/log/console.log
# touch /var/log/console.log and chmod it to mode 600 before it will work
#console.info                                   /var/log/console.log
# uncomment this to enable logging of all log messages to /var/log/all.log
# touch /var/log/all.log and chmod it to mode 600 before it will work
#*.*                                             /var/log/all.log
# uncomment this to enable logging to a remote loghost named loghost
#*.*                                            @loghost
# uncomment these if you're running inn
# news.crit                                     /var/log/news/news.crit
# news.err                                      /var/log/news/news.err
# news.notice                                   /var/log/news/news.notice
# Uncomment this if you wish to see messages produced by devd
# !devd
# *.>=notice                                    /var/log/devd.log
!ppp
*.*                                             /var/log/ppp.log
!*
include                                         /etc/syslog.d
include                                         /usr/local/etc/syslog.d

Noticed some commented out lines that look like when uncommented out, allow all logs to go over syslog.

---- Later ----

I noted making changes to etc/syslog.conf (or the cp of the file to syslog.conf.backup) didn't stick.  Turns out FreeNAS has the "real" riles somewhere else, the changes I make are in RAM I guess, it will not be persistent

The path to do so (and really break stuff) is /conf/base/etc
in there will be syslog-ng.conf

Also look up beadm.  Its a way to create a new boot environment.  Or in this case backup the current boot environment before editing stuff.

I input the following:

Now, time to edit /conf/base/etc/local/syslog-ng.conf to look at /var/log/samba4/log.smbd to syslog output.  (backup syslog.ng with cp syslog.ng syslog.ng.backup)

Taken from a forum post, need to change paths and names to match my environment:
"destination m_samba_audit { file("/mnt/ie/logs/smb/smb.log"); };
log { source(src); filter(f_local5); destination(m_samba_audit);

Mine should be more like:

"destination m_samba_audit { file("/var/log/samba4/log.smbd"); };

log { source(src); filter(f_local5); destination(m_samba_audit);

The difference in path is I believe that other poster has the samba logs going into a smb share foler first- then to be take from there to go to syslog. This is probably best practice for storage space and data retention concerns.

Buuuut...

Files in this path are unwritable even to root.  Is there a chmod to fix this?  Or a mount command?

The fix seems a bit heavy handed but the advice given was to upgrade from 11.2-U7 to 11.3-U2.  After the upgrade, the paths above will be writeable.



Manual here (11.7?  I'm on 11.2, is FreeNAS even on 11.7?)  is pretty interesting- has a section on enabling syslog-ng

https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-syslog.html

When you enable syslog in the WebUI, it appears it just enables ```syslog_ng_ebable="YES"``` and below that, ```nginx_enable="YES"```

This explains that the only logs into Splunk so far have nginx inside the raw data.

In theory, I need to edit rc.conf found in /conf/base/etc to have syslog-ng enabled, the stanza is
```syslogd_ebable="YES"```.  It is currently at default ```syslogd_enable="NO"```

Then edit the syslog-ng.conf in /conf/base/etc/local to have the direction and log stanzas calling out the samba log.


After the mods just like Linux, ```service syslogd restart```

Also, link on how to send test syslog traffic to check connectivity:
https://www.freebsd.org/cgi/man.cgi?query=logger&sektion=1&manpath=freebsd-release-ports

Default syslog-ng.conf for reference.  What my noob brain takes away is that even though destinations are not commented out, I am not seeing logs from those destinations when syslog is enabled because further down their respective 'log' lines of the path are commented out (though the log specifications below are not, I assume without context as to what destination they apply to it gets skipped over).clea

root@freenas:/conf/base/etc/local # more syslog-ng.conf
@version:3.19
@include "scl.conf"

#
# This sample configuration file is essentially equilivent to the stock
# FreeBSD /etc/syslog.conf file.
#
# $FreeBSD: head/sysutils/syslog-ng/files/syslog-ng.conf.sample 340872 2014-01-24 00:14:07Z mat $
#

#
# options
#
options { chain_hostnames(off); flush_lines(0); threaded(yes); };

#
# sources
#
source src { system();
             udp(); internal(); };

#
# destinations
#
destination messages { file("/var/log/messages"); };
destination security { file("/var/log/security"); };
destination authlog { file("/var/log/auth.log"); };
destination maillog { file("/var/log/maillog"); };
destination lpd-errs { file("/var/log/lpd-errs"); };
destination xferlog { file("/var/log/xferlog"); };
destination cron { file("/var/log/cron"); };
destination debuglog { file("/var/log/debug.log"); };
destination consolelog { file("/var/log/console.log"); };
destination all { file("/var/log/all.log"); };
destination newscrit { file("/var/log/news/news.crit"); };
destination newserr { file("/var/log/news/news.err"); };
destination newsnotice { file("/var/log/news/news.notice"); };
destination slip { file("/var/log/slip.log"); };
destination ppp { file("/var/log/ppp.log"); };
destination console { file("/dev/console"); };
destination allusers { usertty("*"); };
#destination loghost { udp("loghost" port(514)); };

#
# log facility filters
#
filter f_auth { facility(auth); };
filter f_authpriv { facility(authpriv); };
filter f_not_authpriv { not facility(authpriv); };
#filter f_console { facility(console); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_ftp { facility(ftp); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_security { facility(security); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };
filter f_local0 { facility(local0); };
filter f_local1 { facility(local1); };
filter f_local2 { facility(local2); };
filter f_local3 { facility(local3); };
filter f_local4 { facility(local4); };
filter f_local5 { facility(local5); };
filter f_local6 { facility(local6); };
filter f_local7 { facility(local7); };

#
# log level filters
#
filter f_emerg { level(emerg); };
filter f_alert { level(alert..emerg); };
filter f_crit { level(crit..emerg); };
filter f_err { level(err..emerg); };
filter f_warning { level(warning..emerg); };
filter f_notice { level(notice..emerg); };
filter f_info { level(info..emerg); };
filter f_debug { level(debug..emerg); };
filter f_is_debug { level(debug); };

#
# program filters
#
filter f_ppp { program("ppp"); };
filter f_slip { program("startslip"); };

#
# *.err;kern.warning;auth.notice;mail.crit              /dev/console
#
log { source(src); filter(f_err); destination(console); };
log { source(src); filter(f_kern); filter(f_warning); destination(console); };
log { source(src); filter(f_auth); filter(f_notice); destination(console); };
log { source(src); filter(f_mail); filter(f_crit); destination(console); };

#
# *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages
#
log { source(src); filter(f_notice); filter(f_not_authpriv); destination(messages); };
log { source(src); filter(f_kern); filter(f_debug); destination(messages); };
log { source(src); filter(f_lpr); filter(f_info); destination(messages); };
log { source(src); filter(f_mail); filter(f_crit); destination(messages); };
log { source(src); filter(f_news); filter(f_err); destination(messages); };

#
# security.*                                            /var/log/security
#
log { source(src); filter(f_security); destination(security); };

#
# auth.info;authpriv.info                               /var/log/auth.log
log { source(src); filter(f_auth); filter(f_info); destination(authlog); };
log { source(src); filter(f_authpriv); filter(f_info); destination(authlog); };

#
# mail.info                                             /var/log/maillog
#
log { source(src); filter(f_mail); filter(f_info); destination(maillog); };

#
# lpr.info                                              /var/log/lpd-errs
#
log { source(src); filter(f_lpr); filter(f_info); destination(lpd-errs); };

#
# ftp.info                                              /var/log/xferlog
#
log { source(src); filter(f_ftp); filter(f_info); destination(xferlog); };

#
# cron.*                                                /var/log/cron
#
log { source(src); filter(f_cron); destination(cron); };

#
# *.=debug                                              /var/log/debug.log
#
log { source(src); filter(f_is_debug); destination(debuglog); };

#
# *.emerg                                               *
#
log { source(src); filter(f_emerg); destination(allusers); };

#
# uncomment this to log all writes to /dev/console to /var/log/console.log
# console.info                                          /var/log/console.log
#
#log { source(src); filter(f_console); filter(f_info); destination(consolelog); };

#
# uncomment this to enable logging of all log messages to /var/log/all.log
# touch /var/log/all.log and chmod it to mode 600 before it will work
# *.*                                                   /var/log/all.log
#
#log { source(src); destination(all); };

#
# uncomment this to enable logging to a remote loghost named loghost
# *.*                                                   @loghost
#
#log { source(src); destination(loghost); };

#
# uncomment these if you're running inn
# news.crit                                             /var/log/news/news.crit
# news.err                                              /var/log/news/news.err
# news.notice                                           /var/log/news/news.notice
#
#log { source(src); filter(f_news); filter(f_crit); destination(newscrit); };
#log { source(src); filter(f_news); filter(f_err); destination(newserr); };
#log { source(src); filter(f_news); filter(f_notice); destination(newsnotice); };

#
# !startslip
# *.*                                                   /var/log/slip.log
#
log { source(src); filter(f_slip); destination(slip); };

#
# !ppp
# *.*                                                   /var/log/ppp.log
#
log { source(src); filter(f_ppp); destination(ppp); };

BRO on a Raspberry Pi + Splunk

So the onset of this project was spurred by a few goals-
- I have a wordpress site that might as well be a honeypot because wordpress on a DMZ offering services to the interwebs is, well, pretty much asking for it.
- I had a Raspberry Pi 3B just sitting there- it had a few roles before but I just couldn't find a new one after giving up on Home Assistant
- I want to increase my auditing and analysis fu

One could go about installing BRO from source, and a great guide (though meant for Ubuntu) is, of course, found on digital ocean:
https://www.digitalocean.com/community/tutorials/how-to-install-bro-on-ubuntu-16-04
And I might go down this path later to have more control and understanding of the setup.

But to saver yourself a little headache, and to add additional features, sneakymonk3y on github made the foxhound-nsm build.  It did not install correctly, but I reached out and gebhard73 had a fork that did work (except for critical stack who seemed to drop ARM support).

Check out sneakymonk3y's blog post on the build (pay attention to the critical stack account, hopefully critical stack supports ARM again as it looks awesome):
https://www.sneakymonkey.net/2016/10/30/raspberrypi-nsm/

One day I hope to be able to make a build like that.

So step one:  foxhound build done.

Step two:  Mirror/Span port to Pi from router of DMZ interface

My home router is a pfSense box (thank you pfSense and netgate crew for everything!)- using a generic appliance box from Amazon, I highly recommend pfSense to learn routing, firewall ACLs, Snort and other network fundamentals.  After googling how to bridge the DMZ to an available interface as a mirror port, packets were streaming into the Foxhound BRO pi.

Step three:  Look at Bro logs.  Going to var/log/bro/current shows that Bro was indeed getting information in!  But how good of a grep'er or regex'er are you?  I wanted pretty SIEM stuff, so now to-

Step four: installing a splunk ARM universal forwarder.  You will need an account with Splunk, and using wget with username and password then the URL of the "download" button will just download the html of the page.  I had to download the tar to my workstation, then scp the file to the pi (after making a tmp folder in the foxhound directory).

https://www.raspberrypi.org/documentation/remote-access/ssh/scp.md

The support for the Splunk UF for the raspberry pi is great at the dev level, the forward is the latest 7.0.3 build which was great to see, but the supporting documentation is next to nothing.  Following the Linux tar install instructions:

Install from a tar file

1. Expand the tar file into an appropriate directory using the tar command. The default installation location is splunk in the current working directory.

   tar xvzf splunkforwarder-<…>-Linux-x86_64.tgz
   

2. To install into /opt/splunkforwarder, run:

   tar xvzf splunkforwarder-<…>-Linux-x86_64.tgz -C /opt

Then received the, Couldn't determine $SPLUNK_HOME, perhaps it should be set in environment"  when typing ./splunk start in the opt/splunk/bin path, so this thread was a help:

 https://answers.splunk.com/answers/553373/couldnt-determine-splunk-home-perhaps-it-should-be.html

Next, my Splunk server is using a free license, so I have to manually configure the forwarder's outputs.conf, and manually install any add-on to give the forwarder the functionality an app on the server requires (such as data input from the bro logs, ingesting and conducting source typing and field extractions for the app's dashboard).






Go backs: I would like to make an Ubuntu VM BRO build simply to get critical stack working as well, but my ESXi machine would need another NIC that it does not have, and its such an old box the BIOS doesn't have enough memory to support a PCI NIC I have... maybe there is some neat networking trick to get the pfSense mirror port output through the LAN NIC of the hypervisor and into a BRO VM.

Adding Snort for Splunk app

Log into Snort, go to add app, search snort- add Snort for Splunk

Follow this instructions:

App Installation

1.) To install the app, download the app to a suitable download location.
2.) Open Splunk and click on the Manage Apps icon.
3.) Click on the Install app from file button.
4.) In the Upload app window, select the Browse button under File and locate the SnortforSplunk.spl file in the download location in step 1.
5.) Click the Upload button to install the app.
6.) Once the app is installed follow the next steps to setup the Data Input.
7.) Under Splunk -> Settings -> Data Inputs -> Local Inputs -> UDP -> Click the New button.
8.) In the Port field under Add Data -> Select Source, enter 514 for the port to be used.
9.) In the Only accept connection from field under Add Data -> Select Source, enter the IP address of the pfSense appliance
(in the format XXX.XXX.XXX.XXX)and click Next.
10.) From the Source Type dropdown under Add Data -> Input Settings, select Network and Security -> snort.
11.) From the App Context dropdown under Add Data -> Input Settings, select Snort for Splunk.
12.) Click the Review button.
13.) Once satisfied with the settings, click the Submit button.

pfSense Setup

1.) The setup assumes that pfSense version 2.3.2-RELEASE-p1 is being used as a firewall, along with pfSense-pkg-snort version 3.2.9.2_16 (which includes Barnyard2 version 1.13 and Snort version 2.9.8.3) and that this has been properly setup.
2.) Select Services -> Snort from the main menu and this will show the Snort Interfaces page.
3.) Select the Edit option (Pencil icon) under the Actions column on the page adjacent to the interface to be captured.
4.) Under the submenu, select the {Interface} Barnyard2 (substitute {interface} for either WAN or LAN or as has been setup on pfSense).
5.) Under General Barnyard2 Settings, make sure the following are checked:-
- Enable Barnyard2
- Show Year
- Archive Unified2 Logs
and leave the rest of these settings on their default values.
6.) Scroll down to Syslog Output Settings and select Enable Syslog
7.) Under Remote Host enter the IP address of the Splunk server that is receiving the log files from Barnyard2.
8.) Under Remote Port enter the port of the Splunk server that is receiving the log files from Barnyard2 (default is port 514). **
9.) Change Log Facility from default to LOG_AUTH.
10.) Change Log Priority from default to LOG_ALERT.
11.) With all the settings done click on the Save button at the bottom.
12.) Click on the Snort Interfaces menu item and under the Snort Status column, click on the icon to start/restart the Snort interface.
13.) Check on the Splunk server that the information logged by Barnyard2 is captured by the app.

** Exception, I already had PFsense syslogs going into UDP 514, so made another data input in splunk for port 992/UDP 

*** added port 992/UDP to public interface firewall on CentOS 7 server hosting Splunk indexer.

**** Specified a new index created, index = snort, used for snort app

* Is PFSsense blocking barnyard logs?

building a relatively secure raspberry pi server- save yourself the time

** This is an insanely unorganized clump of notes I've taken while setting up a server for my own personal re-use when I may have to do it again, so I store it in the cloud for ease of access- but hopefully there may be some helpful tidbits of others found within, as it was other people's blogs that really helped me get through it.

Years ago I built a working Wordpress server using my domain name following most of pestermeester's tutorial below:

https://pestmeester.nl/index.html#10.0

I had taken the server down sometime ago, played with some other Pi projects and upon trying to follow the tutorial again, there have been too many updates and the commands are no longer relevant.  Going to raspberry Pi forums is not much help, you cannot post without waiting on moderators, and their very own tutorials are also outdated.  Though the appeal to use a Pi remains strong due to no moving parts and low power draw, I wanted to get something up and working, so I leaned on my power hungry ESXi machine.

Rather than this be a tutorial that goes step by step and be invalid within months, this is just general links and notes as to the sites I used that were most current at the time for my particular setup (pfSense 2.4.2, Ubuntu headless server 16.x with LAMP, email and SSH selected during install).  Then notes on the various 'trip wires' I ran into that had to be figured out or worked around.

So after a very frustrating day with the kid/education friendly (ie: should have solid documentation) Pi, I spun up a VM with Ubuntu Server 16.x- wow- the install GUI asks you what packages you want installed- it installed LAMP, webmail, SSH, new user account, all in one shot!  Its not like this sidesteps the learning process, you will still learn what a LAMP stack is when you have to set it all up, but it saves you all the "apt-get" lines that inevitably gets outdated.

Since this is a VM, added:

sudo apt-get install open-vm-tools

** make a snapshot here so its easy to start over when one inevitably messes things up beyond what $ --purge can fix.

Then follow this tutorial that nails the basics that pestermeester had for account setup and SSH private key login:
https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-16-04

Follow the digitalocean pre-req for Lets Ecrypt by configuring the virtual host setup in Apache:
https://www.digitalocean.com/community/tutorials/how-to-set-up-apache-virtual-hosts-on-ubuntu-16-04
** to get lets encrypt to install correctly, the router/firewall needs to be setup correctly after having setup my domain hosting service to point to my IPS public IP.  I had to use port forwarding as I could not get 1:1 NAT to work.  Once all setup, lets encrypt does not suffer from verification issues or time-out issues.
* enable NAT reflection drop down to NAT + proxy so that viewing the server/site works from machines on your LAN.

To even test port forwarder working I had to use incongnito tabs in chrome so that chrome would not automatically force https.

Lets Encrypt will print a URL to test your cert after its completed, its actually really fun to see this URL validate your certificate and give it a grade:https://www.ssllabs.com/ssltest/analyze.html?d=yourdomainnamehere

Now I can access my apache test page even when chrome forces https.  Finally getting somewhere!

* remember to 'fix' any firewall rules in your router/firewall and local rules on the Ubuntu box (ufw rules) to tighten things down if you had made any/any rules and/or ufw disable for testing purposes.

* take another ESXi snapshot, because why not?

Then the below tutorial is pretty good for installing wordpress:

https://www.techrepublic.com/article/how-to-install-wordpress-on-ubuntu-16-04/

But I'm partial to digitalocean tutorials, so their wordpress install has a few more details:
https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-with-lamp-on-ubuntu-16-04

A more applicable digital ocean wordpress tutorial as the lets encrypt tutorial had us setup apache/ubuntu to be able to host multible sites on one host.
https://www.digitalocean.com/community/tutorials/how-to-set-up-multiple-wordpress-sites-on-a-single-ubuntu-vps

* note, hate to say this but the digitalocean tutorial was no-joy, the techrepublic one gets me up and running.

*** tip in the digital ocean comments:  If using a one-click LAMP image, be sure to remove /var/www/html/index.html file (or change the configuration) because the server defaults to .html files before .php files.

*** key point #2 from the comments:  Turns out on MySQL 5.7 the line:

GRANT ALL ON wordpress.* TO 'wordpressuser'@'localhost' IDENTIFIED BY 'password';

*** still had database access issues, followed this site:

https://chartio.com/resources/tutorials/how-to-grant-all-privileges-on-a-database-in-mysql/

did not not work for me; this command seems to be deprecated in MySQL. I used:

CREATE USER 'wordpressuser';

Then:

ALTER USER 'wordpressuser'@'localhost' IDENTIFIED BY 'password';

* yet another digital ocean tidbit that helps explains the organization of apache2 and wordpress files to make your site appear:
https://www.digitalocean.com/community/tutorials/how-to-move-an-apache-web-root-to-a-new-location-on-ubuntu-16-04

You will be up and running in a minutes!!!

Follow the raspberry pi tutorial or this other linked tutorial to add the security aspects-
https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-16-04

sudo apt-get install php7.0 php7.0-mysql libapache2-mod-php7.0 php7.0-cli php7.0-cgi php7.0-gd

OMG, logging into mysql with a password works!

Added a virtual NIC to connect the Ubuntu server to the DMZ, followed this guide:

https://ubuntuforums.org/showthread.php?t=1400504

** after everything was installed, wordpress put into the write folder, the right directories listed in the right .confs etc, the site still would not come up.  Apache worked when I reverted to an apache only snapshot, but not when wordpress was installed (inside the LAN i did get the wordpress site, just not from outside).  DNS and firewall settings were good- the issue is due to setting up/installing wordpress before the DNS A record is done, so the private IP was inputted.  Go to the wordpress admin gui, general settings, and replace the IP addresses to your DNS.

https://www.digitalocean.com/community/questions/domain-redirects-to-ip-address-and-also-loading-from-ip-address

With wordpress and woocommerce running, the next issue is uploading and/or cropping 

images will become an issue, so php7-gd needs to be installed

$ sudo apt-get install php7.0-gd

$ sudo systemctl restart apache2.service

## remaining tasks

- change permission settings so that plugins can be updated and installed

- fine tune firewall and ufw rules

- install email client/GUI

LG G3 Lineage OS install

- This is just a pointers thread, as most any google'd "install TWRP/Lineage OS" finding will give you a great tutorial- but specific to my goals and the LG G3 there were some key lessons.

- LG G3 official 3.0.2-0 and even the un-official 3.0.3.2 does not support updating via OS GUI (must download from website, manually update via TWRP) nor decryption (if you encrypted the phone) to update when updating manually.  3.0.3.2 does at least prompt you for the swipe/pin to decrypt, but it just doesn't work.

- If you have already encrypted and now stuck in a loop where you can't format the internal storage to start over (as you can't mount it to start with), in TWRP go to command line and:

"recovery --wipe_data --set_filesystem_encryption=off"

- Don't encrypt the phone, you will not be able to update the OS and the LG G3's snapdragon 801 does not have hardware optimization for encryption, you take a huge read/write speed hit.

- I did not choose the superuser/root option, I actually wanted to run un-rooted this time to use android pay.

- Android pay works!  Its great going to a store and not caring if their network or POS device is compromised, my info and CC is still safe!  And its convenient (and much faster than chip and pin).

- My LG G3 isn't super fast/snappy, but Lineage has breathed new life into my old phone- saving me from having to ditch it and buy a new phone.

- Part of that 'new breath' is SUPER fast security updates- faster than Nexus/Pixel!  I got the WPA2 Krack patch the first Monday after the announcement of the hack!

Enable trim on PFsense

I started to bookmark or save URLs of helpful tutorials, but started to find sometimes the sites/blogs etc would be taken down and the info lost.  So I found a great tutorial on how to enable trim for SSD in PFSense and want to save it for the long run, as it was a pain sifting through google results for a tutorial of this quality.

URL:   https://gist.github.com/mdouchement/853fbd4185743689f58c

Thank you mdouchement for the below:

nstallation

Use memstick-licecd to install pfsense on your SSD.

Enable TRIM

• Initialize fstab:

[2.2.4-RELEASE][root@pfSense.localdomain]/root: /usr/local/sbin/ufslabels.sh

It may do nothing because all is already initialize but it's only in a case of

• Enable AHCI by adding ahci_load="YES" in th following file:

[2.2.4-RELEASE][root@pfSense.localdomain]/root: vi /boot/loader.conf.local

• Get your device ID by showing the fstab file:

[2.2.4-RELEASE][root@pfSense.localdomain]/root: cat /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ufsid/563dee910aa6a80a             /               ufs     rw              1       1
/dev/label/swap0                none            swap    sw              0       0

Check the Device ID that is mounted on /, in this example it's /dev/ufsid/563dee910aa6a80a

• Before enabling TRIM, the partition/device must be unmounted so you have to reboot pfsense in Single User mode.

[2.2.4-RELEASE][root@pfSense.localdomain]/root: reboot

• Add your USB memstick-livecd and boot on it and select 2 then use tunefs to enable TRIM:

          __ ____
   _ __  / _/ ___|  ___ _ __  ___  ___
  | '_ \| |_\___ \ / _ \ '_ \/ __|/ _ \
  | |_) |  _|___) |  __/ | | \__ \  __/
  | .__/|_| |____/ \___|_| |_|___/\___|
  |_|


 +------------Welcome to pfSense-----------+
 |                                         |                 ______
 |  1. Boot Multi User [Enter]             |                /      \
 |  2. Boot [S]ingle User                  |          _____/    f   \
 |  3. [Esc]ape to loader prompt           |         /     \        /
 |  4. Reboot                              |        /   p   \______/  Sense
 |                                         |        \       /      \
 |  Options:                               |         \_____/        \
 |  5. [K]ernel: kernel (1 of 2)           |               \        /
 |  6. Configure Boot [O]ptions...         |                \______/
 |                                         |
 |                                         |
 |                                         |
 +-----------------------------------------+


...


Enter full pathname of shell or RETURN for /bin/sh:
# /sbin/tunefs -t enable /dev/ufsid/563dee910aa6a80a
tunefs: issue TRIM to the disk set
# /sbin/reboot

• Boot pfsense in normal mode then go in shell and you can check that TRIM is now activated:

[2.2.4-RELEASE][root@pfSense.localdomain]/root: tunefs -p /
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       enabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         enabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            6408
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)
[2.2.4-RELEASE][root@pfSense.localdomain]/root: tunefs -p /var
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       enabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         enabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            6408
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)

Links that help me to write this tutorial

• https://forum.pfsense.org/index.php?topic=97554.0 (thanks to ThePOO)
• https://forum.pfsense.org/index.php?topic=66622.15

ESXi 6 hangs/freezes up

Troubleshooting is so fun when you made tens of changes at once haha.  So my ole' trusty ESXi 6 HPz800 machine started to hang up after a day or so of operation.  Looking into the ESXi logs, they are not much help as you find what you would think are red alarms, but after lots of googling they seem to be normal logs.  But skipping over to vmdkwarning logs and I got a lot of (copied and pasted from another forum member's issues, too lazy to SSH into the system again):

Lost access to volume

4bcce772-3bfe7a35-dceb-001b21541d90 (1_5WD

1) due to connectivity issues. Recovery attempt

is in progress and outcome will be reported

shortly.

And then:

Successfully restored access to volume 4bcce772-

3bfe7a35-dceb-001b21541d90 (1_5WD_1_)

following connectivity issues.

info

4/21/2010 1:16:16 PM

Then according to a few forum posts, it might be HDD failing, or overheating, or get in and tweak heart beat settings (IMO the latter a band-aid to symptoms of something needing to get fixed).

In denial that it is already time to replace my WD Black 1TB I have the datastore on, I thought about what things have changed on my system:

- put some buffer material along my HDDs in software RAID to reduce noise, but this could be reducing their airflow too

- pulled out a Hauppauge HVR card from a defunct MythTV build

- pulled out 24 gigs of 'original' RAM and put in 48 gigs of Ebay ECC RAM

Software side

- upgraded to FreeNAS 10

- Ubuntu server with Plex media server, fstab FreeNAS10 CIF of Movies share

- New Splunk server on CentOS7

Oh joy, plenty to look into.  But after lots of forums and poking around I was finding the Ubuntu Plex server to be getting hung up, particularly the kswapd process eating all of the resources- typical Linux forums leads one on a chase is it the kernel?  Kind of known yet unknownish bug? You the installer is just dumb etc.  What I settled on is I didn't give the Ubuntu server enough RAM to run plex effectively (though I watched two movies without issue, it spins out of control randomly at idle)- hopefully that is the fix, or else I will just have to start over on a CentOS server build (Ubuntu is showing it's desktop user bias as it has not been good at 'services' jobs like running splunk server or now in this case Plex).  This thread doesn't give me much optimism:

http://serverfault.com/questions/316560/how-do-i-tell-what-process-is-causing-kswapd-to-be-in-use/316636