Splunk Dev license, Splunk cloud for android app, password, breaks pfsense TA, fun

So I wanted to take my Splunking another level up and use the mobile app to get alerts.

I believe the older implementation was based off of expecting a strong IT/sysadmin role at the business so that a proxy, port forwarding etc could be setup.  But now Splunk has you install a Cloud Gateway app, it seems to operate like a Synology account or Unifi account where there is a bit of cloud assistance to connect you to your instance outside of your network, and does things for you like SSL etc.

Kind of sucks for me because part of the project to use something like HA-Proxy and learn the security and connectivity parts myself.

Anyhow, not reading the documentation like we all do (right?) I installed it on a splunk free instance- oups.  It tries to work, no warnings about your license (should I submit that fix this?) but worse yet, now my TA_pfsense extractions are totally broken so no logs are coming in (every few hours something small gets through), meanwhile FreeNAS, tasmota and Suricata is still flowing in fine.

Logs show that the data input I have pfsense firewalls coming over has some huge time stamp issues, thinking logs with time stamps from 2001 are coming in.  

Looking at more logs there is:

Invalid key in stanza [pfsense] in blah/blah/blah props.conf, line blah, KV Store = false

Invalid key ins stanza blah ... ... TIMEPREFIX (value: ^)

Invalid key in stanza blah .... TIMEFORMAT (value: %b %d %H:%M:%S).

Invalid key in stanza blah ... MAXTIMESTAMPLOOKAHED (value: 16).

So the KV store thing kind of made sense, that error log was happening before the Cloud Gateway install and I guess false is not even an option, options wer... ( I lost the URL to the splunk doc of what are actual valid stanzas), I went with KV_MODE = none

The inputs for timeprefix, timeformat and maxtimestamplookahead are the same they have always been, and work, but now Splunk docs show underscores...  I don't know why or how it can be upset.

Went to verify the Snort for Splunk app I'm using to ingest Suricata logs (since pfsense snort update purposely removed Barnyard2 logging) and it has "TIME_PREFIX=," so I guess thats why its ok.

Again, super weird TA-pfsense could work in some kind of legacy mode until the Gateway app installed- maybe something to do with the gateway app starting up and enabling KV Store?

Still perplexed how this went to sh*t with a Cloud Gateway app install, it must install something in the backend and changed the default datetime.xml?  Its almost like it updated Splunk itself and broke something- oh and ironically with all the Splunk warnings about Python 3, the Gateway app uses Python 2 and needs you to input a 'run python 2' stanza:

URL needed.

So off to using nano like normal people do, and as a sanity check I downloaded the pfsense TA all over again, un-packed it and made sure that the stanzas matched, that it didn't somehow 'mutate' on me.

So restart:

Also, installing the license, I had no account/password, so this:

https://docs.splunk.com/Documentation/Splunk/7.1.0/Security/Secureyouradminaccount#Create

fu*k, what now...

typo

fixed to MAX_TIMESTAMP_LOOKAHEAD

All this to let Splunk into my Splunk (yuck) and try and make it so that my phone can get a notification when my garage door opens (via Tasmota logs coming in and an alert).

WHAT

THE

FUCK

All errors gone, even the index and inputs consistent thing + splunk restart.  STill, nothing coming in.

fuck my life

Cool side note, with the splunk free version, URL/debug/refresh wouldn't work.  With the dev license, it works now.

So, pfsense is now coming in, but the transforms and props that did the neat trick of looking past timestamp and assigning a sourcetype:  ex:  pfsense:filterlog, pfsense:openvpn is broken.

Its because I had it commented out.  I am my worst enemy.

Debug refresh (love getting this feature)

We are back in business.  YAS QUEEN!!!

Snort update to 3.2.9.14_1 breaks Barnyard2 syslog output in pfsense, Suricata time and modifying the Snort for Splunk app

Boy thats a big title right?

So the Snort update 3.2.9.14_1 (or heck maybe an earlier one?  I hadn't updated for some time) breaks Barnyard2's syslog output to a defined server (in this case Splunk).

I was using Snort for Splunk's app- an amazing app with great field extractions and great dashboard.  I was panicking to get some kind of threat feed back in.

Enter Suricata.  I had mixed success with early Suricata builds on pfsense so I had stuck to Snort.  With this issue popping up, I decided to install Suricata again.

After following some guides to get Snort rules inputted into Suricata, and with Suricata's WebUI very simular to Snorts, I was off and running way faster than I expected.

Very quickly I had Suricata data feeding into Splunk, and for fun into the same data input Snort was using.  To the log formatting credit, it was getting extracted pretty well right out of the gate.

Instead of installing a Suricata app, I see an opportunity here to use a transforms.conf stanza to sourcetype data from the Suricata feed to sub-devide into Suricata or Snort rule.  With that done, I can apply a different props.conf stanza to the Splunk sourcetyped data to fix the extractions for description and other fields.

What is the path to FreeNAS plugins?

Plugins install in jails, the path will be mnt/Pool1/iocage/jails

This is running after boot however.  To make .conf changes that survive reboot, I believe one needs to go into a different path.

Put spinning rust to sleep on a Raspi

After getting my SMB data backed up on a pi connected to two 5TB HDDs (Seagates) using mhddfs to combine both drives/partitions into one mount (without LVM) and getting FreeNAS to do an rysnc push (disable delay-update + lots of private/public key fun) I noticed the 5TB drives just keep spinning and spinning, despite being at idle.

To fix this, thankfully these drives are supported by hdparm.  Install hdparm, read up the arch wiki on it:
https://wiki.archlinux.org/index.php/Hdparm

I ran hdparm -B /dev/sda1 and for sdb1

Both drives showed APM_level as 128.  A value of 127 is needed to have drives stop spinning at idle.  I ran hdparm -B 127 /dev/sda (and again for sdb1) and after about 10 seconds both drives stopped spinning.

And to do a quick spin-up test, I kicked off the rsync job manually- I could hear the drives spin up for the sync portion and then take about 20 seconds to spin down after finishing the sync portion (no writes, was already up to date).

This is a great alternative to the idea of setting up a cron job to unmount the drives between rysnc jobs, especially since I have no idea how unmounting two drives tied into mhddfs might affect things.

But not out of the woods yet- these settings do not persists after reboot- and in typical linux fashion there are different answers on how to set this up to persist after reboot.  This thread was Pi specific but uses hdparm's timing parameter instead of power parameter:
https://maker-tutorials.com/en/spin-down-magage-hard-drive-on-raspberry-pi-hd-idle/

So use their example:

But replace with your uuid's of course, but also change spindown_time = 240 with apm (in this case apm = 127 is what makes my drives spin down).

And then save, and then reboot to see if they persisted.

And of course if you know anything about this blogger, of course not.  In fact it made it worse.  The default settings of APM being 128 (and 127 is what makes the drives allow themselves to spin down) is now at 254!!

So maybe its better to go fully by the tutorial LOL.  I noticed when I would hdparm -B /dev/sba1 it would print out the results "APM_level = 128" so I tried this in the .conf file vs "apm = 128" but no joy.

The tutorial above using "spindown_time" also didn't work though.  Gotta live raspi, this is not an issue with most any other operating system.

Last try is this:  When you go to edit hdparm.conf it pretty much has the man file in it but commented out.  If follow that man file, there is no feature of "by-uuid", it has you go by /dev/sba1 etc.  So trying it that way:  Nope.  Defaults to 254 again.

So with editing the hdparm.conf six ways from Sunday and it still didn't work, the googles says one option is to edit the etc/rc.local file- whatever is there is run during boot up.
https://www.dexterindustries.com/howto/run-a-program-on-your-raspberry-pi-at-startup/

Typed in the two commands I want run:
hdparm -B 127 /dev/sda1

hdparm -B 127 /dev/sdb1

Still- NOPE

So created a script in /mnt called "hdparm.sh"
Inside this script is:
"
hdparm -B 127 /dev/sda1
hdparm -B 127 /dev/sdb1

Tested this script by typing in from /mnt "hdparm.sh"  It printed out the APM_level as 127.

Inputted into rc.local
"# make HDDs sping down at idle
sh /mnt/hdparm.sh &

And Bob's your uncle- now after reboot the hard drives settle down and stop spinning.


Things that may have helped that I did along side these other things so I don't know if they had effect or fixed the issue rather than what I thought did:
- set rc.local to executable
-

fuse file system join several file systems together to form a larger single one

I had a unique situation of wanting a larger jbod but on a Pi, and you shouldn't use LVM to accomplish that and RAID on a pi3 is not really an option.  Found mhddfs.

https://romanrm.net/mhddfs

BASH commands explained

Have to save a place for a really cool website that explains the syntax in a bash command.  A great example:

https://explainshell.com/explain?cmd=rsync+-avz+--progress+--partial+-e

Just wow.

Raspberry Pi 3 as an rsync backup server

So in my other post I started down the road of editing /conf/etc files in FreeNAS in the pursuit of getting SMB logs sent via syslog-ng.  This is a scary endever, a bricked FreeNAS and I loose... all of my data including VM storage via NFS.

I've had a Raspberry Pi 3 sitting on my table for what feels like a year now.  It's only good use-case for me was to be a hass.io home-assistant server, but I have since moved that to a VM.

Though a zfs replication backup would be the ideal setup, I do not have the hardware (nor want to pay more on my powerbill) to spin up another ZFS built just for this backup.

Enter the Raspberry Pi 3 and a big external HDD.

I searched around for a good tutorial to hold my hand along the way as I want something to refresh my memory to:

- new Pi headless on an SD card
  -  Used a GUI disk manager to get the old SD card back into one volume
- Not only setup SSH, but with a key (not password setup)
- Setup FreeNAS account that can do this backup, but only this backup
- not just mnt but FSTAB the external HDD into the Pi
- First time playing with rsync, set it up to get all of my SMB shares and VM NFS
-

First things first, install Raspberry Lite (no need for GUI) but in typical fashion, off to a broken start.  Etcher was showing an error, file mismatch (what was burned vs. source image).  Googling later found this:
https://forums.balena.io/t/checksums-do-not-match/36537/48

Which said to go to this post:
https://superuser.com/questions/1199823/how-to-prevent-creation-of-system-volume-information-folder-in-windows-10-for/1199824#1199824

To go into local policy (or if you can't, registry edit) to change "Do not allow locations on removable drives to be added to libraries".  This still didn't fix the error.  If you have a linux laptop handy, its a good use-case to get this done.  Or before blowing the dust off the *nix box I tried to use Raspberry's very own Imager v1.2.  That worked.

Next, setup SSH- the smart thing is to make the folder before booting, but if you already booted up you need a screen and keyboard to setup SSH.  The goal here is to do everything headless (hence using MAC to figure out IP via ARP on your router)

https://www.raspberrypi.org/documentation/remote-access/ssh/README.md#3-enable-ssh-on-a-headless-raspberry-pi-add-file-to-sd-card-on-another-machine

Taken from memory from a person's headless server build, you can grab memory from the GPU for the system, since this headless lite build will not output to a monitor or have a GUI.

https://www.raspberrypi.org/forums/viewtopic.php?t=181262

According to that thread, 16mb is fine.  This can be done in the "sudo raspi-config"

Confirmed by https://www.pestmeester.nl/   :  A very dated tutorial but one of the first I ever used for my pi and amazingly written to setup a pi LAMP stack with security in mind.  Sadly its dated and a lot of the packages and even some of the commands are now gone/not functional.

Apparently Windows 10 just touching the SD adds a file.  Not good for this situation.  Strangely not an issue when I create Linus bootable USBs, but I find anything non x64 Linux to be a pain like this- be in freeBSD or ARM, there is always some nuance.

My ARP table is lame and not showing a hostname, so this was a useful post to get the MAC range of Pis
https://raspberrypi.stackexchange.com/questions/28365/what-are-the-possible-ouis-for-the-ethernet-mac-address

Now do your standard apt-get update and apt-get upgrade -y.

Also passwd to change the default password.  Its a good time to make a different user too specifically for the backup job.

----  Finally Pi is setup, time to attack the list to make it an rsync server -----

A very very good place to pickup after getting Raspberry Lite installed is the Linux Automated Backup series by carpie.net on Youtube:

https://www.youtube.com/watch?v=iPK1TYOwzXI

Command notes:
- dmesg to get drive info (sda)

This gets blurry for me, how is the e1xternal drive sda and not the SD card?  And in his vid he will label the drive sda1, but if you use blkid, you will see sda1 is already the SD card, sda2 is the external drive.

- blkid to get more detailed info (/dev/sda2:  LABEL="Seagate Backup Plus Drive"  UUID"blah blah"

-  I'm going to stick with sda2.

- I was hard pressed, do I keep the ntfs format, do I format to ext4?  If I keep NTFS then in theory I can connect to windows, but if the files within are compressed by rsync, is there any advantage?  Maybe better to use a linux native format like ext4 because reasons.  Then I found this guy's blog and for sure will ditch ntfs, but has me doubting ext4 too!  I have lots of files that likely have the bad character issue.

http://therandymon.com/index.php?/archives/285-Backing-Up-FreeNAS-to-an-external-hard-drive.html

His issues might be specific to the fact he plugged the HDD directly into the NAS though, vs running through a server like the Pi.

This now has me doubting to even use the Pi.  Its kind of nice to think of just putting this external HDD in the server rack plugged into my Freenas, set it up and forget it.  It will work a lot faster, not load my network and be out of site, out of mind and free up the Pi for any other project... oh read down to the end of the post, my current version of FreeNAS probably doesn't support UFS file system anyhow, maybe rsync is the most reliable setup given how frequent FreeNAS updates, and how huge the changes are when it does update.

TLDR, just use ext4.  Can always spin up a nix box to view the contents if the Pi dies or I need it to be faster.

Back to the youtube vid:

- sudo parted -a optimal -s /dev/sda2 mklabel gpt mkpart backup ext4 0% 100%

    - didn't work so used fdisk to format, device is now /dev/sda2pl
    - weird, now sda1 does not show up...

- sudo mkfs.ext4 -F /dev/sda2

- sudo parted /dev/sda2 print

- sudo mkdir /opt/backup

- sudo blkid (to get the UUID for fstab)

- sudo nano /etc/fstab     input UUID and other stanza info

- sudo mount /opt/backup
- ls opt/backup

THis whole thing is hosed- reminds me why I hate raspian so much.  Every single tutorial is different, every result is different from teh tutorials.  Is this why its a beginners device?  It forces one to RTFM and re-do opperations all day that would take only 5 minutes on a legit linux build?

Linux laptop + gparted = 2 minutes to un-f#ck the raspi's sh*t show of trying to format over the old format and inability to combine two partitions into one.  (I say innability because if you can't google how to do it, its as good as not able to, especially on something so "beginner friendly").

Note because of the fstab change made earlier, the SD card install of raspbian lite is now hosed.  But I wanted to re-do it anyhow (lost confidence of loss of sda1 partition of SD drive) and this time add the ssh folder (in hindsight maybe its simply due to reboot and on that reboot raspian expands the primary partition to take up the whole SD card).

--------- starting over ------------

Fresh raspian lite on the SD card via the Raspi burning software
- added "ssh.txt" to SD card after burn and re-insert into USB
- booted up, ssh worked, yay.
- raspi-config for GPU memory mod and timezone setup
- started following this tutorial as ext4 seems rare:
https://gordonlesti.com/mount-ext4-usb-flash-drive-to-raspberry-pi/
- sudo chown -R pi: /opt/backup

honorable mention, great site on some of the commands you can use within rsync:  https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/

A part of me wants to make a different user on the pi, but this pi will stay behind my NAT and only use an SSH key to access freenas.



When you go to google this, most results are of making the FreeNAS the rsync server, not the client.  Or lots of results of rysinc from one FreeNAS to another.  Its a little harder to find info on an external non-FreeNAS system being the rsync server to pull from FreeNAS as teh client.  The FreeNAS documentation is bad for this goal, it just addresses FreeNAS as the server BUT a good read to understand the high level process of rsync is in 8.3.1

So off to watch a bunch of vids, tutorials and RTFM'ing as its confusing how to setup the FreeNAS to be just a client and the Pi the server.  If I understand correctly, the Pi's private SSH key will be on the FreeNAS, the FreeNAS will have the cron job to run rsync and the Pi is simply listening.

It looks as simple as going into FreeNAS > Tasks > Rsync Tasks > Add > select PUSH.

FreeNAS will only allow using a user with a private key set up in home dir- this will be an account on the Pi that has rights to the backup path specified above and has had a private/public keygen made, export the private to FreeNAS.  Teh part 3 video shows how to make that key only work for rysnc tasks as added security.

Back to the youtube tutorial part 3, rsync seems to already be installed on the current version of Raspian Lite.

The tutorial has the youtuber's linux laptop (client/PUSH) setup an ssh keygen, and moves the private key to the Pi...  The FreeNAS guide in 7.3.2 is more complicated, with private./public generated on the PUSH (FreeNAS) and both Pi and FreeNAS need to exchange public keys due to MITM mitigation.

Had to follow a guide to enable root login via SSH in the raspi

Then need to perform this setup from FreeNAS except no need to create a folder we already have on the pi after generating keys (was not necessary).

cat ~/.ssh/id_rsa.pub | ssh <USERNAME>@<IP-ADDRESS> 'mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys'

Setup script on Pi and cron job to run script to use rsync once a week (first do a manual rsync as first run might take longer than a week, do not want to start a new sync while original still running).

- Edit, so with backups running the 5TB drive will soon reach it's limit.  Enter mhddf, a really neat way of avoiding making an LVM partition between two drives, but having more of a jbod like setup.

* edit.  Removed "delay updates" from rsync syntax as it seems to lock my files (at least of a very large and long backup) into the ~tmp~ folder, in purgatory.  Hoping there is a script out there that can move the already backed up files out of this ~tmp~ folder and into their perspective folders recursively.

*  Need to create cron job to unmount the HDDs between rsyncs to spin down the drives- otherwise they are always spinning and humming away.