Splunk Dev license, Splunk cloud for android app, password, breaks pfsense TA, fun

So I wanted to take my Splunking another level up and use the mobile app to get alerts.

I believe the older implementation was based off of expecting a strong IT/sysadmin role at the business so that a proxy, port forwarding etc could be setup.  But now Splunk has you install a Cloud Gateway app, it seems to operate like a Synology account or Unifi account where there is a bit of cloud assistance to connect you to your instance outside of your network, and does things for you like SSL etc.

Kind of sucks for me because part of the project to use something like HA-Proxy and learn the security and connectivity parts myself.

Anyhow, not reading the documentation like we all do (right?) I installed it on a splunk free instance- oups.  It tries to work, no warnings about your license (should I submit that fix this?) but worse yet, now my TA_pfsense extractions are totally broken so no logs are coming in (every few hours something small gets through), meanwhile FreeNAS, tasmota and Suricata is still flowing in fine.

Logs show that the data input I have pfsense firewalls coming over has some huge time stamp issues, thinking logs with time stamps from 2001 are coming in.  

Looking at more logs there is:

Invalid key in stanza [pfsense] in blah/blah/blah props.conf, line blah, KV Store = false

Invalid key ins stanza blah ... ... TIMEPREFIX (value: ^)

Invalid key in stanza blah .... TIMEFORMAT (value: %b %d %H:%M:%S).

Invalid key in stanza blah ... MAXTIMESTAMPLOOKAHED (value: 16).

So the KV store thing kind of made sense, that error log was happening before the Cloud Gateway install and I guess false is not even an option, options wer... ( I lost the URL to the splunk doc of what are actual valid stanzas), I went with KV_MODE = none

The inputs for timeprefix, timeformat and maxtimestamplookahead are the same they have always been, and work, but now Splunk docs show underscores...  I don't know why or how it can be upset.

Went to verify the Snort for Splunk app I'm using to ingest Suricata logs (since pfsense snort update purposely removed Barnyard2 logging) and it has "TIME_PREFIX=," so I guess thats why its ok.

Again, super weird TA-pfsense could work in some kind of legacy mode until the Gateway app installed- maybe something to do with the gateway app starting up and enabling KV Store?

Still perplexed how this went to sh*t with a Cloud Gateway app install, it must install something in the backend and changed the default datetime.xml?  Its almost like it updated Splunk itself and broke something- oh and ironically with all the Splunk warnings about Python 3, the Gateway app uses Python 2 and needs you to input a 'run python 2' stanza:

URL needed.

So off to using nano like normal people do, and as a sanity check I downloaded the pfsense TA all over again, un-packed it and made sure that the stanzas matched, that it didn't somehow 'mutate' on me.

So restart:

Also, installing the license, I had no account/password, so this:

https://docs.splunk.com/Documentation/Splunk/7.1.0/Security/Secureyouradminaccount#Create

fu*k, what now...

typo

fixed to MAX_TIMESTAMP_LOOKAHEAD

All this to let Splunk into my Splunk (yuck) and try and make it so that my phone can get a notification when my garage door opens (via Tasmota logs coming in and an alert).

WHAT

THE

FUCK

All errors gone, even the index and inputs consistent thing + splunk restart.  STill, nothing coming in.

fuck my life

Cool side note, with the splunk free version, URL/debug/refresh wouldn't work.  With the dev license, it works now.

So, pfsense is now coming in, but the transforms and props that did the neat trick of looking past timestamp and assigning a sourcetype:  ex:  pfsense:filterlog, pfsense:openvpn is broken.

Its because I had it commented out.  I am my worst enemy.

Debug refresh (love getting this feature)

We are back in business.  YAS QUEEN!!!