Saturday, August 8, 2020

The pfSense Snort or Suricata unified2 log Apocalypse

 Over the last month (Aug 2020) Snort, and then shortly by Suricata have dropped Barnyard2 logging support on pfSense because reasons:

https://forum.netgate.com/topic/154632/attention-barnyard2-users-for-snort-or-suricata-please-read-this-notice

Snippet because URLs die sometimes:


I like logs... well more a love/hate of logs.  I have syslog checked to 'all' (all the things) in pfSense to output to Splunk.  I use the TA-pfsense from Splunk to index this data (some small tweaks noted in this blog to tune it up).

I then USED to use Barnyard2 and the webGUI that came with it within pfSense to send snort (then later Suricata) logs to splunk over a different port.  This was GREAT- it made using separate TAs or Apps easy within Splunk to ingest this data, add value etc.


WELP, not anymore.


So off to fingering out how to make this work?


First Snort does support the old Unified2 but rumor is it will deprecate that for JSON.  Suricata already supports eve json out, so I went to Suricata.

Next, the Suricata logs 'inject' into the system logs so that syslog can send them out.  This introduced a few problems:

- TApfsense props.conf to handle this json data- I couldn't figure out a solution

- TA_suricata- how the heck do I transforms and move over the eve json logs from the pfSeense data input and over into the TA_Suricata add on (leaving the Snort apps as they are not build for json)

I tried some things, a stanza here, a hack there- none worked and I guess thank-goodness they didn't because while digging deeper into the data during testing I found that the pfsense syslog output has a byte limit and it was cutting off/truncating large parts of the log anyhow!  Another sucky part was that the IDS/IPS alerts were still going over in Unified2, the eve json also going over syslog with repeat of that data (plus the addition flow data of other events).


IT was just a big, hot mess.


So some brave souls are using freebsd package manager to install filebeats

https://forum.netgate.com/topic/141027/problems-disabling-payload-in-suricata-json-eve-alert-logs/6

https://forum.netgate.com/topic/136998/how-to-send-snort-alert-logs-to-graylog-without-barnyard2/11

That is beyond me... 


Here is to hoping that a filebeats package is coming to pfSense very soon- but if history is to be used as a reference, its not likely...


But as I get antsy to get those logs back, and my Splunk skills are FAAAAAAAR from whats needed to somehow make it all work within ONE TA(add-on)--- AND even if my skillz were there, the logs are getting truncated by pfsense anyhooooooow, I see myself bricking my pfsense box trying to install and run filebeats....


Hot dog!  Maybe a splunk forwarder is the way

https://elatov.github.io/2017/02/installing-splunk-forwarder-on-pfsense/


I've spent hours trying to make the forwarder work- changing intputs to monitor the folder the json is in, opening the firewall, going into SPlunk data inputs and enabling 9997 etc.  For a second I thought I had some weird index cert or SSL issue but thanfully I found this post, and this guys Splunk CLI saved the day.


https://community.splunk.com/t5/Getting-Data-In/Cannot-See-Universal-Forwarder-from-Splunk-Enterprise/td-p/307601