DrayTek UK Users' Community Forum

Help, Advice and Solutions from DrayTek Users

For splunk users

  • robingb
  • Topic Author
  • Offline
  • Junior Member
  • Junior Member
More
20 Aug 2009 20:35 #57325 by robingb
For splunk users was created by robingb
I though I would start a thread on the use of splunk as it is very complicated for new users but also very powerfull, so well worth it in the long run.

On my setup I have a Vigor 2820Vn outputting logging data to a NAS running syslog-ng. I run splunk on my PC. I load the previous days log files into splunk to analyse the reports, checking for hack attempts and or compromises in network security.

When you first install splunk it starts logging everything the local host (PC)is doing. If you don't want this data, stop splunk from doing so else it slows down searches. To stop splunk logging PC data go to "Manager" > "Data Inputs" and disable everything in there. There are many layers so it takes 5-10 minutes. By the time you have finished splunk will already have many thousands of entries.

To clear all these entries (and so speed up searches on your router data) go to the command prompt and...

cd c:\program files\splunk\bin

Then stop splunk, e.g. "splunk stop"
Then clear eventdata e.g. "splunk clean eventdata"
Then start splunk again, e.g. "splunk start"
Now you have a clean non logging installation (splunk need an automated way for this!)

Now add your syslog files as a data source, e.g. go to "Manager" > "Data Inputs" > "Files & Directories" > "New" > "Upload a local file" and then select the syslog-ng log file(s) and set host to say "2820", leave source type as "automatic" and set index to "main".

Now you have added your Draytek log data to splunk you can go to the "search" app and search the data. However to really see the benefit of splunk you need to define fields in the data.

When in "search" and looking at some results there is a small white down arrow in a small gray box next to each log entry, click on this and select "Extract Fields".

You can now define a regex for fields, I use the following..

(?i) Vigor: (?P<FIELDNAME>(\[\w+\])|([^:-]*)) = Draytek_Entry_Type
(?i)\[FILTER\]\[(?P<FIELDNAME>[^\]]*)\] = Draytek_Filter_Action
(?i)\[FILTER\]\[\w+\]\[(?P<FIELDNAME>[^,]*), = Draytek_Filter_Direction
(?i)Magic Number: (?P<Draytek_Magic_Number>.*?)\s+# = Draytek_Magic_Number
(?i) UpSpeed=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_UpSpeed
(?i) DownSpeed=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_DownSpeed
(?i) SNR=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_SNR
(?i) Atten=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_Attenuation
(?i) States=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_States
(?i)\[Mode=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_Mode
(?i) inquire (?P<FIELDNAME>.*) = Draytek_DNS_Inquire_Address
(?i) DNS \-> (?P<FIELDNAME>[^ ]*)[ ] = Draytek_DNS_Server_For_Inquire
(?i) User: (?P<FIELDNAME>[^: ]*)[: ] = Draytek_Local_User_IP
(?i)[^a-zA-Z]-[> |>](?P<FIELDNAME>[^:]*): = Draytek_Destination_IP

Hope that helps

Please Log in or Create an account to join the conversation.

More
21 Aug 2009 12:46 #57337 by thrain
Replied by thrain on topic For splunk users
Splunk is the mutts nuts.

It makes is so easy to locate the data and graph it.

If you can run the syslog server & splunk on the same box.

Just one word of warning, some versions of syslog report the following if the syslog message is repeated :-

Nov 21 04:42:34 router last message repeated 68 times
Nov 21 04:43:35 router last message repeated 153 times
Nov 21 04:44:31 router last message repeated 143 times
Nov 21 04:45:33 router last message repeated 133 times

There is an option to switch the last message repeated off, but I can't remember what it is.

Please Log in or Create an account to join the conversation.

Moderators: ChrisSami