I have been playing lately with Splunk and let me tell you… WOW! Awesome tool. You can have N servers forwarding logs to a main server where you can search using SPL to query all over those inputs.
Here I will demostrate how to make a simple stat usage of your website that you can aggregate later with Splunk to check how your users are using your stuff.
First you have to change the way you log. Splunk likes eating key=values all over the place. So feed them
action=user_searching_stuff , age=Some , email@example.com , gender=M... etc
Now that you know how to log lets imagine a scenario like this. You have a screen with a few filters and you want to know what filters are used the most. So your log will be something like this..
User searching by email. Only email field was filled.
action=user_searching_stuff , name= , firstname.lastname@example.org , gender= , ... etc
User searching by gender. Only gender field was filled.
action=user_searching_stuff , name= , email= , gender=M , ... etc
So after a while of users using it you come up with the following Splunk search string.
* index="test" statistic "action=user_search_stuff" | stats count(eval(name!="")) as name, count(eval(email!="")) as email, count(eval(gender!="")) as gender
This will give you a table with counts per event per this user_searching_stuff action. Super useful info to keep track of how your users use your product.