I have a message thread, these messages are coming on splunk.The chain consists of ten different messages: five messages from one system, five messages from another (backup) system.
Messages from the primary system use the same SrcMsgId value, and messages from the backup system are combined with a common SrcMsgId.Messages from the standby system also have a Mainsys_srcMsgId value - this value is identical to the main system's SrcMsgId value.
The message chain from the backup system enters the splunk immediately after the messages from the main system.Tell me how can I display a chain of all ten messages? Perhaps first messages from the first system (main), then from the second (backup) with the display of the time of arrival at the server.
With time, I understand, I will include _time in the request. I got a little familiar with the syntax of queries, but still I still have a lot of difficulties with creating queries.Please help me with an example of the correct request.Thank you in advance!
Best Answer
You're starting with quite a challenging query! :-)
To combine the two chains, they'll need a common field. The SrcMsgId field won't do since it can represent different message chains. What you can do is create a new common field using Mainsys_srcMsgId, if present, and SrcMsgId. Then link the messages via that field using streamstats. Finally sort by the common field to put them together. Here's an untested sample query:
index=foo```Get Mainsys_srcMsgId, if it exists; otherwise, get SrcMsgId```| eval joiner = coalesce(Mainsys_srcMsgId, SrcMsgId)| streamstats count by joiner```Find the earliest event for each chain so can sort by it later```| eventstats min(_time) as starttime by joiner```Order the results by time, msgId, sequence| sort starttime joiner count```Discard our scratch fields```| fields - starttime joiner count