Log consumers should be able to differentiate sampled from unsampled traces

[carterkozak]

What happened?

Log lines inside a traced operation contain a traceId MDC key. Tracing output may be loaded by ID in our logging pipeline, however most traceIds are not sampled, and as such may not be viewed. It's difficult to tell which traceIds one can search for.

What did you want to happen?

It would be helpful if we set a second MDC key-value pair to differentiate sampled and unsampled traces.

Proposal

Tracer should set sampled:true when setting the traceId key for a sampled request.

MDC key name: sampled, traceSampled, better ideas? sampled is simple and easy to remember.

Should we set sampled:false for traces that are not sampled? This would allow us to differentiate versions which have not yet upgraded, however it would also slightly increase the overhead of every log line. While it may be helpful today, I don't think it's necessary long term.

[carterkozak]

@dsd987 Mind taking a look at this?

[carterkozak]

After some thought, I think we should use key traceSampled in order to make collisions with existing parameter names less likely, and provide more context describing the value.

[carterkozak]

Update: Caught up with @ellisjoe who is looking into what it would take to expose this information using indexed traces without the need for additional data attached to service events.
Potential upsides:

• Less data logged per event
• Could work with request logs as well as service logs
• May give us direct UI links to trace info from service/request logs

[raiju]

This PR should unblock it for Jetty filters: #137