Tracing not showing in grafana

[titogarrido]

Hi Folks,

I am trying to use Loki + Tempo but when I check my logs in Loki the trace_id and span_id are missing.

Code related to logging:

[...]
from opentelemetry.instrumentation.logging import LoggingInstrumentor
from logging_loki import LokiHandler, emitter

emitter.LokiEmitter.level_tag = "level"

LoggingInstrumentor().instrument(set_logging_format=True)
handler = LokiHandler(
    url="http://ipadpi.local:3100/loki/api/v1/push",
    tags={"application": "backend"},
    version="1",
)
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)

[...]

@app.get("/test")
async def test():
    logger.info(
        "Info em test2",
        extra={"tags": {"service": "backend"}},
    )

    return Status(message='test!')

The result in loki is:

and I am running my uvicorn using:
opentelemetry-instrument uvicorn main:app

What am I missing here?

[fullonic]

Hey @titogarrido, have you configured the grafana datasource? as explained tempo and loki

[titogarrido]

Hi @fullonic ! Yes, it should be connect... Is there a way to debug what python-logging-loki is sending to Loki?

[titogarrido]

@app.on_event("startup")
async def startup():
    Instrumentator().instrument(app).expose(app)
    LoggingInstrumentor().instrument(set_logging_format=True)

emitter.LokiEmitter.level_tag = "level"


lokiHandler = LokiHandler(
    url="http://ipadpi.local:3100/loki/api/v1/push",
    tags={"application": "backend"},
    version="1",
)

FILE_FORMAT = logging.Formatter('%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] [trace_id=%(otelTraceID)s span_id=%(otelSpanID)s resource.service.name=%(otelServiceName)s] - %(message)s')
LOKI_FORMAT = logging.Formatter('%(asctime)s %(levelname)s %(name)s %(filename)s:%(lineno)d trace_id=%(otelTraceID)s span_id=%(otelSpanID)s resource.service.name=%(otelServiceName)s %(message)s')


fileHandler = logging.FileHandler("backend.log")

fileHandler.setFormatter(FILE_FORMAT)
lokiHandler.setFormatter(LOKI_FORMAT)
logger = logging.getLogger(__name__)
logger.addHandler(lokiHandler)
logger.setLevel(logging.DEBUG)
logger.addHandler(fileHandler)

Ok, that works, I have to add the format to loki handler.

[frobnitzem]

I had this same issue. When viewing logs using Grafana's "Logs visualizer" documented here, the label browser shows labels that were definitely unique to the logs I sent, but no entries appear when I run the LogQL query. (query returned 0 results errors)/

Once I changed the python formatter to '%(asctime)s %(levelname)s %(name)s %(message)s', the log messages started appearing in the Log visualizer. I later realized I could also see both early and late logs through the "Explorer".

Note the early messages didn't include the header, and show up in grey. Recent messages include %(levelname)s, and are now colorful.

The need for a formatter should be mentioned in the main README!