Since you ended up on this page, chances are that at this particular moment you feel just as mystified as I did when I got OpenTelemetry (OTel) in Python working with X-Ray, but somehow the CloudWatch logs weren’t linked to my X-Ray traces in the AWS Console, although “the Logs-section is right there, and omg why don’t you correlate already, pls!” — Fear not, I’ve got you covered 🎉
For this post I presume you already have a Python application running on AWS and instrumented it with some OpenTelemetry traces that nicely show up in X-Ray. Whatsmore, you’ve sprinkled some log messages inside your spans and they nicely end up in CloudWatch — great! Now why on earth does the message in the Logs-section of ServiceLens under your X-Ray trace straightup lie and tell you “No logs to display for these resources”?!
In a nutshell the problem boils down to either the AWS-XRAY-TRACE-ID missing from the log message, so we need to slap that on every message we send to CloudWatch. Or we didn’t tell X-Ray which logs to scan for the given AWS-XRAY-TRACE-ID. Or very possibly a combination of both.
So if we’d create an OTel Trace Id to AWS-XRAY-TRACE-ID converter like so:
And we were to apply it like this:
In combination with a Resource attribute aws.log.group.names configured like in the snippet below, where config[“x_ray_matching_log_groups”] contains a list of CloudWatch absolute log group names:
Then your spans in X-Ray would be correlated to the tagged logs, and we’d be done!
If that’s what you came here for: happy to help, have a nice one! But if you’re looking for a little more sophisticated approach, don’t leave just yet — the following section is for you 😄
Combining X-Ray, CloudWatch and Structlog
Since I wasn’t too keen on the idea of having to remember to manually add a span-specific AWS-XRAY-TRACE-ID to every log message, I figured that I should somehow be able to harness the capabilities of the Structlog library I was already using, and sure enough I was!
There were roughly two approaches I was thinking of, both by leveraging the Processors-functionality of Structlog. Approach (1) would be simply adding the custom processor displayed below to the structlog.config(processors=[... statement in my logging bootstrapper, and solution (2) would be the instrumenting the Structlog configuration after bootstrapping.
I went with the latter, because it seemed less intrusive than the former — plus it fits nicely with all existing instrumentation extensions that are available for the Python OpenTelemetry library. Then again: is it really less intrusive when you implicitly and perhaps unexpectedly overwrite a Structlog configuration from out of nowhere? I’ll leave that up to you — you can still always opt for approach (1), which saves you the 10-ish lines of code of the _prepend_processor as well.
In conclusion
Although out-of-the-box the official OpenTelemetry Python Sdk doesn’t seem to be fully compatible with X-Ray — the spans show up, but the log messages are not correlated in ServiceLens — I demonstrated in this blog post that with a little help, correlating CloudWatch logs with their respective X-Ray traces works absolutely fine!
Relevant posts and acknowledgements
- https://github.com/aws/aws-xray-sdk-python/issues/188 problem and solution are pretty much discussed here, but focus is on X-Ray Sdk instead of OTel sdk
- https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/24634 discussion about the problem in the OTel Python sdk
- https://stackoverflow.com/a/72178525 probably adapted the
_convert_otel_trace_id_to_xrayfrom here, but I’m really not sure anymore
