How to Configure and Deploy OpenTelemetry Collector?

[jnicholls]

I apologize in advance for using GitHub issues as a channel for what is a support question, but I'm not sure where else to turn.

I'm using the CloudWatch Observability EKS add-on, which is fantastic. It obviously is installing this helm chart under-the-hood with all the reasonable defaults needed. However the agent itself is installed as a DaemonSet w/ host networking to capture node-level information. So while the agent itself supports the ability to configure itself to be an OTel collector, it isn't making itself addressable by the application pods in the cluster as best as I can tell. Unless I need to use the HOST_IP and address it that way...?

Is running the CloudWatch agent as an OTel collector supported? If so, how would I go about utilizing it? I see the chart supports accepting an otelConfig so I presume it's meant to be ran with this capability.

Alternatively, I could install the ADOT EKS add-on in addition to the CloudWatch Observability add-on and use that instead—I'm not certain how that deployment will look, whether it will be a cluster-wide service or some sort of operator to inject side-car containers into every pod, or something like that—but I am not certain both CloudWatch Observability and ADOT add-ons can or should run side-by-side, and I am just curious if it's possible to configure the former so that I can use its OTel capabilities.

Thanks in advance!

[jefchien]

We do not recommend running the two add-ons side by side as there is potential for conflicts. The CloudWatch Observability EKS add-on does not support side-car deployments of the CloudWatch Agent, but this is something that can be done separately and manually. What specific OTel capabilities are you looking to use?

The otelConfig field supports a supplemental OTel configuration that supports all the components that are built into the CloudWatch Agent.

[jnicholls]

@jefchien Thanks so much for the response! Ok that is good to know, I will not run the add-ons together. I am mostly interested in two things from the collector: exporting metrics as EMF logs, and exporting traces to X-Ray. As far as log collection, the Fluent that's coupled with the CloudWatch Observability add-on is perfect. If the collector runs as a daemon and not a side-car container, I'd be less inclined to bother with the EMF logging of metrics, and just create my own OTel metrics exporter that writes out EMF directly so that Fluent can pick it up and ship it over to CloudWatch. I'd still be interested in traces being exported to X-Ray.

When you say the agent can be installed as a side-car separately and manually, do you mean in addition to the CloudWatch agent being installed as a DaemonSet? Honestly I think I'd be fine with reusing the existing DaemonSet, I'd just need a way to address the agent; like creating a Service that routes requests to it, or if it's binding to the Host network I could probably address it via mounting the HOST_IP variable into my containers. Does that make any sense?

Thanks again for your attention!

[jefchien]

When you say the agent can be installed as a side-car separately and manually, do you mean in addition to the CloudWatch agent being installed as a DaemonSet?

Yes. It's possible to install the agent as a sidecar, but it would not be managed by or through the add-on.

Honestly I think I'd be fine with reusing the existing DaemonSet, I'd just need a way to address the agent; like creating a Service that routes requests to it, or if it's binding to the Host network I could probably address it via mounting the HOST_IP variable into my containers.

The add-on manages a Service (cloudwatch-agent) in the amazon-cloudwatch namespace to route metrics/traces to the CloudWatch agent pods that it manages. You can configure your CloudWatch agents in the add-on configuration to enable trace collection (so it opens the ports required) and send your traces to the agent from your containers through the Service.