> ## Documentation Index
> Fetch the complete documentation index at: https://docs.unomiq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sending Traces

> Learn how to send OpenTelemetry traces to the Unomiq Gateway API using the Unomiq SDK, a Collector Sidecar, or directly from your application

This guide describes how to send OpenTelemetry traces to the [Unomiq OTel Gateway API](/api-reference/gateway/export-traces).

***

## Prerequisites

### API Credentials

Create API credentials from the [Unomiq Dashboard](https://dashboard.unomiq.com). The credentials must have the `write:traces` permission.

This will give you an API key (**Client ID**) and secret (**Client Secret**).

***

## With `unomiq-sdk`

The [`unomiq-sdk`](https://pypi.org/project/unomiq-sdk/) Python package handles OAuth2-authenticated OTLP export and [unit attribute attachment](/guides/attaching-unit-to-traces). It can be used on its own (direct export) or alongside an OTel Collector sidecar.

### Configuration

Set the following environment variables for your application:

| Variable               | Description                                           |
| ---------------------- | ----------------------------------------------------- |
| `UNOMIQ_CLIENT_ID`     | API Key from the Unomiq Management API / dashboard    |
| `UNOMIQ_CLIENT_SECRET` | API Secret from the Unomiq Management API / dashboard |

The SDK uses these credentials internally to acquire and refresh OAuth2 tokens and export traces to the Unomiq Gateway.

### Direct Export (No Sidecar)

In this approach, `unomiq-sdk` handles everything: it creates a `TracerProvider`, configures an OAuth2-authenticated OTLP exporter, and attaches unit attributes to spans.

```
Your App ──(OAuth2 Bearer token)──▶ Gateway
```

This is useful for serverless environments (Cloud Run, Lambda) or when you want fewer infrastructure components.

```python theme={null}
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
from unomiq.sdk import Unomiq


def configure_tracing(service_name: str, service_version: str) -> trace.Tracer:
    """Set up tracing with Unomiq SDK (handles OAuth and export)."""

    resource = Resource.create({
        SERVICE_NAME: service_name,
        SERVICE_VERSION: service_version,
    })

    Unomiq.init(
        app_name=service_name,
        resolve_unit=lambda: "my-unit-id",
        resolve_parent_unit=lambda: "my-org-id",
        resource_attributes=dict(resource.attributes),
    )

    return trace.get_tracer(service_name, service_version)
```

With this setup, the SDK reads `UNOMIQ_CLIENT_ID` and `UNOMIQ_CLIENT_SECRET` from the environment, creates an authenticated OTLP exporter, and sets up the global `TracerProvider`. No additional exporter or provider configuration is needed.

For frameworks like Flask or Django, you can use OpenTelemetry instrumentor libraries after calling `Unomiq.init()`:

```python theme={null}
from opentelemetry.instrumentation.flask import FlaskInstrumentor

tracer = configure_tracing("my-service", "1.0.0")
FlaskInstrumentor().instrument_app(app)
```

### With OTel Collector Sidecar

You can also use `unomiq-sdk` alongside a Collector sidecar. In this case, the sidecar handles OAuth and export, while `unomiq-sdk` only handles [unit attribute attachment](/guides/attaching-unit-to-traces).

```
Your App ──(no auth)──▶ OTel Collector Sidecar ──(OAuth2 Bearer token)──▶ Gateway
```

```python theme={null}
from opentelemetry import trace
from unomiq.sdk import Unomiq

# The TracerProvider is already created by opentelemetry-instrument or your
# own setup. Unomiq.init() only adds the UnitSpanProcessor.
Unomiq.init(
    resolve_unit=lambda: "my-unit-id",
    resolve_parent_unit=lambda: "my-org-id",
)

tracer = trace.get_tracer(__name__)
```

Since auth and export are handled by the collector, the SDK only needs `resolve_unit` / `resolve_parent_unit`. No `app_name` or `resource_attributes` are required.

Point your application's OTLP exporter at the local collector:

```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
```

See the [OTel Collector Sidecar](#otel-collector-sidecar-configuration) section below for the collector configuration.

***

## Without `unomiq-sdk`

If you are not using `unomiq-sdk`, you can send traces by managing the OAuth2 flow and OTLP export yourself. There are two approaches:

1. **Collector Sidecar** -- an OTel Collector runs alongside your app, handles OAuth2, and forwards traces to the gateway.
2. **Direct from Application** -- your application code manages OAuth2 tokens and sends traces to the gateway.

Both approaches use the **OAuth2 client credentials** grant type, with automatic **refresh token** support when available.

### Configuration

Set the following environment variables:

| Variable               | Description                          |
| ---------------------- | ------------------------------------ |
| `UNOMIQ_CLIENT_ID`     | API Key from the Unomiq dashboard    |
| `UNOMIQ_CLIENT_SECRET` | API Secret from the Unomiq dashboard |
| `UNOMIQ_TOKEN_URL`     | `https://oauth-api.unomiq.com/token` |
| `UNOMIQ_AUDIENCE`      | `https://gateway-api.unomiq.com`     |
| `UNOMIQ_OTLP_ENDPOINT` | `https://gateway-api.unomiq.com`     |

### Direct from Application (Python)

In this approach, your application manages OAuth2 tokens and sends authenticated traces directly to the gateway. No sidecar is needed.

```
Your App ──(OAuth2 Bearer token)──▶ Gateway
```

#### Step 1: OAuth2 Token Manager

Create a token manager that handles the client credentials flow and automatic refresh.
If the token endpoint returns a `refresh_token`, subsequent renewals use the lighter
`refresh_token` grant instead of re-sending client credentials:

```python theme={null}
"""OAuth2 token manager for OTLP exporters."""

import threading
import logging
from typing import Optional
from datetime import datetime, timedelta

import requests

logger = logging.getLogger(__name__)


class OAuthTokenManager:
    """Manages OAuth2 token retrieval and automatic refresh."""

    def __init__(
        self,
        client_id: str,
        client_secret: str,
        token_url: str,
        scopes: str,
        audience: str,
        refresh_buffer: int = 300,  # Refresh 5 minutes before expiry
    ):
        self.client_id = client_id
        self.client_secret = client_secret
        self.token_url = token_url
        self.scopes = scopes
        self.audience = audience
        self.refresh_buffer = refresh_buffer

        self._token: Optional[str] = None
        self._refresh_token: Optional[str] = None
        self._token_expiry: Optional[datetime] = None
        self._lock = threading.Lock()
        self._stop_refresh = threading.Event()

        # Get initial token
        self._fetch_token()

        # Start background refresh
        self._refresh_thread = threading.Thread(
            target=self._refresh_loop, daemon=True, name="oauth-token-refresh"
        )
        self._refresh_thread.start()

    def _fetch_token(self) -> None:
        """Fetch a new access token using client credentials."""
        response = requests.post(
            self.token_url,
            data={
                "grant_type": "client_credentials",
                "client_id": self.client_id,
                "client_secret": self.client_secret,
                "scope": self.scopes,
                "audience": self.audience,
            },
            headers={"Content-Type": "application/x-www-form-urlencoded"},
            timeout=10,
        )
        response.raise_for_status()

        token_data = response.json()
        self._token = token_data["access_token"]
        self._refresh_token = token_data.get("refresh_token")
        expires_in = token_data.get("expires_in", 3600)
        self._token_expiry = datetime.now() + timedelta(seconds=expires_in)
        logger.info("OAuth token fetched, expires in %d seconds", expires_in)

    def _refresh_token_grant(self) -> None:
        """Use the refresh token to obtain a new access token."""
        try:
            response = requests.post(
                self.token_url,
                data={
                    "grant_type": "refresh_token",
                    "client_id": self.client_id,
                    "refresh_token": self._refresh_token,
                    "audience": self.audience,
                },
                headers={"Content-Type": "application/x-www-form-urlencoded"},
                timeout=10,
            )
            response.raise_for_status()

            token_data = response.json()
            self._token = token_data["access_token"]
            # Support refresh token rotation
            if "refresh_token" in token_data:
                self._refresh_token = token_data["refresh_token"]
            expires_in = token_data.get("expires_in", 3600)
            self._token_expiry = datetime.now() + timedelta(seconds=expires_in)
            logger.info("OAuth token refreshed via refresh_token, expires in %d seconds", expires_in)

        except Exception as e:
            logger.warning("Refresh token grant failed: %s, falling back to client_credentials", e)
            self._refresh_token = None
            self._fetch_token()

    def _should_refresh(self) -> bool:
        if self._token is None or self._token_expiry is None:
            return True
        return (self._token_expiry - datetime.now()).total_seconds() <= self.refresh_buffer

    def _refresh_loop(self) -> None:
        while not self._stop_refresh.is_set():
            try:
                if self._should_refresh():
                    with self._lock:
                        if self._should_refresh():
                            if self._refresh_token:
                                self._refresh_token_grant()
                            else:
                                self._fetch_token()
            except Exception:
                logger.exception("Error refreshing OAuth token")
            self._stop_refresh.wait(60)

    def get_headers(self) -> dict:
        """Return authorization headers for the OTLP exporter."""
        with self._lock:
            if self._should_refresh():
                if self._refresh_token:
                    self._refresh_token_grant()
                else:
                    self._fetch_token()
            return {"Authorization": f"Bearer {self._token}"}

    def stop(self) -> None:
        """Stop the background refresh thread."""
        self._stop_refresh.set()
```

#### Step 2: Configure the Tracer

Wire the token manager into the OTLP exporter:

```python theme={null}
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

from oauth_token_manager import OAuthTokenManager


def configure_tracing(service_name: str, service_version: str) -> trace.Tracer:
    """Set up OpenTelemetry tracing with OAuth2-authenticated export."""

    # Create the token manager
    token_manager = OAuthTokenManager(
        client_id=os.environ["UNOMIQ_CLIENT_ID"],
        client_secret=os.environ["UNOMIQ_CLIENT_SECRET"],
        token_url=os.environ["UNOMIQ_TOKEN_URL"],
        scopes="write:traces",
        audience=os.environ["UNOMIQ_AUDIENCE"],
    )

    # Build the OTLP exporter with auth headers
    endpoint = os.environ["UNOMIQ_OTLP_ENDPOINT"].rstrip("/") + "/v1/traces"
    exporter = OTLPSpanExporter(
        endpoint=endpoint,
        headers=token_manager.get_headers(),
    )

    # Assemble the tracer provider
    resource = Resource.create({
        SERVICE_NAME: service_name,
        SERVICE_VERSION: service_version,
    })
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

    return trace.get_tracer(service_name, service_version)
```

#### Step 3: Use the Tracer

```python theme={null}
tracer = configure_tracing("my-service", "1.0.0")

with tracer.start_as_current_span("my-operation") as span:
    span.set_attribute("key", "value")
    # ... your code here
```

For frameworks like Flask or Django, you can also use the OpenTelemetry instrumentor libraries (e.g., `opentelemetry-instrumentation-flask`) after calling `configure_tracing()`.

#### Required Dependencies

```
opentelemetry-api
opentelemetry-sdk
opentelemetry-exporter-otlp-proto-http
requests
```

***

## OTel Collector Sidecar Configuration

This section applies to both the `unomiq-sdk` sidecar approach and the without-SDK sidecar approach. In both cases, the collector handles OAuth2 token acquisition and forwards authenticated traces to the gateway.

```
Your App ──(no auth)──▶ OTel Collector Sidecar ──(OAuth2 Bearer token)──▶ Gateway
```

The collector uses the [`oauth2clientauthextension`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/oauth2clientauthextension) from the OpenTelemetry Collector Contrib distribution.

### Environment Variables

Set the following environment variables for the **collector** (not the application):

| Variable               | Description                                           |
| ---------------------- | ----------------------------------------------------- |
| `UNOMIQ_CLIENT_ID`     | API Key from the Unomiq Management API / dashboard    |
| `UNOMIQ_CLIENT_SECRET` | API Secret from the Unomiq Management API / dashboard |
| `UNOMIQ_TOKEN_URL`     | `https://oauth-api.unomiq.com/token`                  |
| `UNOMIQ_AUDIENCE`      | `https://gateway-api.unomiq.com`                      |
| `UNOMIQ_OTLP_ENDPOINT` | `https://gateway-api.unomiq.com`                      |

### Collector Config

Create an `otel-collector-config.yaml`:

```yaml theme={null}
extensions:
  oauth2client:
    client_id: ${env:UNOMIQ_CLIENT_ID}
    client_secret: ${env:UNOMIQ_CLIENT_SECRET}
    token_url: ${env:UNOMIQ_TOKEN_URL}
    endpoint_params:
      audience: ${env:UNOMIQ_AUDIENCE}
    scopes:
      - write:traces
    timeout: 10s

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 100

exporters:
  otlphttp:
    endpoint: ${env:UNOMIQ_OTLP_ENDPOINT}
    auth:
      authenticator: oauth2client

service:
  extensions: [oauth2client]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]
```

Key points:

* The `oauth2client` extension acquires a token using the client credentials grant and automatically refreshes it before expiry.
* The `otlphttp` exporter references the extension via `auth.authenticator`, so every outgoing request includes the `Authorization: Bearer <token>` header.
* The receiver listens on standard OTLP ports (4317 for gRPC, 4318 for HTTP) without requiring any authentication from your application.

### Running the Collector

Use the **contrib** distribution of the collector, which includes the `oauth2clientauthextension`. The core distribution does not include it.

**Docker Compose example:**

```yaml theme={null}
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
    environment:
      - UNOMIQ_CLIENT_ID
      - UNOMIQ_CLIENT_SECRET
      - UNOMIQ_TOKEN_URL
      - UNOMIQ_AUDIENCE
      - UNOMIQ_OTLP_ENDPOINT
    ports:
      - "4317:4317"
      - "4318:4318"
```

### Configuring Your Application

Point your application's OTLP exporter at the local collector. No authentication configuration is needed in the application itself.

**Python (automatic instrumentation):**

```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
opentelemetry-instrument python app.py
```

**Python (SDK):**

```python theme={null}
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(endpoint="http://otel-collector:4318/v1/traces")
```

**Any language:** Set the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable to `http://localhost:4318` (or the collector's hostname in Docker/Kubernetes).

***

## Choosing an Approach

|                      | `unomiq-sdk` (Direct)          | `unomiq-sdk` (Sidecar)                        | Without SDK (Sidecar)                                  | Without SDK (Direct)                                   |
| -------------------- | ------------------------------ | --------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------ |
| **Auth handling**    | SDK manages tokens             | Collector manages tokens                      | Collector manages tokens                               | Application manages tokens                             |
| **Unit attachment**  | Automatic via SDK              | Automatic via SDK                             | Manual ([see guide](/guides/attaching-unit-to-traces)) | Manual ([see guide](/guides/attaching-unit-to-traces)) |
| **Code changes**     | Minimal                        | Minimal                                       | None (env vars only)                                   | Token manager + exporter setup                         |
| **Infrastructure**   | No extra infrastructure        | Requires running a collector                  | Requires running a collector                           | No extra infrastructure                                |
| **Language support** | Python                         | Python + any (collector is language-agnostic) | Any (language-agnostic)                                | Requires per-language implementation                   |
| **Best for**         | Serverless (Cloud Run, Lambda) | Docker, Kubernetes                            | Docker, Kubernetes (non-Python)                        | Serverless (non-Python or without SDK)                 |

***

## Monitoring Sent Traces

Once your traces are flowing to the gateway, you can monitor them in two ways:

* **API** -- Use the [Get Live Traces](/api-reference/engine/get-live-traces) endpoint to programmatically retrieve and inspect incoming traces in real time.
* **Dashboard** -- View and explore traces visually from the [Unomiq Dashboard](https://dashboard.unomiq.com).

***

## Troubleshooting

### Common Issues

**401 Unauthorized from the gateway**

* Verify that your client ID and secret are correct (`UNOMIQ_CLIENT_ID` / `UNOMIQ_CLIENT_SECRET`).
* Check that the scopes match what the gateway expects (`write:traces`).
* Ensure the token endpoint is reachable from your environment.

**No traces appearing at the gateway**

* Confirm the OTLP endpoint is set to the correct gateway URL.
* For the sidecar approach, check collector logs for export errors.
* For direct export, enable debug logging: `logging.getLogger('opentelemetry').setLevel(logging.DEBUG)`.

**Token refresh failures**

* The sidecar collector and the `unomiq-sdk` / Python token manager all refresh tokens automatically before expiry. Check logs for errors from the token endpoint.
* Ensure your OAuth2 client has not been revoked or rate-limited.

### Viewing Collector Logs (Sidecar Approach)

Add the `debug` exporter to your collector pipeline for verbose output:

```yaml theme={null}
exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      exporters: [otlphttp, debug]
```