Traces

This document covers how SDKs should add support for Performance Monitoring with Distributed Tracing.

This should give an overview of the APIs that SDKs need to implement, without mandating internal implementation details.

Reference implementations:

This document uses standard interval notation, where [ and ] indicates closed intervals, which include the endpoints of the interval, while ( and ) indicates open intervals, which exclude the endpoints of the interval. An interval [x, y) covers all values starting from x up to but excluding y.

This section describes the options SDKs should expose to configure tracing and performance monitoring.

Tracing is enabled by setting either a tracesSampleRate or tracesSampler. If not set, these options default to undefined or null, making tracing opt-in.

This option is deprecated and should be removed from all SDKs.

This should be a floating-point number in the range [0, 1] and represents the percentage chance that any given transaction will be sent to Sentry. So, barring outside influence, 0.0 is a guaranteed 0% chance (none will be sent) and 1.0 is a guaranteed 100% chance (all will be sent). This rate applies equally to all transactions; in other words, each transaction has an equal chance of being marked as sampled = true, based on the tracesSampleRate.

See more about how sampling should be performed below.

This should be a callback function, triggered when a transaction is started. It should be given a samplingContext object and should return a sample rate in the range of [0, 1] for the transaction in question. This sample rate should behave the same way as the tracesSampleRate above. The only difference is that it only applies to the newly-created transaction and that different transactions can be sampled at different rates. Returning 0.0 should force the transaction to be dropped (set to sampled = false) and returning 1.0 should force the transaction to be sent (set to sampled = true).

Historically, the tracesSampler callback could have also returned a boolean to force a sampling decision (with false equivalent to 0.0 and true equivalent to 1.0). This behavior is now deprecated and should be removed from all SDKs.

See more about how sampling should be performed below.

Sentry SDKs propagate trace information to downstream SDKs via headers on outgoing HTTP requests. The tracePropagationTargets option gives users a mechanism of controlling to which outgoing HTTP requests these headers should be attached. For example, users can specify this property to keep trace propagation within their infrastructure, thereby preventing data within the headers from being sent to third party services.

This option takes an array of strings and/or regular expressions. SDKs should only add trace headers to an outgoing request if the request's URL matches the regex or, in the case of string literals, contains at least one of the items from the array. String literals do not have to be full matches, meaning the URL of a request is matched when it contains a string provided through the option.

SDKs may choose a default value which makes sense for their use case. Most SDKs default to the regex .* (meaning they attach headers to all outgoing requests), but deviation is allowed if necessary. For example, because of CORS, browser-based SDKs default to only adding headers to domain-internal requests.

See sentry-trace, traceparent and baggage for more details on the individual headers which are attached to outgoing requests.

The following example shows which URLs of outgoing requests would (not) match a given tracePropagationTargets array:

Copied
// Entries can be strings or regex
tracePropagationTargets: ['localhost', /^\// ,/myApi.com\/v[2-4]/]

URLs matching: 'localhost:8443/api/users', 'mylocalhost:8080/api/users', '/api/envelopes', 'myApi.com/v2/projects'
URLs not matching: 'someHost.com/data', 'myApi.com/v1/projects'

This must be a boolean value. Default is false. This option controls trace continuation from unknown 3rd party services that happen to be instrumented by a Sentry SDK.

If the SDK is able parse an org ID from the configured DSN, it must be propagated as a baggage entry with the key sentry-org. Given a DSN of https://1234@o1.ingest.us.sentry.io/1, the org ID is 1, based on o1.

Addiotnally, the SDK must be configurable with an optional org: <org-id> setting that takes precedence over the parsed value from the DSN. This option should be set when running a self-hosted version of Sentry or if a non-standard Sentry DSN is used, such as when using a local Relay.

On incoming traces, the SDK must compare the sentry-org baggage value against its own parsed value from the DSN or org setting. Only if both match, the trace is continued. If there is no match, neither the trace ID, the parent sampling decision nor the baggage should be taken into account. The SDK should behave like it is the head of trace in this case, and not consider any propagted values.

This behavior can be disabled by setting strictTraceContinuation: false in the SDK init call. Initially, SDKs must introduce the this option with a default value of false. Once the majority of SDKs have introduced this option, we'll change the default value to true (in a major version bump), making it opt-out.

Regardless of strictTraceContinuation being set to true or false, if the SDK is either configured with a org or was able to parse the value from the DSN, incoming traces containing an org value in the baggage that does not match the one from the receiving SDK, the trace is not continuned.

Examples:

baggage: sentry-org: 1, SDK config: org: 1, strictTraceContinuation: false -> continue trace baggage: sentry-org: none, SDK config: org: 1, strictTraceContinuation: false -> continue trace baggage: sentry-org: 1, SDK config: org: none, strictTraceContinuation: false -> continue trace baggage: sentry-org: none, SDK config: org: none, strictTraceContinuation: false -> continue trace baggage: sentry-org: 1, SDK config: org: 2, strictTraceContinuation: false -> start new trace

baggage: sentry-org: 1, SDK config: org: 1, strictTraceContinuation: true -> continue trace baggage: sentry-org: none, SDK config: org: 1, strictTraceContinuation: true -> start new trace baggage: sentry-org: 1, SDK config: org: none, strictTraceContinuation: true -> start new trace baggage: sentry-org: none, SDK config: org: none, strictTraceContinuation: true -> continue trace baggage: sentry-org: 1, SDK config: org: 2, strictTraceContinuation: true -> start new trace

This should be a boolean value. Default is false. When set to true transactions should be created for HTTP OPTIONS requests. When set to false NO transactions should be created for HTTP OPTIONS requests. This configuration is most valuable on backend server SDKs. If this configuration does not make sense for an SDK it can be omitted.

Because transaction payloads have a maximum size enforced on the ingestion side, SDKs should limit the number of spans that are attached to a transaction. This is similar to how breadcrumbs and other arbitrarily sized lists are limited to prevent accidental misuse. If new spans are added once the maximum is reached, the SDK should drop the spans and ideally use the internal logging to help debugging.

The maxSpans should be implemented as an internal, non-configurable, constant that defaults to 1000. It may become configurable if there is justification for that in a given platform.

The maxSpans limit may also help avoiding transactions that never finish (in platforms that keep a transaction open for as long as spans are open), preventing OOM errors, and generally avoiding degraded application performance.

As of writing, transactions are implemented as an extension of the Event model.

The distinctive feature of a Transaction is type: "transaction".

Apart from that, the Event gets new fields: spans, contexts.TraceContext.

In memory, spans build up a conceptual tree of timed operations. We call the whole span tree a transaction. Sometimes we use the term "transaction" to refer to a span tree as a whole tree, sometimes to refer specifically to the root span of the tree.

Over the wire, transactions are serialized to JSON as an augmented Event, and sent as envelopes. The different envelope types are for optimizing ingestion (so we can route "transaction events" differently than other events, mostly "error events").

In the Sentry UI, you can use Discover to look at all events regardless of type, and the Issues and Performance sections to dive into errors and transactions, respectively. The user-facing tracing documentation explains more of the concepts on the product level.

The Span class stores each individual span in a trace.

The Transaction class is like a span, with a few key differences:

  • Transactions have name, spans don't.
  • Transactions must specify the source of its name to indicate how the transaction name was generated.
  • Calling the finish method on spans record the span's end timestamp. For transactions, the finish method additionally sends an event to Sentry.

The Transaction class may inherit from Span, but that's an implementation detail. Semantically, transactions represent both the top-level span of a span tree as well as the unit of reporting to Sentry.

  • Span Interface

    • When a Span is created, set the startTimestamp to the current time
    • SpanContext is the attribute collection for a Span (Can be an implementation detail). When possible SpanContext should be immutable.
    • Span should have a method startChild which creates a new span with the current span's id as the new span's parentSpanId and the current span's sampled value copied over to the new span's sampled property
    • The startChild method should respect the maxSpans limit, and once the limit is reached the SDK should not create new child spans for the given transaction.
    • Span should have a method called toSentryTrace which returns a string that could be sent as a header called sentry-trace.
    • Span should have a method called toW3CTrace which returns a string that could be sent as a header called traceparent.
    • Span should have a method called iterHeaders (adapt to platform's naming conventions) that returns an iterable or map of header names and values. This is a thin wrapper containing return {"sentry-trace": toSentryTrace(), "traceparent": toW3CTrace()} right now. See continueFromHeaders as to why this exists and should be preferred when writing integrations.

  • Transaction Interface

    • A Transaction internally holds a flat list of child Spans (not a tree structure)
    • Transaction has additionally a setName method that sets the name of the transaction
    • Transaction receives a TransactionContext on creation (new property vs. SpanContext is name)
    • Since a Transaction inherits a Span it has all functions available and can be interacted with like it was a Span
    • A transaction is either sampled (sampled = true) or unsampled (sampled = false), a decision which is either inherited or set once during the transaction's lifetime, and in either case is propagated to all children. Unsampled transactions should not be sent to Sentry.
    • TransactionContext should have a static/ctor method called fromSentryTrace which prefills a TransactionContext with data received from a sentry-trace header value
    • TransactionContext should have a static/ctor method called fromW3CTrace which prefills a TransactionContext with data received from a traceparent header value
    • TransactionContext should have a static/ctor method called continueFromHeaders(headerMap) which is really just a thin wrapper around fromSentryTrace(headerMap.get("sentry-trace")) right now. This should be preferred by integration/framework-sdk authors over fromSentryTrace as it hides the exact header names used deeper in the core sdk, and leaves opportunity for using additional headers (from the W3C) in the future without changing all integrations.

  • Span.finish()

    • Accepts an optional endTimestamp to allow users to set a custom endTimestamp on the finished span
    • If an endTimestamp value is not provided, set endTimestamp to the current time (in payload timestamp)

  • Transaction.finish()

    • super.finish() (call finish on Span)
    • Send it to Sentry only if sampled == true
    • Like spans, can be given an optional endTimestamp value that should be passed into the span.finish() call
    • A Transaction needs to be wrapped in an Envelope and sent to the Envelope Endpoint
    • The Transport should use the same internal queue for Transactions / Events
    • The Transport should implement category-based rate limiting →
    • The Transport should deal with wrapping a Transaction in an Envelope internally

Each transaction has a sampling decision, that is, a boolean which declares whether or not it should be sent to Sentry. This should be set exactly once during a transaction's lifetime, and should be stored in an internal sampled boolean.

There are multiple ways a transaction can end up with a sampling decision:

  • Random sampling according to a static sample rate set in tracesSampleRate
  • Random sampling according to a dynamic sample rate returned by tracesSampler
  • Absolute decision (100% chance or 0% chance) returned by tracesSampler
  • If the transaction has a parent, inheriting its parent's sampling decision
  • Absolute decision passed to startTransaction

If more than one option could apply, the following rules determine which takes precedence:

  1. If a sampling decision is passed to startTransaction (startTransaction({name: "my transaction", sampled: true})), that decision will be used, regardlesss of anything else
  2. If tracesSampler is defined, its decision will be used. It can choose to keep or ignore any parent sampling decision, or use the sampling context data to make its own decision or choose a sample rate for the transaction.
  3. If tracesSampler is not defined, but there's a parent sampling decision, the parent sampling decision will be used.
  4. If tracesSampler is not defined and there's no parent sampling decision, tracesSampleRate will be used.

If defined, the tracesSampler callback should be passed a samplingContext object, which should include, at minimum:

  • The transactionContext with which the transaction was created
  • A float/double parentSampleRate which contains the sampling rate passed down from the parent
  • A boolean parentSampled which contains the sampling decision passed down from the parent, if any
  • Data from an optional customSamplingContext object passed to startTransaction when it is called manually

Depending on the platform, other default data may be included. (For example, for server frameworks, it makes sense to include the request object corresponding to the request the transaction is measuring.)

A transaction's sampling decision should be passed to all of its children, including across service boundaries. This can be accomplished in the startChild method for same-service children and using the senry-trace header for children in a different service.

To improve the likelihood of capturing complete traces when backend services use a custom sample rate via tracesSampler, the SDK propagates the same random value used for sampling decisions across all services in a trace. This ensures consistent sampling decisions across a trace instead of generating a new random value for each service.

If no tracesSampler callback is used, the SDK fully inherits sampling decisions for propagated traces, and the presence of sample_rand in the DSC doesn't affect the decision. However, this behavior may change in the future.

The random value is set according to the following rules:

  1. When an SDK starts a new trace, sample_rand is always set to a random number in the range of [0, 1]. This explicitly includes traces that aren't sampled, as well as when the tracesSampleRate is set to 0.0 or 1.0.
  2. It is recommended to generate the random number deterministically using the trace ID as seed or source of randomness. The exact method by which the random number is created is implementation defined and may vary between SDK implementations. See 4. on why this behaviour is desirable.
  3. On incoming traces, an SDK assumes the sample_rand value along with the rest of the DSC, overriding an existing value if needed.
  4. If sample_rand is missing on an incoming trace, the SDK creates and from now on propagates a new random number on-the-fly, based on the following rules:
    1. If sample_rate and sampled are propgated, create sample_rand so that it adheres to the invariant. This means, for a decision of True generate a random number in half-open range [0, rate) and for a decision of False generate a random number in range [rate, 1].
    2. If the sampling decision is missing, generate a random number in range of [0, 1], like for a new trace.

The SDK should always use the stored random number (sentry-sample_rand) for sampling decisions and should no longer rely on math.random() or similar functions in tracing code:

  1. When the tracesSampler is invoked, this also applies to the return value of traces sampler. That is, trace["sentry-sample_rand"] < tracesSampler(context)
  2. Otherwise, when the SDK is the head of a trace, this also applies to sample decisions based on tracesSampleRate. That is, trace["sentry-sample_rand"] < config.tracesSampleRate
  3. There is no more direct comparison with math.random() during the sampling process.

When using a tracesSampler, the proper way to inherit a parent's sampling decision is to return the parent's sample rate, instead of leaving the decision as a float (for example, 1.0). This way, Sentry can still extrapolate counts correctly.

Copied
tracesSampler: ({ name, parentSampleRate }) => {
  // Inherit the trace parent's sample rate if there is one. Sampling is deterministic
  // for one trace, i.e. if the parent was sampled, we will be sampled too at the same
  // rate.
  if (typeof parentSampleRate === "number") {
    return parentSampleRate;
  }

  // Else, use default sample rate (replacing tracesSampleRate).
  return 0.5;
},

If the SDK supports backpressure handling, the overall sampling rate needs to be divided by the downsamplingFactor from the backpressure monitor. See the backpressure spec for more details.

The header is used for trace propagation. SDKs use the header to continue traces from upstream services (incoming HTTP requests), and to propagate tracing information to downstream services (outgoing HTTP requests).

sentry-trace = traceid-spanid-sampled

sampled is optional. So at a minimum, it's expected:

sentry-trace = traceid-spanid

To offer a minimal compatibility with the W3C traceparent header (without the version prefix) and Zipkin's b3 headers (which consider both 64 and 128 bits for traceId valid), the sentry-trace header should have a traceId of 128 bits encoded in 32 hex chars and a spanId of 64 bits encoded in 16 hex chars. To avoid confusion with the W3C traceparent header (to which our header is similar but not identical), we call it simply sentry-trace. No version is being defined in the header.

The sentry-trace header should only be attached to an outgoing request if the request's URL matches at least one entry of the tracePropagationTargets SDK option or this options is set to null.

To simplify processing, the value consists of a single (optional) character. The possible values are:

Copied
  - No value means defer

0 - Don't sample

1 - Sampled

Unlike with b3 headers, a sentry-trace header should never consist solely of a sampling decision, with no traceid or spanid values. There are good reasons to always include the traceid and spanid regardless of the sampling decision, and doing so also simplifies implementation.

Besides the usual reasons to use *defer,* in the case of Sentry, a reason would be if a downstream system captures an error event with Sentry. The decision could be done at that point to sample that trace in order to have tracing data available for the reported crash.

sentry-trace = sampled

Which in reality is useful for proxies to set it to 0 and opt out of tracing.

The header is used for trace propagation. SDKs use the header to continue traces from upstream services (e.g. incoming HTTP requests), and to propagate tracing information to downstream services (e.g. outgoing HTTP requests).

traceparent = version-traceid-spanid-traceflags

We can assume a version of 00, as well as traceflags being either -00 or -01. A deferred sampling decision is not part of the specfication. See W3C traceparent header for more information.

The traceparent header should only be attached to an outgoing request if the request's URL matches at least one entry of the tracePropagationTargets SDK option or this option is set to null or not set.

The Sentry.startTransaction function should take two arguments - the transactionContext passed to the Transaction constructor and an optional customSamplingContext object containing data to be passed to tracesSampler (if defined). It creates a Transaction bound to the current hub and returns the instance. Users interact with the instance for creating child spans and, thus, have to keep track of it themselves.

With Sentry.span users can attach spans to an already ongoing transaction. This property returns a SpanProtocol if a running transaction is bound to the scope; otherwise, it returns nil. Although we recommend users keep track of their own transactions, the SDKs should offer a way to expose auto-generated transactions. SDKs shall bind auto-generated transactions to the scope, making them accessible with Sentry.span. If the SDK has global mode enabled, which specifies whether to use global scope management mode and should be true for client applications and false for server applications, Sentry.span shall return the active transaction. If the user disables global mode, Sentry.span shall return the latest active (unfinished) span.

  • Introduce a method called traceHeaders

    • This function returns a header (string) sentry-trace
    • The value should be the trace header string of the Span that is currently on the Scope

  • Introduce a method called startTransaction

    • Takes the same two arguments as Sentry.startTransaction
    • Creates a new Transaction instance
    • Should implement sampling as described in more detail in the 'Sampling' section of this document

  • Modify the method called captureEvent or captureTransaction

    • Don't set lastEventId for transactions

The Scope holds a reference to the current Span or Transaction.

  • Scope Introduce setSpan
    • This can be used internally to pass a Span / Transaction around so that integrations can attach children to it
    • Setting the transaction property on the Scope (legacy) should overwrite the name of the Transaction stored in the Scope, if there is one. With that we give users the option to change the transaction name even if they don't have access to the instance of the Transaction directly.

The beforeSend callback is a special Event Processor that we consider to be of most prominent use. Proper Event Processors are often considered internal.

Transactions should not go through beforeSend. However, they are still processed by Event Processors. This is a compromise between some flexibility in dealing with the current implementation of transactions as events, and leaving room for different lifetime hooks for transactions and spans.

Motivations:

  1. Future-proofing: if users rely on beforeSend for transactions, that would complicate eventually implementing individual span ingestion without breaking user code. As of writing, a transaction is sent as an event, but that is considered an implementation detail.

  2. API compatibility: users have their existing implementation of beforeSend that only ever had to deal with error events. We introduced transactions as a new type of event. As users upgrade to a new SDK version and start using tracing, their beforeSend would start seeing a new type that their code was not meant to handle. Before transactions, they didn't have to care about different event types at all. There are several possible consequences: breaking user apps; silently and unintentionally dropping transactions; transaction events modified in surprising ways.

  3. In terms of usability, beforeSend is not a perfect fit for dropping transactions like it is for dropping errors. Errors are a point-in-time event. When errors happen, users have full context in beforeSend and can modify/drop the event before it goes to Sentry. With transactions the flow is different. Transactions are created and then they are open for some time while child spans are created and appended to it. Meanwhile outgoing HTTP requests include the sampling decision of the current transaction with other services. After spans and the transaction are finished, dropping the transaction in a beforeSend-like hook would leave orphan transactions from other services in a trace. Similarly, modifying the sampling decision to "yes" at this late stage would also produce inconsistent traces.

Previous
Sessions
Next
Backpressure Management
Help improve this content
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").