#support

Telemetry Challenges in Orleans Silo with ActivitySource

TLDR Brian was struggling with telemetry across his system. The issue was tracing activities in methods. After a series of failed measures, they've found the mistake of improper instance declarations of "activity".

Powered by Struct AI
7
3mo
Solved
Join the chat
Jun 28, 2023 (3 months ago)
Brian
Photo of md5-be4dd17aaeb44a89495650071f137e6c
Brian
12:34 AM
continuing to struggle getting all of the telemetry to come across. in fact, there is only one method that does produce telemetry in the backend.
12:35
Brian
12:35 AM
configuring Orleans silo like this:

        silo.Services.AddOpenTelemetry()
            .WithTracing(trace =>
            {
                trace
                    .AddSource("*")
                    .SetResourceBuilder(
                        ResourceBuilder
                            .CreateDefault()
                            .AddEnvironmentVariableDetector()
                            .AddTelemetrySdk()
                            .AddService(serviceName: serviceName, serviceVersion: serviceVersion));
                trace.AddOtlpExporter(otlp => otlp.Endpoint = new Uri($"http://{hostContext.Configuration["Otel:Collector"]}"));
                trace.AddConsoleExporter();
            });
12:35
Brian
12:35 AM
I can only see the console output for the following method:

public Task<IList<Election>> GetElections(ElectionId? electionId = null, bool openOnly = true)

    {
        ActivityContext? activityContext = null;
        using var activity = _activities.StartActivity(ActivityKind.Server);
        IList<Election> elections;
        try
        {
            activity?.SetTag("electionId", electionId?.ToString());
            activity?.SetTag("openOnly", openOnly.ToString());
            elections = electionId == null
                ? State.Elections.Where(e => !e.Deleted &&
                                             (e.Status == "Active" || !openOnly)).ToList()
                : State.Elections.Where(x => x.Id == electionId && !x.Deleted).ToList();
            activity?.SetStatus(ActivityStatusCode.Ok);
        }
        catch (Exception ex)
        {
            activity?.RecordException(ex);
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            throw;
        }
        return Task.FromResult(elections);
    }
12:36
Brian
12:36 AM
here's one that doesn't work:

public async Task Create(ElectionId id, Title title, Description? description)
    {
        var activity = _activities.StartActivity(ActivityKind.Consumer);
        try
        {
            activity?.SetTag("id", id);
            activity?.SetTag("title", title);
            activity?.SetTag("description", description);
            var @event = new Vote.Election.Created
            {
                Id = this.GetPrimaryKey(),
                Title = title,
                Description = description
            };
            var activityEvent = new ActivityEvent(@event.GetType().FullName ?? @event.GetType().Name, tags: new ActivityTagsCollection
            {
                { "Id", @event.Id },
                { "Title", @event.Title },
                { "Description", @event.Description }
            });
            RaiseEvent(@event);
            activity?.AddEvent(activityEvent);
            _subscribeToElections = this.GetStreamProvider("StreamProvider").GetStream<object>("Election", this.GetPrimaryKey());
            if (_publishToApplication != null) await _publishToApplication.OnNextAsync(@event);
            await ConfirmEvents();
            activity?.SetStatus(ActivityStatusCode.Ok);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            throw;
        }
    }
12:37
Brian
12:37 AM
I guess the only other thing worth mentioning is that I am injecting the ActivitySource as follows: silo.Services.AddSingleton(activitySource);
12:38
Brian
12:38 AM
so all grains have the same instance of ActivitySource, all activities are derived from that source, and yet only one method's activities are successfully generated?
01:13
Brian
01:13 AM
found it:

var activity = ... vs using var activity = ...