Telemetry Challenges in Orleans Silo with ActivitySource
TLDR Brian was struggling with telemetry across his system. The issue was tracing activities in methods. After a series of failed measures, they've found the mistake of improper instance declarations of "activity".
Brian
Wed, 28 Jun 2023 00:35:07 UTCconfiguring Orleans silo like this: ``` silo.Services.AddOpenTelemetry() .WithTracing(trace => { trace .AddSource("*") .SetResourceBuilder( ResourceBuilder .CreateDefault() .AddEnvironmentVariableDetector() .AddTelemetrySdk() .AddService(serviceName: serviceName, serviceVersion: serviceVersion)); trace.AddOtlpExporter(otlp => otlp.Endpoint = new Uri($"http://{hostContext.Configuration["Otel:Collector"]}")); trace.AddConsoleExporter(); });```
Brian
Wed, 28 Jun 2023 00:35:50 UTCI can only see the console output for the following method: ```public Task<IList<Election>> GetElections(ElectionId? electionId = null, bool openOnly = true) { ActivityContext? activityContext = null; using var activity = _activities.StartActivity(ActivityKind.Server); IList<Election> elections; try { activity?.SetTag("electionId", electionId?.ToString()); activity?.SetTag("openOnly", openOnly.ToString()); elections = electionId == null ? State.Elections.Where(e => !e.Deleted && (e.Status == "Active" || !openOnly)).ToList() : State.Elections.Where(x => x.Id == electionId && !x.Deleted).ToList(); activity?.SetStatus(ActivityStatusCode.Ok); } catch (Exception ex) { activity?.RecordException(ex); activity?.SetStatus(ActivityStatusCode.Error, ex.Message); throw; } return Task.FromResult(elections); }```
Brian
Wed, 28 Jun 2023 00:36:29 UTChere's one that doesn't work: ```public async Task Create(ElectionId id, Title title, Description? description) { var activity = _activities.StartActivity(ActivityKind.Consumer); try { activity?.SetTag("id", id); activity?.SetTag("title", title); activity?.SetTag("description", description); var @event = new Vote.Election.Created { Id = this.GetPrimaryKey(), Title = title, Description = description }; var activityEvent = new ActivityEvent(@event.GetType().FullName ?? @event.GetType().Name, tags: new ActivityTagsCollection { { "Id", @event.Id }, { "Title", @event.Title }, { "Description", @event.Description } }); RaiseEvent(@event); activity?.AddEvent(activityEvent); _subscribeToElections = this.GetStreamProvider("StreamProvider").GetStream<object>("Election", this.GetPrimaryKey()); if (_publishToApplication != null) await _publishToApplication.OnNextAsync(@event); await ConfirmEvents(); activity?.SetStatus(ActivityStatusCode.Ok); } catch (Exception ex) { activity?.SetStatus(ActivityStatusCode.Error, ex.Message); throw; } }```
Brian
Wed, 28 Jun 2023 00:37:59 UTCI guess the only other thing worth mentioning is that I am injecting the `ActivitySource` as follows: `silo.Services.AddSingleton(activitySource);`
Brian
Wed, 28 Jun 2023 00:38:43 UTCso all grains have the same instance of `ActivitySource`, all activities are derived from that source, and yet only one method's activities are successfully generated?
Brian
Wed, 28 Jun 2023 01:13:25 UTCfound it: `var activity = ...` vs `using var activity = ...`
SigNoz Community
Indexed 1061 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Brian
Wed, 28 Jun 2023 00:34:44 UTCcontinuing to struggle getting all of the telemetry to come across. in fact, there is only one method that does produce telemetry in the backend.