SigNoz Community

SigNoz Community

#support

Creating an Alert for Error Percentage in Node.js App

TLDR Paulo needed help creating an alert for error percentage in a Node.js app. Srikanth guided them through using the query builder and selecting the correct operations and filters.

Powered by Struct AI

38

8mo

Mar 29, 2023 (8 months ago)

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

01:02 PM

Hello! I need help to create a alert for Error Percentage (10%+) for a app/service called: nodejs-example-app-for-signoz
How i made this query?

Image 1 for Hello! I need help to create a alert for Error Percentage (10%+) for a app/service called: nodejs-example-app-for-signoz
How i made this query?

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

02:38 PM

Do you want to make an alert similar to this?

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

02:41 PM

No, just use thist chart to alert on slack channel

02:43

Paulo

02:43 PM

This is one of default chart on defaults panes.

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

02:45 PM

This is made with SUM_RATE of spans with status_code=STATUS_CODE_ERROR for a service as query A to SUM_RATE of all spans from same service query B and using Formula of (A/B)*100 .

02:45

Srikanth

02:45 PM

You can achieve this with query builder

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

02:46 PM

Metric based or traced-based alert?

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

02:47 PM

No, there is a metric called signoz_calls_total .

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

02:47 PM

Ok i will try

02:50

Paulo

02:50 PM

I use IN or Like?

02:51

Paulo

02:51 PM

Who is equal?

02:51

Paulo

02:51 PM

Status_code IN status_code_error
Status_code LIKE status_code_error

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

03:08 PM

You can use Status_code IN status_code_error

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

06:04 PM

hey bro Srikanth

Image 1 for hey bro <@4K1456>

06:04

Paulo

06:04 PM

The result is 2.88

Image 1 for The result is 2.88

06:05

Paulo

06:05 PM

But on default panel the result is 15%

Image 1 for But on default panel the result is 15%

06:05

Paulo

06:05 PM

what i`m doing wrong?

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

06:07 PM

Select the operations which are the API endpoints; otherwise, it includes all the operations. We call these operations as service entry spans i.e in a trace call there are some operations which are the actual starting points and there are sub operations so when we work with service we are generally interested in top-level/entry-spans for service.

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

06:14 PM

Sorry, but i did not urderstand.

06:15

Paulo

06:15 PM

So, its not possible create a alert for the Error Percentage, from the default (overview metrics) panel.

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

06:15 PM

It is possible. Your filter criteria is not the same as in overview page.

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

06:16 PM

I just want create a alert for this chart

Image 1 for I just want create a alert for this chart

06:16

Paulo

06:16 PM

How i made the correct. filter?

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

06:19 PM

That is exactly what I am referring to. You are halfway through creating the alert. You need to add an operations filter. In the list of operations for the service, there are going to be some operations which are the entry point for the service, and you need to select those operations. For the example trace -> service-a /api/users/{user_id} -> service-a db.operation -> return. In this case, the /api/users/{user_id} is the top-level span i.e span where the trace journey started for the service. There can be a number of sub-operations under that such as db.operation etc...

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

06:23 PM

I can create this here?
https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1680113061665989?thread_ts=1680094940.711099&cid=C01HWQ1R0BC

06:24

Paulo

06:24 PM

And another question, how the default do the service discovery of all, if i understand correctly, i will need to create one by one.

Mar 30, 2023 (8 months ago)

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

02:15 AM

> I can create this here?
Yes
> how the default do the service discovery of all
Are you talking about the service entry spans or service discovery in k8s?

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

04:47 PM

hey Srikanth. Thanks!! I found the operation you mentioned.

Image 1 for hey <@4K1456>. Thanks!! I found the operation you mentioned.

04:47

Paulo

04:47 PM

But has a option to select all?

04:48

Paulo

04:48 PM

Or all that the error percentage chart on overview metrics show as default.

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

04:53 PM

Your question was not clear to me. In this list, you need to select those are the entry-level spans.

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

04:56 PM

Here, i think the chart from home page have a filter that select all pertinent to that app

Image 1 for Here, i think the chart from home page have a filter that select all pertinent to that app

Image 2 for Here, i think the chart from home page have a filter that select all pertinent to that app

04:57

Paulo

04:57 PM

I will need do it manually one by one?

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

04:57 PM

No, key operations are not necessarily top-level operations.

04:58

Srikanth

04:58 PM

For example the tcp.connect is most likely some inner span in service not the top level/entry span

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

04:59 PM

Who operations the error percentage chart form home page uses?

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

05:00 PM

It has an API and queries to fetch those top-level operations. You can open devtools and see it.

Photo of md5-a0f48b46df36888f7a94b0c0f1f05cc5

Paulo

05:09 PM

You are the best. I will configure and back you as soon. Thanks again

Image 1 for You are the best. I will configure and back you as soon. Thanks again

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Setting Up Alerts for p95 and p99 Latencies Using Clickhouse Query

Arnab asked for a Clickhouse query to set up alerts for p95 and p99 latencies. Srikanth provided guidance on using the query builder and clarification on service entry spans. Arnab will check and get back after verifying OpenTelemetry metrics.

1

Issues with Signoz 10.2: Grouping, Missing Traces, and Capturing Requests

Anil reported issues with Signoz 10.2, including grouping, missing traces, and capturing requests. Srikanth suggested upgrading to a newer version. The issue was resolved after upgrading the node version to 18 and additional clarifications were provided.

Customizing Signoz Error Alert Descriptions

Sumanth needed help with making Signoz error alert descriptions more descriptive. Amol suggested using {{.Labels.serviceName}} and discussed limitations of adding labels in queries.

Extracting Custom Fields as Attributes from Log in SigNoz

Harald wants to have custom fields from their body parsed as attributes in the SigNoz logger. They have tried implementing suggestions from nitya-signoz and Prashant, but the issue remains unsolved due to a potential race condition encountered while executing the code. They have provided a full example using Kind for further assistance.

4

Incorrect Alert Firing Issue in Threshold Condition

Harshith.R.S is experiencing an issue with alerts firing when the value is above the threshold. The issue remains unresolved after trying different query modifications suggested by Amol.

1

Powered By