Troubleshooting and Adding Log Files to SigNoz POC
TLDR Noor has requested help with incorporating log files into their SigNoz POC. In collaboration with vishal-signoz and nitya-signoz, they managed to successfully setup and resolve their issues.
vishal-signoz
Thu, 17 Aug 2023 07:02:19 UTCnitya-signoz Can help
nitya-signoz
Thu, 17 Aug 2023 07:04:13 UTCUse this
Noor
Thu, 17 Aug 2023 11:49:12 UTCHello Nitya Thanks will this work ```receivers: filelog:``` include: [ /Untiled/Users/nooramin-ali/Downloads/error_log ```] operators: - type: regex_parser regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$' timestamp: parse_from: attributes.time layout: '%Y-%m-%d %H:%M:%S' severity: parse_from: attributes.sev``` I can run script in Signoz diretory on the same mac
Noor
Thu, 17 Aug 2023 13:45:52 UTCDo I need to run script in docker desktop please forward me to location to run this script . Thanks
Noor
Thu, 17 Aug 2023 16:00:23 UTCAlso please check if my path is correct too
Noor
Thu, 17 Aug 2023 16:14:42 UTCI think I figured it out I will try it now
Noor
Thu, 17 Aug 2023 17:41:39 UTCI have saved the logs here
Noor
Thu, 17 Aug 2023 17:41:49 UTCPlease let me know if it will work
Noor
Thu, 17 Aug 2023 17:46:09 UTCAny help will be help full to make this POC successful
Noor
Thu, 17 Aug 2023 20:40:32 UTCI made changes to the log files as seen
Noor
Thu, 17 Aug 2023 20:41:05 UTCwill the show up on Signox using your script . Thanks
Noor
Thu, 17 Aug 2023 21:49:20 UTCI have restarted the signoz how long till I see these logs in the front end .
Noor
Thu, 17 Aug 2023 21:54:49 UTCHere is restart for signoz
Noor
Thu, 17 Aug 2023 21:56:11 UTCPlease let me know if I have done it correctly
Noor
Fri, 18 Aug 2023 02:59:59 UTCI did not get reply back from your team member can you reply to my slack by Friday please. Thanks
Noor
Fri, 18 Aug 2023 03:40:33 UTCDo you why I am un able to login into Signoz I get the following error something_went_wrong
nitya-signoz
Fri, 18 Aug 2023 04:19:49 UTCHow are you running your otel collector ? If you are running it in docker, then you will have to mount these files to otel-collector.
Noor
Fri, 18 Aug 2023 12:22:33 UTCI had to uninstall and reinstall everything Nitya
Noor
Fri, 18 Aug 2023 12:26:06 UTCI am using local host to log in it will not let me log in any longer
Noor
Fri, 18 Aug 2023 12:30:22 UTCNitya where do I mount files in docker
nitya-signoz
Fri, 18 Aug 2023 12:54:18 UTCHere
Noor
Fri, 18 Aug 2023 13:03:11 UTCwhere do I save this ```receivers: filelog:``` include: [ /Untiled/Users/nooramin-ali/Documents/Log file/access.log ```] operators: - type: regex_parser regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$' timestamp: parse_from: attributes.time layout: '%Y-%m-%d %H:%M:%S' severity: parse_from: attributes.sev``` in order for the logs to show up on Signoz we are doing POC without the Agent install
Noor
Fri, 18 Aug 2023 13:07:48 UTCI am also unable to open signoz running on my mac with IP address I get following error
nitya-signoz
Fri, 18 Aug 2023 13:09:41 UTCyou will have to mount your file to otel-collector.
then you have to add the path to the mounted file in `include:` and save the receiver here
nitya-signoz
Fri, 18 Aug 2023 13:09:53 UTCI am not sure about the ip, isn’t localhost working ?
Noor
Fri, 18 Aug 2023 13:14:26 UTCI will add the path file first . I will try trouble shoot the local host Ip address issue I had reboot my mac yesterday evening after that local host via ip address stop working I am not able log in to sig noz front end at all. Thanks Nitya
Noor
Fri, 18 Aug 2023 13:37:56 UTCHello Nitya please let me know if I did this correctly .
Noor
Fri, 18 Aug 2023 16:34:38 UTCAlso can you please send best way to restart collector I restarted my Mac also to fix local host issue where i cannot log into Signoz front end any longer to view the logs any reason why
Noor
Fri, 18 Aug 2023 18:14:31 UTCNow I am able see the local host and front end us fine all is good I can see the alerts, logs, and dashboard when you say restart your otel collector.What is command for it or how I do it please . Thanks for your help I am so close to my POC is now . Thanks
Noor
Fri, 18 Aug 2023 19:44:53 UTCis this command to restart systemctl restart otelcol-contrib
Noor
Fri, 18 Aug 2023 19:53:21 UTCor this the right command sudo service otelcol restrate. where do I run this command please
Noor
Sat, 19 Aug 2023 12:32:13 UTCI hope some one in your team can really make my POC success full . Every time I make change in signoz on my mac my front host crashes now I got this . It is an old version can some one please delete my login Id from your end Noor-Ul-Amin.Ali I want recreate my log ID again and see if it works. If the POC does not work they might not consider getting the product . Thanks
nitya-signoz
Sat, 19 Aug 2023 12:39:08 UTCPlease select a slot here , we can get on a call
Noor
Mon, 21 Aug 2023 11:29:16 UTCLet me know if this time works for you or not for today 7 am this morning. Thanks
nitya-signoz
Mon, 21 Aug 2023 11:31:10 UTCLet connect on the agreed time.
Noor
Mon, 21 Aug 2023 11:40:48 UTCok I did not see phone # in meeting invite
Noor
Mon, 21 Aug 2023 11:41:20 UTCI just saw it thanks
Noor
Mon, 21 Aug 2023 12:19:26 UTCreceivers:
include: [ /Untiled/Users/nooramin-ali/Documents/Log file/access.log
tcplog/docker:
listen_address: "0.0.0.0:2255"
operators:
- type: regex_parser
regex: '^<([0-9]+)>[0-9]+ (?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]+)?([zZ]|([\+-])([01]\d|2[0-3]):?([0-5]\d)?)?) (?P<container_id>\S+) (?P<container_name>\S+) [0-9]+ - -( (?P<body>.*))?'
timestamp:
parse_from: attributes.timestamp
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
- type: move
from: attributes["body"]
to: body
- type: remove
field: attributes.timestamp
# please remove names from below if you want to collect logs from them
- type: filter
id: signoz_logs_filter
expr: 'attributes.container_name matches "^signoz-(logspout|frontend|alertmanager|query-service|otel-collector|otel-collector-metrics|clickhouse|zookeeper)"'
opencensus:
endpoint: 0.0.0.0:55678
otlp/spanmetrics:
protocols:
grpc:
endpoint: localhost:12345
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_http:
endpoint: 0.0.0.0:14268
# thrift_compact:
# endpoint: 0.0.0.0:6831
# thrift_binary:
# endpoint: 0.0.0.0:6832
hostmetrics:
collection_interval: 30s
scrapers:
cpu: {}
load: {}
memory: {}
disk: {}
filesystem: {}
network: {}
prometheus:
config:
global:
scrape_interval: 60s
scrape_configs:
# otel-collector internal metrics
- job_name: otel-collector
static_configs:
- targets:
- localhost:8888
labels:
job_name: otel-collector
processors:
logstransform/internal:
operators:
- type: trace_parser
if: '"trace_id" in attributes or "span_id" in attributes'
trace_id:
parse_from: attributes.trace_id
span_id:
parse_from: attributes.span_id
output: remove_trace_id
- type: trace_parser
if: '"traceId" in attributes or "spanId" in attributes'
trace_id:
parse_from: attributes.traceId
span_id:
parse_from: attributes.spanId
output: remove_traceId
- id: remove_traceId
type: remove
if: '"traceId" in attributes'
field: attributes.traceId
output: remove_spanId
- id: remove_spanId
type: remove
if: '"spanId" in attributes'
field: attributes.spanId
- id: remove_trace_id
type: remove
if: '"trace_id" in attributes'
field: attributes.trace_id
output: remove_span_id
- id: remove_span_id
type: remove
if: '"span_id" in attributes'
field: attributes.span_id
batch:
send_batch_size: 10000
send_batch_max_size: 11000
timeout: 10s
signozspanmetrics/prometheus:
metrics_exporter: prometheus
latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
dimensions_cache_size: 100000
dimensions:
- name: service.namespace
default: default
- name: deployment.environment
default: default
# This is added to ensure the uniqueness of the timeseries
# Otherwise, identical timeseries produced by multiple replicas of
# collectors result in incorrect APM metrics
- name: 'signoz.collector.id'
# memory_limiter:
# # 80% of maximum memory up to 2G
# limit_mib: 1500
# # 25% of limit up to 2G
# spike_limit_mib: 512
# check_interval: 5s
#
# # 50% of the maximum memory
# limit_percentage: 50
# # 20% of max memory usage spike expected
# spike_limit_percentage: 20
# queued_retry:
# num_workers: 4
# queue_size: 100
# retry_on_failure: true
resourcedetection:
# Using OTEL_RESOURCE_ATTRIBUTES envvar, env detector adds custom labels.
detectors: [env, system] # include ec2 for AWS, gcp for GCP and azure for Azure.
timeout: 2s
extensions:
health_check:
endpoint: 0.0.0.0:13133
zpages:
endpoint: 0.0.0.0:55679
pprof:
endpoint: 0.0.0.0:1777
exporters:
clickhousetraces:
datasource:
nitya-signoz
Mon, 21 Aug 2023 12:22:35 UTC```receivers:
filelog:
include: [ /tmp/access.log, /tmp/error.log ]
start_at: beginning
tcplog/docker:
listen_address: "0.0.0.0:2255"
operators:
- type: regex_parser
regex: '^<([0-9]+)>[0-9]+ (?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]+)?([zZ]|([\+-])([01]\d|2[0-3]):?([0-5]\d)?)?) (?P<container_id>\S+) (?P<container_name>\S+) [0-9]+ - -( (?P<body>.*))?'
timestamp:
parse_from: attributes.timestamp
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
- type: move
from: attributes["body"]
to: body
- type: remove
field: attributes.timestamp
# please remove names from below if you want to collect logs from them
- type: filter
id: signoz_logs_filter
expr: 'attributes.container_name matches "^signoz-(logspout|frontend|alertmanager|query-service|otel-collector|otel-collector-metrics|clickhouse|zookeeper)"'
opencensus:
endpoint: 0.0.0.0:55678
otlp/spanmetrics:
protocols:
grpc:
endpoint: localhost:12345
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_http:
endpoint: 0.0.0.0:14268
# thrift_compact:
# endpoint: 0.0.0.0:6831
# thrift_binary:
# endpoint: 0.0.0.0:6832
hostmetrics:
collection_interval: 30s
scrapers:
cpu: {}
load: {}
memory: {}
disk: {}
filesystem: {}
network: {}
prometheus:
config:
global:
scrape_interval: 60s
scrape_configs:
# otel-collector internal metrics
- job_name: otel-collector
static_configs:
- targets:
- localhost:8888
labels:
job_name: otel-collector
processors:
logstransform/internal:
operators:
- type: trace_parser
if: '"trace_id" in attributes or "span_id" in attributes'
trace_id:
parse_from: attributes.trace_id
span_id:
parse_from: attributes.span_id
output: remove_trace_id
- type: trace_parser
if: '"traceId" in attributes or "spanId" in attributes'
trace_id:
parse_from: attributes.traceId
span_id:
parse_from: attributes.spanId
output: remove_traceId
- id: remove_traceId
type: remove
if: '"traceId" in attributes'
field: attributes.traceId
output: remove_spanId
- id: remove_spanId
type: remove
if: '"spanId" in attributes'
field: attributes.spanId
- id: remove_trace_id
type: remove
if: '"trace_id" in attributes'
field: attributes.trace_id
output: remove_span_id
- id: remove_span_id
type: remove
if: '"span_id" in attributes'
field: attributes.span_id
batch:
send_batch_size: 10000
send_batch_max_size: 11000
timeout: 10s
signozspanmetrics/prometheus:
metrics_exporter: prometheus
latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
dimensions_cache_size: 100000
dimensions:
- name: service.namespace
default: default
- name: deployment.environment
default: default
# This is added to ensure the uniqueness of the timeseries
# Otherwise, identical timeseries produced by multiple replicas of
# collectors result in incorrect APM metrics
- name: 'signoz.collector.id'
# memory_limiter:
# # 80% of maximum memory up to 2G
# limit_mib: 1500
# # 25% of limit up to 2G
# spike_limit_mib: 512
# check_interval: 5s
#
# # 50% of the maximum memory
# limit_percentage: 50
# # 20% of max memory usage spike expected
# spike_limit_percentage: 20
# queued_retry:
# num_workers: 4
# queue_size: 100
# retry_on_failure: true
resourcedetection:
# Using OTEL_RESOURCE_ATTRIBUTES envvar, env detector adds custom labels.
detectors: [env, system] # include ec2 for AWS, gcp for GCP and azure for Azure.
timeout: 2s
extensions:
health_check:
endpoint: 0.0.0.0:13133
zpages:
endpoint: 0.0.0.0:55679
pprof:
endpoint: 0.0.0.0:1777
exporters:
clickhousetraces:
datasource:
nitya-signoz
Mon, 21 Aug 2023 12:29:47 UTC```/Untiled/Users/nooramin-ali/Documents/Log file/access.log:/tmp/access.log``` mount the file
Noor
Mon, 21 Aug 2023 12:46:02 UTCI am getting ready the mount the files now. Please give me the line # I will add it
Noor
Mon, 21 Aug 2023 13:30:53 UTCIs this correct
Noor
Mon, 21 Aug 2023 15:04:12 UTCI see this now how long it will take to for logs to show up in Signoz
Noor
Mon, 21 Aug 2023 15:25:36 UTCLet me know if I need to move this in different location
Noor
Mon, 21 Aug 2023 16:25:04 UTCLet me know what else I need to do now
Noor
Mon, 21 Aug 2023 16:25:27 UTCI have not see any alerts show up yet
Noor
Mon, 21 Aug 2023 16:59:29 UTCSo now I need to this udo systemctl restart otelcol-contrib.service what is best command to restart otecol and where should it be run from . Thanks
Noor
Mon, 21 Aug 2023 18:37:09 UTCHave see this error : * error decoding 'volumes[1]': invalid spec: /var/lib/docker/containers:/var/lib/docker/containers:ro /Untiled/Users/nooramin-ali/Documents/Log file/access.log:/tmp/access.log: too many colons
Noor
Mon, 21 Aug 2023 18:49:43 UTCI need to tweak this a little if you guide why I am getting this error now .It will help full I am very close now to get this POC moving. Thanks
Noor
Mon, 21 Aug 2023 18:50:16 UTCI am getting this error when I try to run docker-compose restart
Noor
Mon, 21 Aug 2023 19:21:24 UTCI need to please request one more meeting to make this work right for my POC I am getting close I am new to Signoz platform I don't have that much time left I want get this POC done by Thursday. Thanks
Noor
Mon, 21 Aug 2023 20:41:09 UTCWell I had to redo it again to get logs showing up . Now I want get some input why it keeps doing.
Noor
Mon, 21 Aug 2023 20:43:51 UTCI still don't now why it is not working correctly for me I need to make sure these location edit is correct please
Noor
Mon, 21 Aug 2023 20:44:27 UTCThis I really really want it to pull logs from and show it to me . Thanks
Noor
Mon, 21 Aug 2023 21:22:20 UTCWell I had to deleted and reinstalled it looks good now
Noor
Mon, 21 Aug 2023 21:23:01 UTCSo fare so good I don't why it has given me so many issue in last few days on my mac
Noor
Mon, 21 Aug 2023 21:39:52 UTCPlease connect with me so I can get this going all I want to make sure after adding this part ``` include: [ /tmp/access.log, /tmp/error.log ]``` start_at: beginning and this now /Untiled/Users/nooramin-ali/Documents/Log file/access.log:/tmp/access.log. What else I need to do in order to log show up on front end please it is very important for this POC to become successful . Thanks
nitya-signoz
Tue, 22 Aug 2023 04:16:31 UTCThe mounting path doesn’t seem to be correct. Please add a `-` at the beginning as seen in other mounts and also reconfirm the path of the file once again. There might be some changes as thare are spaces in between your folder names
Noor
Tue, 22 Aug 2023 11:24:19 UTCok will do
Noor
Tue, 22 Aug 2023 11:28:51 UTC-/Untiled/Users/nooramin-ali/Documents/Logfile/access.log:/tmp/access.log this look ok now
Noor
Tue, 22 Aug 2023 11:37:25 UTCI still don't see log showing up on front end side
Noor
Tue, 22 Aug 2023 12:44:21 UTCLet me know what else I need to do to make sure these logs show up
nitya-signoz
Tue, 22 Aug 2023 12:45:02 UTCIt’s looks okay, did you restart the signoz-otel-collector container ?
Noor
Tue, 22 Aug 2023 12:45:56 UTCthis is what it looks like now
Noor
Tue, 22 Aug 2023 12:46:12 UTCI did run docker ps twice
Noor
Tue, 22 Aug 2023 12:46:23 UTCwhat other commands can I run
nitya-signoz
Tue, 22 Aug 2023 12:46:45 UTCplease format it properly, there should be a space after -
nitya-signoz
Tue, 22 Aug 2023 12:47:43 UTC`docker container restart signoz-otel-collector`
Noor
Tue, 22 Aug 2023 12:48:35 UTChere you go
nitya-signoz
Tue, 22 Aug 2023 12:50:36 UTCHere you mentioned that you have renamed the name of the folder and removed space, the above screenshot has a space
Noor
Tue, 22 Aug 2023 12:54:09 UTCyou space between Log and file
Noor
Tue, 22 Aug 2023 12:55:16 UTCI have removed the space now
Noor
Tue, 22 Aug 2023 12:55:33 UTCrun the command also now
Noor
Tue, 22 Aug 2023 12:57:43 UTCwhere do I run this command signoz, deploy or docker docker container restart signoz-otel-collector
nitya-signoz
Tue, 22 Aug 2023 13:02:29 UTCanywhere
Noor
Tue, 22 Aug 2023 13:03:44 UTCI did
Noor
Tue, 22 Aug 2023 13:06:50 UTChow long should it take to pull log file from the location of the path
Noor
Tue, 22 Aug 2023 14:35:39 UTCAlso what is best way find out if logs are showing up after collector gets restarted
nitya-signoz
Tue, 22 Aug 2023 14:40:01 UTCShare the logs of signoz-otel-collector
Noor
Tue, 22 Aug 2023 14:43:28 UTCwhat I mean Nitya is how do I find out here after I have restarted the collector few times.
nitya-signoz
Tue, 22 Aug 2023 14:48:42 UTCNoor please share a private github repository with me, I will send you the setup which you can run directly
nitya-signoz
Tue, 22 Aug 2023 14:49:08 UTCAlso add the log files there
Noor
Tue, 22 Aug 2023 14:49:17 UTCok
Noor
Tue, 22 Aug 2023 15:05:46 UTCI just added you here let me know if have access to it
nitya-signoz
Tue, 22 Aug 2023 15:06:38 UTCGot it, please upload the log file there.
Noor
Tue, 22 Aug 2023 15:06:49 UTCok
Noor
Tue, 22 Aug 2023 15:09:26 UTCI am getting this error Yowza, that’s a big file. Try again with a file smaller than 25MB.
Noor
Tue, 22 Aug 2023 15:10:24 UTCmy log file size is Log File - 741.9 MB
Noor
Tue, 22 Aug 2023 15:11:08 UTCother size is Log File - 217.6 MB
Noor
Tue, 22 Aug 2023 15:12:39 UTCso will it not work is the size of the file is too large
nitya-signoz
Tue, 22 Aug 2023 15:15:19 UTCCan you get a trimmed down version of your log file ?
Noor
Tue, 22 Aug 2023 15:15:48 UTCok I will try
Noor
Tue, 22 Aug 2023 15:22:30 UTCdone for POC demo purpose and reason this will do
Noor
Tue, 22 Aug 2023 15:24:52 UTCThank you very much for your time and effort on it I am learning Signoz now in very short time
Noor
Tue, 22 Aug 2023 15:28:25 UTCPlease let me know if I need to do anything on my end
Noor
Tue, 22 Aug 2023 17:13:54 UTCHello Nitya my POC is on Thursday you think I have it done by that time. Thanks
Noor
Wed, 23 Aug 2023 11:23:26 UTCHello Nitya I hope and pray all is well with you and your team. Please let me what else I can do on my side for this POC . Thanks
nitya-signoz
Wed, 23 Aug 2023 11:38:00 UTCOkay you want the `testaccess.rtf` logs in signoz right ?
Noor
Wed, 23 Aug 2023 11:39:48 UTCYes for this POC I just want to show how Signoz will work for log correlation
Noor
Wed, 23 Aug 2023 11:41:44 UTCI have already created rest of docs for Signoz now all I want to do is live demo as POC with my team and our Sr Architect
nitya-signoz
Wed, 23 Aug 2023 11:43:13 UTCI will send you the setup on github
Noor
Wed, 23 Aug 2023 11:43:22 UTCok
Noor
Wed, 23 Aug 2023 11:46:44 UTCdo you how long will it take someone to learn Signoz what are common commands to work on alerts, dashboard , traces, and logs. Does your company have docs like that or not? Just wondering that is all. Thanks
nitya-signoz
Wed, 23 Aug 2023 11:51:44 UTCCheck your repo, and follow the commands in readme.
nitya-signoz
Wed, 23 Aug 2023 11:52:21 UTC>
Noor
Wed, 23 Aug 2023 11:55:30 UTCdo you mean hear
nitya-signoz
Wed, 23 Aug 2023 11:55:44 UTCYes please refresh
Noor
Wed, 23 Aug 2023 12:23:04 UTCI did refresh my local host this
Noor
Wed, 23 Aug 2023 12:29:19 UTCor do you me this one
Noor
Wed, 23 Aug 2023 12:34:07 UTCjust got this error
Noor
Wed, 23 Aug 2023 12:55:53 UTCDo you know why we the error
nitya-signoz
Wed, 23 Aug 2023 13:01:46 UTCPlease follow the instructions in the error message, you will have to make changes to your docker
Noor
Wed, 23 Aug 2023 13:07:27 UTCok I will do that thanks
Noor
Thu, 24 Aug 2023 12:05:12 UTCHello Nitya is never worked for me I am having a POC\ Demo on Signoz with my logs showing up in front end side . My POC\Demo will be on Friday. Mean while find out an answer why I was not able move logs from my . Right now I am going to touch anything till my Demo and POC is done. I will share you few screen shots with you so you can share with your team members . Thanks
SigNoz Community
Indexed 1060 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Noor
Wed, 16 Aug 2023 21:49:44 UTCWhat is best way to add two of log file from my Macbook pro to signoz POC I down loaded the log file from HTTP server we are not allowed install signoz agent I would like to know if POC can be done like that I already have Signox running on my Macbook pro for POC . Thanks