#support

Customizing Clickhouse Tables to Save Application Logs

TLDR Shivam sought advice on creating custom Clickhouse tables for application logs. vishal-signoz suggested moving data to S3 to save costs, and also confirmed the required changes in query layers. nitya-signoz noted that this would entail an entire codebase rewrite.

Powered by Struct AI
Aug 21, 2023 (3 months ago)
Shivam
Photo of md5-dfe2ccb640343537dbb50f03db6268f1
Shivam
02:09 PM
can we create custom clickhouse tables to store application logs?
Aug 22, 2023 (3 months ago)
vishal-signoz
Photo of md5-f936d3e5743d23344d6c60813189716f
vishal-signoz
01:57 AM
Sorry I didn’t understand why you want to create custom Clickhouse tables for logs? What’s your usecase?
cc: nitya-signoz
Shivam
Photo of md5-dfe2ccb640343537dbb50f03db6268f1
Shivam
07:07 AM
Because we are using our own format for logs and don’t need extra things that signoz save
07:07
Shivam
07:07 AM
I think we can save space if we use our own format for logs instead of generic one provided by signoz
vishal-signoz
Photo of md5-f936d3e5743d23344d6c60813189716f
vishal-signoz
07:09 AM
You will need current schema to support queries.
You go with space optimized schema then the performance will not be as good.
07:10
vishal-signoz
07:10 AM
You can move data to S3 to save costs.
Shivam
Photo of md5-dfe2ccb640343537dbb50f03db6268f1
Shivam
07:11 AM
If ,I create custom schema to store logs .Then ,We need to do required changes in query layers also??
vishal-signoz
Photo of md5-f936d3e5743d23344d6c60813189716f
vishal-signoz
07:11 AM
Yes and at otel collector too
nitya-signoz
Photo of md5-a52b9d6c34f193d9a1ff940024f36f77
nitya-signoz
07:11 AM
Few questions:-
• What is the different in space occupied that you are seeing between your and signoz schema
• How do you plan to ingested data in custom schema.
• Our query layer and frontend is written for custom schema, do you not intend to use signoz ui in that case ?
Shivam
Photo of md5-dfe2ccb640343537dbb50f03db6268f1
Shivam
07:14 AM
{
            timestamp : '',
            requestId : '',
            functionName : "",
            thisnameSpace : '',
            message : '',
} this are the only fields we want to store in clickhouse customLogs table

07:15
Shivam
07:15 AM
We are using pino logger in our application and transport to signoz otel collector .
07:16
Shivam
07:16 AM
And need to transport these logs to clickhouse customLogs table
07:17
Shivam
07:17 AM
and then we are able to filter our logs on requestId,functionname, and partinal string serach on message
07:18
Shivam
07:18 AM
Just want to know how much efforts it will take to setup this configguration through signoz ??
nitya-signoz
Photo of md5-a52b9d6c34f193d9a1ff940024f36f77
nitya-signoz
07:19 AM
I don’t think space wise there will be much difference.
Shivam
Photo of md5-dfe2ccb640343537dbb50f03db6268f1
Shivam
07:19 AM
And changes that we need to do on signoz code base??
nitya-signoz
Photo of md5-a52b9d6c34f193d9a1ff940024f36f77
nitya-signoz
07:19 AM
it will be mostly entire codebase rewrite.
Shivam
Photo of md5-dfe2ccb640343537dbb50f03db6268f1
Shivam
07:21 AM
okay

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

59
8mo

Configuring Logs in Signoz and Clickhouse

Varaprasad queries on pushing logs to ClickHouse and analyzing from SigNoz. Srikanth advises using `signoz-otel-collector` and shares a guide. Varaprasad explains their plan of action. Ankit confirms vector parsing and sharding setup in ClickHouse.

7
1mo

Issues with Installing Signoz on Standalone VM with Two Additional Disks

vvpreo is having difficulty installing Signoz on a standalone VM due to an issue with OTEL collector not starting because of migration problems. Ankit and vishal-signoz proposed various solutions, but the issue remained unresolved. They planned a screen-sharing session to resolve it.

1

30
2mo

Querying ClickHouse Database and Exporting Data for Analysis

Ashish asked about querying certain data from ClickHouse database. nitya-signoz clarified the purpose of usage table and the storage of metrics. nitya-signoz offered guides and pointers on where to export data for external analysis.

9
1w

Services UI Goes Blank Due to Retention Issue with S3 Bucket

oluchi reports issue with services UI going blank after a while. Conversation explores possible reasons, such as S3 connection problems and disk space, but no resolution is reached.

4

46
7mo