[NEW] Add some easy-to-use features and optimizations (eliminate elk)

[fushengruomengzhang]

Description

The following optimizations have been made to the logs menu

1. Select index: I hope to provide an * option
2. Select Available Fields Optimization: After clicking the search button, the selected Available Fields will be initialized
3. Display optimization: When displaying content, the original text should be displayed, with new lines or spaces displayed

New features added

Because my log volume is large, averaging 5GB per day. So I want to keep only the logs from the recent period of time, and I occasionally need to view these historical data as well

1. Support importing and exporting historical data
2. Support cleaning historical data

Another optimization

Can obtain logs through Kafka。
filebeat -> kafka -> siglens

Benefits

Easy to use
Replace elk

[nkunal]

@fushengruomengzhang

1. We are adding the * option in the logs search menu, in 0.2.5 release we added a * or wildcard selection option in the dashboard menu
2. "After clicking the search button, the selected Available Fields will be initialized" ==> Can you elaborate a little bit what you mean by this? The "Available Fields" dropdown menu that we have is used to hide the columns that you don't want to see
3. "When displaying content, the original text should be displaye" ==> For this in the logs menu , we already have 3 different views: single-line view, wrap-line view and tabular view. You can use the single-line view which will show the whole log line.
4. "Support importing and exporting historical data" ==> What do you mean by this ? And how would you like this to work ? Any examples of how this works with other products would be helpful
5. "Support cleaning historical data" ==> We already have retention setting, which automatically cleans historical data. Are you asking to have it in a way to delete this manually ?
6. "..Kafka--> siglens" ==> thanks for the suggestion, could you submit a separate enhancement request just for this. This is the kafka connector to ingest logs into siglens. Based on the popularity request of this enhancement, we can increase the priority of this

Thanks for all the fantastic suggestions

[fushengruomengzhang]

@nkunal

version

I deployed siglens 0.2.5 through Docker

use *

As shown in the figure, I would like to add the option of * to represent all indexes at this location

I am using version 0.2.5, there is no * option here

Available Fields

select Fields,only message field

But after clicking the search button

I hope only select message field

When displaying content, the original text should be displaye

My individual content has line breaks and tabs, and I hope to be able to view them like I would in a log file.
When the content is too long, wrap it up.
Similar to the display of ELK.

• The current display is shown in the picture

* I hope it can be presented like this, so that it is clearer

Support importing and exporting historical data AND Support cleaning historical data

I think these two questions should be looked at together.
Let me talk about my daily scenarios as follows:
My project generates a large number of logs every day. If these logs persist on Siglens servers, it will lead to insufficient hard disk resources. Therefore, on the 1st of each month, I will migrate the logs from the previous month to another server to ensure Siglens' hard disk resources. (Export and Cleanup)
But sometimes I still need to check these migrated logs, so I need to import the logs that are brought out again into Siglens. (Import)
In summary:
Assuming today is April 1st, 2024, I need to export the data from April 1st to March 31st, 2024 to a file or compressed file, and clean up the data during this period. I also need to be able to support me in re importing this file or compressed file into Siglens