Bulk importing of uptime monitoring servers to the system

[gorkem-bwl]

Here are some details and use-cases:

To make migrations easier, we can add an automatic server addition feature to Checkmate. This will allow admins to upload a CSV file with server details (URL, display name, and monitor frequency) directly from the UI, saving time and effort.

How it works:

• Admins click the "Add Servers (Uptime)" button on the dashboard.
• They upload a CSV file with server info.
• Checkmate processes the file and adds the servers to the database quickly.

Benefits are faster migration for users switching from other tools and improved user interface.

PS: We already have a function in Settings that adds up to 300 servers at once using a static file. This can be adapted for CSV uploads, making implementation faster and easier. It will also give you an understanding of how adding servers work in general.

(For the grant, this is a "medium project", but with some hurdles to tackle)

[fluellenarman]

Commenting here so I can be assigned to this issue.

[gorkem-bwl]

Assigned. You can reach out to me on the Discord channel (link is in readme.md) to discuss any questions you may have.

[ajhollid]

@fluellenarman the bare minimum of data need to create a Monitor is as follows:

{
   name: "Your Monitor Name"
   url: "www.your-url.com"
}

Everything else is optional, but can be included.

Important to note that the correct teamId must be provided or users will not be able to view their monitors.

Monitor schema for reference:

import mongoose from "mongoose";

const MonitorSchema = mongoose.Schema(
	{
		userId: {
			type: mongoose.Schema.Types.ObjectId,
			ref: "User",
			immutable: true,
			required: true,
		},
		teamId: {
			type: mongoose.Schema.Types.ObjectId,
			ref: "Team",
			immutable: true,
			required: true,
		},
		name: {
			type: String,
			required: true,
		},
		description: {
			type: String,
		},
		status: {
			type: Boolean,
			default: undefined,
		},
		type: {
			type: String,
			required: true,
			enum: [
				"http",
				"ping",
				"pagespeed",
				"hardware",
				"docker",
				"port",
				"distributed_http",
			],
		},
		url: {
			type: String,
			required: true,
		},
		port: {
			type: Number,
		},
		isActive: {
			type: Boolean,
			default: true,
		},
		interval: {
			// in milliseconds
			type: Number,
			default: 60000,
		},
		uptimePercentage: {
			type: Number,
			default: undefined,
		},
		thresholds: {
			type: {
				usage_cpu: { type: Number },
				usage_memory: { type: Number },
				usage_disk: { type: Number },
			},
			_id: false,
		},
		notifications: [
			{
				type: mongoose.Schema.Types.ObjectId,
				ref: "Notification",
			},
		],
		secret: {
			type: String,
		},
	},
	{
		timestamps: true,
	}
);

export default mongoose.model("Monitor", MonitorSchema);

[fluellenarman]

Thank you for the info. This was very useful to me in implementing this feature.

---

Update:

user can now use a .csv file to add servers to checkmate.
The .csv file is in this form.
url, name, interval

• Tested it with this .csv file, no issues and worked as expected.
  google.com,GOOGLE,1
  youtube.com,YOUTUBE,2
  en.wikipedia.org,WIKI,3
  github.com,GITHUB,4
  wayback.archive.org,ARCHIVE,5

• Tested it with same .csv file, but with 50 entries (same 5 entries repeated 10 times).
  Some requests were dropped. Repeated the test a few times and dropped requests varied from 1 to 11.
  Server had the below errors.

error: monitorController connect ECONNREFUSED 207.241.237.3:80
or this error
error: StatusService No matching document found for id "6798738aa861cb9fe4187085" version 0 modifiedPaths "uptimePercentage" (details: Error inserting check for monitor: 6798738aa861cb9fe4187085)

In addition, importing servers through the .csv is not as fast as adding demo monitors in the settings

---

method

• Created a handleBulkImport function modeled after handleCreateMonitor in CreateUptime/index.jsx.
  Parses the .csv in a for loop, formats and sends the data in the same way as handleCreateMonitor.

---

Todo

• Investigate why requests are dropped when the .csv grows larger.
• Investigate why handleBulkImport is not as fast as add demo monitors and make fixes/changes as needed.
• Add error handling when the .csv file contains an invalid entry.

Please let me know if there's anything that I missed or anything else I should be working on in relation to this issue.

Sorry for the late update.

[gorkem-bwl]

I think that should be it. If you can figure out the items under todo, we can have a look.