New Machine requirement: Replacement for Equinix x64 servers #3597
Comments
|
The Joyent/equinix_mnx machines are in a separate account (Nodecore) -- I think MNX are paying for those so would hopefully be unaffected (cc @bahamat). |
|
FWIW the Jenkins workspace machines are c3.small.x86 which are: I think we're only using one of the two disks. By contrast the third non-Equinix hosted jenkins-workspace machine hosted on IBM Cloud is: So I think the takeaway here is disk space. Also |
|
@richardlau the Nodecore systems referenced are also on an account that's currently 100% subsidized, and that subsidy is ending. I'm currently investigating what I can do about pricing discounts, but I know that "free" is not continuing for these. |
|
We'd like to offer hosting those instances on mnx.io. This would be like when they were hosted at Joyent. We'd set up a dedicated Triton account with individual instances (rather than two dedicated physical servers). The account billing will be covered by us (MNX). I will assist in getting everything set up and provide credentials to anyone that needs it. We're also adding another datacenter which will be publicly available in the coming months for the offsite backup instance. |
|
Thank you @bahamat - that's great to hear! Let us know when that's in place. |
|
@bahamat That's sounds great. For clarity, would that include the two machines in the Node.js account or just the ones in the Nodecore one? |
|
@richardlau The NodeCore account is the only one I have access to, so that’s the one I meant. For the others, I’d need to know what the requirements are, then I need to check if we have available capacity for it. If you have VMs there, I need the cpu/ram/storage for them, then I can see how much more we can provide. |
|
@bahamat Details are in #3597 (comment). There are two machines with that configuration (c3.small.x86 in Equinix). I think we might not need as much CPU/RAM, but they are consuming disk space, i.e. jenkins-workspace-7: root@test-equinix-ubuntu2204-x64-1:/home/iojs# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 439G 354G 63G 86% /
root@test-equinix-ubuntu2204-x64-1:/home/iojs# du -hs /home/iojs/build/workspace/
240G /home/iojs/build/workspace/
root@test-equinix-ubuntu2204-x64-1:/home/iojs# du -hs /home/iojs/build/binary_tmp.git
56G /home/iojs/build/binary_tmp.git
root@test-equinix-ubuntu2204-x64-1:/home/iojs#jenkins-workspace-8: root@test-equinix-ubuntu2204-x64-2:/home/iojs# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 438G 107G 309G 26% /
root@test-equinix-ubuntu2204-x64-2:/home/iojs# du -hs /home/iojs/build/workspace/
101G /home/iojs/build/workspace/
root@test-equinix-ubuntu2204-x64-2:/home/iojs# |
|
@bahamat Have you been able to review #3597 (comment) and #3597 (comment) as to whether the two Jenkins workspace machines could be included in the mnx offer, or if we'll need to source replacements elsewhere? |
|
What is the timeline for this issue? Is there a deadline from Equinix? |
I was checking the notes and seems like start of April is the current deadline. Do we feel confortable with the given deadline? |
|
Reading through the meeting minutes now, thanks. @UlisesGascon . |
|
Regarding deadlines, I want to give the message of a sense that we want the project to continue to succeed and we don't want to disrupt operations, and also it's important now to have a plan in place for transition on a timeline. We'll support you through that timeline. (and if you need more time, let me know, but don't delay unnecessarily). |
|
This is a list of the affected machines: In the Nodecore organization (these VMs are spread over two instances in Equinix Metal):
In the Node.js organization (these are separate instances in Equinix Metal, current specs #3597 (comment)):
|
|
Oh, I've just noticed we already listed these in the issue description up top, except the release machines are missing from that list -- I'll update the list in the description as well🙂. |
|
As mentioned in the call today, I'd like to add the [email protected] account to the equinix accounts in question so we can get a handle on all the details. |
|
@nodejs/build can you chime in with your ok, concern/objections with adding the linuxIT operations account to the equinix accounts? +1 from me. |
|
On Fri, 8 Mar 2024 at 17:30, Michael Dawson ***@***.***> wrote:
@nodejs/build <https://github.com/orgs/nodejs/teams/build> can you chime
in with your ok, concern/objections with adding the linuxIT operations
account to the equinix accounts?
+1
regards
--
Richard Lau
Software Engineer, Runtimes
Red Hat <https://www.redhat.com/>
***@***.***
<https://www.redhat.com/>
|
|
@richardlau @mhdawson I've been able to confirm that MNX is happy to host the additional machines from the Node.js org from Equinix Metal, as well as the NodeCore instances. |
|
Thank you @bahamat glad to see this. |
|
@ryanaslett we talked about this one in the build WG meeting today. Is it possible to get an update on the planned migration, migration times. |
|
The mnx.io account is established, and instances can be provisioned there. I still need access to the equinix accounts, so I havent been able to use information from that to help inform a plan. Not sure what the next steps are after the +1’s above. I also do not have admin on either the test or release jenkins, so once machines are provisioned Im not able to add the machines as workers or set their jenkins secret, and will need to coordinate with somebody to either get jenkins admin access or help add these to jenkins. Jenkins Workspace Machines:
Need to confirm the actual necessary size for these machines, as well as work with @bahamat to define a package(cpu/ram/disk) that properly mirrors the actual workload of these machines. Once that is set up I can provision these so they can be added to the list of potential jenkins-worker machines. Once we determine that they are successfully handling jobs, we would be clear to turn off the other machines. Infra Machines:I don't have access to the infra secrets, so I'm unable to thoroughly investigate anything regarding the 3 infra machines.
This appears to be down. Unsure of its status. I was hoping to peruse some stats about the worker machines to get proper sizing using the grafana data.
No access to this, but my assumption is that there is major amounts of data that could take a long time to transfer.
Not sure what this machine is used for, or its status. SmartOS test/release machines:These machines are running very old versions of smartOS.
SmartOS 18 and 20 are both no longer supported. Im not sure what node’s testing policy is, but it seems as though it may have been lowest supported/highest supported, which seems like we should target: SmartOS 21.4.1 (Lowest) And continue the pattern of 1 release machine/2 testing machines for each version. Miscellaneous machines
Two containers running on this machine:
Without access to the release jenkins, I'm not really sure whether or not these could exist somewhere else or need to be on their own singular host. It seems as though we should be able to find a docker host somewhere else to run these.
Appears to be a secondary testing node for ubuntu 18. This can be replicated on mnx fairly easily, if its still deemed necessary. I'm not sure what the policy is for EOL distro testing (and for the release container above). As far as a timeline goes, some of these are low hanging fruit, while others may be major undertakings. I was originally under the impression that the two jenkins worker nodes were the only two instances that were of concern, but I now understand that everything in the nodecore account needs to find a new home. Ideally, If I can get access to
That would expedite getting things migrated. |
|
|
@ryanaslett I've invited you to the "Node.js" Equinix Metal organization as an owner (this is the only one of the accounts owned by the Build WG) which contains the two Jenkins workspace machines. The other machines to be migrated are in the "Nodecore" Equinix Metal organization, which I think @bahamat would need to invite you to. We should open a separate issue solely around additional access (i.e. admin on both Jenkins). I can't ssh into grafana. backup, as you surmise, does contain a lot of data. unencrypted is a mirror of www (hosted on Digital Ocean) and is configured to be the failover server in Cloudflare should www be unavailable. For smartos versions, talk to @bahamat and the folks at mnx -- they're likely to be in a much better position to advise on which smartos versions we should be testing on. |
@richardlau I looked at the grafana instance today. Looks like it crashed with a full disk and consequently didn't boot properly. I cleared the boot prompt so that it would come up and ssh is available now, but I didn't do anything to address the disk issue so I don't think Grafana is healthy yet. I figured addressing the disk space was better left to your team. I discussed this with @ryanaslett earlier today, so this may be old news to some folks already. |
Equinix have been sponsoring our infrastructure by providing a generous amount of capacity for the Node.js infrastructure. This is now coming to and end and we need to make a plan for migrating our systems away from Equinix (Note: This does not affect the aarch64 Altras which are supplied as part of the Works On Arm project, but are hosted by Equinix)
debian10-x64-1: {ip: 147.28.162.110, alias: grafana}
smartos15-x64-1: {ip: 147.28.183.83, alias: backup}
ubuntu1604-x64-1: {ip: 147.28.162.105, alias: unencrypted}
smartos18-x64-2: {ip: 147.28.162.101}
smartos20-x64-2: {ip: 147.28.162.108}
ubuntu1804_docker-x64-1: {ip: 147.28.162.104, user: ubuntu}
ubuntu2204-x64-1: {ip: 147.75.72.255, alias: jenkins-workspace-7}
ubuntu2204-x64-2: {ip: 145.40.96.123, alias: jenkins-workspace-8}
smartos18-x64-3: {ip: 147.28.162.102}
smartos18-x64-4: {ip: 147.28.162.103}
smartos20-x64-3: {ip: 147.28.162.107}
smartos20-x64-4: {ip: 147.28.162.109}
ubuntu1804-x64-1: {ip: 147.28.162.99, user: ubuntu}
The joyent and equinix_mnx ones are in the
nodecoreproject in the portal, the two test ones are inNode.jsThe text was updated successfully, but these errors were encountered: