Nearform can no longer host machines #3615
Comments
|
Hi Michael, We have a Fibre line that needs to be removed that controls the fixed ip addresses that are currently on the servers. We can keep this in place for a period of time, but we would be hoping to shut down in under 2 months. So if we could set the 1st of April as the deadline, would that give enough time to address the above? Thanks, |
|
@nodejs/performance FYI |
|
Hi Michael, Ryan Aslett from LF IT here. I've been doing a bit of background research on this and wanted to make sure I understood what the requirements are. If I understand the situation correctly, these are physical machines that Nearform is hosting for nodejs in their datacenter, which they can no longer continue to support having on their network. My understanding of the 2 Windows on ARM machines, (Surface Pro) The OSX machines: One of the x86 mac mini's VM's were split between release and test for 10.15, and the other has 2 vm's dedicated to test of 10.15-x64 The release machine was retired because 10.15 isnt able to run xcode13 and notarize. It looks like there were some recent experiments to get 10.15 x64 tests to run on orka: https://ci.nodejs.org/computer/test%2Dorka%2Dmacos10.15%2Dx64%2D1/builds . for the 2 ARM based mac minis, it seems like the test one has been unused for the last 11 days: I do not have access to the release jenkins, so Im not sure what the status is of the nearform ARM release machine, other than it seems like we now have two functioning release machines for macos11-x64-1 #3179 (comment). Given that, I wonder if we already have capacity at Macstadium an Orka to handle the roles these Nearform OSX machines are performing ? (Though, perhaps we might need another additional orka testrunner for 10.15/x64) Would the goal in pursuing MacInCloud or another provider (i.e. sponsored https://aws.amazon.com/pm/ec2-mac/ ) be mostly for redundancy and resiliency against provider outages? ** Large Benchmarking Machines ** Would changing the benchmark infra be an option? can those be virtual/cloud based machines, or is bare metal a requirement? In any case I look forward to helping get this figured out. |
Virtual/is really not an option. However any "bare metal" host would do. I personally use a Hetzner machine for similar purposes (it's significantly worse/slower, but we need the consistency of results, not the actual speed). In terms of resources, we could do with similar specs (I don't have those handy), or even something a bit less powerful. My 2 cents is that those machines are likely near end-of-life. |
|
Is there something speaking against using github runners? |
|
Anything running on VMs have too much interference and the standard deviation between runs is too high to measure bytecode level optimizations, e.g. microbenchmarks. |
Github actions supports self hosted runners, even bare metal ones, but converting the Jenkins CI infrastructure to a Github Actions infrastructure is an ambitious undertaking that would be unlikely to succeed in the timeframe of this immediate need. #2247 seems like a good place to continue discussing whether or not that's an eventual or possible outcome. |
2 Windows on ARM machinesI agree it seems we don't need them anymore. The OSX machinesCurrently almost unused because the current version of Node.js doesn't support macOS 10.15. These machines could be updated to macOS 12, 13, or even 14. The Intel benchmarking machinesThe systems we have now are based on dual-CPU Intel Xeon E5-2699 v4. |
Thanks for confirming - that was one of the questions I had when discussing this with some members of build yesterday - I figured that was likely the case. Obviously the implication is that we won't be able to compare "old" runs with "new" runs without re-running them, but that shouldn't be too much of a problem (we can always re-run if required). Does the performance team require two systems or would one be adequate for the capacity needs?
I believe that's the primary driver, yes. AWS should also be a viable option if they were willing to sponsor us. |
|
Regarding OSX testing: Based on everything I've been able to glean from issues and meeting notes, it seems like a good path forward would be to lean into what we're doing with MacStadium for the short term, with an eye on having a secondary provider longer term.
Regarding Benchmark testing:
|
In order to land any performance related PR, we run the benchmarks. Some of those jobs lasts 6-8 hours, and in the most extreme cases days.
The lack of benchmarking machines would slow down progress on most things performance related.
They are not part of our release process.
I's guess a few times per week.
One of the key strategies we employ is to rely on previous runs to compare. I'd not really trust this setup, because the actual machine would change every time. On top, AWS spot instances cost for c5.metal (seems a good choice in terms of resources) is likely 3x (or more) compared to a provider like Hetzner. |
Do we? I thought each Benchmark CI job that is ran runs through the requested benchmark(s) twice -- once with the base branch (i.e. what is being compared to) and once with the PR being tested. |
Our goal across platforms has been to have at least two providers for any platform. So while we might be able to use 1 for a short period of time, the plan should be to find a second provider if at all possible. |
I'd say the relationship is good and we are happy with the machines they have provided. I believe most of the common issues we have relate to OSX itself versus the host. Many thanks to MacStadium for their continued support. |
Yes. We typically run the benchmark across different commits as a PR evolves. I'm not convinced that those result would be comparable across different HW. |
|
Are only benchmarks run on those machines? |
Expand> I'm not sure we have enough capacity to replace them at Macstadium (we already struggle with disk space), but I would be more happy if we find other providers to donate resources (for example, Scaleway have bare metal M1 and M2 Pro mac minis).Pardon me if I'm intruding here, but if there is a need M1 or M2 runners for GitHub Actions, may I suggest giving FlyCI a try? We offer MacOS M1 and M2 runners (ARM64). For public repos, we offer 500 mins/month of free M1 usage (4 vCPUs, 7 GB RAM, 28 GB storage). The setup is super easy:
jobs:
ci:
- runs-on: macos-latest
+ runs-on: flyci-macos-large-latest-m1
steps:
- name: 👀 Checkout repo
uses: actions/checkout@v4Do you think this might be a good option for nodejs / build? Web: flyci.net Update: I apologize, I just realized you guys are using Jenkins, not GitHub Actions. Please ignore my comments above! |
|
Issues with proposal from Linux IT for how to move forward on replacing NearForm OSX machines - #3638 @UlisesGascon FYI |
|
Update: @efrisby we've selected some machines at Hetzner to act as a replacement for the benchmark machines, but are waiting on an internal fiscal process to complete so we can purchase them and get them set up, based on the machine sizing from @mcollina and @mhdawson. I've reached out to @jasnell to see if he still had a contact at ARM so we can ask what to do with their Surface Pro machines. Im still trying to get access to Macstadium to assess what our options with them are. One thing we have not resolved is that once the rack is decommissioned, what should happen to that hardware? |
|
In terms of the hardware I think we should try to see if there are any Node.js collaborators who are interested and could pick up locallly. |
I'm not sure we have many collaborators in Ireland, but definitely a few past collaborators and wide community members: @GlenTiki @No9 wdyt? There might also be the option for the go to a local charity. |
|
If Nearform are decommissioning the hardware and no longer have use of them themselves, they may know of other good local philanthropic uses, so I’d trust @efrisby take the lead there before me. However, my thoughts…
I would like to see the hardware go to someone that needs it - students, etc. 💯 No tech charities immediately come to mind except for coderdojo. I don’t know how active that is here in Waterford (where both NearForm and I are based). I am a member of the organiser team of the local monthly Waterford tech meetup, and I would love to offer something like the Mac minis as a raffle prize - lots of students, researchers, college staff and local devs attend, so I could see them put to good use that way, if no other local collaborators need them or suggest anything. |
|
@GlenTiki agree not many charities take desktop type machines now - mainly laptops as they are easier to manage for everyone in the charities and the end users. |
|
Hi all, If the equipment needs to be packaged up and delivered to a location, that is no problem at all. Nearform work with a charity that unused computers go to. The charity is based in UK / Ireland, who then transport machines to schools in Vietnam. We have sent mac mini's in the past, so I don't think there are any issues providing. Also we have provided to Coderdojo in the past, so no reason not to reach out again. The only ones that we may need to returned are the Intel Xeon servers. If anyone would have any suggestions, or if someone is in the position to host these, we can arrange to ship. Thanks, |
|
I've marked all of the Nearform hosted machines in ci.nodejs.org and ci-release.nodejs.org as offline so no new jobs will be scheduled on to them. @efrisby We're no longer using the Nearform hosted machines in the Node.js CI. Thanks once again to Nearform for hosting these machines for us for all of these years! |
|
@richardlau @mhdawson thanks for the kind words. I will pass that on to the team here. Regarding the machines hosted here, are we now in a situation that we can turn all these off, 2 x Intel Severs If you can confirm, I will power these down tomorrow and disconnect. If anyone has any suggestions what to do with these also, can you please get in contact with me to arrange also. If you wish to donate to charity we can look at those options or if you wish to send them to someone just let me know. We can wipe and run hard drive cleans on them before any donations are made also. The two intel servers however might either need to be sent back or if you wish we have a recycle company as a supplier that can recycle old computer hardware, https://vyta.com/ that we work with to collect and green recycle and reuse of equipment. Wiping is also done to a certified standard. Thanks again to everyone that helped the Nearform team here also with the support of these devices and to the community for the effort put in to move this whole area to a new solution, especially within the time we had. All the best, Eamonn |
|
@efrisby Yes, those listed machines can be disconnected and powered down. |
|
The surface pro machines were on loan from ARM to NearForm. Y'all would need to contact nearform about those, as I have details on what is happening with. |
|
I'd maybe give it until the end of April to wait for a response from ARM, then maybe ping them again. If there's still no answer after that, then I'd suggest shipping the surface pros to either myself or @mcollina for storage (because we were both around when the agreement to lend the machines was made). I'll keep trying with Arm and if they don't respond, I'll donate the devices to a charity. |
|
@jasnell I think it might be better to ship those to me because of import duties. |
|
It's been so long that I can't remember the details and everything about the benchmark machine was in my old NearForm email inbox that I no longer have access to. I know the machine was on loan only but that's all I remember. |
|
@bensternthal, @ryanaslett maybe you can help out here in terms of the Intel machines. There seems to be no retained context in terms of the loan from Intel as the people from Neaform who worked with intel to bring the machines in are no longer at Nearform and there was nobody from the build WG who was involved in setting up the loan. I don't think Intel is a Foundation member anymore so don't know who to reach out to. Could you two handle figuring out what to do with the machines? |
|
I don't think they are on loan, more of a donation. I don't have access to those emails anymore. |
|
@mhdawson based on reading this thread I would say the intel machines can be donated. |
|
@mhdawson @mcollina @jasnell I will take out the intel servers and see what we can do. We have a company that recycle old equipment that I will contact as donating servers like this is harder to find a home for. If you have anyone has any suggestions of anyone local in Ireland let me know as shipping outside of Ireland will be difficult due to the size and weight. Thanks |
|
I'd take server hardware if it's going to just be stripped for parts - could use it in a home lab. |
|
@GlenTiki If you can make it to Tramore tomorrow, I should be onsite, else if you wish we can arrange another day perhaps. I have support person calling to help with taking out some equipment so we can hopefully take these out. It might be late in the day that we will be ready, perhaps around 4.30 however. Thanks. |
|
@efrisby I'm away for a bit and won't be around until after the 11th May - gimme a time that suits after that and I'll be out :) |
|
@GlenTiki do you still have my email, the nearform one. Just drop me an email when your back and we can arrange a time that suits us. Catch up with you soon :) |
Creating this to capture/track as opposed to email discussion which is harder to pull people into.
Nearform has let the build WG know through email that they can no longer host the machines they had in our datacenter. These include
They have proposed moving then to another hoster which would cost $3856 Euros as a move cost and then $850 Euro per month as an ongoing cost.
From informal discussion so far we believe we don't need the Windows on ARM machines as they have been replaced by machines in Azure. That may make the cost a bit lower.
The options going forward at a high level would be:
outlined above which would require foundation Funding.
Initial discussion is that we don't believe we can/should just create the larger machines in existing hosters. As part of this process we should also confirm with the performance team what size machines are actually needed.
Given that there have been discussions with the Foundation/Linux IT team about them helping to manage machines and their stated approach of "fully owning" what they manage it would be good to see if Linux IT can take on solving this time sensitve issue for the project.
@bensternthal could to you take on getting Linux IT to give us a yes/no in terms of taking this on, ideally in a timeframe needed by Nearform.
@efrisby could you share what the required timeframe for a move is?
The text was updated successfully, but these errors were encountered: