Choose object storage provider for haikuports

[kallisti5]

With haikuporter's support of s3, we need to choose a object storage provider.
For context, this will be replacing our Digital Ocean volume block attachment which is $25 month / 250GiB

Assuming ~400GiB stored... 2TiB of egress a month (which gives us a lot of head room)
Assuming 35 million API ops a month (17M Class A, 17M Class B)

• Wasabi $16-35 / month likely as we grow. Risk of pulling too much egress and getting shut off.
  • Pro: We use them today, "known evil"
  • Pro: Cheap as long as we don't pull more than we store. ($5.99 per TiB)
  • Con: Weird "storage amounts count for 3 months" system.
  • Con: Storage pricing is cheap, but inconsistent. If we use more egress then we store, they get upset and cut us off. (and have done it before). There's a lot of risk here of sudden cut-offs without clear resolution paths.
  • Con: We probably should put them behind some caching CDN. bunny.net is 0.005/per GiB. +$10/mo
• telnyx - Likely $11-20 / month, $ per API operations a big risk. Haikuporter, hpkgbouncer, all hit APIs
  • Pro: Flat $0.006 per GiB stored. At 400 GiB that's $2.4
  • Pro: No egress fees (We started on Wasabi in this model... so I question it long term)
  • Con: Change operations (REST PUT COPY POST LIST) $0.5 per million.
  • Con: Read operations (REST GET SELECT) $0.04 per million
  • Con: New player
• storj.io - Likely $12-20 / month
  • Pro: Flat $0.004 per GiB stored. At 400 GiB that's $1.6/month
  • Pro: Flat $0.006 per GiB egress. At 2TiB that's $12 / month
  • Pro: No per-operation costs
  • Pro: Lots of interesting use cases like hosting our own file gateways
  • Con: Not the cheapest.
  • Con: Would need CDN for + 10 / mo
• Backblaze B2 - Likely $16-24 / month
  • Pro: Flat $6 per TiB
  • Pro: Free egress up to 3x their average monthly storage amount. Egress over average stored is $0.01/GiB. 2TiB - 400GiB = 1600GiB * 0.01 = $16 / month
  • Pro: Big company, been around for a long time
  • Con: Most expensive "cheap" s3 storage provider.
  • Con: Need for CDN likely at scale. Unlimited free egress with CDN partners: Fastly, Cloudflare, bunny.net, CacheFly, Coreweave, Equinix Metal, Vultr, PhoenixNAP. (I've looked at bunny.net before for DNS... they're pretty good. 2TiB $10/mo)
• Digital Ocean Spaces - ~$23 per month for 400GiB + 2TiB xfer We don't like having to have multiple buckets to get reasonable pricing.
  • Pro: Flat $5 for 250GiB.
  • Pro: Can have multiple buckets for $5 / per 250GiB
  • Pro: 1TiB egress included.
  • Pro: Local to our nodes. May cut down egress bandwidth usage.
  • Con: Expensive 0.01/GiB egress after 1TiB. 2TiB total is $10
  • Con: Would need branch haikuports buckets for reasonable pricing, needing hpkgbouncer changes.
  • Con: Expensive 0.05 per GiB after 250GiB in a single bucket. 400GiB is $12.50/month

Notes: We don't have to go all-in on a single S3 provider. Haiku can remain at wasabi, haikuports can be "where ever". We can run one deployment of hpkgbouncer per repo.

[kallisti5]

My preference:

1. Storj.io - Cheap, does some cool things being based on storj. Would need a CDN for egress.
2. Backblaze B2 - not the cheapest.. but if we grow we can toss a reasonably priced CDN in front of it and get free egress (minus how much the CDN costs... and bunny.net is cheap)
3. Wasabi - Continue with Wasabi... they have served us well. However, we should likely throw a CDN in front of them to reduce risk of getting cut off.
4. Digital Ocean - Risks being expensive, but can't beat "local" object storage access (which will hopefully cut down on bandwidth usage)
5. Telnyx - Cheapest storage at scale... we just have to be mindful of API transaction limits from hpkgbouncer and haikuporter. New provider though.

[kallisti5]

I've reached out to wasabi to try and get "actual" bandwidth utilization numbers. They don't publish it in our portal (but I sure as hell know they look at it since they have cut us off before due to egress)

[waddlesplash]

Why do you have Backblaze as "20-25 a month"? If we factor in the CDN with free egress then shouldn't it be storage costs only, and thus be equivalent to Wasabi + CDN?

[kallisti5]

Why do you have Backblaze as "20-25 a month"? If we factor in the CDN with free egress then shouldn't it be storage costs only, and thus be equivalent to Wasabi + CDN?

"Free egress up to 3x their average monthly storage amount. Egress over average stored is $0.01/GiB.
2TiB - 400GiB = 1600GiB * 0.01 = $16 / month"
16 + 6 = $21

EDIT: I did that math wrong. Lets re-run the cost numbers. Assuming 6TiB egress, and 400GiB storage.

Backblaze:

• Storage: $6
• Egress: 6000GiB (haikuports transfer below) - 1200 (400GiB * 3) = 4800GiB * 0.01 = $48 / month
• Egress + Bunny CDN: 6000*0.005 = $30 / month
• Daily API Call limits. 2,500 Class B free 0.004 per 10,000 after , 2,500 Class C, 0.004 per 1,000 after.
  • If the Wasabi bill is right, we make 900k a month. That's around 30k a day. ~+$4 - $8 / month

Storj:

• Storage: $1.6 / month
• Egress: 6000GiB $36
• CDN: 6000*0.005 $30 (probably not worth it beyond caching down requests to Storj)

Wasabi:

• haha, we'll "cut you off" for using more than 400GiB egress. CDN might cut down bandwidth since it caches popular artifacts........ but man, the risks are big and 400GiB vs 6TiB is a big gap.
• I think we might need to strike off Wasabi for haikuports.

Telnyx:

• Storage: 0.006 * 400 = $2.4 / month
• No egress fees
• $0.5 per million change operations (PUT, COPY, POST, LIST)
• $0.04 per million read operations (GET, SELECT)
  • Big unknown costs since we don't track api call types + counts

Backblaze + bunny.net CDN seems like the best deal tbh with controlled risk. The Bunny.net CDN could cut that 6TiB way down to a "a few TiB or less" on all providers, but it's an unknown how efficient their caching is in our use-case

[kallisti5]

EDIT - Actual worst-case egress bandwidth numbers:

• haiku repo at Wasabi averages 54GiB / month
• haikuports is some percentage of our overall Digital Ocean egress... 5.3 - 5.9 TiB 🫨

[nielx]

For me, while I think reliability is important, it is not the end of the world if we get cut off and need to relocate. However, how do we keep control of our packages? I.e. is there going to be a backup or a primary source for them?
Another factor is the odds of hidden surprises, i.e. I do not want to be surprised by a sudden change of rates if we cross some sort of threshold, so any provider with a 'flat' rate that scales linearly is preferred over a provider that requires us to closely monitor some sort of threshold.
Finally, I would keep things as easy as possible, so the Digital Ocean Spaces where we will need to do additional data design is off the table for me.

[kallisti5]

For me, while I think reliability is important, it is not the end of the world if we get cut off and need to relocate. However, how do we keep control of our packages? I.e. is there going to be a backup or a primary source for them?

The nice thing about s3 is it actually gets easier to back things up. Today we have the automatic "compress all the artifacts, encrypt them, and upload to an s3 bucket" backup system. That doesn't work for huge things though since I really don't want to work with 300GiB tar delta's 😅

In the model where some object storage provider is the source of truth, we really just need to rclone the bucket "somewhere" else. Historically i've just rcloned to a dedicated bit of local storage at my house as a cold backup (you could do the same). rclone works off of deltas like rsync, so it's bandwidth consumption friendly after the initial clone.

rclone also lets you sync between storage providers... and it supports a TON

We actually have an rclone container today ready to go that will do that to storj. We can make some fixes though to make it more generic.

I also have rclonefs which will (theoretically) let us mount s3 buckets as fuse storage mounts on each k8s node so we can (theoretically) offer s3 buckets over rsync to mirrors from pods running on any k8s node. (fuse in k8s is weird though, and we need elevated security context).

Another factor is the odds of hidden surprises, i.e. I do not want to be surprised by a sudden change of rates if we cross some sort of threshold, so any provider with a 'flat' rate that scales linearly is preferred over a provider that requires us to closely monitor some sort of threshold.

Agree. Definitely the biggest pain point of object storage. I really like the pricing of Telnyx, but the whole "per million API hits" thing makes me nervous on something complex and large like haikuports.

Finally, I would keep things as easy as possible, so the Digital Ocean Spaces where we will need to do additional data design is off the table for me.

Agree. Lets strike DO off the list. They had some appealing things to them, but needing a whole gaggle of buckets to groom to get reasonable pricing is too much lift. I'm tired of forming infrastructure "around" providers weird limitations.

[kallisti5]

I updated #141 (comment) with the pricing based on the actual worst case bandwidth numbers I saw on digital ocean.

[kallisti5]

Oh, and I just looked at the Wasabi bill.. it does list "908.40 API requests" for the month. I'm guessing that's 1000's though given the decimal point.. so 908,400 makes more sense.

[kallisti5]

Here's some data on the bunny.net cdn. It definitely cuts down our bandwidth usage ~50% on a single haiku nightly repo.

I'm sure the savings will be less for haikuports (more random packages, etc)

[nielx]

Looks like the preferred is backblaze + bunny then?

[kallisti5]

Agree. I think backblaze + bunny are going to be the cheapest combo. Bunny will cut down the xfer 50%, so that $30 / month should be "worst case"

[kallisti5]

Ryan went ahead and entered our billing info. I went ahead and deployed a temporary VM @ digital ocean to use to shovel artifacts over to backblaze.

I'm going to start with the Haiku repos themselves since it's an easy (smaller) test of data before moving on to haikuports.

[kallisti5]

Aaaand.. Backblaze just crapped the bed.

API calls are NOT free.

EDIT: I guess +$4-8 a month extra for API calls isn't horrible... however it adds risk to Backblaze.

[nielx]

API calls are NOT free.

EDIT: I guess +$4-8 a month extra for API calls isn't horrible... however it adds risk to Backblaze.

That's not great and definitely false advertising...

[kallisti5]

I went ahead and put the haiku repo over onto backblaze. We already blew past the "free tier" of class C api calls during the last sync. 😮‍💨

I'm about to head out of town and will be back Sunday.. so here are important facts:

• I set a cap of $3/day on API calls just in-case the API calls begin to be too many. If we hit the cap, hpkgbouncer will stop updating and other bad stuff ™️

If the 💩 hits the fan, you can take the following actions to undo the migration to backblaze:

• Revert this, and apply via kubectl: 74a0979
• Revert this and apply via concourse: 49cda45

[infrastation]

Other possible alternatives (not mutually exclusive):

• Apply for Digital Ocean's credits for open source projects
• See if there are any deals remaining at LEB and LET BF/CM, get a dedicated server for $20-40/month or a VM with sufficiently high monthly transfer for $20-30/year and serve the contents from there. (Since this would not be a high-end resilient service, it would be a good idea to have a backup option if the cheap server/VM fails.)

[kallisti5]

I'm not super happy with backblaze. The pricing is ok, but we ran into a few weird gotchas which are annoying enough to make me want to swerve:

• Backblaze's API calls are NOT free as originally advertised (mentioned before)
• Backblaze requires us to have an account for each geographic region we want data in. (and a payment method on each)
• Bunny.net CDN (which gives us reasonable pricing when paired with B2) is unreliable.
  • Bunny.net doesn't pick up when files change on B2.
  • If a build runs multiple times, checksums in our repo file doesn't match the package files causing errors in pkgman.
  • We solve this by using the API to tell Bunny.net to clear caches for updated files in our repo on B2
    • But, Bunny.net's API to clear the cache of individual files doesn't work.
      • Bunny.net's first level support confirmed the issue we saw, and raised it to engineering...
        • ...2 months ago. Zero updates since 😠

Given the above, it's kinda soured my opinions on B2. On the bright side, since it's just S3 we can move easily.

With Backblaze crossed off, I looked to https://www.s3compare.io/ to find which provider is cheapest.

• 0.75 TiB storage (I adjusted to 750 GiB in this comparison to account for future growth)
• 2 TiB egress (to be safe)
• 1 month

• iDrive E2 - Looks more for personal use than server?
• Backblaze - lol
• Hetzner - Not against, but peering outside of Europe is still SLOW
• Storj - One of my personal favorites still given the cool tech. Though rsync gateway (gateway-rsync) storj/storj#7324 would make it a clear winner.

Of course, we could go colo / dedicated and run our own S3 server if we're looking to be cheap. Since we want to offer rsync mirroring this might be a good compromise. I just want to keep our business logic away from our storage servers.

[kallisti5]

* Apply for Digital Ocean's [credits for open source projects](https://www.digitalocean.com/open-source/credits-for-projects)

I can check into this. My only fear is we use quite a bit of space since we're a "Whole OS". Digital Ocean only offers ~50GiB of "free" Object Storage, and places a high valuation on it when you scale up towards 1TiB.

* See if there are any deals remaining at [LEB](https://lowendbox.com/) and [LET BF/CM](https://lowendtalk.com/discussion/200000/2024-black-friday-cyber-monday-flash-sale-megathread), get a dedicated server for $20-40/month or a VM with sufficiently high monthly transfer for $20-30/year and serve the contents from there. (Since this would not be a high-end resilient service, it would be a good idea to have a backup option if the cheap server/VM fails.)

Yeah, i'm honestly not against this at this point. We could run our own S3 and rsync server on a dedicated / colo box.

One of my goals has been to keep "business logic" and "data" separate. Previous historic infrastructure designs have mixed mission critical services with storage which means when things go wrong they go WRONG. It's nice to have the s3 API level of separation between things if we do have a dedicated server / colo system.