Skip to content

Releases: neondatabase/neon

release-5493

13 May 16:04
c2e7cb3
Compare
Choose a tag to compare

What's Changed

  • Add option allowing running multiple endpoints on the same branch. by @arssher in #7581
  • Make 'neon.protocol_version = 2' the default by @hlinnaka in #7616
  • pageserver: call maybe_freeze_ephemeral_layer from a dedicated task by @jcsp in #7594
  • pageserver: label tenant_slots metric by slot type by @jcsp in #7603
  • Fix grammar in provider.rs error message by @tristan957 in #6864
  • refactor(pageserver): remove --update-init flag by @problame in #7612
  • test: less flaky test_synthetic_size_while_deleting by @koivunej in #7622
  • Fix test_ts_of_lsn_api flakiness by @arpad-m in #7599
  • On-demand WAL download for walsender by @save-buffer in #6872
  • proxy: reduce number of concurrent connections by @khanova in #7620
  • use less neon_local --pageserver-config-override / pageserver -c by @problame in #7613
  • pageserver: post-shard-split layer trimming (1/2) by @jcsp in #7572
  • feat: Timeline detach ancestor by @koivunej in #7456
  • build(deps): bump werkzeug from 3.0.1 to 3.0.3 by @dependabot in #7625
  • build(deps): bump jinja2 from 3.1.3 to 3.1.4 by @dependabot in #7626
  • Add Postgres commit sha to Postgres version by @bayandin in #4603
  • build(deps): bump flask-cors from 3.0.10 to 4.0.1 by @dependabot in #7633
  • fix(test suite): forward compat test is not using latest neon_local by @problame in #7637
  • refactor(rtc): remove excess cloning by @koivunej in #7635
  • remove neon_local --pageserver-config-override by @problame in #7614
  • feat(pageserver): generate basebackup from aux file v2 storage by @skyzh in #7517
  • pageserver: include generation number in local layer paths by @jcsp in #7609
  • proxy: cull http connections by @conradludgate in #7632
  • compute_ctl: Break up main() by @sharnoff in #7577
  • compute_ctl: Fix unused variable on non-Linux by @sharnoff in #7646
  • tests: common log allow list for ancestor detach tests by @jcsp in #7645
  • neon_local init: write pageserver.toml directly; no pageserver --init --config-override by @problame in #7638
  • proxy: do not invalidate cache for permit errors by @conradludgate in #7652
  • Tiered compaction: add order asserts after delta key k-merge by @arpad-m in #7648
  • Improve tiered compaction tests by @arpad-m in #7643
  • build(deps): bump moto from 4.1.2 to 5.0.6 by @bayandin in #7653
  • neon_local: use pageserver.toml as source of truth for struct PageServerConf by @problame in #7642
  • Fix flaky test_gc_of_remote_layers by @petuhovskiy in #7647
  • Expose LSN and replication delay as metrics by @save-buffer in #7610
  • chore(neon_test_utils): restrict installation to superuser by @skyzh in #7624
  • pageserver: local layer path followups by @jcsp in #7640
  • Ignore page header when comparing VM pages in test_vm_bits.py by @knizhnik in #7499
  • pageserver: fix division by zero in layer counting metric by @VladLazar in #7662
  • remote_storage: AWS_PROFILE with endpoint overrides in ~/.aws/config (updates AWS SDKs) by @problame in #7664
  • tests: adjust log allow list since reqwest upgrade by @jcsp in #7666
  • storage controller: fix handing of tenants with no timelines during scheduling optimization by @jcsp in #7673
  • Fix tiered compaction k-merge bugs and use in-memory alternative by @arpad-m in #7661
  • Revert "chore(neon_test_utils): restrict installation to superuser" by @skyzh in #7679
  • build(deps): bump Npgsql from 8.0.2 to 8.0.3 in /test_runner/pg_clients/csharp/npgsql by @dependabot in #7680
  • fix(test): update the config for neon_binpath in from_repo_dir by @skyzh in #7684
  • Proxy added per ep rate limiter by @khanova in #7636
  • pageserver: don't call get_vectored with empty keyspace by @jcsp in #7686
  • use own arm64 gha runners by @cicdteam in #7373
  • Fix permissions for safekeeper failpoints by @petuhovskiy in #7669
  • openapi: resolve the synthetic_size duplication by @koivunej in #7651
  • draw_timeline_dir: draw branch points and gc cutoff lines by @koivunej in #7657
  • build: run doctests by @koivunej in #7697
  • Tiered compaction: fix early exit check in main loop by @arpad-m in #7702
  • Fix checkpoint metric by @lubennikovaav in #7701
  • feat: allow detaching from ancestor for timelines without writes by @koivunej in #7639
  • Handmade Release 2024-05-13 by @VladLazar in #7735
  • Release 2024-05-13 by @vipvap in #7713

Full Changelog: release-5434...release-5493

release-5434

07 May 08:18
c4d7d59
Compare
Choose a tag to compare

What's Changed

  • neon_local: add "tenant import" by @jcsp in #7399
  • storage controller: send startup compute notifications in background by @jcsp in #7495
  • proxy: Exclude compute and retries by @khanova in #7529
  • s3_scrubber: add tenant-snapshot by @jcsp in #7444
  • proxy: Adjust retry wake compute by @khanova in #7537
  • proxy: Create disconnect events by @khanova in #7535
  • pagserver: use vectored read path in benchmarks by @VladLazar in #7498
  • chore(vm-image): specify sql exporter listen port by @skyzh in #7526
  • pageserver: tweak vec get validation for ancestor lsn wait by @VladLazar in #7533
  • Update azure_* crates to 0.19 by @arpad-m in #7539
  • chore(pageserver): improve in-memory layer vectored get by @skyzh in #7467
  • pageserver: shard-aware keyspace partitioning by @jcsp in #6778
  • pageserver: remove workarounds from #7454 by @jcsp in #7550
  • docs: fix unintentional file link by @fcdm in #7506
  • feat(pageserver): separate sparse and dense keyspace by @skyzh in #7503
  • pageserver: reduce runtime of init_tenant_mgr by @jcsp in #7553
  • Make test_random_updates and test_read_at_max_lsn compatible with new compaction by @arpad-m in #7551
  • Make s3_scrubber for safekeepers useful. by @arssher in #7297
  • storage controller: test for large shard counts by @jcsp in #7475
  • tests: tweak log allow list in test_sharding_split_failures by @jcsp in #7549
  • Makefile: DISABLE_HOMEBREW variable by @prepor in #7556
  • Discover safekeepers via broker request by @petuhovskiy in #7279
  • chore(pageserver): concise error message for layer traversal by @skyzh in #7565
  • chore(pageserver): categorize basebackup errors by @skyzh in #7523
  • Add retry loops and bump test timeout in test_pageserver_connection_stress by @save-buffer in #7281
  • update to reqwest 0.12 by @conradludgate in #7561
  • proxy: keep track on the number of events from redis by type. by @khanova in #7582
  • Fix usage of pg_waldump --ignore option by @lubennikovaav in #7578
  • fix(pageserver): properly propagate missing key error for vectored get by @skyzh in #7569
  • storage controller: make the initial database wait configurable by @mattpodraza in #7591
  • vm-image: add sqlexporter for autoscaling metrics by @skyzh in #7514
  • chore(pageserver): plumb through RequestContext to VirtualFile write methods by @problame in #7566
  • Remove tenant_id from TenantLocationConfigRequest by @arpad-m in #7469
  • Use streaming downloads for Azure as well by @arpad-m in #7579
  • Proxy reconnect pubsub before expiration by @khanova in #7562
  • Store operation identifier in IdLockMap on exclusive lock by @jbajic in #7397
  • fix(pageserver): compile warning of download_object.ctx on macos by @skyzh in #7596
  • refactor(update_gc_info): split GcInfo to compose out of GcCutoffs by @koivunej in #7584
  • Increase Azure parallelism limit to 100 by @arpad-m in #7597
  • tests: relax test download remote layers api by @VladLazar in #7604
  • storage_controller: fix non-timeline passthrough GETs by @jcsp in #7602
  • fix: find gc cutoff points without holding Tenant::gc_cs by @koivunej in #7585
  • pageserver: jitter secondary periods by @jcsp in #7544
  • Update rust to 1.78.0 by @arpad-m in #7598
  • test_runner: remove unused NeonPageserver.config_override field by @problame in #7605
  • storage controller: check warmth of secondary before doing proactive migration by @jcsp in #7583
  • feat(pageserver): add scan interface by @skyzh in #7468
  • proxy: add connect compute concurrency lock by @conradludgate in #7607
  • fix(pageserver): remove update_gc_info calls in tests by @skyzh in #7608
  • refactor: move NodeMetadata to pageserver_api; use it from neon_local by @problame in #7606
  • compute_ctl: Add support for swap resizing by @sharnoff in #7434
  • compute_ctl: Be more precise in how startup time is calculated by @hlinnaka in #7601
  • Allow bad state (not active) pageserver error/warns in walcraft test. by @arssher in #7592
  • Properly initialize first WAL segment on safekeepers. by @arssher in #7574
  • Release 2024-05-06 by @vipvap in #7615

New Contributors

Full Changelog: release-5373...release-5434

release-5373

29 Apr 13:45
0de1e1d
Compare
Choose a tag to compare

What's Changed

  • Make test less sensitive to exact WAL positions by @hlinnaka in #7436
  • Make another walcraft test more robust by @hlinnaka in #7439
  • refactor(Timeline): simpler metadata updates by @koivunej in #7422
  • relaxation: allow using layers across timelines by @koivunej in #7453
  • proxy: Make retries configurable and record it. by @khanova in #7438
  • pageserver: fix unlogged relations with sharding by @jcsp in #7454
  • pagectl draw-timeline-dir: include layer file name as an SVG comment by @problame in #7455
  • chore(pageserver): separate missing key error by @skyzh in #7393
  • Refactor updating relation size cache on reads by @hlinnaka in #7376
  • pageserver: remove import/export script previously used for breaking format changes by @VladLazar in #7458
  • Don't pass tenant_id in location_config requests from storage controller by @arpad-m in #7476
  • pageserver: fix vectored read aux key handling by @VladLazar in #7404
  • pageserver: add read depth metrics and test by @VladLazar in #7464
  • storage_controller: wait for db on startup by @jcsp in #7479
  • feat(pageserver): add metadata key range and aux key encoding by @skyzh in #7401
  • remote_storage: fix prefix handling in remote storage & clean up by @jcsp in #7431
  • pageserver: change pitr_interval=0 behavior by @jcsp in #7423
  • get_lsn_by_timestamp: clamp commit_lsn to be >= min_lsn by @arpad-m in #7488
  • fix: avoid starving background task permits in eviction task by @koivunej in #7471
  • Fix a bug with retries by @khanova in #7494
  • pageserver: publish the same metrics from both read paths by @VladLazar in #7486
  • pageserver: finish vectored get early by @VladLazar in #7490
  • fix(metrics): correct maxrss metrics on macos by @skyzh in #7487
  • Ability to specify the upload_storage_class in S3 bucket configuration by @arpad-m in #7461
  • controller: limit Reconciler concurrency by @jcsp in #7493
  • reduce complexity of proxy protocol parse by @conradludgate in #7078
  • Update connect to compute and wake compute retry configs by @khanova in #7509
  • proxy: Fix cancellations by @khanova in #7510
  • pageserver: coalesce read paths by @VladLazar in #7477
  • proxy: Workaround switch to the regional redis by @khanova in #7513
  • chore(pageserver): temporary metrics on ingestion time by @skyzh in #7515
  • Compute <-> pageserver protocol version 2 by @hlinnaka in #7377
  • storage controller: log hygiene & better error type by @jcsp in #7508
  • refactor(owned_buffers_io::BufferedWriter): be generic over the type of buffer by @problame in #7482
  • refactor(owned_buffer_io::util::size_tracking_writer): make generic over underlying writer by @problame in #7483
  • refactor(ephemeral_file): reuse owned_buffers_io::BufferedWriter by @problame in #7484
  • Remove unused import by @jcsp in #7519
  • perf!: use larger buffers for blob_io and ephemeral_file by @problame in #7485
  • pageserver: fix image layer creation check that inhibited compaction by @VladLazar in #7420
  • Return Past instead of Present or Future when commit_lsn < min_lsn by @arpad-m in #7520
  • feat(pageserver): add aux-file-v2 flag on tenant level by @skyzh in #7505
  • chore(pageserver): shrink aux keyspace to 0x60-0x7F by @skyzh in #7502
  • fix(virtual_file): compile warnings on macos by @skyzh in #7525
  • metrics: record time to update gc info as a per timeline metric by @koivunej in #7473
  • Release 2024-04-29 by @vipvap in #7530

Full Changelog: release-5324...release-5373

release-5324

22 Apr 13:44
271598b
Compare
Choose a tag to compare

What's Changed

  • pageserver: improve readability of shard.rs by @jcsp in #7330
  • storage release: stop using no-op deployProxy / deployPgSniRouter by @problame in #7382
  • proxy: Exclude private ip errors from recorded metrics by @khanova in #7389
  • add async walredo mode (disabled-by-default, opt-in via config) by @problame in #6548
  • pageserver: return ACCEPTED when deletion already in flight by @jcsp in #7384
  • storage_controller: suppress misleading log by @jcsp in #7395
  • proxy: rate limit authentication by masked IPv6. by @conradludgate in #7316
  • pageserver/http: remove status code boilerplate from swagger spec by @VladLazar in #7385
  • feat(neon_local): support listen addr for safekeeper by @skyzh in #7328
  • chore(compute_ctl): add error context to apply_spec by @skyzh in #7374
  • CI(check-build-tools-image): fix getting build-tools image tag by @bayandin in #7402
  • proxy: Cleanup unused rate limiter by @khanova in #7400
  • Add for add-help-for-timeline-arg for timeline command by @jbajic in #7361
  • pageserver: demote range end fallback log by @VladLazar in #7403
  • proxy: Improve logging by @khanova in #7405
  • proxy: Record role to span by @khanova in #7407
  • proxy: move endpoint rate limiter by @conradludgate in #7413
  • proxy: Record only valid rejected events by @khanova in #7415
  • feat: copy delta layer prefix or "truncate" by @koivunej in #7228
  • pageserver: fix secondary download scheduling by @jcsp in #7396
  • Add a docs page for storage controller by @jcsp in #7392
  • fix(Layer): metric regression with too many canceled evictions by @koivunej in #7363
  • build(deps): bump aiohttp from 3.9.2 to 3.9.4 by @dependabot in #7429
  • pageserver: fix cont lsn jump on vectored read path by @VladLazar in #7412
  • storcon_cli: tenant-warmup command by @jcsp in #7432
  • CI: fix docker/setup-buildx-action & update rustls by @bayandin in #7445
  • Bump vm-builder v0.23.2 -> v0.28.1 by @sharnoff in #7433
  • Release 2024-04-22 by @vipvap in #7447

Full Changelog: release-5294...release-5324

release-5294

15 Apr 13:22
c213373
Compare
Choose a tag to compare

What's Changed

New Contributors

Full Changelog: release-5269...release-5294

release-5269

11 Apr 17:28
e0addc1
Compare
Choose a tag to compare

What's Changed

Full Changelog: release-5267...release-5269

release-5267

08 Apr 12:46
5da39b4
Compare
Choose a tag to compare

What's Changed

  • update tokio 1.37 by @conradludgate in #7276
  • Standardizing Logging Format for Improved Middleware Integration by @macdoos in #7194
  • CI(benchmarking): Add Sharded Tenant for pgbench by @bayandin in #7186
  • tests: clean up compat test workarounds by @jcsp in #7097
  • pageserver: drop the layer map lock after planning reads by @VladLazar in #7215
  • proxy: upload consumption events to S3 by @khanova in #7213
  • controller: add storcon-cli by @jcsp in #7114
  • remove http timeout by @conradludgate in #7291
  • pageserver: even more debug for test_secondary_downloads by @jcsp in #7295
  • refactor(walreceiver): eliminate task_mgr usage by @problame in #7260
  • wal_ingest: global counter for bytes received by @koivunej in #7240
  • Schedule image layer uploads in tiered compaction by @arpad-m in #7282
  • pageserver: on-demand activation of tenant on GET tenant status by @jcsp in #7250
  • Upload partial segments by @petuhovskiy in #6530
  • pageserver: replace the locked tenant config with arcsawps by @VladLazar in #7292
  • refactor(Timeline::shutdown): rely more on Timeline::cancel; use it from deletion code path by @problame in #7233
  • update ordered-multimap by @conradludgate in #7306
  • tests: reinstate some syntethic size tests by @VladLazar in #7294
  • Fix bug in prefetch cleanup by @knizhnik in #7277
  • Fix length by @khanova in #7308
  • storage controller: grant 'admin' access to all APIs by @jcsp in #7307
  • storage_controller: setup sentry reporting by @VladLazar in #7311
  • fix(walredo spawn): coalescing stalls other executors std::sync::RwLock by @problame in #7310
  • pageserver: timeout on deletion queue flush in timeline deletion by @jcsp in #7315
  • pageserver: handle WAL gaps on sharded tenants by @jcsp in #6788
  • pageserver: update mgmt_api to use TenantShardId by @jcsp in #7313
  • storage controller: fix repeated location_conf returning no shards by @jcsp in #7314
  • tests: extend log allow list in test_storcon_cli by @jcsp in #7321
  • Fix zero segment upload by @petuhovskiy in #7318
  • Update test-context dependency to 0.3 by @arpad-m in #7303
  • proxy: report metrics based on cold start info by @conradludgate in #7324
  • Clean up 'attachment service' names to storage controller by @jcsp in #7326
  • storage controller: rename TenantState to TenantShard by @jcsp in #7329
  • tests: improve stability of test_deletion_queue_recovery by @jcsp in #7325
  • pageserver: apply shard filtering to blocks ingested during initdb by @jcsp in #7319
  • fixup: PR #7319 defined workload.py def stop() twice by @problame in #7333
  • control_plane: revise compute_hook locking (don't serialise all calls) by @jcsp in #7088
  • tests: further stabilize test_deletion_queue_recovery by @jcsp in #7335
  • fix(utilization): return used as does df by @koivunej in #7337
  • Release 2024-04-08 by @vipvap in #7338

New Contributors

Full Changelog: release-5227...release-5267

release-5227

02 Apr 16:27
82027e2
Compare
Choose a tag to compare

What's Changed

  • Drop connections with all shards invoplved in prefetch in case of error by @knizhnik in #7249
  • Support compaction_threshold=1 for tiered compaction by @arpad-m in #7257
  • storage controller: tenant scheduling policy by @jcsp in #7262
  • pageserver: fix oversized key on vectored read by @VladLazar in #7259
  • fix: drop replication slot causes postgres stuck on exit by @skyzh in #7192
  • Specify caller in 'unexpected response from page server' error by @lubennikovaav in #7272
  • CI(deploy): Do not deploy storage controller to preprod for proxy releases by @bayandin in #7269
  • spec: allow neon extension auto-upgrade + softfail upgrade by @skyzh in #7231
  • storage controller: be more tolerant of control plane blocking notifications by @jcsp in #7268
  • pageserver: check for new image layers based on ingested WAL by @VladLazar in #7230
  • storage controller: scheduling optimization for sharded tenants by @jcsp in #7181
  • neonvm: add LFC approximate working set size to metrics by @skyzh in #7252
  • fixup(#7278/compute_ctl): remote extension download permission by @skyzh in #7280
  • Release 2024-04-01 by @vipvap in #7284

Full Changelog: release-5213...release-5227

release-proxy-5201

27 Mar 17:21
2a88889
Compare
Choose a tag to compare

What's Changed

  • tests: log hygiene checks for storage controller by @jcsp in #6710
  • fixup(#7168): neon_local: use pageserver defaults for known but unspecified config overrides by @skyzh in #7166
  • Move functions for creating/extracting tarballs into utils by @tristan957 in #7132
  • storage controller: tech debt by @jcsp in #7165
  • test: fix on demand activation test flakyness by @VladLazar in #7180
  • proxy: enable sha2 asm support by @conradludgate in #7184
  • fix: ResidentLayer::load_keys should not create INFO level span by @koivunej in #7174
  • refactor(layer): use detached init by @koivunej in #7152
  • pageserver: extra debug for test_secondary_downloads failures by @jcsp in #7183
  • fix(Layer): always init after downloading in the spawned task by @koivunej in #7175
  • Dump layer map json in test_gc_feedback.py by @arpad-m in #7179
  • fix: add safekeeper team to pgxn codeowners by @skyzh in #7170
  • safekeeper: correctly handle signals by @skyzh in #7167
  • Add state diagram for compute by @tristan957 in #7172
  • fix(heavier_once_cell): take_and_deinit should take ownership by @koivunej in #7185
  • fix(layer): remove the need to repair internal state by @koivunej in #7030
  • Force LSN order with wrapper structure by @jbajic in #7071
  • proxy: async aware password validation by @conradludgate in #7176
  • storage_controller: add metrics by @VladLazar in #7178
  • storage controller: fixes to secondary location handling by @jcsp in #7169
  • pageserver: quieten log on shutdown-while-attaching by @jcsp in #7177
  • pageserver: extend /re-attach response to include tenant mode by @jcsp in #6941
  • proxy: simplify password validation by @conradludgate in #7188
  • walredo benchmark: throughput-oriented rewrite by @problame in #7190
  • Update Rust to 1.77.0 by @arpad-m in #7198
  • proxy: connect redis with AWS IAM by @khanova in #7189
  • Fix ephemeral file warning on secondaries by @jcsp in #7201
  • proxy: unit tests for auth_quirks by @conradludgate in #7199
  • CI: deploy release version to a preprod region by @bayandin in #6811
  • pageserver: write consumption metrics to S3 by @jcsp in #7200
  • Remove Get/SetZenithCurrentClusterSize from Postgres core by @knizhnik in #7196
  • fixup(#7204 / postgres): revert IsPrimaryAlive checks by @skyzh in #7209
  • proxy: fix stack overflow in cancel publisher by @conradludgate in #7212
  • pageserver: use a single tokio runtime by @problame in #6555
  • test: fix test_pageserver_recovery flakyness by @VladLazar in #7207
  • storage controller: tighten up secrets handling by @jcsp in #7105
  • pageserver: track total ephemeral layer bytes by @jcsp in #7182
  • Try to fix test_crafted_wal_end flakiness. by @arssher in #7143
  • tests: stabilize compat tests by @jcsp in #7227
  • chore: remove repetitive words by @availhang in #7206
  • refactor(remote_timeline_client): infallible stop() and shutdown() by @problame in #7234
  • tests: tolerate transient unavailability in test_sharding_split_failures by @jcsp in #7223
  • tests: wait for uploads in test_secondary_downloads by @jcsp in #7220
  • proxy: reuse sess_id as request_id for the cplane requests by @khanova in #7245
  • Revert "pageserver: use a single tokio runtime (#6555)" by @problame in #7246
  • pageserver: drop layer map lock in Timeline::get by @VladLazar in #7217
  • test_runner/performance: skip test_bulk_insert by @bayandin in #7238
  • pageserver: return 429 on timeline creation in progress by @jcsp in #7225
  • pageserver: limit total ephemeral layer bytes by @jcsp in #7218
  • storage controller: make direct tenant creation more robust by @jcsp in #7247
  • tests: add a log allow list to stabilize benchmarks by @jcsp in #7251
  • pageserver: remove bare mgr::get_tenant, mgr::list_tenants by @jcsp in #7237
  • add authentication rate limiting by @conradludgate in #6865
  • Proxy release 2024-03-27 by @vipvap in #7254

New Contributors

Full Changelog: release-proxy-5147...release-proxy-5201

release-compute-only-5213

27 Mar 19:24
c431e2f
Compare
Choose a tag to compare

What's Changed

  • pageserver: drop layer map lock in Timeline::get by @VladLazar in #7217
  • test_runner/performance: skip test_bulk_insert by @bayandin in #7238
  • pageserver: return 429 on timeline creation in progress by @jcsp in #7225
  • pageserver: limit total ephemeral layer bytes by @jcsp in #7218
  • storage controller: make direct tenant creation more robust by @jcsp in #7247
  • tests: add a log allow list to stabilize benchmarks by @jcsp in #7251
  • pageserver: remove bare mgr::get_tenant, mgr::list_tenants by @jcsp in #7237
  • add authentication rate limiting by @conradludgate in #6865
  • fix(walreceiver): Timeline::shutdown can leave a dangling handle_walreceiver_connection tokio task by @problame in #7235
  • CI(trigger-e2e-tests): set e2e-platforms by @bayandin in #7229
  • Revert "Revoke REPLICATION" by @save-buffer in #7261
  • Release 2024-03-27 - compute only release by @vipvap in #7263

Full Changelog: release-5201...release-5213