Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] - do-not-merge - Add test PlacementTableVersionNotEventuallyIncreased #7571

Open
wants to merge 106 commits into
base: master
Choose a base branch
from

Conversation

fowlerlee
Copy link

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.

Description

Add a new test called PlacementTableVersionNotEventuallyIncreased

Signed-off-by: Lee Fowler [email protected]

Issue reference

Issue #7490_#7490 (comment)

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

@fowlerlee fowlerlee requested review from a team as code owners February 29, 2024 09:18
@fowlerlee fowlerlee marked this pull request as draft February 29, 2024 09:18
kiterrors "github.com/dapr/kit/errors"
)

func PlacementTableVersionNotEventuallyIncreased(tableVersion int, msg string) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @fowlerlee - so you'll want to call this func in place of where the relevant err is thrown. Here is a direct example of me doing that for the pubsub api, but youll need to find where this error is thrown in dapr and do the same thing. You'll want to pass in all relevant err details like maybe the appID (which you can typically find off the universal struct) and any other metadata that may be relevant to a user. Specifically, for the kiterrors code - you'll need to open a PR to dapr/kit to add the placement kiterrors.CodePrefixPlacement similar to this PR. Then you'll want to use that kit error prefix for the .WithErrorInfo().

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cicoyle : I added the error handling to this PR, and fixed some typos I had that were not conforming to the format tests. The issue still requires integration tests, but I'm not certain the form that these should take, since many already exist? I also 'git rebase' to get the master into the feature branch, hence the many commits from the master to this issue PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mikeee and others added 11 commits March 5, 2024 22:15
* ci: add 'mikeee' to dapr-bot owners

Signed-off-by: mikeee <[email protected]>

* fix: alphabetise list of bot owner gh handles

Signed-off-by: mikeee <[email protected]>

---------

Signed-off-by: mikeee <[email protected]>
* Make injector resilient to sentry unavailability (dapr#7507)

* make injector resilient to sentry unavailability

Signed-off-by: yaron2 <[email protected]>

* remove redundant line

Signed-off-by: yaron2 <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>

* sentry retry up to 30s (dapr#7508)

Signed-off-by: yaron2 <[email protected]>

* Update contrib to 1.13.0-rc.3 (dapr#7509)

* update contrib to 1.13.0-rc.2

Signed-off-by: yaron2 <[email protected]>

* update to rc.3

Signed-off-by: yaron2 <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>

* Injector: add option to add `DAPR_HOST_IP` env var to daprd (dapr#7511)

The `DAPR_HOST_IP` env var is used in various places in Dapr for a sidecar to know its own IP address, for example for service invocation or actor invocation.

When using the Dapr injector to add the daprd container, we can use the downstream APIs to add the `DAPR_HOST_IP` env var based on data from the controller

This option can be enabled by setting the Helm option `dapr_sidecar_injector.enableK8sDownwardAPIs=true`

Signed-off-by: ItalyPaleAle <[email protected]>

* Fix state encryption regression + add integration test (dapr#7517)

* fix state encryption regression + add intg test

Signed-off-by: yaron2 <[email protected]>

* linter

Signed-off-by: yaron2 <[email protected]>

* review feedback

Signed-off-by: yaron2 <[email protected]>

* linter

Signed-off-by: yaron2 <[email protected]>

* just use sha256, remove nolint:gosec

Signed-off-by: yaron2 <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>

* Test Integration: speed up tests 10% (dapr#7528)

* Test Integration: speed up tests 10%

Speed up integration tests by changing poll intervals
`100*time.Millisecond` to `10*time.Millisecond`.

~4.50m to ~4.20m

Signed-off-by: joshvanl <[email protected]>

* Wait for operator healthz before exiting to ensure no exit error

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* Revert selfhosted disk loader to not respect namespace (dapr#7527)

* Revert selfhosted disk loader to not respect namespace

Revert the selfhosted component disk loader to not respect the
namespace. This will allow the selfhosted disk loader to load components
from any namespace from file.

Fixes dapr#7523

Signed-off-by: joshvanl <[email protected]>

* Revert component disk loader behaviour to respect component namespace
when NAMESPACE env var is set.

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* [1.13] Add warning that Dapr state store encryption could lead to catastrophic failures (dapr#7524)

* [1.13] Add warning that Dapr state store encryption could lead to catastrophic failures

If the same key is used to encrypt more than 2^32 values (ie. more than 2^32 "Save" operations), it can lead to the private keys being exposed.

See: dapr#6027
Signed-off-by: ItalyPaleAle <[email protected]>

* Changed per review feedback

Signed-off-by: ItalyPaleAle <[email protected]>

---------

Signed-off-by: ItalyPaleAle <[email protected]>

* Add content-length to http channel (dapr#7537)

* add content-length to http channel

Signed-off-by: yaron2 <[email protected]>

* update tests

Signed-off-by: yaron2 <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>

* Updates components-contrib to 1.13.0-rc.4 (dapr#7538)

* Updates components-contrib to 1.13.0-rc.3

Signed-off-by: joshvanl <[email protected]>

* mod-tidy-all

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* Subscriptions: Fix panic when match rule is empty (dapr#7539)

Prevents a panic resulting when a subscription route rule match
(`spec.routes.rules.match`) is empty. An empty match rule is valid and
is considered "default". Error occurs because of Go interface vs
implementation struct pointer nil check foot-gun.

Adds integration tests for both HTTP and gRPC subscribers.

Signed-off-by: joshvanl <[email protected]>

* Hot Reloading: don't watch files if not enabled (dapr#7521)

* Hot Reloading: don't watch files if not enabled

Update hot reloading so that we don't setup file watchers on the
component directory if hot reloading is not enabled.

Signed-off-by: joshvanl <[email protected]>

* Remove `Close` paradigm from hot reloader

Signed-off-by: joshvanl <[email protected]>

* Linting

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* Remove pubsub content-length test which has been removed from (dapr#7550)

release-1.13

Signed-off-by: joshvanl <[email protected]>

* Revert ApiLevel controlling vnodes back to context metadata (dapr#7547)

* Revert ApiLevel controlling vnodes back to context metadata

Signed-off-by: Elena Kolevska <[email protected]>

* Lint

Signed-off-by: Elena Kolevska <[email protected]>

* Fixes after review

Signed-off-by: Elena Kolevska <[email protected]>

* Set back api level to 10, just for completeness purposes

Signed-off-by: Elena Kolevska <[email protected]>

* Don’t specify APILevelSpecify api level

Signed-off-by: Elena Kolevska <[email protected]>

* Apply suggestions from code review

Co-authored-by: Josh van Leeuwen <[email protected]>
Signed-off-by: Elena Kolevska <[email protected]>

* Cleanup after review

Signed-off-by: Elena Kolevska <[email protected]>

* Small refactor

Signed-off-by: Elena Kolevska <[email protected]>

* Refactor multiple parameters into a request object for placement table dissemination

Signed-off-by: Elena Kolevska <[email protected]>

---------

Signed-off-by: Elena Kolevska <[email protected]>
Signed-off-by: Elena Kolevska <[email protected]>
Co-authored-by: Josh van Leeuwen <[email protected]>
Co-authored-by: Artur Souza <[email protected]>

* Actor Reminders: Default JSON serialization. (dapr#7548)

* Actor Reminders: Default JSON serialization.

To support downgrades to 1.12 from 1.13, this PR changes the reminder
serialization storage format back to JSON by default. This means a 1.12
actor reminder client can read reminders written by 1.13 actors.

1.13 will continue to understand both JSON and protobuf. Protobuf
serialization can be enabled with the `ActorReminderStorageProtobuf`
feature gate. The actor "API Level" has been changed back to 10.

Adds test to ensure the default serialization is JSON.

Signed-off-by: joshvanl <[email protected]>

* Remove ActorReminderStorageProtobuf feature gate in favour of using API
level

Signed-off-by: joshvanl <[email protected]>

* Fix api level tests

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* Update contrib to 1.13.0-rc.6 (dapr#7553)

* update contrib to 1.13.0-rc.5

Signed-off-by: yaron2 <[email protected]>

* update to rc.6

Signed-off-by: yaron2 <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>

* Updates components-contrib to 1.13.0-rc.7 (dapr#7562)

Signed-off-by: joshvanl <[email protected]>

* update contrib to 1.13.0-rc.8 (dapr#7567)

* Add metadata in binding response even in case of error (dapr#7572)

* Add binding metadata even in case of error.

Signed-off-by: Artur Souza <[email protected]>

* ADD IT.

Signed-off-by: Artur Souza <[email protected]>

* Fix lint.

Signed-off-by: Artur Souza <[email protected]>

---------

Signed-off-by: Artur Souza <[email protected]>

* [Release-1.13] Upgrade to contrib `v1.13.0-rc.10` (dapr#7577)

* Upgrade to contrib 1.13.0-rc.9

Signed-off-by: Bernd Verst <[email protected]>

* Use contrib 1.13.0-rc.10 for latest kafka sarama patch version

Signed-off-by: Bernd Verst <[email protected]>

---------

Signed-off-by: Bernd Verst <[email protected]>

* Fix for issue 7576 (dapr#7581) (dapr#7587)

Signed-off-by: Guido Spadotto <[email protected]>
Co-authored-by: Guido Spadotto <[email protected]>
Co-authored-by: Guido Spadotto <[email protected]>

* Adds v1.13.0 release notes (dapr#7586)

* Adds v1.13.0 release notes

Adds docs/release_notes/v1.13.0.md

Signed-off-by: joshvanl <[email protected]>

* Update docs/release_notes/v1.13.0.md

Co-authored-by: Paul Yuknewicz <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>

* Move Actor Reminder Performance higher in notes

Signed-off-by: joshvanl <[email protected]>

* Update v1.13.0.md

Signed-off-by: Artur Souza <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>
Signed-off-by: Artur Souza <[email protected]>
Co-authored-by: Paul Yuknewicz <[email protected]>
Co-authored-by: Artur Souza <[email protected]>

* chore: bump go to 1.21.8 and protobuf lib to 1.33.0 (dapr#7591)

* ci: force go1.21.8 in workflows instead of go.mod

Signed-off-by: mikeee <[email protected]>

* chore: bump google.golang.org/protobuf to 1.33.0

Signed-off-by: mikeee <[email protected]>

* make modtidy-all

Signed-off-by: Artur Souza <[email protected]>

---------

Signed-off-by: mikeee <[email protected]>
Signed-off-by: Artur Souza <[email protected]>
Co-authored-by: Artur Souza <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>
Signed-off-by: ItalyPaleAle <[email protected]>
Signed-off-by: joshvanl <[email protected]>
Signed-off-by: Elena Kolevska <[email protected]>
Signed-off-by: Elena Kolevska <[email protected]>
Signed-off-by: Artur Souza <[email protected]>
Signed-off-by: Bernd Verst <[email protected]>
Signed-off-by: Guido Spadotto <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>
Signed-off-by: mikeee <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>
Co-authored-by: Alessandro (Ale) Segala <[email protected]>
Co-authored-by: Elena Kolevska <[email protected]>
Co-authored-by: Artur Souza <[email protected]>
Co-authored-by: Bernd Verst <[email protected]>
Co-authored-by: Guido Spadotto <[email protected]>
Co-authored-by: Guido Spadotto <[email protected]>
Co-authored-by: Paul Yuknewicz <[email protected]>
Co-authored-by: Mike Nguyen <[email protected]>
* make injector resilient to sentry unavailability

Signed-off-by: yaron2 <[email protected]>

* remove redundant line

Signed-off-by: yaron2 <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>
* update contrib to 1.13.0-rc.2

Signed-off-by: yaron2 <[email protected]>

* update to rc.3

Signed-off-by: yaron2 <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>
The `DAPR_HOST_IP` env var is used in various places in Dapr for a sidecar to know its own IP address, for example for service invocation or actor invocation.

When using the Dapr injector to add the daprd container, we can use the downstream APIs to add the `DAPR_HOST_IP` env var based on data from the controller

This option can be enabled by setting the Helm option `dapr_sidecar_injector.enableK8sDownwardAPIs=true`

Signed-off-by: ItalyPaleAle <[email protected]>
daixiang0 and others added 28 commits March 25, 2024 09:28
chore: convert crypto APIs to net/http
Before v1.12, Daprd's would connect to the sentry service on port 80. This
was changed to 443 in v1.12, but we kept the port 80 open for backwards
compatibility. This PR removes the port 80 from the sentry service.

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
* Refactor api/errors into builders

Refactors api/errors into functional builders. The current API for
api/errors is heavily string concatenation based with long parameter
arguments. This is fragile/error prone to the developer and difficult to
understand for the consumer.

This change should make it clearer to the reader/consumer what the
intended errors are and how to use them. Moving implementation details
of the error protocol codes into the error package also helps to clean
up API implementation code.

Signed-off-by: joshvanl <[email protected]>

* Remove use of `stderror` `errors` alias

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
Signed-off-by: ItalyPaleAle <[email protected]>
* Test Integration: reserve ports in ordered blocks

Dapr opens a lot of ports. Integration tests use upwards of a 1000 ports
through a run. Tests are run in parallel and ports are reserved on
demand through port 0 then being released before being used by a test.
There was a high chance that a port would attempted to be used by
multiple tests/processes at the same time causing conflict errors and
false test failures.

To prevent these conflicts, the ports utils has been moved to a process
which will reserve 300 blocks of ports to be used by tests. These ports
are reserved by incrementing through port ranges 1024 to 65535 in a
ring. This always prevents conflicts on this test runner, and should
also decrease port conflicts across multiple simultaneous test runners.

Signed-off-by: joshvanl <[email protected]>

* Rename intPort to port

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Loong Dai <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
Adds youtube link to a talk about the integration tests which might be
useful for developers to learn more.

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
* Runtime Manifest Loader Refactor

The current implementation of Component and HTTPEndpoint disk &
Kubernetes manifest loader spanned across disjointed packages with semi
generic functionality. PR continues the existing generic manifest loader
pattern with a shared internal package. Moves both Component and
HTTPEndpont loader implementations into shared internal generic package.

Existing Subscription, Configuration, and Resiliency loaders are left
as-is but should will be brought into this shared generic loader in
future PRs.

Signed-off-by: joshvanl <[email protected]>

* Adds namespace filter to disk manifest loader

Signed-off-by: joshvanl <[email protected]>

* Fix disk loader unit test

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
It is sometimes the case that Dapr services are suspended for a period
of time, for example when developer closes their laptop over night,
before being resumed after a significant period of time. Due to the
renewal timer being a Go Timer, it will also be suspended during this
time and only resume after the process is resumed.

Because X.509 certificates expiration validity follow wall-clock time
and the renewal timer follows process up-time time, it is possible for
a Dapr identity certificates to expire and not be renewed until enough
observed time has passed for the renewal timer to trigger.

This commit introduces a safety check whereby the renewal timer will
always trigger either every minute or at renewal time, whichever is
sooner. Once ticked, the renew loop will check to ensure whether the
renewal time has actually been reached before renewing.

This change helps out developers by not needing to restart the Dapr
cluster when they suspend their Dapr clusters for a period of time.
While it is not expected that this change be useful in production, it is
possible users are suspending and resuming virtual machines in
production.

Adds a renewal integration test to ensure Dapr identity certificate is
renewed based on expiration.

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
Co-authored-by: Loong Dai <[email protected]>
Adds support for injector process in integration tests.

Adds basic healthz & ports tests for injector.

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
Use correct free port import and interface in injector integration test
process.

Signed-off-by: joshvanl <[email protected]>
Signed-off-by: mikeee <[email protected]>
Co-authored-by: Josh van Leeuwen <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
…ders

chore: Convert actor HTTP APIs off fasthttp
…IRE TLS clients and servers. (dapr#7037)

* Removes legacy SPIFFE TLS clients and servers in favour of the new SPIRE TLS clients and servers.

Signed-off-by: joshvanl <[email protected]>

* Fix sentry int tests, and adds test to ensure legacy ID is not longer
accepted

Signed-off-by: joshvanl <[email protected]>

* String match on sentry Kubernetes validator longname test

Signed-off-by: joshvanl <[email protected]>

* Fix namespace of sentry in operator tests

Signed-off-by: joshvanl <[email protected]>

* Linting

Signed-off-by: joshvanl <[email protected]>

* Update integration kubernetes process to use leaf certificate with
cluster.local

Signed-off-by: joshvanl <[email protected]>

* Fix setting correct control plane trust domain on daprd

Signed-off-by: joshvanl <[email protected]>

* Remove SENTRY_LOCAL_IDENTITY form expected env var

Signed-off-by: joshvanl <[email protected]>

* Fix control plane trust domain setting in test

Signed-off-by: joshvanl <[email protected]>

* Fixes int version skew tests using legacy client/server

Signed-off-by: joshvanl <[email protected]>

* Fix int version-skew patch on v1.13.0

Signed-off-by: joshvanl <[email protected]>

* Use correct namespace for sentry in injector integration tests

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>
…apr#7657)

When block shutdown is enabled, Dapr will block the shutdown sequence
for a configurable amount of time or until the app becomes unhealthy.
During this time the out-bound Dapr APIs remain available but all
in-bound requests are blocked. Currently, in-flight PubSub messages are
canceled when Dapr receives a SIGTERM, which can result in lost
messages.

Patch updates the pubsub subscribe message handler policy runner to
execute sending the message to the app channel in a background context.
This ensures that when Dapr cancels the Subscription due to a blocking
shutdown, the in-flight message is not effected.

Enables the `dapr_component_pubsub_egress_bulk_count` and
`dapr_component_pubsub_egress_bulk_latency` metrics for bulk pubsub
egress which were previously not enabled, as they are used to test that
an in-flight message is correctly sent after Dapr has begun to shutdown.

Signed-off-by: joshvanl <[email protected]>
…pr#7659)

* Fix incorrect content-length being sent to HTTP published message

PR dapr#7537 reverse revered the change which removed the content-length
from being set on HTTP headers on sent messages. We still need to remove
the content-length from messages from pubsub subscriptions as the pubsub
may report a message with a content-length which does not actually
match the size of the delivered message to the app.

content-length is only removed on HTTP published messages.

Change should be backported to 1.13

Signed-off-by: joshvanl <[email protected]>

* Adds grpc app subscriber to content-length tests

Signed-off-by: joshvanl <[email protected]>

* Adds tests for content-length gRPC subscribed app

Signed-off-by: joshvanl <[email protected]>

* framework/socket: skip if test is windows

Signed-off-by: joshvanl <[email protected]>

* Fix socket runtime GOOS import

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
On Darwin (MacOS), `os.TempDir()` nicely returns a truly ephemeral
process directory is `/var/folders/...` however in order to reap the
benefits of Go cache magic in integration tests we need a slightly more
permanent directory, i.e. `/tmp`. This directory will persist across
integration test runs and speed up the test execution boot times from
s to ms.

Signed-off-by: joshvanl <[email protected]>
@fowlerlee fowlerlee marked this pull request as ready for review April 2, 2024 14:35
@cicoyle
Copy link
Contributor

cicoyle commented May 7, 2024

It looks like the git rebase might have messed up. I am seeing a file diff of 364 files changed, which I wouldn't expect for this change. Can you try again, either a merge or rebase? I can better assist once I can see just the changes intended for this PR 🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet