[pull] main from alipay:main #19
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? <!-- Describe the purpose of this PR. --> Add a new util called `data2Description`, which can reduce the amount of code ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? Remove Guava's Collection usages ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> #1113 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? Lazy map become slightly more lazy =) <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? <!-- Describe the purpose of this PR. --> Support CopyOnWriteArrayListSerializer ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? Add serialization language field for Fury Header in xlang_spec. ## Related issues #1607 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [x] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> --------- Signed-off-by: LiangliangSui <[email protected]>
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? Removes Guava's Concurrency utils usages ## Related issues #1113 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? fix fastutil Object2ObjectOpenHashMap serialization ## Related issues Closes #1615 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? Native stream is not feasible for every cases, this PR add blocked stream utils to adapt to streaming API. This is s serialization helper as the fallback of streaming serialization/deserialization in FuryInputStream/FuryReadableChannel. FuryInputStream/FuryReadableChannel will buffer and read more data, which makes the original passed stream when constructing FuryInputStream not usable. If this is not possible, use this BlockedStreamUtils instead for streaming serialization and deserialization. Note that this mode will disable streaming in essence. It's just a helper for make the usage in streaming interface more easily. The deserialization will read whole bytes before do the actual deserialization, which don't have any streaming behaviour under the hood. ## Related issues #1451 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? <!-- Describe the purpose of this PR. --> This PR introduces a validation method to ensure that all input strings to the `MetaString` encoder are ASCII. ## Related issues <!-- Is there any related issue? Please attach here. - #1619 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> --------- Signed-off-by: Jason Mok <[email protected]>
## What does this PR do? <!-- Describe the purpose of this PR. --> before guava 19.0, of() return EmptyImmutableSet/EmptyImmutableBiMap/EmptyImmutableSortedMap/EmptyImmutableSortedSet. check class and register serializer for them. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? <!-- Describe the purpose of this PR. --> ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [x] Does this PR introduce any public API change? - [x] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> --------- Co-authored-by: weijiang.wj <[email protected]>
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? <!-- Describe the purpose of this PR. --> we use `Sets.newConcurrentHashSet()` to create set, with lastest guava version it use `ConcurrentHashMapKeySetView` and old version it use `Collections.newSetFromMap(map)`. remove `CopyOnWriteArrayListSerializer` from native-image.properties. #1614 forget delete. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…zer ctr (#1625) ## What does this PR do? <!-- Describe the purpose of this PR. --> it has a problem when we use `ThreadSafeFury` and custom Serializer which special constructor, so we need a method provider `Fury` to create Serializer and add it. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? <!-- Describe the purpose of this PR. --> when subclass without fields. `org.apache.fury.meta.ClassDefEncoder#getClassFields` will skip subclass so className will be superclass. however we should save class even fields is empty. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…mentation (#1624) ## What does this PR do? <!-- Describe the purpose of this PR. --> ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> #1113 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change?
…classLoaderFuryPooled (#1628) ## What does this PR do? <!-- Describe the purpose of this PR. --> ThreadPoolFury#factoryCallback don't work when create new classLoaderFuryPooled. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? <!-- Describe the purpose of this PR. --> This PR enhances the current ASCII check (before meta string encoding) I implemented in #1620 to return a UTF-8 encoded `MetaString` early if the input is non-ASCII. This improves efficiency and saves time on `computeEncoding` and `encode`. Unit tests are also added. ## Related issues #1619 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> --------- Signed-off-by: Jason Mok <[email protected]>
…zeNonexistentAsNull (#1634) ## What does this PR do? rename deserializeUnexistentEnumValueAsNull to deserializeNonexistentAsNull ## Related issues ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? remove gpg pinentry-mode ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? fix wildcard capturer capture NullPointerException ## Related issues Closes #1633 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? This PR fix inlineable expression codegen and reduce class load cost in generated code. With those changes, it fixed abstract collection elems same type serialization in #1640 ## Related issues Closes #1640 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? remove soft/weak ref values from thread safe fury ## Related issues Closes #1632 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…ode (#1646) ## What does this PR do? support nonexistent class deserialization in meta share mode ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? This PR bumps release version to 0.5.1 since fury 0.5.1 has been released ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? fix example code error.  ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? fix scala object type codegen ## Related issues Closes #1658 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…lity (#1660) ## What does this PR do? This PR implements scoped meta share mode for type forward/backward compaibility ## Related issues #202 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark Perf increased from `1900102.586` to `2430410.064` ``` Before: Benchmark (bufferType) (objectType) (references) Mode Cnt Score Error Units fury_deserialize array MEDIA_CONTENT false thrpt 10 2734151.212 ± 253921.628 ops/s fury_deserialize_compatible array MEDIA_CONTENT false thrpt 10 1900102.586 ± 62176.872 ops/s furymetashared_deserialize_compatible array MEDIA_CONTENT false thrpt 10 3011439.327 ± 260518.752 ops/s After: Benchmark (bufferType) (objectType) (references) Mode Cnt Score Error Units fury_deserialize array MEDIA_CONTENT false thrpt 10 2661186.814 ± 279377.198 ops/s fury_deserialize_compatible array MEDIA_CONTENT false thrpt 10 2430410.064 ± 164165.865 ops/s furymetashared_deserialize_compatible array MEDIA_CONTENT false thrpt 10 3098083.064 ± 259391.053 ops/s ``` Size decreased from **732 to 577**: ``` Before 2024-05-30 01:00:49 INFO FuryState:157 [fury_deserialize_compatible-jmh-worker-1] - ======> Fury | MEDIA_CONTENT | false | array | 732 | After 2024-05-30 12:57:00 INFO FuryState:157 [fury_deserialize_compatible-jmh-worker-1] - ======> Fury | MEDIA_CONTENT | false | array | 577 | ``` The
## What does this PR do? <!-- Describe the purpose of this PR. --> Add IssueNavigationLink and icon[1] for IDEA [1] https://fury.apache.org/img/navbar-logo.svg ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> Co-authored-by: red <[email protected]>
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/incubator-fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? <!-- Describe the purpose of this PR. --> - Resolve deprecate warnings from `chrono` dep - Remove patch version in Cargo.toml ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> N/A ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/incubator-fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? No ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> Signed-off-by: Ruihang Xia <[email protected]>
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? java.util.Date and its subclasses are mutable,but in version <= 0.10.0,all TimeSerializers extends from ImmutableSerializer. This pr reorganized the inheritance relationships of TimeSerializers.Moved and added test case to adapte new version. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> [issus:](url)#2071 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? 1. Rename `TypeDescription` to `ClassInfo` to keep consistent with Java/Python. 2. Replace `Tag` by `Namespace` and `TypeName` which is used for marking structs in xlang protocol. 3. Add MetaStringResolver to manage the reading and writing of`Namespace` and `TypeName`. 4. Add `TypeMetaResolver` which is defined in (https://fury.apache.org/docs/specification/fury_xlang_serialization_spec#type-meta). Note that this part of protocol may change in the furture. 5. Add `Evoluation Mode` which is defined in (https://fury.apache.org/docs/specification/fury_xlang_serialization_spec#schema-evolution), Now we can deserialize in `Evoluation Mode` 6. Add a `constructClass` config to control the construction of Class. Constructor often introduces a remote code execution vulnerability, if `constructClass` is disabled, we will construct the object to a anonymous object without call the constructor of the class. 7. Refactor the MetaStringEncoder MetaStringDecoder and MetaString to make it perfectly consistent with Java/Python 8. Bump version to 0.11.0-SNAPSHOT constructClass Disabled: ```javascript @type.struct("example.foo") class Foo { @Type.int32() a: number; } const fury = new Fury({ refTracking: true }); fury.registerSerializer(Foo); const foo = new Foo({constructClass: false}); foo.a = 123; const input = fury.serialize(foo) const result = fury.deserialize(input); expect(!(result instanceof Foo)); // The result is constructed from Object, not Foo. expect(result).toEqual({ a: 123 }) // We can access the props normality ``` Evoluation Usage: ```javascript const fury = new Fury({ mode: Mode.Compatible }); @type.struct("example.foo") class Foo { @type.string() bar: string; @Type.int32() bar2: number; setBar(bar: string) { this.bar = bar; return this; } setBar2(bar2: number) { this.bar2 = bar2; return this; } } const { serialize } = fury.registerSerializer(Foo); const bin = serialize(new Foo().setBar("hello").setBar2(123)); @type.struct("example.foo") class Foo2 { @type.string() bar: string; } const fury2 = new Fury({ mode: Mode.Compatible, hooks: { afterCodeGenerated: (code: string) => { return beautify.js(code, { indent_size: 2, space_in_empty_paren: true, indent_empty_lines: true }); } } }); const { deserialize } = fury2.registerSerializer(Foo2); const r = deserialize(bin); expect(r).toEqual({ bar: "hello", bar2: 123, }) ``` It's too much code for a single PR, Todos in the codebase will be resolve by other PRS.
Update the version in docs to latest
## What does this PR do? <!-- Describe the purpose of this PR. --> ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? Optimize the writing of float and bool type sequences via the C API ## Related issues ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark ``` fury_large_float_tuple: Mean +- std dev: [base] 32.8 ms +- 0.9 ms -> [opt] 28.4 ms +- 0.8 ms: 1.16x faster fury_large_boolean_tuple: Mean +- std dev: [base] 22.1 ms +- 0.2 ms -> [opt] 13.6 ms +- 0.6 ms: 1.63x faster Geometric mean: 1.37x faster ```
…2074) <!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? fix ImmutableCollections$SubList duplicate registration java.util.ImmutableCollections$SubList has been registered twice, resulting in different numbers of classIdGenerator across different JDK versions, which ultimately leads to serialization failures when crossing JDK versions. <!-- Describe the purpose of this PR. --> ## Related issues Closes #2070 <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> --------- Co-authored-by: 吴怡帆 <[email protected]>
## What does this PR do? The PR (nodejs/node@6cb0690) for string detection committed in the node repository is merged. Now we can speed up string processing without hps module. There are now three ways of string serialization, the first is with hps module which has super high performance; the second is for higher versions of nodejs which can detect string encoding at zero cost; the third is the normal way which is used for lower version of nodejs and browser env.
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? <!-- Describe the purpose of this PR. --> ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> --------- Co-authored-by: hening <[email protected]>
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? Add extra entry to disallowed file. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
<!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? <!-- Describe the purpose of this PR. --> After testing, it was observed that ThreadPoolFury experiences prolonged blocking during cold starts under high-concurrency scenarios. Analysis revealed that improper usage of locks in ClassLoaderFuryPooled was the root cause. This PR refactors the implementation of ClassLoaderFuryPooled by significantly reducing the granularity of locks, thereby drastically minimizing blocking time during cold starts. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> [](url)#2087 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> CPU:9950X Class:org.apache.fury.benchmark.ThreadPoolFurySuite.java old: Percentiles, ms/op: p(0.0000) = 0.001 ms/op p(50.0000) = 0.001 ms/op p(90.0000) = 1587.388 ms/op p(95.0000) = 1587.388 ms/op p(99.0000) = 1587.388 ms/op p(99.9000) = 1587.388 ms/op p(99.9900) = 1587.388 ms/op p(99.9990) = 1587.388 ms/op p(99.9999) = 1587.388 ms/op p(100.0000) = 1587.388 ms/op new: Percentiles, ms/op: p(0.0000) = 0.001 ms/op p(50.0000) = 0.001 ms/op p(90.0000) = 62.746 ms/op p(95.0000) = 62.746 ms/op p(99.0000) = 62.746 ms/op p(99.9000) = 62.746 ms/op p(99.9900) = 62.746 ms/op p(99.9990) = 62.746 ms/op p(99.9999) = 62.746 ms/op p(100.0000) = 62.746 ms/op
…ween byte arrays (#2101) ## What does this PR do? Try to use `System.arraycopy` when copying between byte arrays. This significantly increases performance, in our use case serialization is twice as fast (serializing/deserializing 1M complex strucutures using (de)serializeJavaObject in parallel). When using Platform.copyMemory, the JDK is doing a lot of type checking to see if it's a byte array, causing a visible slowdown. `Unsafe.memoryCopy` implementation was changed in Java 9 from a native method to a method calling the internal unsafe method containing the reflection checks. There are a few places where System.arraycopy is already used, referring to it being faster on certain JDK implementations. Tested our use case on a macbook with arm64 cpu against zulu 21 and open-jdk 23.  ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark I tried to reproduce the performance impact through one on existing benchmarks `UserTypeSerializeSuite.fury_serialize`, however I didn't see a difference here. Signed-off-by: Seppe Volkaerts <[email protected]>
## What does this PR do? use sha256 to check disallowed.txt tamper ## Related issues Closes #2100 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…formed field names and avoid redundant computation (#2091) <!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? This PR refactors the field sorting in `StructSerializer.java` by caching transformed field names to avoid redundant computations during sorting. The `lowerCamelToLowerUnderscore` transformation is now applied once per field instead of multiple times. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…thon (#2108) <!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? ### Problem The struct hash calculation implementations in Java and Python are inconsistent. When `classinfo` is `None` in the Python implementation, the method returns early without updating the hash value, whereas the Java implementation continues with a default hash value of 0. ### Solution Modified the `visit_customized` method in the Python implementation to initialize `hash_value` to 0 before checking if `classinfo` is `None`, ensuring the hash computation continues regardless of the `classinfo` status. This approach aligns with the Java implementation behavior. Changes: - Initialize `hash_value` to 0 at the beginning of the method - Replace the early return with a conditional assignment to `hash_value` - Always call `_compute_field_hash` at the end of the method ### Testing Verified the fix by running the `CrossLanguageTest.java` tests that were previously failing due to this issue. The struct hash values now match between Java and Python implementations. ## Related issues - #2107 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…2111) ## What does this PR do? ## Related issues Closes #2109 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
#2113) ## What does this PR do? This pr supports passed tracking ref meta when building serializers. The meta can be pssed by Type Annotation in #2036 or by classdef encoded into binary. ## Related issues ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? <!-- Describe the purpose of this PR. --> ## Related issues Closes #2089 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? <!-- Describe the purpose of this PR. --> ## Related issues Closes #2093 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
#2116) ## What does this PR do? support nested bean in array/collection/map for row format ## Related issues Closes #2106 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? ## Related issues Closes #2119 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? remove required_linear_history ## Related issues Closes #2121 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? This PR supports xlang map chunk serialization between java/python ## Related issues Closes #2125 Closes #2126 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? - add windows ci for java21 - Fix sha256 check error on windows:  ## Related issues Closes #2100 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…2130) ## What does this PR do? This PR implements homogeneous xlang collection serialization between java/python. Changes include: - Xlang homogeneous collection serialization in cython/python - Xlang chunk map serialization in pure python for debug - Use homogeneous collection serialization in java - homogeneous collection serialization between java and python ## Related issues Closes #2131 Closes #2132 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
## What does this PR do? <!-- Describe the purpose of this PR. --> ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…sts (#2136) ## What does this PR do? fix nested chunk map serialization error when generics exists ## Related issues Closes #2135 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
# Add Fury Dart Support
## Overview
This PR adds Dart language support to Apache Fury, implementing a
comprehensive serialization solution for Dart and Flutter applications.
Fury Dart consists of approximately 15,000 lines of code and provides an
efficient serialization mechanism that works within Flutter's reflection
limitations.
## Implementation Approach
Dart supports reflection, but Flutter explicitly prohibits it. To
address this constraint, Fury Dart uses a combination of:
1. Core serialization/deserialization logic
2. Static code generation for type handling
This approach ensures compatibility with Flutter while maintaining the
performance and flexibility expected from Fury.
## Features
- XLANG mode support for cross-language serialization
- Reference tracking for handling object graphs
- Support for primitive types, collections, and custom classes
- Serializer registration system
- Code generation for class and enum serialization
- Support for nested collections with automatic generic type conversion
- Custom serializer registration
- Support for using ByteWriter/ByteReader as serialization sources
## Usage Examples
### Basic Class Serialization
```dart
import 'package:fury_core/fury_core.dart';
part 'some_class.g.dart';
@furyClass
class SomeClass with _$SomeClassFury {
late int id;
late String name;
late Map<String, double> map;
SomeClass(this.id, this.name, this.map);
SomeClass.noArgs();
}
```
After annotating your class with `@furyClass`, run:
```bash
dart run build_runner build
```
This generates the necessary code in `some_class.g.dart` and creates the
`_$SomeClassFury` mixin.
### Serializing and Deserializing
```dart
Fury fury = Fury(
xlangMode: true,
refTracking: true,
);
fury.register($SomeClass, "example.SomeClass");
SomeClass obj = SomeClass(1, 'SomeClass', {'a': 1.0});
// Serialize
Uint8List bytes = fury.toFury(obj);
// Deserialize
obj = fury.fromFury(bytes) as SomeClass;
```
### Enum Serialization
```dart
import 'package:fury_core/fury_core.dart';
part 'enum_foo.g.dart';
@furyEnum
enum EnumFoo {
A,
B
}
```
Registration is similar to classes:
```dart
fury.register($EnumFoo, "example.EnumFoo");
```
## Type Support
Fury Dart currently supports the following type mappings in XLANG mode:
| Fury Type | Dart Type |
|----------------------------|------------------------------------------------|
| bool | bool |
| int8 | fury.Int8 |
| int16 | fury.Int16 |
| int32 | fury.Int32 |
| var_int32 | fury.Int32 |
| int64 | int |
| var_int64 | int |
| sli_int64 | int |
| float32 | fury.Float32 |
| float64 | double |
| string | String |
| enum | Enum |
| named_enum | Enum |
| named_struct | class |
| list | List |
| set | Set (LinkedHashSet, HashSet, SplayTreeSet) |
| map | Map (LinkedHashMap, HashMap, SplayTreeMap) |
| timestamp | fury.TimeStamp |
| local_date | fury.LocalDate |
| binary | Uint8List |
| bool_array | BoolList |
| int8_array | Int8List |
| int16_array | Int16List |
| int32_array | Int32List |
| int64_array | Int64List |
| float32_array | Float32List |
| float64_array | Float64List |
## Project Structure
The implementation is organized into three main components:
1. **SourceGen**: Located at
`dart-fury/packages/fury-core/lib/src/code_gen`
Handles static code generation for serialization/deserialization.
2. **FuryCore**: Located at `dart-fury/packages/fury-core/lib/src`
Contains the core serialization and deserialization logic.
3. **FuryTest**: Located at `dart-fury/fury_test`
Comprehensive test suite for Fury Dart functionality.
## Testing Approach
The test suite is inspired by Fury Java's testing approach and includes:
- **Data Type Tests**: Validates custom data types implemented for Dart
- **Code Generation Tests**: Ensures correctness of the generated static
code
- **Buffer Tests**: Validates correct memory handling for primitive
types
- **Cross-Language Tests**: Tests functionality against other Fury
implementations
- **Performance Tests**: Simple benchmarks for
serialization/deserialization performance
### Running Tests
Tests use the standard [dart test](https://pub.dev/packages/test)
framework.
To run tests:
```bash
# First, generate necessary code
cd fury-test
dart run build_runner build
# Run all tests
dart test
# For more options (skipping tests, platform-specific tests, etc.)
# See: https://github.com/dart-lang/test/blob/master/pkgs/test/README.md
```
## Code Quality
Fury Dart maintains high code quality standards. You can verify this
using:
```bash
dart analyze
dart fix --dry-run
dart fix --apply
```
## Current Limitations
- Only supports XLANG mode (priority was given to cross-language
compatibility)
- No out-of-band buffer functionality
- No data type compression (e.g., String compression)
- Generic parameters in user-defined types can be serialized but require
manual type conversion after deserialization
## Development Information
- **Dart SDK**: 3.6.1
## Dependencies
### fury-core package:
```
analyzer: '>=6.5.0 <8.0.0'
build: ^2.4.1
build_config: ^1.1.0
collection: ^1.17.0
meta: ^1.14.0
source_gen: ^2.0.0
source_helper: ^1.3.4
glob: ^2.1.3
decimal: ^3.2.1
lints: ^5.0.0
build_runner: ^2.4.6
source_gen_test: ^1.0.6
test: ^1.24.4
```
### fury-test package:
```
lints: ^5.0.0
build: ^2.4.2
build_runner: ^2.4.15
test: ^1.24.0
checks: ^0.3.0
build_test: ^2.2.3
analyzer: ^7.3.0
collection: ^1.19.1
```
---
Thank you for considering this PR. I'm available to address any
questions or feedback regarding the implementation.
## What does this PR do? - remove non-existent v8-fast-api-calls.h from license - remove duplicate notice text - remove binary image - add license header for main document (I don't add license header for readme.md) ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…rialization in java (#2139) <!-- **Thanks for contributing to Fury.** **If this is your first time opening a PR on fury, you can refer to [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fury (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fury/blob/main/CONTRIBUTING.md). - Fury has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## What does this PR do? <!-- Describe the purpose of this PR. --> ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
…t serializer (#2140) ## What does this PR do? <!-- Describe the purpose of this PR. --> ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )