Conversation
fixed: Need to fix boot1 as `audit` does not show semver$ tio /dev/ttyACM0
[13:25:30.227] tio 3.9
[13:25:30.228] Press ctrl-t q to quit
[13:25:30.228] Connected to /dev/ttyACM0
audit
Board type reads as: Dabao
Boot partition is: Ok(PrimaryPartition)
Semver is:
Description is: Towards Beta-0
Device serializer: 00000000-00000000-0000e500-e5000069
Public serial number: JMGSNA
UUID: a2e2cfac-6a5d1193-dd1e6ad5-594f9426
...UpdateFixed by patching (just during nix build) xtask logic when git describe is called, instead we read an env var set in $ tio /dev/ttyACM0
[17:44:52.774] tio 3.9
[17:44:52.774] Press ctrl-t q to quit
[17:44:52.775] Connected to /dev/ttyACM0
audit
Board type reads as: Dabao
Boot partition is: Ok(PrimaryPartition)
Semver is: v0.9.16-9871-g631dc42
Description is: Towards Beta-0
... |
f9d9c16 to
9b76b9f
Compare
ef375de to
1484607
Compare
Rahzael
left a comment
There was a problem hiding this comment.
@bunnie I wasn't quite sure if you wanted me doing a review on this, but figured it couldn't hurt to give my two cents. I put some comments below on how the nix stuff works and where it's getting code from. I'm much less familiar with the github actions portion, though.
@sbellem I had some questions below, specifically in regards to the github actions, as I'm not very familiar with them and want to make sure I understand where outside code is being pulled in from.
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Install Nix | ||
| uses: DeterminateSystems/determinate-nix-action@v3 |
There was a problem hiding this comment.
Forgive me if I'm a little unfamiliar with github actions. If I'm not mistaken, this runs the actions found here, correct? https://github.com/DeterminateSystems/determinate-nix-action/blob/main/action.yml.
| uses: DeterminateSystems/determinate-nix-action@v3 | ||
|
|
||
| - name: Setup cache | ||
| uses: cachix/cachix-action@v15 |
There was a problem hiding this comment.
Same with this one but for this url? https://github.com/cachix/cachix-action/blob/master/action.yml
There was a problem hiding this comment.
I imagine these changes are to help the docker cache not fill up as fast?
There was a problem hiding this comment.
Yes, the commit is not meant to make it through with this PR, but to lessen the load of on GitHub's cache temporarily. The docker-based builds and related ci have been refactored since then, and there's a PR at #787.
| let secs: i64 = sde.parse().expect("SOURCE_DATE_EPOCH must be a valid Unix timestamp"); | ||
| Utc.timestamp_opt(secs, 0).single().expect("Invalid SOURCE_DATE_EPOCH timestamp").to_rfc2822() | ||
| } else { | ||
| Local::now().to_rfc2822() |
There was a problem hiding this comment.
Any reasoning on why to use local time rather than UTC for the timestamp?
There was a problem hiding this comment.
This is what the original code does, and I left it as is, given that the purpose here is not to make modifications to the existing code/logic but just to add a conditional branch in which the builder of the firmware sets SOURCE_DATA_EPOCH in order have deterministic builds.
flake.nix
Outdated
| inputs = { | ||
| nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/0.2511.906247"; | ||
| flake-utils.url = "github:numtide/flake-utils/11707dc2f618dd54ca8739b309ec4fc024de578b"; | ||
| rust-xous.url = "github:sbellem/rust-xous-flake?rev=39eebf47342faf50a2892e9dfadee895068157b8"; |
There was a problem hiding this comment.
@bunnie These all specify outside flakes (build files) that are run on build when using nix. Basically every build input, including these and their inputs, recursively, is versioned by a hash that is captured by the flake.lock file above, so any changes can be detected. (You'd want to look for changes here as well as the flake.lock file above.)
Do you trust @sbellem to host the main rust-xous flake, or would you rather it lived here with you? Either way, it probably wouldn't hurt to have a backup in case anything happened to it.
Most of the other flakes I recognize:
Nixpkgs is just the main package repository for the the nix package manager and build system. Although, it does appear to be using a https://flakehub.com/ url, which is a private flake repository and binary cache. I've not used them myself, but they are run by Determinate Systems, which is a big and well-known player in the Nix ecosystem. I'll leave whether or not you want to trust them to serve the binary cache up to you.
Numtide's flake-utils are included in a lot of flakes, and help provide tools that make writing these files easier.
Oxelica's rust overlay is a common tool for downloading specific Rust toolchains in binary form (but should still be versioned using a hash that's stored in the flake.lock above), such as the Xous toolchain and a generic RISCV target later below. (It's worth noting that this is also a Flakehub url, so the flake and it's build output will be cached and served by them.)
The last one is Ipetkov's crane, which I had to look up. However, it seems to be pretty commonly used for building rust projects, and after finding it I do remember it as being one of the options mentioned here for building rust projects: https://nixos.wiki/wiki/Rust#Packaging_Rust_projects_with_nix.
Overall, nothing overtly malicious seems to be going on here, at least to my still learning eyes.
There was a problem hiding this comment.
Thanks @Rahzael for the review!
I made some updates to flake.nix such that only github urls are used to source the inputs. Simplified version shown here for reference (see full change at commit 2d6d159).
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/078d69f03934859a181e81ba987c2bb033eebfc5";
flake-utils.url = "github:numtide/flake-utils/11707dc2f618dd54ca8739b309ec4fc024de578b";
crane.url = "github:ipetkov/crane/0314e365877a85c9e5758f9ea77a9972afbb4c21";
rust-overlay.url = "github:oxalica/rust-overlay/e9bcd12156a577ac4e47d131c14dc0293cc9c8c2";
rust-xous.url = "github:sbellem/rust-xous-flake/24959275c25e5bc903c85979a9794e1937495ec6";
};I don't have a strong opinion on whether strictly using github versus flakehub is best, although from a few prompts with Claude, relying strictly on github may be best, but very frankly I don't know. I'm posting here a brief summary comparing both, github and flakehub as sources for nix packages (flakes).
FlakeHub vs nixpkgs as Nix Flake Inputs
(by Claude AI)
URL Format
# Direct from GitHub
inputs.nixpkgs.url = "github:NixOS/nixpkgs/078d69f03934859a181e81ba987c2bb033eebfc5";
# FlakeHub (0.2511.906333 points to commit 078d69f03934859a181e81ba987c2bb033eebfc5)
inputs.nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/0.2511.906333";Key Differences
| Aspect | nixpkgs (GitHub) | FlakeHub |
|---|---|---|
| Versioning | Branch or commit-based | Semver constraints (e.g., 0.1.*) |
| Updates | Latest commit from branch | Within version constraint |
| Caching | GitHub infrastructure | Dedicated CDN with tarball caching |
| Private flakes | Requires GitHub auth setup | Built-in token-based auth |
| Discoverability | N/A | Registry UI to browse flakes |
When to Use
- FlakeHub: Semver guarantees, controlled updates, using other FlakeHub flakes
- Direct nixpkgs: Simpler setup, no external dependency, comfortable with branch pinning
Open Source Project Considerations
| Factor | nixpkgs (GitHub) | FlakeHub |
|---|---|---|
| Contributor friction | None—everyone has GitHub access | Minor learning curve for unfamiliar contributors |
| Transparency | Lock file shows raw git commit | Adds layer of indirection |
| Long-term availability | GitHub is stable, widely mirrored | Depends on Determinate Systems |
| Community norms | Standard convention in Nix ecosystem | Less common in open source |
| CI/CD | Native access in most CI systems | May need extra auth setup |
Recommendation
For open source projects, direct nixpkgs is usually the better default—simpler, no external dependencies, and aligns with community conventions.
FlakeHub is better suited for enterprise or private projects where semver constraints and registry features provide more value.
There was a problem hiding this comment.
I think that I agree with Claude that direct nixpkgs with github (or other public git repo) links for flakes makes the most sense for an open source project.
I think it improves auditability and transparency on where exactly the code is coming from.
There was a problem hiding this comment.
I like this better than the previous one. It makes it more clear where the code is coming from.
9477670 to
b6a0055
Compare
5abf2e3 to
4021fb9
Compare
Problem: there is no reproducible, isolated way to build baochip firmware artifacts. Solution: add a nix flake with build derivations for all targets (dabao, baosec, bootloader, baremetal), a custom Rust 1.93.0 toolchain with xous targets, and dev shells for stable and nightly Rust.
Problem: nix builds are not tested in CI. Solution: add a GitHub Actions workflow that builds all nix packages, with cachix for caching (read-only for fork PRs).
Problem: locales has its own Cargo.lock with different dependency versions (e.g. glob 0.3.0 vs 0.3.1), causing offline builds to fail when the vendor store only contains deps from the root Cargo.lock. Solution: create a merged vendor directory that symlinks crates from both vendor stores, producing a single vendor-config.toml that covers both workspaces.
Problem: flake-utils is less actively maintained and has known sharp edges with system-agnostic outputs. Solution: Migrate to flake-parts using mkFlake + perSystem. All packages, devShells, and build logic are unchanged.
Nix Flake for Reproducible Builds for Baochip targets
This PR adds Nix-based reproducible builds for Baochip firmware artifacts (dabao, baosec, bootloader, baremetal). Related to #57.
TODO:
devbranchWhat is Nix?
Nix is a package manager that provides reproducible builds by isolating dependencies and pinning exact versions. This ensures builds are identical across different machines and over time.
What's Included
flake.nixwith build derivations for all targets1.92.01.93.0) with Xous targets built via rust-xous-flakenix developshell with toolchains (riscv32imac-unknown-xous-elf & riscv32imac-unknown-none-elf)nix develop .#nightlyshell with Rust nightly forcargo fmt.github/workflows/nix.yml) — example runnix build --rebuild)Installing Nix
This installs Nix with flakes enabled. For other options, see Determinate Systems Nix Installer.
Building Packages
Build outputs are placed in a
resultsymlink.Available Packages
nix build .#dabao-helloworld.#dabaonix build .#baosecnix build .#bao1x-baremetal-dabao.#baremetalnix build .#bao1x-boot0nix build .#bao1x-boot1nix build .#bao1x-alt-boot1nix build .#bootloaderDevelopment Shell
Verifying Reproducibility
nix build --rebuild .#dabao-helloworldThis builds twice and compares results. If they differ, it reports an error.
Challenges Solved
Custom Rust target: The
riscv32imac-unknown-xous-elftarget is built from source using Nix via rust-xous-flake. Note: this is not a bootstrapped build—it uses the pre-built Rust 1.92.0 compiler.No git in Nix sandbox: The following files are patched at build time (via
substituteInPlaceinflake.nix) to use environment variables instead of git commands:tools/src/sign_image.rs— uses hardcoded version instead ofSemVer::from_git()xtask/src/versioning.rs— readsXOUS_VERSIONenv var instead ofgit describetools/src/swap_writer.rs— readsGIT_REVenv var instead ofgit rev-parseReproducibility:
SOURCE_DATE_EPOCH=1,CARGO_INCREMENTAL=0, and--remap-path-prefixfor deterministic buildsxtask/src/versioning.rsto useSOURCE_DATE_EPOCHfor the build timestamp instead ofLocal::now()(commit e30d405)Dependency vendoring: Used Crane to vendor Cargo dependencies. The
locales/workspace required separate vendoring due to its ownCargo.lock.Speeding Up Builds with Cachix (Optional)
First builds compile the Rust toolchain and can be slow. Cachix provides pre-built binaries.
Using the public cache (read-only, no account needed):
To enable CI cache writes (maintainers):
xous-core)gh secret set CACHIX_AUTH_TOKENauthTokenandwarmup-cachesections in.github/workflows/nix.ymlA public cache is available at https://app.cachix.org/cache/xous-core (read-only for PRs from forks).
Tested
Caveats to Consider for Reviewers
1.92.01.93.0 compilerxousVersion) requires manual updates togitTagandgitTagRevCountinflake.nixwhen tags changelocales/Cargo.lockmust be kept in sync separately from mainCargo.lock--no-verifyflag (skips signature verification) like ci (.github/workflows/build.yml) doesnix build --rebuild) runs but failures are non-blocking (continue-on-error: true)xtask/src/versioning.rsmodified to supportSOURCE_DATE_EPOCHfor reproducible timestampsTroubleshooting
"experimental features 'nix-command' and 'flakes' are disabled"
If you installed Nix without the Determinate Systems installer, add to
~/.config/nix/nix.conf:Then restart your shell or run
sudo systemctl restart nix-daemon(Linux).Outdated section
(click to expand)
Still needs some work, but should work to build dabao helloworld, dabao-console, and boot1.
To use it, run
nix developto enter the dev shell, and thencargo xtaskcommands to build uf2 artifacts.nix build(wip at https://github.com/sbellem/xous-core/blob/nix-build/flake.nix)riscv32imac-unknown-xous-elfwith nix (wip at https://github.com/sbellem/rust-xous-flake/tree/main)Some challenges
In sandboxed environments for
nix build,gitis not available, and therefore image signing via xous-semver fails as it relies ongit describeto get the version. The current workaround is to patch the code "dynamically" inflake.nixto read a hardcoded base version combined with the git revision whichnixhas access to (self.shortRev). This still does not exactly match the output ofgit describeand thus needs to be addressed properly.git describefor nix build sandboxed environment wheregitis not available.Current
flake.nixdoes not yield reproducible builds. This can be tested withnix build .#dabao --rebuildfor instance.Also relate to the sandbox environment, which does not have git access, we need to vendor dependencies.
Note that in order to make the above work (build without git access, in offline mode),
Cargo lock files had to be updated, andwe run the xtask commands with--offlineand--no-verify.CI Cache Over-usage Issue
In the meantime work is being done on fork at https://github.com/sbellem/xous-core/tree/nix-troubleshoot.
Answer: Use cachix
A public cache has been setup at https://app.cachix.org/cache/xous-core. For PRs from forks it can only be used to pull (read-only). To support pushes (writes) betrusted-io would need to add an auth token.